google-app-engine


Handle schema change in Google datastore with Go?


I added a new attribute to Go struct, persisted in a datastore entity kind.
I tried to load entities using a filter on the new attribute:
q := datastore.NewQuery("Person").Filter("Employed =", false)
This only worked as intended for persons created after the new attribute was added.
I had expected persons created before the attribute was added to be incleded in the filter, but they didnt have the attribute at all, so they were be excluded by the filter.
I have thought of two ways to handle this:
First load all entities and filter in a second step with a loop. This adds complexity to the code.
Batch load and resave all entities, which adds the property and sets it to false. I have to remember to do this every time I add a new attribute.
Is there a batter way to handle these types of entity schema changes?
The proper way is to re-save old entities if you can afford it.
You can use Python map/reduce library to update old entities.
If you want to stick to Go you can go with simple batch processing using task queues & query cursors.
In our Python app we prefer "in-flight hot update" approach whenever it is possible - new version loads entities and check for missing properties. If some props are obsolete we would keep them for a month or to and then add logic to remove them. This allows to migrate gradually and often rollback without issues (shit happens) or at least limit impact to small % of records. Also it's more cost effective as you do not pay for additional reads/writes. Once we sure we ok we can run a mapreduce. This does not involve downtime. You may need or don't need downtime - it's really depend on your change.
With Go it may be a bit tricky as Go would panic if you try to load entity that has no corresponding fields on a struct. There should be some workarounds about it (I remember something about special interface or a struct where you can load arbitrary entity) but I have no much practical experience with Go on GAE yet.

Related Links

AttributeError on google cloud datastore entity object
Is there any way to handle Google Datastore Kind Property names with spaces in Golang?
Google App Engine: Disable favicon.ico
Google Cloud DataStore. How to serve data?
How to access a cloud storage bucket from app engine without getting the 403 FORBIDDEN error?
Google App Engine Custom Domain - Routing in Go
Best approach for caching lists of objects in memcache
com.google.gcloud.datastore vs com.google.appengine.api.datastore
Google App Engine - Issue with creating a bulkloading config
Solr Timeout error even data is instered
Why does BigQuery fail to parse an Avro file that is accepted by avro-tools?
Insufficient Permission with Appengine Flex service account to access Drive folder
Getting Invalid Key message thrown when creating child records
Where do I find the pricing for the different frontend instance types?
Missing index on specific entities in app engine
Get current deployed timestamp in AppEngine/Go

Categories

HOME
shell
notepad++
google-app-engine
embedded
xml
apache-flink
android-studio-2.2
google-spreadsheet
formal-languages
zend-framework
solaris
dojo
x264
frontend
webix
enthought
libusb
bitnami
simpleitk
cgbitmapcontext
joptionpane
gmock
onload
beautifier
jackson-modules
ceylon
comparable
mql
magento-1.8
complex-numbers
glib
spring-security4
openerp-7
gdata
p4merge
setup.py
mcustomscrollbar
magento-1.9.3
visa
ssrs-2014
apollostack
monkey-testing
httr
puredata
fwrite
textsum
lampp
chai-as-promised
geotiff
datamaps
podscms
reindex
jqchart
lsyncd
mouseleave
flyout
juice-ui
vcloud-director-rest-api
node-request
angular2-testing
pytables
envi
watch-os-2
appscale
mobiledoc-kit
avisynth
django-urls
page.js
skspritenode
mirror
user-permissions
broadband
document-repository
at-job
gii
memset
file-not-found
vbe
mt4j
jeromq
ui-select2
dolby-audio-api
android-icons
preload
redis-py
monotouch.dialog
rmiregistry
subgraph
subproject
inkcanvas
jquery-tools
exponentiation
creole
undefined-index
localizable.strings
ifilter

Resources

Encrypt Message