google-app-engine
Handle schema change in Google datastore with Go?
I added a new attribute to Go struct, persisted in a datastore entity kind. I tried to load entities using a filter on the new attribute: q := datastore.NewQuery("Person").Filter("Employed =", false) This only worked as intended for persons created after the new attribute was added. I had expected persons created before the attribute was added to be incleded in the filter, but they didnt have the attribute at all, so they were be excluded by the filter. I have thought of two ways to handle this: First load all entities and filter in a second step with a loop. This adds complexity to the code. Batch load and resave all entities, which adds the property and sets it to false. I have to remember to do this every time I add a new attribute. Is there a batter way to handle these types of entity schema changes?
The proper way is to re-save old entities if you can afford it. You can use Python map/reduce library to update old entities. If you want to stick to Go you can go with simple batch processing using task queues & query cursors. In our Python app we prefer "in-flight hot update" approach whenever it is possible - new version loads entities and check for missing properties. If some props are obsolete we would keep them for a month or to and then add logic to remove them. This allows to migrate gradually and often rollback without issues (shit happens) or at least limit impact to small % of records. Also it's more cost effective as you do not pay for additional reads/writes. Once we sure we ok we can run a mapreduce. This does not involve downtime. You may need or don't need downtime - it's really depend on your change. With Go it may be a bit tricky as Go would panic if you try to load entity that has no corresponding fields on a struct. There should be some workarounds about it (I remember something about special interface or a struct where you can load arbitrary entity) but I have no much practical experience with Go on GAE yet.
Related Links
AttributeError on google cloud datastore entity object
Is there any way to handle Google Datastore Kind Property names with spaces in Golang?
Google App Engine: Disable favicon.ico
Google Cloud DataStore. How to serve data?
How to access a cloud storage bucket from app engine without getting the 403 FORBIDDEN error?
Google App Engine Custom Domain - Routing in Go
Best approach for caching lists of objects in memcache
com.google.gcloud.datastore vs com.google.appengine.api.datastore
Google App Engine - Issue with creating a bulkloading config
Solr Timeout error even data is instered
Why does BigQuery fail to parse an Avro file that is accepted by avro-tools?
Insufficient Permission with Appengine Flex service account to access Drive folder
Getting Invalid Key message thrown when creating child records
Where do I find the pricing for the different frontend instance types?
Missing index on specific entities in app engine
Get current deployed timestamp in AppEngine/Go