App Engine: Copy live Datastore to local dev Datastore (that still works)
This used to be possible by downloading with the bulkloader and uploading to the local dev server. However, the bulkloader download has been non-functional for several months now, due to not supporting oauth2. A few places recommend downloading from a cloud storage backup, and uploading to the local datastore through either bulkloader or by directly parsing the backup. However, neither of these appear functional anymore. The bulkloader method throws: OperationalError: unable to open database file And the RecordsReader class, which is used to read the backup files, reaches end of file when trying to read the first record, resulting in no records being read. Does there exist a current, functional, method for copying the live datastore to the local dev datastore?
RecordsReader is functioning perfectly on unix. I've tried this https://gist.github.com/jehna/3b258f5287fcc181aacf one day ago and it worked amazing. You should add to the imports your Kinds implementation and run it in the datastore interactive shell. for example: from myproject.kinds_implementations import MyKind I've removed the for pp in dir(a): try: ppp = getattr(a, "_" + pp) if isinstance(ppp, db.Key): ppp._Key__reference.set_app(appname) ppp except AttributeError: """ It's okay """ And it worked well. In my case the backup downloaded in multiple directories so I've modified the access to the directories. for some thing like that: for directory in mypath: full_directory_path = join(mypath, directory) for sub_dir in listdir(directory_full_path): full_sub_dir_path = join(full_directory_path, sub_dir) onlyfiles = [ f for f in listdir(full_sub_dir_path) if isfile(join(mypath,f)) ] for file in onlyfiles: If you're working on windows you're welcome to follow my question about RecordsReader on windows, hopefully someone will answer there Google datastore backup to local dev_appserver edit: Working great on windows if you change the file open permissions from 'r' to 'rb'
The bulkloader is still functional on Python with OAuth2, albeit with some caveats. In downloading from the live app, there is an issue with refreshing of the OAuth2 token so the total download time is limited to 3600 seconds, or 3600+3600 if you manually use a refresh token with --oauth2_refresh_token. When uploading to the development server app, OAuth2 will fail with a 401, so it's necessary to edit google.appengine.ext.remote_api.handler and stub out 'CheckIsAdmin' to always return True as a workaround: def CheckIsAdmin(self): return True user_is_authorized = False ... I upvoted the above answer however, as it looks like a more robust solution at this point.
can not figure out relation between yaml and main page handler in google app engine
NoClassDefFoundError when adding new font in iText on AppEngine
Facebook login in Google Cloud Endpoints
To share a local host for go gae?
App-engine: JAX-RS with Jersey no working
Bi-directional one-to-many relationship in google app engine using JPA
How to check if field with value None is stored in datastore or not stored at all?
DeadlineExceededError in self.response.write
Google Drive invalid credentials
Google App Engine endpointscfg.py command starting 1.8.6 does not accept argument -f
Google Checkout Order Report API -> Google Wallet analog?
Access Denied exception when using google-api-java-client
Exception on trying to access the entity using key in GAE datastore
How to read servlet data from Google map
Search API, how to like query
Google Drive API behavior