google-app-engine


App Engine: Copy live Datastore to local dev Datastore (that still works)


This used to be possible by downloading with the bulkloader and uploading to the local dev server. However, the bulkloader download has been non-functional for several months now, due to not supporting oauth2.
A few places recommend downloading from a cloud storage backup, and uploading to the local datastore through either bulkloader or by directly parsing the backup. However, neither of these appear functional anymore. The bulkloader method throws:
OperationalError: unable to open database file
And the RecordsReader class, which is used to read the backup files, reaches end of file when trying to read the first record, resulting in no records being read.
Does there exist a current, functional, method for copying the live datastore to the local dev datastore?
RecordsReader is functioning perfectly on unix. I've tried this https://gist.github.com/jehna/3b258f5287fcc181aacf one day ago and it worked amazing.
You should add to the imports your Kinds implementation and run it in the datastore interactive shell.
for example:
from myproject.kinds_implementations import MyKind
I've removed the
for pp in dir(a):
try:
ppp = getattr(a, "_" + pp)
if isinstance(ppp, db.Key):
ppp._Key__reference.set_app(appname)
ppp
except AttributeError:
""" It's okay """
And it worked well. In my case the backup downloaded in multiple directories so I've modified the access to the directories. for some thing like that:
for directory in mypath:
full_directory_path = join(mypath, directory)
for sub_dir in listdir(directory_full_path):
full_sub_dir_path = join(full_directory_path, sub_dir)
onlyfiles = [ f for f in listdir(full_sub_dir_path) if isfile(join(mypath,f)) ]
for file in onlyfiles:
If you're working on windows you're welcome to follow my question about RecordsReader on windows, hopefully someone will answer there Google datastore backup to local dev_appserver
edit:
Working great on windows if you change the file open permissions from 'r' to 'rb'
The bulkloader is still functional on Python with OAuth2, albeit with some caveats. In downloading from the live app, there is an issue with refreshing of the OAuth2 token so the total download time is limited to 3600 seconds, or 3600+3600 if you manually use a refresh token with --oauth2_refresh_token.
When uploading to the development server app, OAuth2 will fail with a 401, so it's necessary to edit google.appengine.ext.remote_api.handler and stub out 'CheckIsAdmin' to always return True as a workaround:
def CheckIsAdmin(self):
return True
user_is_authorized = False
...
I upvoted the above answer however, as it looks like a more robust solution at this point.

Related Links

can not figure out relation between yaml and main page handler in google app engine
NoClassDefFoundError when adding new font in iText on AppEngine
Facebook login in Google Cloud Endpoints
To share a local host for go gae?
App-engine: JAX-RS with Jersey no working
Bi-directional one-to-many relationship in google app engine using JPA
How to check if field with value None is stored in datastore or not stored at all?
DeadlineExceededError in self.response.write
Google Drive invalid credentials
Google App Engine endpointscfg.py command starting 1.8.6 does not accept argument -f
Google Checkout Order Report API -> Google Wallet analog?
Access Denied exception when using google-api-java-client
Exception on trying to access the entity using key in GAE datastore
How to read servlet data from Google map
Search API, how to like query
Google Drive API behavior

Categories

HOME
phpword
api
webdriver
google-spreadsheet
datastore
character-encoding
fftw
rds
echo
spring-session
barcode
computer-science
sendgrid-api-v3
drop-down-menu
endeca
emoji
ycsb
opengl-es-3.0
filechannel
automatic-ref-counting
intercom
gkturnbasedmatch
flatpak
libusb
azure-powershell
symfony-2.8
dynamic-sql
feathersjs
forgot-password
template-engine
lilypond
cgbitmapcontext
libigl
pic32
delimiter
iis-express
classcastexception
keras-layer
openproject
reactivekit
jvmti
tcc
drupal-theming
strftime
openresty
ntvs
builder
boxing
flyout
offline.js
baidu
shinobi
filenet
polyline
redbeard-ios
centrifuge
rspec2
simevents
jclouds
omniauth-facebook
php-socket
yajsw
findfirst
design-principles
include-guards
fuelcms
zscript
openbabel
sql-scripts
trdion2011
angular-carousel
mintty
dct
gae-quotas
android-audiorecord
eclipse-kepler
tlf
cffile
mvc-editor-templates
cmmi
nebula
joomla-template
stacky
device-width
web-notifications
wcf-configuration
pyunit
josql
multiplexing
cgimage
ikimagebrowserview
user-tracking
saxparseexception
entitydatasource
rmdir
appledoc
photolibrary
non-clustered-index
binomial-heap
addchild
filesystemobject
strtod
perl5.12
linfu-dynamicproxy
int64
appliance

Resources

Encrypt Message