google-app-engine


GAE : What is the preferred way of ensuring an outside task really finishes?


I have a workflow that consist of doing some heavy work outside of Google AppEngine by adding tasks to a Redis queue.
The outside servers that treats the Redis queue send a POST request back to GAE once the work is done.
The issue is that sometime, the Redis queue fails, or at last the POST request isn't made. This results in a "waiting" state on GAE that never changes.
In order to fix that issue, I plan to implement a "healthcheck" system that will automatically close the state to "invalid" after some time that have elapsed, but I was wondering which one was the best way to do it, in term of resources & pricing.
Option 1 : When I submit a task to the Redis queue, I would also create a new GAE task, called "Healthcheck", that would be run in 5 minutes, and if the current task is not done, switch it to "invalid" and close it.
Option 2 : When submitting the task to the Redis queue, I hang th process using a while True loop and refresh the current state up until it switched to done. I would also set up a Deadline exception watcher that would update the status to "invalid" when the Deadline is called by GAE (in general, 10 minutes).
PROs/CONs :
The option 1 has the advantage of not having a hang process, thus using less resource in my own opinion. The Cons here is that I will have as many "healthcheck" tasks as tasks added to the Redis queue (around 30k/day)
The option 2 has the advantage of not creating a new GAE task, but will consume more resource (useless in my opinion).
What is your suggestion on it, is there a third way to do it I didn't thought about? What would you do?
Thank you in advance :)
Tasks are free to use (except storage space that is very cheap). I don't see why number of tasks is a cons.
The 2nd option is not an option in my opinion as is expensive and process can die at any time.
The 3d option would be to have a cron job to query records that are in "running" state for more then defined deadline. That would require composite index on 2 fields (status, dt_created) - will be more expensive comparing to option #1.
Go with tasks.
By the way this question is opinion based and not best suited for SO so can be closed by moderators.

Related Links

Google app engine Federated (openid) login localhost
Can I receive incoming mail with Google App Engine on a non-default version?
GAE always on feature = static VIP?
GAE/J request log format breakdown
Handling URLs in App Engine
Exceeded soft memory limit with basic SELECT
Appengine application as OpenId provider. Is it possible?
How to set CSS style colors in GWT
How do I suppress App Engine logging while running unit tests?
Google App Engine: Java version of the python deferred library?
Appengine Datastore Advantages [closed]
Are crons executed globally or per app instance (i.e. load dependent)?
AppEngine as WebService and S3 as storage
Setting cookie in LiftFilter
Send Emails From AppEngine Locally
ImportError: cannot import name taskqueue

Categories

HOME
eclipse
deep-learning
webdriver
case
youtube-data-api-v3
swt
vscode-extensions
r-googlesheets
yandex
soa
simulink
switch-statement
dojo
maxscript
azure-container-service
vert.x
webix
32bit-64bit
vimdiff
android-securityexception
jxl
intercom
selenium-ide
scrolltop
symbol
template-engine
video-editing
enterprise-miner
apache2.4
replication
php-mysqlidb
qliksense
epsilon
data.stackexchange.com
android-sharedpreferences
delimiter
has-and-belongs-to-many
openmdao
recurring-events
python-2.6
database-comparison
gojs
graphhopper
android-augmented-reality
nohup
bc
c9.io
unordered-map
strftime
httrack
openresty
builder
user-defined-fields
dbcontext
identification
payara-micro
git-flow
oozie-coordinator
lsyncd
winusb
preprocessor
centrifuge
delta
nio2
grass
jdi
coda
rdflib
graphics2d
canvg
git-ftp
mmwormhole
infosphere-spl
mruby
ghcjs
pseudo-class
ignite-ui
lnk
alternate
dsym
gae-quotas
scringo
flex++
fb.ui
preload
intersect
extreme-programming
xcode3.2
veracity
paginator
automationelement
massive
twitterizer
ie-compatibility-mode
facebook-iframe
jqueryform
blender-2.49
mathematical-notation
fdf
online-storage

Resources

Encrypt Message