Best approach for caching lists of objects in memcache
Our Google AppEngine Java app involves caching recent users that have requested information from the server. The current working solution is that we store the users information in a list, which is then cached. When we need a recent user we simply grab one from this list. The list of recent users is not vital to our app working, and if it's dropped from the cache it's simply rebuilt as users continue to request from the server. What I want to know is: Can I do this a better way? With the current approach there is only a certain amount of users we can store before the list gets to large for memcache (we are currently limiting the list to 1000 and dropping the oldest when we insert new). Also, the list is going to need updating very quickly which involves retrieving the full list from memcache just to add a single user. Having each user stored in cache separately would be beneficial to us as we require the recent user to expire after 30 minutes. At the moment this is a manual task we do to make sure the list does not include expired users. What is the best approach for this scenario? If it's storing the users separately in cache, what's the best approach to keeping track of the user so we can retrieve it?
You could keep in the memcache list just "pointers" that you can use to build individual memcache keys to access user entities separately stored in memcache. This makes the list's memcache size footprint a lot smaller and easy to deal with. If the user entities have parents then pointers would have to be their keys, which are unique so they can be used as memcache keys as well (well, their urlsafe versions if needed). But if the user entities don't have parents (i.e. they're root entities in their entity groups) then you can use their datastore key IDs as pointers - typically shorter than the keys. Better yet, if the IDs are numerical IDs you can even store them as numbers, not as strings. The IDs are unique for those entities, but they might not be unique enough to serve as memcache keys, you may need to add a prefix/suffix to make the respective memcache keys unique (to your app). When you need a user entity data you 1st obtain the "pointer" from the list, build the user entity memcache key and retrieve the entity with that key. This, of course, assumes you do have reasons to keep that list in place. If the list itself is not mandatory all you need is just the recipe to obtain the (unique) memcache keys for each of your entities.
If you use NDB caching, it will take care of memcache for you. Simply request the users with the key using ndb.Key(Model, id).get() or Model.get_by_id(id), where id is the User id.
'Model is not immutable' TypeError
How to store data from SmartGWT ListGrid?
BigQuery OAuth access token on Java App Engine dev server
Google App Engine ancestor query in JPA
Google App Engine Go memory management
How to build Many-to-Many relations in Google App-engine using JPA
Currently experiencing frequent DeadlineExceededExceptions when it was fine a few hours ago
How to get related to db.Model entity from datastore - what API allows it?
GAE-Python: Can Appstats record RPC made in background thread?
Blank Page after granting OAuth2 permission in BigQuery GAE Sample
GAE get Data using JDO with key
Setup GAE project with Resteasy using Maven in Eclipse
Working Maven3 configuration for AppEngine + DataNucleus + JPA 2
Spring Security ACL on App Engine Datastore
JDO 1:N issue (retriving data)
Endpoints Bad Api Configuration