google-app-engine


App Engine Java and EclipseLink: Deadlock on shared-cache access


I'm have a Java Maven Google App Engine project configured like follows:
I'm using EclipseLink as JPA persistence-manager for Cloud SQL. My object contains some simple fields (string, date, ...) and a ManyToMany relationship, which is configured as Lazy-Load
#Entity
#Table(name = "mytable")
public class MyObject1 {
private String nome;
private String descrizione;
#ManyToMany
#JoinTable
(
name = "myobject1_has_myobject2",
joinColumns = { #JoinColumn(name = "object1_id", referencedColumnName = "id") },
inverseJoinColumns = { #JoinColumn(name = "object2_id", referencedColumnName = "id") }
)
private List<MyObject2> relationshipObjects;
}
The project flow works like this:
- A query is made that retrieve x results of type MyObject1 (let's say 10 results)
- The query results list is iterated and each result is given to a different Thread for processing
- Each thread iterate the ManyToMany relationship (the relationshipObjects object), which is Lazy and this in confirmed because the code calls IndirectList.iterator, and do some processing for each MyObject2 item of the list
- When all the threads have finished, the query result of MyObject1 is iterated once again to create a request response
This kind of implementation is giving some trouble regarding the multi-thread implementation and a some sort of deadlock.
Here is the stacktrace
Caused by: Exception [EclipseLink-2001] (Eclipse Persistence Services - 2.6.4.v20160829-44060b6): org.eclipse.persistence.exceptions.ConcurrencyException
Exception Description: Wait was interrupted.
Message: [null]
at org.eclipse.persistence.exceptions.ConcurrencyException.waitWasInterrupted(ConcurrencyException.java:108)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquireDeferredLock(ConcurrencyManager.java:187)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquireDeferredLock(CacheKey.java:210)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireDeferredLock(AbstractIdentityMap.java:84)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireDeferredLock(IdentityMapManager.java:146)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireDeferredLock(IdentityMapAccessor.java:81)
at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5200)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:965)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:899)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:852)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:735)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:689)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:805)
at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:962)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:573)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1175)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:460)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:473)
I cannot reproduce all the time the problem, besides that I can give you all the information I can gather.
Looking inside the EclipseLink documentation I found a section related to this matter
Cache - If using a shared cache, EclipseLink requires locking the cache on reads and writes to ensure consistency.
You will see cache access, such as IdentityMapManager acquireLock or acquireDeferredLock, or WriteLockManager as the last call on the stack.
In my persistence-unit I did not configure the shared-cache behaviour, so it is running on default which is enabled.
Here is my persistence-unit properties
<properties>
<!-- configure the various connection pool properties -->
<!-- http://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_connection_pool.htm -->
<property name="eclipselink.connection-pool.default.initial" value="1" />
<property name="eclipselink.connection-pool.default.min" value="64" />
<property name="eclipselink.connection-pool.default.max" value="64" />
<property name="eclipselink.connection-pool.default.shared" value="true" />
<!-- whether connections in EclipseLink read connection pool should be shared (not exclusive). Connection sharing means the same JDBC connection will be used concurrently for multiple reading threads. -->
<property name="eclipselink.jdbc.connection_pool.read.shared" value="true" />
<!-- specify if JDBC statements should be cached -->
<!-- http://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_jdbc_cachestatements.htm -->
<property name="eclipselink.jdbc.cache-statements" value="true" />
<!-- the number of statements held when using internal statement caching -->
<!-- http://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_jdbc_cachestatements_size.htm#CACBICGG -->
<property name="eclipselink.jdbc.cache-statements.size" value="100" />
</properties>
I can see in my stacktrace that there is indeed the IdentityMapManager.acquireDeferredLock(IdentityMapManager.java:146) row that is referred.
The thing is, this error is thrown by the App Engine request (see last line of stacktrace) when I call the getResultList method.
This call is made by the main request thread, other threads (one for query result) has not been launched yet.
So I started to looking for the shared-cache documentation and I found this part:
The shared cache exists for the duration of the persistence unit (EntityManagerFactory, or server) and is shared by all EntityManagers and users of the persistence unit
My EntityManagerFactory instance is instance-shared (I have a static variabile which is initialized at first query).
So at first access (for each App Engine instance) the variabile is initialized and then shared for all the http requests that will be server by the same instance.
I did this sort of "caching" because the deployment descriptor at the first access of EntityManagerFactory is very slow, and even if I can pre-warmup this object, opening a new EMF at every request costs about 1-2 seconds.
So, I open/close a new EntityManager at each flow (and each thread, because EntityManager is not thread-safe) but the EMF object is shared.
Also, there is another line which says
This is normally related to having relationships that do not use LAZY, ensure all relationship use LAZY.
My ManyToMany relationship is already Lazy, as pointed out before, so even this point cannot be the cause
Basing on that here, I tried to gather all together:
- EclipseLink requires locking the cache on reads and writes to ensure consistency, so the access to this cache is atomic and multiple threads are queued.
- The sharedcache is based on the EMF object
- The EMF object is shared between requests of the same instance
As suggested by EclipseLink documentation I tried to disable the shared-cache and all the flow appaers to works, but it is now very slow.
Anyway, this is another point that confirm the problem here is related to the shared-cache of JPA.
This solution is not suitable because, even w/o considering the speed problem, all those request and threads that concurr of get data from the DBMS (while iterating the Lazy list) consumes all the available connections and the DBMS starts on giving connection errors.
Another suggestion from the documentation
DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;
but the error is still the same, nothing changed (the error is on IdentityMapManager.acquireLock, so the deferredLock is not used anyway)
From the App Engine logs I can see that all these requests are killed after 60s timeout, so the Wait was interrupted message can be related to all the threads that were waiting to access the shared-cache, but at the end the App Engine deadline killed the request.
Because of that I tried to deploy on basic-scaling (which does not have the 60s deadline) to see if the request is only slower than the dealine or it is truly stuck on a deadlock
Inside the logs there is no error... but the longest requests does not even show. At this point I can think that the erroneous requests are stuck indefinitely and the request logs will not be shown at all.
Another test I made is reducing all the persistence-unit configuration, removing all the shared configuration, like follows
<properties>
<property name="eclipselink.connection-pool.default.initial" value="1" />
<property name="eclipselink.connection-pool.default.min" value="64" />
<property name="eclipselink.connection-pool.default.max" value="64" />
</properties>
But the error is still the same. So it is not related to a connection-pool sharing but the multi-thread itself
As ultimate test I tried to remove the multi-thread flow (each query result is processed one-by-one by the main thread) and leaving the shared-cache enabled.
This is working.
At this point I'm wondering... because the shared-cache is synchronized, so there is a "funnel" that block anyway the multi-thread process, should I use the monothread implementation anyway?

Related Links

Why does Google Cloud SQL (using JDBC) take longer to insert records from Google App Engine than from my personal computer?
How do I track Google account logins on python GAE?
Delete and Rename events in google drive
App Engine - How to Clear Cookie
How do I get the path to the Cloud SDK directory using the gcloud command?
Cannot reach Endpoint method with cURL
Find the total instance hours in my Google Apps Engine
Cron Jobs on Google App Engine
Golang GAE, aestest.NewContect not working on local machine. Could not find python interpreter
Downloading my existing project in google cloud using Gcloud SDK
Google app engine css not found/deployed
CI for ASP.Net using Jenkins on Google Cloud
Can the GAE Servlet code be accessest by anyone
Is there a way of checking if a Web Application or Website using the Google App Engine?
google endpoints on flex app engine
Deploy to Google App Engine via a GitHub Repo

Categories

HOME
shell
xml
alexa-skills-kit
formal-languages
battery
components
jsf-2.2
gradient
carousel
cakephp-3.4
google-cloud-datalab
joomla3.0
arangodb
google-schemas
github-enterprise
favicon
code-generation
renjin
dimensional-modeling
pyramid
foreign-keys
squashfs
mongoid6
pylons
oculus
calculation
dst
kudu
sql-server-2000
hibernate-search
arduino-ide
xdocreport
logback-groovy
apollostack
xmldocument
berkeley-db-je
show-hide
jboss-4.2.x
lampp
magento-2.0.7
context-switch
static-cast
git-flow
libjpeg
janus
structuremap4
rsa-archer-grc
rocks
seek
microstation
uicolor
dup
clipboard.js
jtwig
http-status-code-401
grass
design-principles
concurrentmodification
intel-c++
google-books
ember-cli-rails
dulwich
liquid-layout
mirror
google-refine
aspectj-maven-plugin
osc
ejb-2.x
execve
addressing-mode
crowd
sonarqube-5.0
personalization
buffering
mutual-authentication
visual-studio-express
jeromq
wp7test
mantle
alice
preload
redis-py
saleslogix
subgraph
paginator
pinch
update-statement
facebook-iframe
interprocess
purepdf
mathematical-notation
sqlobject
ifilter

Resources

Encrypt Message