Cache and/or Connection problems under load

Question

I have a Kodo web app that's been running just fine in
production for many months now. However, recently the web
traffic has shot up by a huge amount, literally overnight.
But unfortunately, it's caused the app to fail very ungracefully
under the strain.
It's been a crazy few days, and I haven't been able to do
very much analysis because of higher priorities. But from
what I have been able to gleen, it now looks like Kodo is
the most likely culprit. From what I've read in other messages
here, it appears others may have been experiencing similar
problems.
My environment: Redhat Linux 8, Postgres 7.3.4 with the
included JDBC3 driver, Apache 1.3.x, Tomcat 4.1.x and the
webapp connector. Similar behavior was seen with Apache 2.x,
Tomcat 4.1.x and the JK2 connector (that was on the new machine
I setup to handle the new traffic, which, of course, died the
night before).
As I mentioned, this app has been running reliably for
months with no problems. But when placed under heavy load,
it appears to get into some sort of pathological state where
it slows down dramatically (asymptotically?) to the point where
it's effectively locked up. In one case, where the app was
left running for several hours in this state, requests were
taking 90 minutes to complete (normal is 1-5 seconds).
From what I can deduce, there seem to be four things
going on, three of which have been mentioned in recent threads
here:
1) Excessive memory consumption. When the app is
operating normally, I see fairly flat memory usage for
the JVM process. Under load, the JVM steadily expands
until it hits its heap limit. I've gotten OutOfMemory
exceptions with a heap size of 350MB, which should be plenty.
2) Level 2 cache locking issues. I've seen dozens of
threads waiting on a lock in the DataCache code. Not sure
if there's a deadlock happening here or just that the
threads are waiting on a lock that's being held for a long time.
3) Database Connection leaks or contention. I see threads
spinning in the DataSource code trying to get a connection.
I also see dozens of connections from the Postgres side which
seem to be sitting idle, but in the middle of a transaction.
When things get bad, I also see exceptions being thrown because
of timeouts waiting for a connection to become available. It's
a web app, PMs should not be tied up for more than a few seconds.
4) CPU usage pegged or nearly so for the JVM. I suspect
this is related to #3. Something very bad is going on here.
If I stop all inbound requests to the JVM when it's in this
bad state, it will continue to burn CPU at 90%+ for a very
long time. I think it will eventually finish what it's doing,
but I haven't had the luxury of waiting for it. It's definitely
not a linear slowdown proportional to the load.
Attached are my kodo.properties file and some thread stack
traces along with some comments. Any advice would be greatly
appreciated. This is not a complicated app nor am I doing
anything unusual. It doesn't seem logical that Kodo could
breakdown so dramatically under load, so I'm hoping it's some
sort of interaction thing that I can work around.
Thanks.
Ron Hitchens {mailto:[email protected]} RonSoft Technologies
[phone] (Home Office) http://www.ronsoft.com
[phone] (fax) Bit Twiddling At Its Finest
"Born with a broken heart" -Kenny Wayne Shepard

Answer 1

Please read prior posts regarding level 2 cache. It is unusable under stress
as far I am concerned. Basically entire cache gets locked on any database
read. Makes it very unscalable
Are you using 2.5.3? It will request a connection from a pool every time it
resolves reference to a PC even if it is cached in PM and therefore Kodo
does not need to read any. As result if you iterate over 100 objects in your
query and for each object resolve reference to another object (always the
same) kodo will request 100 database connections from the pool (and note
they issue rollback on every time they return a connection to the pool so
getting connection might be fairly expensive)
In conjunction with level 2 cache contention this causes application to go
into a stupor.
Try to go back to 2.5.2 (or may be 2.5.4 they promised in the near future
with a workaround) or use "persistent-manager" connection retention if you
discard PM after each HTTP invocation - it will take care of connection
pooling issue. As far as L2 cache I was unable to find any work around so
far - see if you might be better of without cache. You might if your object
graph is not very complex
"Ron Hitchens" <[email protected]> wrote in message
news:[email protected]...
>
I have a Kodo web app that's been running just fine in
production for many months now. However, recently the web
traffic has shot up by a huge amount, literally overnight.
But unfortunately, it's caused the app to fail very ungracefully
under the strain.
It's been a crazy few days, and I haven't been able to do
very much analysis because of higher priorities. But from
what I have been able to gleen, it now looks like Kodo is
the most likely culprit. From what I've read in other messages
here, it appears others may have been experiencing similar
problems.
My environment: Redhat Linux 8, Postgres 7.3.4 with the
included JDBC3 driver, Apache 1.3.x, Tomcat 4.1.x and the
webapp connector. Similar behavior was seen with Apache 2.x,
Tomcat 4.1.x and the JK2 connector (that was on the new machine
I setup to handle the new traffic, which, of course, died the
night before).
As I mentioned, this app has been running reliably for
months with no problems. But when placed under heavy load,
it appears to get into some sort of pathological state where
it slows down dramatically (asymptotically?) to the point where
it's effectively locked up. In one case, where the app was
left running for several hours in this state, requests were
taking 90 minutes to complete (normal is 1-5 seconds).
From what I can deduce, there seem to be four things
going on, three of which have been mentioned in recent threads
here:
1) Excessive memory consumption. When the app is
operating normally, I see fairly flat memory usage for
the JVM process. Under load, the JVM steadily expands
until it hits its heap limit. I've gotten OutOfMemory
exceptions with a heap size of 350MB, which should be plenty.
2) Level 2 cache locking issues. I've seen dozens of
threads waiting on a lock in the DataCache code. Not sure
if there's a deadlock happening here or just that the
threads are waiting on a lock that's being held for a long time.
3) Database Connection leaks or contention. I see threads
spinning in the DataSource code trying to get a connection.
I also see dozens of connections from the Postgres side which
seem to be sitting idle, but in the middle of a transaction.
When things get bad, I also see exceptions being thrown because
of timeouts waiting for a connection to become available. It's
a web app, PMs should not be tied up for more than a few seconds.
4) CPU usage pegged or nearly so for the JVM. I suspect
this is related to #3. Something very bad is going on here.
If I stop all inbound requests to the JVM when it's in this
bad state, it will continue to burn CPU at 90%+ for a very
long time. I think it will eventually finish what it's doing,
but I haven't had the luxury of waiting for it. It's definitely
not a linear slowdown proportional to the load.
Attached are my kodo.properties file and some thread stack
traces along with some comments. Any advice would be greatly
appreciated. This is not a complicated app nor am I doing
anything unusual. It doesn't seem logical that Kodo could
breakdown so dramatically under load, so I'm hoping it's some
sort of interaction thing that I can work around.
Thanks.
Ron Hitchens {mailto:[email protected]} RonSoft Technologies
[phone] (Home Office) http://www.ronsoft.com
[phone] (fax) Bit Twiddling At Its Finest
"Born with a broken heart" -Kenny Wayne Shepard
With cahce enabled, 2.5.3
Here the app had recently slowed down and then effectively locked up.
There where many outstanding web requests that were not receiving output.
At this point most threads seemed to be waiting at the same location.
There were a large number of active database connections and most of
them had open transactions (according to pg_stat_activity). The app
was not responding to any web requests.
It would seem that db transactions had been started, then the thread
got stuck for a long time on a synchronization lock in the cache lookup.
Below are two randomly chosen thread stack dumps.
Thread-72[1] where
[1] java.lang.Object.wait (native method)
[2] java.lang.Object.wait (Object.java:429)
[3]oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquir
e (WriterPreferenceReadWriteLock.java:169)
[4]com.solarmetric.kodo.runtime.datacache.AbstractCacheImpl.acquireReadLock
(AbstractCacheImpl.java:384)
[5]com.solarmetric.kodo.runtime.datacache.TimedDataCache.acquireReadLock
(TimedDataCache.java:256)
[6] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:595)
[7] com.solarmetric.kodo.runtime.StateManagerImpl.loadFields(StateManagerImpl.java:2,330)
[8] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:897)
[9] com.europeasap.data.City.jdoGetname (null)
[10] com.europeasap.data.City.getName (City.java:39)
[11] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:922)
[12] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
[13] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
[14] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
[15] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[16] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [17]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[18] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[19] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[20] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[21] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[22] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[23]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[24] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[25] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[26] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[27]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[28] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[29]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[30] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[31] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[32] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[33] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[35] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[36]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[37] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[38]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[39] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[40] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[41] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[42]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[43] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[44] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[45] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[46] org.apache.catalina.connector.warp.WarpConnection.run (null)
[47] java.lang.Thread.run (Thread.java:534)
Thread-64[1] where
[1] java.lang.Object.wait (native method)
[2] java.lang.Object.wait (Object.java:429)
[3]oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquir
e (WriterPreferenceReadWriteLock.java:169)
[4]com.solarmetric.kodo.runtime.datacache.AbstractCacheImpl.acquireReadLock
(AbstractCacheImpl.java:384)
[5]com.solarmetric.kodo.runtime.datacache.TimedDataCache.acquireReadLock
(TimedDataCache.java:256)
[6] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:595)
[7] com.solarmetric.kodo.runtime.StateManagerImpl.loadField(StateManagerImpl.java:2,248)
[8] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:899)
[9] com.europeasap.data.HotelPrices.jdoGetseasonalPrices (null)
[10] com.europeasap.data.HotelPrices.normalizeIndex(HotelPrices.java:113)
[11] com.europeasap.data.HotelPrices.getCost (HotelPrices.java:45)
[12] com.europeasap.logic.CostHelper.findLowestHotel(CostHelper.java:181)
[13] com.europeasap.logic.CostHelper.computeBasePackageCost(CostHelper.java:297)
[14] com.europeasap.logic.CostHelper.computeFinalPackageCost(CostHelper.java:246)
[15] com.europeasap.form.CustomerBookingForm.updateDisplayCosts(CustomerBookingForm.java:1,440)
[16] com.europeasap.form.CustomerBookingForm.updateCustomizeDisplayInfo(CustomerBookingForm.java:1,407)
[17] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:68)
[18] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[19] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [20]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[21] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[22] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[23] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[24] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[25] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[26]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[27] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[28] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[29] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[30]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[31] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[32]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[33] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[34] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[35] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[36] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[37]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[38] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[39]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[40] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[42] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[43] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[44] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[45]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[46] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[47] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[48] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[49] org.apache.catalina.connector.warp.WarpConnection.run (null)
[50] java.lang.Thread.run (Thread.java:534)
while running slow, 2.5.3
At this point, the app had been running several hours normally, then
apparently slowed down and locked up while I was away. When looking
at the app threads and database activity, everything appeared idle.
No transactions seemed to be open in the db. But the app was not
behaving normally. Web requests that did not make use of JDO worked
fine (but slow). But requests that hit the db either blocked or were
very slow to respond.
Looking back at the log, there had been a large number of requests
that threw exceptions because they could not get a connection within
five seconds.
Most threads were idle, waiting on read, but some were in the state
shown by the following two stack dumps. Unlike the cache threads above,
they did not seem to be waiting for a lock to be granted, they seemed
to be spinning in the connection management code. Apparently trying
to get a connection. I suspended and resumed the same thread repeatedly
and it always seemd to be doing the same thing. Single stepping was
very difficult because the debugger was slow to respond, apparently
because of other threads also busy spinning.
Postgres indicated that there where lots of connections open and
that they were all idle, so there should not have been a shortage
of connections in the pool. There are two mysteries here: 1) why
can't this thread get a connection? and 2) Why is it busy spinning?
Thread-56[1] where
[1]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
reStatement (PreparedStatementCache.java:184)
[2]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
reStatement (PreparedStatementCache.java:169)
[3] com.solarmetric.datasource.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:199)
[4]com.solarmetric.kodo.impl.jdbc.schema.dict.AbstractDictionary.isClosed
(AbstractDictionary.java:1,912)
[5]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnectionFromFact
ory (SQLExecutionManagerImpl.java:186)
[6] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnection(SQLExecutionManagerImpl.java:147)
[7]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.newSQLExecutionManag
er (JDBCStoreManager.java:828)
[8]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getSQLExecutionManag
er (JDBCStoreManager.java:714)
[9]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getDatastoreConnecti
on (JDBCStoreManager.java:287)
[10]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.getDatastoreCon
nection (DataCacheStoreManager.java:465)
[11] com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.load(DataCacheStoreManager.java:591)
[12] com.solarmetric.kodo.runtime.StateManagerImpl.loadFields(StateManagerImpl.java:2,330)
[13] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:897)
[14] com.europeasap.data.City.jdoGetname (null)
[15] com.europeasap.data.City.getName (City.java:39)
[16] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:922)
[17] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
[18] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
[19] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
[20] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[21] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [22]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[23] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[24] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[25] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[26] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[27] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[28]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[29] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[30] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[31] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[32]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[33] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[35] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[36] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[37] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[38] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[39]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[40] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[42] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[43]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[44] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[45] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[46] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[47]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[48] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[49] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[50] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[51] org.apache.catalina.connector.warp.WarpConnection.run (null)
[52] java.lang.Thread.run (Thread.java:534)
Thread-56[1] where
[1]com.solarmetric.datasource.DataSourceImpl$AbstractPool.findConnection
(DataSourceImpl.java:826)
[2] com.solarmetric.datasource.DataSourceImpl$AbstractPool.getConnection(DataSourceImpl.java:605)
[3] com.solarmetric.datasource.DataSourceImpl.getConnection(DataSourceImpl.java:363)
[4] com.solarmetric.datasource.DataSourceImpl.getConnection(DataSourceImpl.java:356)
[5]com.solarmetric.kodo.impl.jdbc.runtime.DataSourceConnector.getConnection
(DataSourceConnector.java:63)
[6]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnectionFromFact
ory (SQLExecutionManagerImpl.java:185)
[7] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.getConnection(SQLExecutionManagerImpl.java:147)
[8]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.newSQLExecutionManag
er (JDBCStoreManager.java:828)
[9]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getSQLExecutionManag
er (JDBCStoreManager.java:714)
[10]com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.getDatastoreConnecti
on (JDBCStoreManager.java:287)
[11]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.getDatastoreCon
nection (DataCacheStoreManager.java:465)
[12]com.solarmetric.kodo.runtime.datacache.DataCacheStoreManager.initialize
(DataCacheStoreManager.java:519)
[13] com.solarmetric.kodo.runtime.StateManagerImpl.loadInitialState(StateManagerImpl.java:215)
[14]com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectByIdFilter
(PersistenceManagerImpl.java:1,278)
[15] com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectById(PersistenceManagerImpl.java:1,179)
[16]com.solarmetric.kodo.runtime.datacache.query.CacheAwareQuery$CachedResultLis
t.get (CacheAwareQuery.java:432)
[17] java.util.AbstractList$Itr.next (AbstractList.java:421)
[18] com.europeasap.form.CustomerBookingForm.populateDepartureCityInfo(CustomerBookingForm.java:919)
[19] com.europeasap.form.CustomerBookingForm.onetimeInit(CustomerBookingForm.java:871)
[20] com.europeasap.form.CustomerBookingForm.populatePackageInfo(CustomerBookingForm.java:880)
[21] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:66)
[22] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[23] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [24]
org.apache.struts.action.ActionServlet.doGet (ActionServlet.java:492)
[25] javax.servlet.http.HttpServlet.service (HttpServlet.java:740)
[26] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[27] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[28] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[29] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[30]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[31] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[32] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[33] org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
[34]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[35] org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
[36]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[37] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[38] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[39] org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2,415)
[40] org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
[41]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[42] org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
[43]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[44] org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
[45]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:641)
[46] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[47] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[48] org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
[49]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[50] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[51] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[52] org.apache.catalina.connector.warp.WarpRequestHandler.handle (null)
[53] org.apache.catalina.connector.warp.WarpConnection.run (null)
[54] java.lang.Thread.run (Thread.java:534)
With cache disabled 2.4.3
This run was an accident. I inadvertently ran the app with the older
2.4.3 version of Kodo, with the cache disabled. This one got into trouble
almost immediately. It also seemed to lockup with lots of opentransactions
in the db. It's also interesting that these two threads also seem to be
hanging around the same method as in 2.5.3.
Thread-63[1] where 0x9f9
[1]com.solarmetric.datasource.PreparedStatementCache$CacheAwareConnection.prepa
reStatement (PreparedStatementCache.java:184)
[2] com.solarmetric.datasource.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:377)
[3]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.prepareStatementInter
nal (SQLExecutionManagerImpl.java:807)
[4]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executePreparedQueryI
nternal (SQLExecutionManagerImpl.java:761)
[5]com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQueryInternal
(SQLExecutionManagerImpl.java:691)
[6] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQuery(SQLExecutionManagerImpl.java:372)
[7] com.solarmetric.kodo.impl.jdbc.SQLExecutionManagerImpl.executeQuery(SQLExecutionManagerImpl.java:356)
[8] com.solarmetric.kodo.impl.jdbc.ormapping.ClassMapping.loadByPK(ClassMapping.java:950)
[9] com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.initialize(JDBCStoreManager.java:263)
[10] com.solarmetric.kodo.runtime.StateManagerImpl.loadInitialState(StateManagerImpl.java:174)
[11]com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectByIdFilter
(PersistenceManagerImpl.java:1,023)
[12] com.solarmetric.kodo.runtime.PersistenceManagerImpl.getObjectById(PersistenceManagerImpl.java:942)
[13] com.solarmetric.kodo.impl.jdbc.ormapping.OneToOneMapping.load(OneToOneMapping.java:147)
[14] com.solarmetric.kodo.impl.jdbc.runtime.JDBCStoreManager.load(JDBCStoreManager.java:375)
[15] com.solarmetric.kodo.runtime.StateManagerImpl.loadField(StateManagerImpl.java:2,035)
[16] com.solarmetric.kodo.runtime.StateManagerImpl.isLoaded(StateManagerImpl.java:720)
[17] com.europeasap.data.CityMarkup.jdoGetcity (null)
[18] com.europeasap.data.CityMarkup.getCity (CityMarkup.java:30)
[19] com.europeasap.logic.CostHelper.getCityMarkup (CostHelper.java:81)
[20] com.europeasap.logic.CostHelper.computeBasePackageCost(CostHelper.java:289)
[21] com.europeasap.logic.CostHelper.computeFinalPackageCost(CostHelper.java:246)
[22] com.europeasap.form.CustomerBookingForm.updateDisplayCosts(CustomerBookingForm.java:1,440)
[23] com.europeasap.form.CustomerBookingForm.updateCustomizeDisplayInfo(CustomerBookingForm.java:1,407)
[24] com.europeasap.action.CustomizeTrip.perform (CustomizeTrip.java:68)
[25] org.apache.struts.action.ActionServlet.processActionPerform(ActionServlet.java:1,787)
[26] org.apache.struts.action.ActionServlet.process(ActionServlet.java:1,586) [27]
org.apache.struts.action.ActionServlet.doPost (ActionServlet.java:510)
[28] javax.servlet.http.HttpServlet.service (HttpServlet.java:760)
[29] javax.servlet.http.HttpServlet.service (HttpServlet.java:853)
[30] org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
[31] org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
[32] org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
[33]org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext (StandardPipeline.java:643)
[34] org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
[35] org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
[36] org.apache.catal