Query result closing prematurely, Object closed error

Question

Hi guys
I've discovered an unusual behaviour when iterating through a collection
returned from a query.
Scenario
The code executes a query, iterates partially through the resultset and for
each row calls a method which creates another object. Given that there
could be thousands of rows, I commit the new objects every n times and then
go back to the start of the loop which re-executes the query. This code runs
fine when
MAX_SCHEDULES_PER_TRANS <= com.solarmetric.kodo.DefaultFetchBatchSize.
Problem
If the DefaultFetchBatchSize is 10, and I commit every 11 iterations, then
during the 3rd outer loop (ie commit has succeeded twice and the query has
been executed 3 times) an exception is thrown when calling events.next() on
the 10th element. The stack trace is included below, but it appears that
the query resultset is somehow being closed early.
If I increase the DefaultFetchBatchSize to 20, then I can also increase my
MAX_SCHEDULES_PER_TRANS to 20 without getting the error.
Any ideas?
Regards
Nathan
Kodo 2.5.3
SQLServer 2000
Code Snippet
private static final int MAX_SCHEDULES_PER_TRANS = 10;
while (!terminated) {
Collection col = (Collection) getQuery().execute(currentDate); //
getQuery() method Creates query object once and reused in subsequent calls
Iterator events = col.iterator();
int counter = 0;
boolean committed = false;
while (events.hasNext() && !committed && !terminated) {
Event event = (Event) events.next();
event.schedule()
counter++;
if ((counter == MAX_SCHEDULES_PER_TRANS ) || !events.hasNext()) {
try {
context.getPersist().commitTransaction();
committed = true;
catch (OptimisticLockException e) {
context.getPersist().rollbackTransaction();
break;
getQuery().close(col);
Kodo.properties
# Kodo JDO Properties configuration
# DEVELOPMENT
com.solarmetric.kodo.LicenseKey: xxxxxxx
com.solarmetric.kodo.ee.ManagedRuntimeProperties=TransactionManagerName=Tran
sactionFactory
javax.jdo.PersistenceManagerFactoryClass=com.solarmetric.kodo.impl.jdbc.JDBC
PersistenceManagerFactory
javax.jdo.option.ConnectionDriverName=com.microsoft.jdbc.sqlserver.SQLServer
Driver
javax.jdo.option.ConnectionUserName=xxxxx
javax.jdo.option.ConnectionPassword=xxxx
javax.jdo.option.ConnectionURL=jdbc:microsoft:sqlserver://devsnetsql01:1433;
SelectMethod=cursor;DatabaseName=devsnet09
javax.jdo.option.RetainValues=true
javax.jdo.option.RestoreValues=true
javax.jdo.option.Optimistic=true
javax.jdo.option.NontransactionalWrite=false
javax.jdo.option.NontransactionalRead=true
javax.jdo.option.Multithreaded=true
javax.jdo.option.MsWait=5000
javax.jdo.option.MinPool=0
javax.jdo.option.MaxPool=0
javax.jdo.option.IgnoreCache=false
com.solarmetric.kodo.FlushBeforeQueries=with-connection
com.solarmetric.kodo.impl.jdbc.schema.DBSequenceFactory=20
com.solarmetric.kodo.impl.jdbc.StatementCacheMaxSize=0
com.solarmetric.kodo.impl.jdbc.WarnOnPersistentTypeFailure=true
com.solarmetric.kodo.impl.jdbc.SequenceFactoryClass=com.solarmetric.kodo.imp
l.jdbc.schema.DBSequenceFactory
com.solarmetric.kodo.impl.jdbc.FlatInheritanceMapping=true
com.solarmetric.kodo.impl.jdbc.AutoReturnTimeout=10
com.solarmetric.kodo.Logger=stdout
com.solarmetric.kodo.EnableQueryExtensions=true
com.solarmetric.kodo.DefaultFetchThreshold=30
com.solarmetric.kodo.DefaultFetchBatchSize=10
com.solarmetric.kodo.impl.jdbc.TransactionIsolation=READ_COMMITTED
com.solarmetric.kodo.ee.ManagedRuntimeProperties=TransactionManagerMethod=co
m.ibm.ejs.jts.jta.TransactionManagerFactory.getTransactionManager
com.solarmetric.kodo.ee.ManagedRuntimeClass=com.solarmetric.kodo.ee.Invocati
onManagedRuntime
Stack trace
com.solarmetric.kodo.runtime.DataStoreException: java.sql.SQLException:
[Microsoft][SQLServer 2000 Driver for JDBC]Object has been closed.
[code=0;state=HY000]
NestedThrowables:
java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Object has
been closed.
at
com.solarmetric.kodo.impl.jdbc.runtime.SQLExceptions.throwDataStore(SQLExcep
tions.java:64)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.instantiateRow(LazyRes
ultList.java:223)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.get(LazyResultList.jav
a:142)
at java.util.AbstractList$Itr.next(AbstractList.java:431)
at
com.solarmetric.kodo.runtime.objectprovider.ResultListIterator.next(ResultLi
stIterator.java:49)
at com.mallesons.servicenet.ta.shared.BOIterator.next(BOIterator.java:48)
at
com.mallesons.servicenet.ta.event.SchedulerThread.process(Scheduler.java:171
at com.mallesons.servicenet.ta.event.SchedulerThread.run(Scheduler.java:122)
NestedThrowablesStackTrace:
java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Object has
been closed.
at com.microsoft.jdbc.base.BaseExceptions.createException(Unknown
Source)(Inlined Compiled Code)
at com.microsoft.jdbc.base.BaseExceptions.getException(Unknown
Source)(Inlined Compiled Code)
at com.microsoft.jdbc.base.BaseConnection.validateClosedState(Unknown
Source)(Inlined Compiled Code)
at com.microsoft.jdbc.base.BaseStatement.validateClosedState(Unknown
Source)(Inlined Compiled Code)
at com.microsoft.jdbc.base.BaseResultSet.validateClosedState(Unknown
Source)(Compiled Code)
at com.microsoft.jdbc.base.BaseResultSet.getRow(Unknown Source)
at
com.solarmetric.datasource.ResultSetWrapper.getRow(ResultSetWrapper.java:465
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.instantiateRow(LazyRes
ultList.java:186)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.get(LazyResultList.jav
a:142)
at java.util.AbstractList$Itr.next(AbstractList.java:431)
at
com.solarmetric.kodo.runtime.objectprovider.ResultListIterator.next(ResultLi
stIterator.java:49)
at com.mallesons.servicenet.ta.shared.BOIterator.next(BOIterator.java:48)
at
com.mallesons.servicenet.ta.event.SchedulerThread.process(Scheduler.java:171
at com.mallesons.servicenet.ta.event.SchedulerThread.run(Scheduler.java:122)

Answer 1

Thanks for pointing out the optimisticLockException catch error. That would
have been a bug but it hasn't been hit yet as I've only ever run one
instance of this app!!
The BOIterator is just wrapper layer and I have removed it to ensure it is
not the culprit. Still getting the same error.
I have done some more testing and have narrowed down the problem a little.
It seems to only occur when I update one of the columns that is being used
in the query. If I update a column that is not used in the query, it works
fine.
com.solarmetric.kodo.runtime.DataStoreException: java.sql.SQLException:
[Microsoft][SQLServer 2000 Driver for JDBC]Object has been closed.
[code=0;state=HY000]
NestedThrowables:
java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Object has
been closed.
at
com.solarmetric.kodo.impl.jdbc.runtime.SQLExceptions.throwDataStore(SQLExcep
tions.java:64)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.instantiateRow(LazyRes
ultList.java:223)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.get(LazyResultList.jav
a:142)
at java.util.AbstractList$Itr.next(AbstractList.java:431)
at
com.solarmetric.kodo.runtime.objectprovider.ResultListIterator.next(ResultLi
stIterator.java:49)
at
com.mallesons.servicenet.ta.event.SchedulerThread.process(Scheduler.java:197
at com.mallesons.servicenet.ta.event.SchedulerThread.run(Scheduler.java:143)
NestedThrowablesStackTrace:
java.sql.SQLException: [Microsoft][SQLServer 2000 Driver for JDBC]Object has
been closed.
at com.microsoft.jdbc.base.BaseExceptions.createException(Unknown Source)
at com.microsoft.jdbc.base.BaseExceptions.getException(Unknown Source)
at com.microsoft.jdbc.base.BaseConnection.validateClosedState(Unknown
Source)
at com.microsoft.jdbc.base.BaseStatement.validateClosedState(Unknown Source)
at com.microsoft.jdbc.base.BaseResultSet.validateClosedState(Unknown Source)
at com.microsoft.jdbc.base.BaseResultSet.getRow(Unknown Source)
at
com.solarmetric.datasource.ResultSetWrapper.getRow(ResultSetWrapper.java:478
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.instantiateRow(LazyRes
ultList.java:186)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.get(LazyResultList.jav
a:142)
at java.util.AbstractList$Itr.next(AbstractList.java:431)
at
com.solarmetric.kodo.runtime.objectprovider.ResultListIterator.next(ResultLi
stIterator.java:49)
at
com.mallesons.servicenet.ta.event.SchedulerThread.process(Scheduler.java:197
at com.mallesons.servicenet.ta.event.SchedulerThread.run(Scheduler.java:143)
2003-10-27 15:26:40,142 DEBUG - SchedulerThread.run() - exit
"Marc Prud'hommeaux" <[email protected]> wrote in message
news:[email protected]...
Nathan-
I guess you are right that it doesn't look like the query result is
being kept open between commits. I do notice that the block that catches
the OptimisticLockException doesn't break out of the loop though: can
you check to ensure that one isn't getting thrown?
Also, what is the com.mallesons.servicenet.ta.shared.BOIterator stuff?
If it is a wrapper layer that you have around the query execution,
can you try removing it, just to ensure that it isn't misbehaving in
some way?
In article <[email protected]>, nathan boyes wrote:
Hi Marc,
Bug 718 certainly seems similar, however I still don't think it is
necessarily the problem in this case. That particular bug relates to the
resultset that was opened prior to commit, not remaining open after the
commit. However, once I've committed, I call query.closeAll() then re
execute the query. I would expect this to give me a new resultset that
would not be affected by the previous commit.
Also the CURSOR_CLOSE_ON_COMMIT is disabled by default and we haven't
enabled it. A query against the database using one of the SQLServer
functions confirms that this setting is false on the server.
Predictably, instantiating the entire collection also fixes the problem.
I
guess this is similar to setting DefaultFetchThreshold = -1.
Not really sure what else to try.
"Marc Prud'hommeaux" <[email protected]> wrote in message
news:[email protected]...
Nathan-
This sounds to me like
http://bugzilla.solarmetric.com/show_bug.cgi?id=718 , even though it is
not known to happen for SQLServer. If, before you commit, you
instantiate the entire "col" list (with something like
"new LinkedList (col)"), do you still see the problem?
If this is indeed the problem, it seems that this is a server option
that you can configure. If you disable CURSOR_CLOSE_ON_COMMIT on the
server, do you still see the problem? See:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlref/ts_set-set_48kz.asp
>>>
>>>
>>>
>>>
In article <[email protected]>, nathan boyes wrote:
I have upgraded to 2.5.5 and am now calling query.closeAll() neither
of
which had any effect. When I set DefaultFetchThreshold to -1, thenthe
problem disappeared.
For efficiency, if I am iterating through 50 objects beforecommitting
then
it is probably worth setting
DefaultFetchThreshold=50
DefaultFetchBatchSize=50
which will run without problem.
However, that doesn't address the underlying problem that could stilloccur
in other parts of our system.
"Stephen Kim" <[email protected]> wrote in message
news:[email protected]...
First can you upgrade to Kodo 2.5.4 and see if the problem persists?
If so, can you for the sake of debugging try setting
DefaultFetchThreshold to -1?
Also you should be calling either query.closeAll () or query.close(events).
nathan boyes wrote:
Hi guys
I've discovered an unusual behaviour when iterating through a
collection
returned from a query.
Scenario
The code executes a query, iterates partially through the
resultset
and
for
each row calls a method which creates another object. Given that
there
could be thousands of rows, I commit the new objects every n timesand
then
go back to the start of the loop which re-executes the query. This
code
runs
fine when
MAX_SCHEDULES_PER_TRANS <=
com.solarmetric.kodo.DefaultFetchBatchSize.
Problem
If the DefaultFetchBatchSize is 10, and I commit every 11
iterations,
then
during the 3rd outer loop (ie commit has succeeded twice and the
query
has
been executed 3 times) an exception is thrown when calling
events.next()
on
the 10th element. The stack trace is included below, but it
appears
that
the query resultset is somehow being closed early.
If I increase the DefaultFetchBatchSize to 20, then I can also
increase
my
MAX_SCHEDULES_PER_TRANS to 20 without getting the error.
Any ideas?
Regards
Nathan
Kodo 2.5.3
SQLServer 2000
Code Snippet
private static final int MAX_SCHEDULES_PER_TRANS = 10;
while (!terminated) {
Collection col = (Collection) getQuery().execute(currentDate);
getQuery() method Creates query object once and reused insubsequent
calls
Iterator events = col.iterator();
int counter = 0;
boolean committed = false;
while (events.hasNext() && !committed && !terminated) {
Event event = (Event) events.next();
event.schedule()
counter++;
if ((counter == MAX_SCHEDULES_PER_TRANS ) ||
!events.hasNext()) {
>>>>>>
try {
context.getPersist().commitTransaction();
committed = true;
catch (OptimisticLockException e) {
context.getPersist().rollbackTransaction();
break;
getQuery().close(col);
Kodo.properties
# Kodo JDO Properties configuration
# DEVELOPMENT
com.solarmetric.kodo.LicenseKey: xxxxxxx
com.solarmetric.kodo.ee.ManagedRuntimeProperties=TransactionManagerName=Tran
sactionFactory
javax.jdo.PersistenceManagerFactoryClass=com.solarmetric.kodo.impl.jdbc.JDBC
PersistenceManagerFactory
javax.jdo.option.ConnectionDriverName=com.microsoft.jdbc.sqlserver.SQLServer
Driver
javax.jdo.option.ConnectionUserName=xxxxx
javax.jdo.option.ConnectionPassword=xxxx
javax.jdo.option.ConnectionURL=jdbc:microsoft:sqlserver://devsnetsql01:1433;
SelectMethod=cursor;DatabaseName=devsnet09
javax.jdo.option.RetainValues=true
javax.jdo.option.RestoreValues=true
javax.jdo.option.Optimistic=true
javax.jdo.option.NontransactionalWrite=false
javax.jdo.option.NontransactionalRead=true
javax.jdo.option.Multithreaded=true
javax.jdo.option.MsWait=5000
javax.jdo.option.MinPool=0
javax.jdo.option.MaxPool=0
javax.jdo.option.IgnoreCache=false
com.solarmetric.kodo.FlushBeforeQueries=with-connection
com.solarmetric.kodo.impl.jdbc.schema.DBSequenceFactory=20
com.solarmetric.kodo.impl.jdbc.StatementCacheMaxSize=0
com.solarmetric.kodo.impl.jdbc.WarnOnPersistentTypeFailure=true
com.solarmetric.kodo.impl.jdbc.SequenceFactoryClass=com.solarmetric.kodo.imp
l.jdbc.schema.DBSequenceFactory
com.solarmetric.kodo.impl.jdbc.FlatInheritanceMapping=true
com.solarmetric.kodo.impl.jdbc.AutoReturnTimeout=10
com.solarmetric.kodo.Logger=stdout
com.solarmetric.kodo.EnableQueryExtensions=true
com.solarmetric.kodo.DefaultFetchThreshold=30
com.solarmetric.kodo.DefaultFetchBatchSize=10
com.solarmetric.kodo.impl.jdbc.TransactionIsolation=READ_COMMITTED
com.solarmetric.kodo.ee.ManagedRuntimeProperties=TransactionManagerMethod=co
m.ibm.ejs.jts.jta.TransactionManagerFactory.getTransactionManager
com.solarmetric.kodo.ee.ManagedRuntimeClass=com.solarmetric.kodo.ee.Invocati
onManagedRuntime
Stack trace
com.solarmetric.kodo.runtime.DataStoreException:java.sql.SQLException:
[Microsoft][SQLServer 2000 Driver for JDBC]Object has been closed.
[code=0;state=HY000]
NestedThrowables:
java.sql.SQLException: [Microsoft][SQLServer 2000 Driver forJDBCObject
has
been closed.
at
com.solarmetric.kodo.impl.jdbc.runtime.SQLExceptions.throwDataStore(SQLExcep
tions.java:64)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.instantiateRow(LazyRes
ultList.java:223)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.get(LazyResultList.jav
a:142)
at java.util.AbstractList$Itr.next(AbstractList.java:431)
at
com.solarmetric.kodo.runtime.objectprovider.ResultListIterator.next(ResultLi
stIterator.java:49)
atcom.mallesons.servicenet.ta.shared.BOIterator.next(BOIterator.java:48)
>>>>>>
at
com.mallesons.servicenet.ta.event.SchedulerThread.process(Scheduler.java:171
atcom.mallesons.servicenet.ta.event.SchedulerThread.run(Scheduler.java:122)
>>>>>>
NestedThrowablesStackTrace:
java.sql.SQLException: [Microsoft][SQLServer 2000 Driver forJDBCObject
has
been closed.
at com.microsoft.jdbc.base.BaseExceptions.createException(Unknown
Source)(Inlined Compiled Code)
at com.microsoft.jdbc.base.BaseExceptions.getException(Unknown
Source)(Inlined Compiled Code)
at
com.microsoft.jdbc.base.BaseConnection.validateClosedState(Unknown
Source)(Inlined Compiled Code)
atcom.microsoft.jdbc.base.BaseStatement.validateClosedState(Unknown
Source)(Inlined Compiled Code)
atcom.microsoft.jdbc.base.BaseResultSet.validateClosedState(Unknown
Source)(Compiled Code)
at com.microsoft.jdbc.base.BaseResultSet.getRow(Unknown Source)
at
com.solarmetric.datasource.ResultSetWrapper.getRow(ResultSetWrapper.java:465
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.instantiateRow(LazyRes
ultList.java:186)
at
com.solarmetric.kodo.impl.jdbc.runtime.LazyResultList.get(LazyResultList.jav
a:142)
at java.util.AbstractList$Itr.next(AbstractList.java:431)
at
com.solarmetric.kodo.runtime.objectprovider.ResultListIterator.next(ResultLi
stIterator.java:49)
atcom.mallesons.servicenet.ta.shared.BOIterator.next(BOIterator.java:48)
>>>>>>
at
com.mallesons.servicenet.ta.event.SchedulerThread.process(Scheduler.java:171
atcom.mallesons.servicenet.ta.event.SchedulerThread.run(Scheduler.java:122)
>>>>>>
>>>>>>
>>>>>
Stephen Kim
[email protected]
SolarMetric, Inc.
http://www.solarmetric.com
Marc Prud'hommeaux [email protected]
SolarMetric Inc. http://www.solarmetric.com
Marc Prud'hommeaux [email protected]
SolarMetric Inc. http://www.solarmetric.com