ConnectionPool problems with WLS 7.0 and Oracle 9.2

Question

Hi,
We are using WLS 7.0 SP4, and Oracle 9 and the Oracle thin driver type 4. In our
application on the productive system (and only there) we constantly encounter
a whole set of SQLExceptions which have all in common that the Connection from
the pool is not valid any more when the application tries to use it.
Typical, recurring error messages are:
- Exhausted ResultSet
- Connection has already been closed
- Closed Statement
- Transaction is no longer active - status committing
- NullPointerException at
weblogic.jdbc.pool.Connection.prepareStatement()
There are no special Statements which create these errors. They are spread at
random across practically every query the application creates, and the same queries
sometimes succeed and sometimes fail.
I double and triple checked that all Connections, Statements and ResultSets are
closed immediately after use. As an example, I attached a code snippet and a resulting
StackTrace which.
The problem also seems to occur only with an (unknown) minimum of concurrent usern,
since in the approval tests on an almost identical test system these errors never
occurred.
I also followed the advice from Oracle and installed the latest Oracle JDBC driver
(Oracle 10g) - to no avail.
What else can I do?
Another question: Is it correct that my Oracle JDBC driver is in the application
classpath (via a reference in the Manifest file of the application jar), not in
the system classpath? There has never been a problem with that, but in a Newsgroup
answer from Nov 10, 2003 (subject: "ResultSet closes prematurely"), Joe Weinstein
suggested to "get it listed at the
front of the -classpath argument that the startWebLogic script creates for the
java line that starts the server".
I hesitate to do so, since the driver is in a standard WebApp- directory, WEB-INF/lib.
Is it possible and safe to add a jar located there to the system classpath? If
it is possible, why is it necessary?
Best regards,
Andreas Zehrt
[CodeSnippetsAndStackTraceForConnectionPoolProblem.txt]

Answer 1

Andreas Zehrt wrote:
Hi Joe,
Your hint that there is a threading problem was right:
On further investigation of the code I found out that the class that passes the
Connection to the DAO not only stores it as a member at some point (which is not
a good idea anyway) but is also a singleton - then, of course, it's no surprise
that the Connection gets invalid in a incalculable way when concurrent threads
share it.
The singleton instantiation was not so obvious because the way of instantiation
is controlled by a configuration parameter that can be overridden at different
levels.
I changed it and the productive logfiles indicate that the SQLExceptions related
to that class have disappeared.I am happy to have helped.
So, thanks a lot for the advice.
But I am still wondering why this code has worked for so long a time with WLS
5.1 and Oracle 8 (the system has been productive for over 2 years). Even in the
approval tests with WLS 7.0 and Oracle 9, we did not run into problems, although
it was multi-user environment.Mo idea.
I still believe that there is a difference between WLS 5.1 and 7.0 in the way
it treats pooled Oracle JDBC Connections. I wished both Oracle and Bea could be
a little more explicit about those changes and possible version incompatabilities
beyond the general advice "use the latest thin driver".Though I can think of no change to our pooling which would have had any material
effect in this case, I will certainly do what I can to see that our documentation
is explicit about changes.
Joe
Best regards, Andreas
Joe Weinstein <[email protected]> wrote:
Hi Andreas.
Andreas Zehrt wrote:
Hi,
We are using WLS 7.0 SP4, and Oracle 9 and the Oracle thin driver type4. In our
application on the productive system (and only there) we constantlyencounter
a whole set of SQLExceptions which have all in common that the Connectionfrom
the pool is not valid any more when the application tries to use it.
Typical, recurring error messages are:
- Exhausted ResultSetThat is typically if the statement that created it is either re-executed
or closed.
- Connection has already been closedAs described. If you give a stacktrace, we could make a debug patch which
would show
where it was originally closed.
- Closed Statementsame as above.
- Transaction is no longer active - status committingThat implies your code is obtaining a connection from a transactional
datasource,
and then later trying to use it after the transaction which it was associated
with,
is finished.
- NullPointerException at
weblogic.jdbc.pool.Connection.prepareStatement()Maybe any of the above.
There are no special Statements which create these errors. They arespread at
random across practically every query the application creates, andthe same queries
sometimes succeed and sometimes fail.
I double and triple checked that all Connections, Statements and ResultSetsare
closed immediately after use. As an example, I attached a code snippetand a resulting
StackTrace which.
The problem also seems to occur only with an (unknown) minimum of concurrentusern,
since in the approval tests on an almost identical test system theseerrors never
occurred.
I also followed the advice from Oracle and installed the latest OracleJDBC driver
(Oracle 10g) - to no avail.
What else can I do?
Another question: Is it correct that my Oracle JDBC driver is in theapplication
classpath (via a reference in the Manifest file of the applicationjar), not in
the system classpath? There has never been a problem with that, butin a Newsgroup
answer from Nov 10, 2003 (subject: "ResultSet closes prematurely"),Joe Weinstein
suggested to "get it listed at the
front of the -classpath argument that the startWebLogic script createsfor the
java line that starts the server".
I hesitate to do so, since the driver is in a standard WebApp- directory,WEB-INF/lib.
Is it possible and safe to add a jar located there to the system classpath?If
it is possible, why is it necessary?I was only concerned to ensure we know which driver we are working with.
We also ship
an oracle thin driver, which becomes obsolete soon...
I am concerned that your code creates pool connections to be used later.
The problems
can arise if more than one thread ever gets the same connection, or if
the connection
is used in the same thread, spanning transactions. It does also seem
that there may
be a threading issue, because if two threads each call the code to create
a connection,
and two connections are made, but one over-writes the other, the two
threads can
end up using the same connection, and closing it. The over-written one
never gets closed,
resulting in that leak message you got...
Joe
The Connection parameter is opened by a business component class, ComaServiceProviderClassicImpl.It is propagated through
several classes in the business layer, but not used, until the DAOtakes it to make the query.
So, the Connection is closed where it was opened, not in the DAO class.
public class ConcernDAOImpl extends BaseDAO {
     public Collection getConcernsForIncidents(Connection conn, Collectionincidents)
throws DataAccessException, ConstraintException, ComaParseException{>
sqlMessage.append(")");
String sqlStmt = sqlMessage.toString();
PreparedStatement pstmt = null;
ResultSet rs = null;
try {
pstmt = conn.prepareStatement(sqlStmt);
rs = pstmt.executeQuery();
while (rs != null && rs.next()) {
final Concern concern =
new Concern(DAOUtil.getComaOID(rs, ComaDBNames.KDANR));
concern.setIncidentOID(DAOUtil.getComaOID(rs, ComaDBNames.KDAVGENR));
return concerns;
} catch (SQLException sqle) {
// Wrapps real SQL exception
String[] message = new String[]{sqle.getMessage(), sqlStmt};
throw new DataAccessException(ExpCode.S_ORACLE_SQL, message,
sqle);
} finally {
closeAll(rs, pstmt);
_logger.exitDebug(method);
Here, the Connection is acquired and finally closed
public class ComaServiceProviderClassicImpl {
     public void updateComplaint(
final Request updateRequest,
final ResponseSingleElement response,
final Principal principal)
throws SystemException {
try {
logger.info("updateComplaint", "store incident");
// store the incident in the database
incidentManager.storeIncident(getConnection(), updateIncident);
// reload the incident from Cache and / or the databaseto get the ContactReferences.
Incident returnIncident = incidentManager.loadIncident(//IncidentManager passes the Connection to the DAO
getConnection(), updateIncident.getOID());
} catch (RemoteException rex) {
// remote exceptions
rollbackIfNecessary();
CoreUtils.unwrapRemoteException(rex, logger);
} catch (SystemException e) {
// all other exceptions --> rollback if necessary and rethrow
rollbackIfNecessary();
throw e;
} finally {
removeConnection();
logger.exitDebug("updateComplaint");
This is the resulting StackTrace:
sql exception: [Closed Statement: next] - sql statement: [select *
     at de.deutschepost.ubbrief.coma.persistence.dao.ConcernDAOImpl.getConcernsForIncidents(ConcernDAOImpl.java:363)
     at de.deutschepost.ubbrief.coma.persistence.dao.CachingConcernDAOImpl.getConcernsForIncidents(CachingConcernDAOImpl.java:129)
     at de.deutschepost.ubbrief.coma.persistence.incidentmanager.IncidentManagerImpl.loadConcernStructuresIntoIncidents(IncidentManagerImpl.java:1067)
     at de.deutschepost.ubbrief.coma.persistence.incidentmanager.IncidentManagerImpl.loadStructureForIncident(IncidentManagerImpl.java:320)
     at de.deutschepost.ubbrief.coma.persistence.incidentmanager.IncidentManagerImpl.loadIncidents(IncidentManagerImpl.java:264)
     at de.deutschepost.ubbrief.coma.persistence.taskmanager.TaskManagerImpl.selectTasksForUser(TaskManagerImpl.java:299)
     at de.deutschepost.ubbrief.coma.service.z2.ComaServiceProviderZ2Impl.getTaskList(ComaServiceProviderZ2Impl.java:113)
     at de.deutschepost.ubbrief.coma.service.z2.ComaServiceProviderZ2Bean_1dhrj7_EOImpl.getTaskList(ComaServiceProviderZ2Bean_1dhrj7_EOImpl.java:154)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList.runServiceMethod(CMPGetTaskList.java:64)
     at de.deutschepost.ubbrief.coma.sbbx.sp.BasicMethodProvider.execute(BasicMethodProvider.java:145)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList_9b9mv5_EOImpl.execute(CMPGetTaskList_9b9mv5_EOImpl.java:46)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList_9b9mv5_EOImpl_WLSkel.invoke(UnknownSource)
     at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:159)
     at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:263)
     at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:230)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList_9b9mv5_EOImpl_WLStub.execute(UnknownSource)
     at de.deutschepost.ubbrief.backbone.jazz.impl.core.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:115)
     at de.deutschepost.ubbrief.backbone.common.impl.core.rpc.server.ServerKernelImpl.handleTransportMessage(ServerKernelImpl.java:270)
     at de.deutschepost.ubbrief.backbone.common.impl.core.messaging.MessageTransport.handleMessage(MessageTransport.java:454)
     at de.deutschepost.ubbrief.backbone.common.impl.core.KernelFacade.handleMessage(KernelFacade.java:209)
     at de.deutschepost.ubbrief.backbone.jazz.impl.backbone.BackboneBean.messageArrived(BackboneBean.java:637)
     at de.deutschepost.ubbrief.backbone.jazz.impl.backbone.BackboneBean_ina9d7_ELOImpl.messageArrived(BackboneBean_ina9d7_ELOImpl.java:105)
     at de.deutschepost.ubbrief.backbone.jazz.impl.transport.receive.LocalQueueReceiveBean.deliverMessage(LocalQueueReceiveBean.java:43)
     at de.deutschepost.ubbrief.backbone.jazz.impl.transport.receive.AbstractMessageReceiveBean.onMessage(AbstractMessageReceiveBean.java:127)
     at weblogic.ejb20.internal.MDListener.execute(MDListener.java:377)
     at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:311)
     at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:286)
     at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2351)
     at weblogic.jms.client.JMSSession.execute(JMSSession.java:2267)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:234)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:210)
####<May 26, 2004 12:18:43 PM CEST> <Warning> <JDBC> <S0048016> <REMA20Z><Finalizer> <kernel identity> <> <001074> <A JDBC pool connection leak
was detected. A Connection leak occurs when a connection obtained from
the pool was not closed explicitly by calling close() and then was disposed
by the garbage collector and returned to the connection pool. The following
stack trace at create shows where the leaked connection was created.
Stack trace at connection create:
     at weblogic.jdbc.pool.Connection.<init>(Connection.java:66)
     at weblogic.jdbc.pool.Driver.allocateConnection(Driver.java:294)
     at weblogic.jdbc.pool.Driver.connect(Driver.java:210)
     at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:373)
     at weblogic.jdbc.jts.Driver.connect(Driver.java:129)
     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:287)
     at de.deutschepost.ubbrief.coma.core.ComaComponentImpl.getConnectionFromPool(ComaComponentImpl.java:163)
     at de.deutschepost.ubbrief.coma.core.ComaComponentImpl.getConnectionInternal(ComaComponentImpl.java:135)
     at de.deutschepost.ubbrief.coma.core.ComaComponentImpl.getConnection(ComaComponentImpl.java:99)
     at de.deutschepost.ubbrief.coma.persistence.customermanager.CurryCustomerManagerImpl.findCustomers(CurryCustomerManagerImpl.java:73)
     at de.deutschepost.ubbrief.coma.service.z2.ComaServiceProviderZ2BaseImpl.resolveCustomerInstances(ComaServiceProviderZ2BaseImpl.java:808)
     at de.deutschepost.ubbrief.coma.service.z2.ComaServiceProviderZ2Impl.getTaskList(ComaServiceProviderZ2Impl.java:213)
     at de.deutschepost.ubbrief.coma.service.z2.ComaServiceProviderZ2Bean_1dhrj7_EOImpl.getTaskList(ComaServiceProviderZ2Bean_1dhrj7_EOImpl.java:154)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList.runServiceMethod(CMPGetTaskList.java:64)
     at de.deutschepost.ubbrief.coma.sbbx.sp.BasicMethodProvider.execute(BasicMethodProvider.java:145)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList_9b9mv5_EOImpl.execute(CMPGetTaskList_9b9mv5_EOImpl.java:46)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList_9b9mv5_EOImpl_WLSkel.invoke(UnknownSource)
     at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:159)
     at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:263)
     at weblogic.rmi.cluster.ReplicaAwareRemoteRef.invoke(ReplicaAwareRemoteRef.java:230)
     at de.deutschepost.ubbrief.coma.sbba.z2.CMPGetTaskList_9b9mv5_EOImpl_WLStub.execute(UnknownSource)
     at de.deutschepost.ubbrief.backbone.jazz.impl.core.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:115)
     at de.deutschepost.ubbrief.backbone.common.impl.core.rpc.server.ServerKernelImpl.handleTransportMessage(ServerKernelImpl.java:270)
     at de.deutschepost.ubbrief.backbone.common.impl.core.messaging.MessageTransport.handleMessage(MessageTransport.java:454)
     at de.deutschepost.ubbrief.backbone.common.impl.core.KernelFacade.handleMessage(KernelFacade.java:209)
     at de.deutschepost.ubbrief.backbone.jazz.impl.backbone.BackboneBean.messageArrived(BackboneBean.java:637)
     at de.deutschepost.ubbrief.backbone.jazz.impl.backbone.BackboneBean_ina9d7_ELOImpl.messageArrived(BackboneBean_ina9d7_ELOImpl.java:105)
     at de.deutschepost.ubbrief.backbone.jazz.impl.transport.receive.LocalQueueReceiveBean.deliverMessage(LocalQueueReceiveBean.java:43)
     at de.deutschepost.ubbrief.backbone.jazz.impl.transport.receive.AbstractMessageReceiveBean.onMessage(AbstractMessageReceiveBean.java:127)
     at weblogic.ejb20.internal.MDListener.execute(MDListener.java:377)
     at weblogic.ejb20.internal.MDListener.transactionalOnMessage(MDListener.java:311)
     at weblogic.ejb20.internal.MDListener.onMessage(MDListener.java:286)
     at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:2351)
     at weblogic.jms.client.JMSSession.execute(JMSSession.java:2267)
     at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:234)
     at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:210)