JDBC Threads stuck on reserving connections

Question

Hello,
I am using WLS 10gR3 single server on Solaris 5.10, with Oracle 10g DB that resides on the same server as weblogic and is dedicated to WLS only.
Under load, I see several threads (10-12) that are stuck on :
"[ACTIVE] ExecuteThread: '53' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010968e800 nid=0x9d waiting for monitor entry [0xfffffffd7e0f3000..0xfffffffd7e0ff840]
java.lang.Thread.State: BLOCKED (on object monitor)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:343)
- waiting to lock <0xfffffffdb1ca0940> (a java.util.HashSet)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:106)
at weblogic.jdbc.pool.Driver.connect(Driver.java:149)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:642)
at weblogic.jdbc.jts.Driver.connect(Driver.java:124)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:338)
at com.xx.yy.security.HibernateManagerProvider.getConnection(HibernateManagerProvider.java:69)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:421)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1538)
at org.hibernate.loader.Loader.doQuery(Loader.java:661)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
at org.hibernate.loader.Loader.loadEntity(Loader.java:1851)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48)
"[ACTIVE] ExecuteThread: '42' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010aaf4400 nid=0x92 waiting for monitor entry [0xfffffffd7f6f3000..0xfffffffd7f6ff6c0]
java.lang.Thread.State: BLOCKED (on object monitor)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:343)
- waiting to lock <0xfffffffdb1ca0940> (a java.util.HashSet)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:106)
at weblogic.jdbc.pool.Driver.connect(Driver.java:149)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:642)
at weblogic.jdbc.jts.Driver.connect(Driver.java:124)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:338)
at com.xx.yy.security.HibernateManagerProvider.getConnection(HibernateManagerProvider.java:69)
The thread that holds the lock is :
[ACTIVE] ExecuteThread: '51' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x0000000105ee8800 nid=0x9b runnable [0xfffffffd7e4f4000..0xfffffffd7e4ff740]
java.lang.Thread.State: RUNNABLE
at java.util.AbstractCollection.contains(AbstractCollection.java:89)
at java.util.AbstractCollection.containsAll(AbstractCollection.java:278)
at java.util.AbstractSet.equals(AbstractSet.java:78)
at weblogic.security.acl.internal.AuthenticatedSubject.equals(AuthenticatedSubject.java:428)
at java.util.HashMap.getEntry(HashMap.java:349)
at java.util.HashMap.containsKey(HashMap.java:335)
at java.util.HashSet.contains(HashSet.java:184)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:343)
- locked <0xfffffffdb1ca0940> (a java.util.HashSet)
at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:106)
at weblogic.jdbc.pool.Driver.connect(Driver.java:149)
at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:642)
at weblogic.jdbc.jts.Driver.connect(Driver.java:124)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:338)
at com.xx.yy.security.HibernateManagerProvider.getConnection(HibernateManagerProvider.java:69)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:421)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1538)
at org.hibernate.loader.Loader.doQuery(Loader.java:661)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
at org.hibernate.loader.Loader.doList(Loader.java:2211)
at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2127)
at org.hibernate.loader.Loader.list(Loader.java:2087)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1569)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283)
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:305)
at com.xx.yy.db.HibernateTemplateEx$10.doInHibernate(HibernateTemplateEx.java:537)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:369)
*The datasource is configured within the EAR with :*
<?xml version="1.0" encoding="UTF-8"?>
<jdbc-data-source
xmlns="http://www.bea.com/ns/weblogic/jdbc-data-source"
xmlns:sec="http://www.bea.com/ns/weblogic/90/security"
xmlns:wls="http://www.bea.com/ns/weblogic/90/security/wls"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bea.com/ns/weblogic/jdbc-da
ta-source http://www.bea.com/ns/weblogic/jdbc-data-source/1.0/jdbc-data-source.xsd">
<name>OracleDS</name>
<jdbc-driver-params>
<url>jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=thehost)(PORT=1521))(CONNEC
T_DATA=(SERVER=DEDICATED)(SERVICE_NAME=ora10)))</url>
<driver-name>oracle.jdbc.driver.OracleDriver</driver-name>
<properties>
<property>
<name>user</name>
<value>myuser</value>
</property>
</properties>
<password-encrypted>{3DES}nhYBET5FfK9uUUJZNI4pVg==</password-encrypted>
</jdbc-driver-params>
<jdbc-connection-pool-params>
<initial-capacity>300</initial-capacity>
<max-capacity>300</max-capacity>
<capacity-increment>5</capacity-increment>
<test-table-name>SQL SELECT 1 FROM DUAL</test-table-name>
<statement-cache-size>10</statement-cache-size>
<statement-cache-type>LRU</statement-cache-type>
</jdbc-connection-pool-params>
<jdbc-data-source-params>
<jndi-name>OracleDS</jndi-name>
<scope>Global</scope>
<global-transactions-protocol>LoggingLastResource</global-transactions-protocol>
</jdbc-data-source-params>
</jdbc-data-source>
Monitoring the pool I see that barely 50 connections are used at peak load, and there are no waiters for connections nor wait time. DBA says there's no issue on the DB side and there's no limit to the number of connections WLS can make.
I cannot see any reason why a thread would stuck on reserving for the pool, as said before, there are plenty of free connections.
Can anyone advise please ?

Answer 1

Hi Joe and thanks for having a look at this.
During 6 thread dumps, taken at 10 seconds interval, the thread did stay there.
The JVM is SUN's 1.6 64bit that comes with the Solaris Installation, I believe. I'll try to upgrade or switch to Jrockit.