Hung Threads (Toplink 10.1.3)

We are experiencing hung threads (waiting in TopLink code) in our app server, each one consuming a db connection so we eventually run out of db connections and die.
We've got 9 threads waiting over 10 minutes on ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
And 1 thread waiting over 10 minutes on WriteLockManager.acquireLocksForClone(WriteLockManager.java:66)
Thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):
"Servlet.Engine.Transports : 12" daemon prio=5 tid=0x02976988 nid=0x5a waiting on condition [5e57c000..5e57fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
"Servlet.Engine.Transports : 11" daemon prio=5 tid=0x028181a0 nid=0x59 in Object.wait() [5e67c000..5e67fc30]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e5bc300> (a oracle.toplink.internal.helper.ConcurrencyManager)
at java.lang.Object.wait(Object.java:429)
at oracle.toplink.internal.helper.WriteLockManager.acquireLocksForClone(WriteLockManager.java:66)
- locked <0x9e5bc300> (a oracle.toplink.internal.helper.ConcurrencyManager)
at oracle.toplink.publicinterface.UnitOfWork.cloneAndRegisterObject(UnitOfWork.java:715)
at oracle.toplink.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:154)
at oracle.toplink.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:99)
at oracle.toplink.internal.sessions.IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:257)
at oracle.toplink.publicinterface.UnitOfWork.registerExistingObject(UnitOfWork.java:3465)
- locked <0x9e97ac68> (a oracle.toplink.internal.sessions.UnitOfWorkImpl)
at oracle.toplink.publicinterface.UnitOfWork.registerExistingObject(UnitOfWork.java:3427)
- locked <0x9e97ac68> (a oracle.toplink.internal.sessions.UnitOfWorkImpl)
at oracle.toplink.queryframework.ObjectLevelReadQuery.registerIndividualResult(ObjectLevelReadQuery.java:1717)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:449)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:413)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:376)
at oracle.toplink.queryframework.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:451)
at oracle.toplink.queryframework.ObjectLevelReadQuery.registerIndividualResult(ObjectLevelReadQuery.java:1701)
at oracle.toplink.queryframework.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:686)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:466)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:825)
at oracle.toplink.publicinterface.UnitOfWork.internalExecuteQuery(UnitOfWork.java:2532)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingUowOld(BaseDAO.java:256)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOsForUpdate(BaseDAO.java:485)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOByPK(BaseDAO.java:291)
at com.saic.ct.sys.persistence.schedule.ScheduleDAO.findWorkGroupByPK(ScheduleDAO.java:1248)
"Servlet.Engine.Transports : 10" daemon prio=5 tid=0x029b0168 nid=0x58 waiting on condition [5e77c000..5e77fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
"Servlet.Engine.Transports : 9" daemon prio=5 tid=0x02c880c8 nid=0x57 waiting on condition [5e87c000..5e87fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
"Servlet.Engine.Transports : 8" daemon prio=5 tid=0x0257e900 nid=0x56 waiting on condition [5ec7c000..5ec7fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
"Servlet.Engine.Transports : 5" daemon prio=5 tid=0x02d99b40 nid=0x53 waiting on condition [5e97c000..5e97fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
"Servlet.Engine.Transports : 3" daemon prio=5 tid=0x02407580 nid=0x4f waiting on condition [5f17c000..5f17fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
"Servlet.Engine.Transports : 2" daemon prio=5 tid=0x027d1bf0 nid=0x49 waiting on condition [5ef7c000..5ef7fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:443)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:413)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:376)
at oracle.toplink.queryframework.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:451)
at oracle.toplink.queryframework.ObjectLevelReadQuery.registerIndividualResult(ObjectLevelReadQuery.java:1701)
at oracle.toplink.queryframework.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:686)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:466)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:825)
at oracle.toplink.publicinterface.UnitOfWork.internalExecuteQuery(UnitOfWork.java:2532)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingUowOld(BaseDAO.java:256)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOsForUpdate(BaseDAO.java:485)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOByPK(BaseDAO.java:291)
at com.saic.ct.sys.persistence.schedule.ScheduleDAO.findWorkGroupByPK(ScheduleDAO.java:1248)
"Servlet.Engine.Transports : 1" daemon prio=5 tid=0x029773e0 nid=0x46 waiting on condition [664fc000..664ffc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
"Servlet.Engine.Transports : 0" daemon prio=5 tid=0x0241ce38 nid=0x44 waiting on condition [5fe7c000..5fe7fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

Zev,
The patch I applied was distributed in the file p5206620_101300_GENERIC.zip and contains the ObjectBuilder and Version classes. I'm pretty sure it is appplied correctly, I will verify again.
This are the Toplink startup logs:
[TopLink Info]: 2006.05.18 02:29:32.737--TopLink, version: Oracle TopLink - 10g Release 3 (10.1.3.0.0) (Build Patch for Bugs 5145690 and 5156075)
[TopLink Info]: 2006.05.18 02:29:32.775--Server: WebLogic Server 8.1 SP4 Mon Nov 29 16:21:29 PST 2004 471647
[TopLink Config]: 2006.05.18 02:29:32.807--connecting(DatabaseLogin(
     platform=>Oracle9Platform
     user name=> "epharm_owner"
     connector=>JNDIConnector datasource name=>com/pfizer/pgrd/epharm/ds1
[TopLink Config]: 2006.05.18 02:29:35.390--Connected: jdbc:oracle:thin:@nlsunk770b:1521:clnval3
     User: EPHARM_OWNER
     Database: Oracle Version: Oracle9i Enterprise Edition Release 9.2.0.5.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.5.0 - Production
     Driver: Oracle JDBC driver Version: 9.2.0.4.0
[TopLink Config]: 2006.05.18 02:29:35.392--connecting(DatabaseLogin(
     platform=>Oracle9Platform
     user name=> "epharm_owner"
     connector=>JNDIConnector datasource name=>com/pfizer/pgrd/epharm/ds1
[TopLink Config]: 2006.05.18 02:29:35.405--Connected: jdbc:oracle:thin:@nlsunk770b:1521:clnval3
     User: EPHARM_OWNER
     Database: Oracle Version: Oracle9i Enterprise Edition Release 9.2.0.5.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.5.0 - Production
     Driver: Oracle JDBC driver Version: 9.2.0.4.0
Thomas Buchan

Similar Messages

Hung Threads with multi thread processing

We use TopLink v10.1.3.5
One of our applications uses multi-core processing and when multiple threads try to access lazily loaded 1-m relationships at the same time, the threads just hang forever. There are no errors but the thread dump says it’s waiting on a condition.
We do not use cache Synchronization.
The 1-m relationship is privately owned, uses Batch reading and Indirection.
The problem occurs intermittently and we cannot reproduce it at will.
Found this discussion, but since we are on 10.3.5, I think we should already have the patch?
Hung Threads (Toplink 10.1.3)
Can anyone please provide any help on this.
Thanks for the help.
Thread dump -
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:167(Compiled Code))
at oracle/toplink/internal/helper/ConcurrencyManager.acquire(ConcurrencyManager.java:76(Compiled Code))
at oracle/toplink/internal/identitymaps/CacheKey.acquire(CacheKey.java:85(Compiled Code))
at oracle/toplink/internal/identitymaps/IdentityMap.acquireLock(IdentityMap.java:85(Compiled Code))
at oracle/toplink/internal/identitymaps/IdentityMapManager.acquireLock(IdentityMapManager.java:101(Compiled Code))
at oracle/toplink/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:68(Compiled Code))
at oracle/toplink/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:58(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:502(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:382(Compiled Code))
at oracle/toplink/mappings/OneToOneMapping.valueFromRow(OneToOneMapping.java:1020(Compiled Code))
at oracle/toplink/mappings/DatabaseMapping.readFromRowIntoObject(DatabaseMapping.java:1045(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:245(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:564(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:382(Compiled Code))
at oracle/toplink/internal/descriptors/ObjectBuilder.buildObjectsInto(ObjectBuilder.java:678(Compiled Code))
at oracle/toplink/internal/queryframework/DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:142(Compiled Code))
at oracle/toplink/queryframework/ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:483(Compiled Code))
at oracle/toplink/queryframework/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:813(Compiled Code))
at oracle/toplink/queryframework/DatabaseQuery.execute(DatabaseQuery.java:620(Compiled Code))
at oracle/toplink/queryframework/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:781(Compiled Code))
at oracle/toplink/queryframework/ReadAllQuery.execute(ReadAllQuery.java:451(Compiled Code))
at oracle/toplink/publicinterface/Session.internalExecuteQuery(Session.java:2089(Compiled Code))
at oracle/toplink/publicinterface/Session.executeQuery(Session.java:993(Compiled Code))
at oracle/toplink/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:62(Compiled Code))
at oracle/toplink/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:55(Compiled Code))
at oracle/toplink/internal/indirection/DatabaseValueHolder.getValue(DatabaseValueHolder.java:61(Compiled Code))
at oracle/toplink/indirection/IndirectList.buildDelegate(IndirectList.java:202(Compiled Code))
at oracle/toplink/indirection/IndirectList.getDelegate(IndirectList.java:359(Compiled Code))
at oracle/toplink/indirection/IndirectList.size(IndirectList.java:703(Compiled Code))

All I can tell from the stack shown is that you are triggering indirection on a collection to an object that has an eagerly fetched 1:1 mapping to an object that is locked in the cache. The thread is waiting on the lock owner to finish building the object before it can be returned.
To tell what is going wrong, you will need to look at the complete thread dump to see what other threads might be building the object with its cache key locked. For instance, a long running query might make it appear the system is hung when it is just a bottleneck in the application. If you can, you should also try to call session.getIdentityMapAccessor().printIdentityMapLocks(); at the time of the problem as it will print off information on the locks such as the object involved and which threads hold them. As for patches; 10.1.3 is an older stream but I am unaware of fixes that are not included in 10.1.3.5 that might resolve this - you should go through support to be sure.
Best Regards,
Chris

OIM 9.1.0.1870.1 with Oracle database 11.1.0.7 - Hung threads

Anyone seen issues like hung threads using OIM9.1.0 with Oracle Database 11.1.0.7?

Thanks guys for the quick responses.
But I have tried both the options for putting the jar in Third Party and also by uploading using the OOTB utility UploadJar.sh.
But it is giving the same error.
I have tired rebouncing the server and also Purge cached, but no success.
Just to mention again, I have tried with all the last 3 postgres JDBC4 driver available on the site (u mentioned).
So, any other clue?
Thanks in advance.

Weblogic Hung threads

What exactly in the WL log files indicate a hung thread? I'm looking to write a script to monitor WL logs and notify by email when a hung thread situation happens.
TIA
Vinayan Menon

Stuck thread. Weblogic logs stuck thread in log file and you can specify stuck thread interval in admin console.
http://e-docs.bea.com/wls/docs92/ConsoleHelp/taskhelp/tuning/TuningExecuteThreads.html
jayesh@yagnasys

Kill/stop hung thread to generate an exception

In production certain requests cause the thread executing them to hang. We are unable to reproduce that problem outside of production. One thought I had was to call stop on the thread executing that request which should cause a ThreadDeath which should be logged by our application so I could hopefully find the source of the problem. I made a way to call stop on a Thread and that seems to cause the Thread to die on my machine but not production. Any suggestions on that tactic or on a better tactic to use?

See http://www.experts-exchange.com/Programming/Programming_Languages/Java/Q_20599112.html for the resolution.

Hung CFDocument threads

Hi! I'm using ColdFusion 7 and I've been experiencing intermittent hung threads related to CFDocument. I finally got around to configuring SeeFusion to give me thread dumps of running threads, and I caught one in action today. The thread had been "running" for a couple of hours, even though the requestTimeout on the page was set to 500 seconds.
Here's the top of the thread dump:
Full thread dump Java HotSpot(TM) Server VM (1.4.2_13):
"jrpp-2833" runnable
        - locked <257> (a java.lang.Object)
        at ice.pilots.html4.CSSMatcher.OEAB(OEAB)
        at ice.pilots.html4.CSSMatcher.I(OEAB)
        at ice.pilots.html4.CSSMatcher.build(OEAB)
        at ice.pilots.html4.CSSLayout._maxCharacters(OEAB)
        at ice.pilots.html4.CSSLayout.S(OEAB)
        at ice.pilots.html4.CSSLayout.layout(OEAB)
        at ice.pilots.html4.ThePrinter.getPageCount(Unknown source)
        at ice.storm.print.AbstractStormPrinter.setStormPageFormat(OEAB)
        at ice.pilots.html4.ThePrinter.setStormPageFormat(Unknown source)
        at com.icesoft.util.PageDecorator.HeaderFooterDecoration.paint(HeaderFooterDecoration.java:2 03)
        at ice.pilots.html4.ThePrinter.printPage(Unknown source)
        at coldfusion.document.DocumentSection.process(DocumentSection.java:225)
Since I can't see the internals of these classes, I can't really troubleshoot any further. Can an Adobe person with access to the source code help to shed some light on what might be going on here?
I will continue to try to reproduce this problem and attempt to boil it down to a simple-ish test case. I just don't know how successful I'll be since it seems to be an intermittent problem.
Oh, and obviously I know that CF 7 is getting pretty long in the tooth, but we can't afford to pay thousands to upgrade right now and then spend hours and hours setting it up, testing, and fixing the inevetible upgrade issues.
Thanks for reading! Hope someone can help...
CF Version:                 7,0,2,142559
Update Level: cfusion.ear/cfusion.war/WEB-INF/cfusion/lib/updates/hf702-70523.jar

As far as I can tell, running the exact same request (that hung before) works for me, and quickly (<10 sec). As I suspected, this problem is intermittent and non-deterministic, so I don't think I'm going to be able to come up with a simplified test case.

JAX-WS Client hangs thread

Hello
I have a web application which consumes a remote web service, it has been in live use for more than a year.
The web service consumed is part of an off the shelf application which fails approx 6 times a year. When it fails, the GlassFish server hosting my web application is also liable to fail. Thus a failure of the purchased application results in the failure of a whole set of other applications.
The Web Service client was written using JAX-WS. When the web service fails, calls to the client are liable to hang and never return. There is no timeout and no error, just a hung thread in GlassFish. In GlassFish the default maximum for request processing threads is 5, if all of them hang Glassfish is dead.
jstack output shows JAX-WS gets stuck, it does a read without a timeout and never regains control. I have seen threads stuck for in excess of 1 hour 45 minutes, they remain stuck until the application server is re-started.
To work around this issue I have written a class to predict if a connection is likely to be healthy or not, if not it attempts to replace the connection with a new one. I am reasonably confident this will reduce the likelihood of failures but I believe there must be a better way to avoid the overhead of getting a new connection every time.
1) Has anybody else suffered this same issue, if so how did you get round it ?
2) Does anybody agree with me that the reads done by JAX-WS should timeout if necessary and failure to do so is a weakness/bug in the JAX-WS client ?
What follows is an example of jstack output showing a hung thread.
----------------- t@162 -----------------
0xff2cc360     _read + 0x8
0xfead8e18     JVM_Read + 0xe4
0xfadbc154     Java_java_net_SocketInputStream_socketRead0 + 0x1fc
0xf8c0c280     * java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) bci:-1830175952 (Interpreted frame)
0xf8c0c224     * java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) bci:0 (Interpreted frame)
0xf8c058b8     * java.net.SocketInputStream.read(byte[], int, int) bci:84 line:129 (Interpreted frame)
0xf8c50588     <C2IAdapter>
0xf949c8fc     0xf949c8fc     * java.io.BufferedInputStream.fill() bci:175 line:218 (Compiled frame)
0xf8fbac24     0xf8fbac24     * java.io.BufferedInputStream.read1(byte[], int, int) bci:44 line:256 (Compiled frame)
0xfa889c6c     0xfa889c6c     * sun.net.www.http.HttpClient.parseHTTPHeader(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) bci:51 line:681 (Compiled frame)
0xf8cbfc44     <I2CAdapter>
0xf8c057a8     * sun.net.www.http.HttpClient.parseHTTP(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) bci:30 line:626 (Interpreted frame)
0xf8c057a8     * sun.net.www.protocol.http.HttpURLConnection.getInputStream() bci:246 line:983 (Interpreted frame)
0xf8c05874     * java.net.HttpURLConnection.getResponseCode() bci:16 line:367 (Interpreted frame)
0xf8c058b8     * com.sun.xml.ws.transport.http.client.HttpClientTransport.checkResponseCode() bci:5 line:201 (Interpreted frame)
0xf8c05764     * com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(com.sun.xml.ws.api.message.Packet) bci:345 line:149 (Interpreted frame)
0xf8c05d3c     * com.sun.xml.xwss.XWSSClientPipe.process(com.sun.xml.ws.api.message.Packet) bci:76 line:118 (Interpreted frame)
0xf8c05d3c     * com.sun.xml.ws.api.pipe.helper.PipeAdapter.processRequest(com.sun.xml.ws.api.message.Packet) bci:6 line:115 (Interpreted frame)
0xf8c52980     <C2IAdapter>
0xfab36e58     0xfab36e58     * com.sun.xml.ws.api.pipe.Fiber.__doRun(com.sun.xml.ws.api.pipe.Tube) bci:206 line:595 (Compiled frame)
0xf8c547bc     <I2CAdapter>
0xf8c05874     * com.sun.xml.ws.api.pipe.Fiber._doRun(com.sun.xml.ws.api.pipe.Tube) bci:31 line:554 (Interpreted frame)
0xf8c05874     * com.sun.xml.ws.api.pipe.Fiber.doRun(com.sun.xml.ws.api.pipe.Tube) bci:84 line:539 (Interpreted frame)
0xf8c05874     * com.sun.xml.ws.api.pipe.Fiber.runSync(com.sun.xml.ws.api.pipe.Tube, com.sun.xml.ws.api.message.Packet) bci:48 line:436 (Interpreted frame)
0xf8c05874     * com.sun.xml.ws.client.Stub.process(com.sun.xml.ws.api.message.Packet, com.sun.xml.ws.client.RequestContext, com.sun.xml.ws.client.ResponseContextReceiver) bci:155 line:248 (Interpreted frame)
0xf8c05874     * com.sun.xml.ws.client.sei.SEIStub.doProcess(com.sun.xml.ws.api.message.Packet, com.sun.xml.ws.client.RequestContext, com.sun.xml.ws.client.ResponseContextReceiver) bci:4 line:135 (Interpreted frame)
0xf8c05874     * com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(java.lang.Object, java.lang.Object[], com.sun.xml.ws.client.RequestContext, com.sun.xml.ws.client.ResponseContextReceiver) bci:64 line:109 (Interpreted frame)
0xf8c05874     * com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(java.lang.Object, java.lang.Object[]) bci:14 line:89 (Interpreted frame)
0xf8c05874     * com.sun.xml.ws.client.sei.SEIStub.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) bci:24 line:118 (Interpreted frame)
0xf8c05d3c     * $Proxy64.updateProblem(com.peregrine.servicecenter.pws.pm.UpdateProblemRequest) bci:16 (Interpreted frame)
0xf8c05d3c     * com.reuters.gsm.sf.consumePeregWs.PeregPm.addUpdate(java.lang.String, int, int, java.lang.String, java.lang.String, java.util.Date) bci:341 line:129 (Interpreted frame)
Regards, Vince

[http://stackoverflow.com/questions/2592303/how-do-i-set-the-jax-ws-client-request-timeout-programatically-on-jboss]

Toplink 10g to 11g migration advice.

We have application that was completely developed JDeveloper 10.1.3/4 with Toplink, JSF, ADF, and Session Facade; extremely similar to the SRDemo.
I want to upgrade to JDeveloper 11g, and will probably just let JDeveloper to automatically migrate the 10g application to 11g.
Does anyone have any tips or lessons learned concerning Toplink and/or ADF when using the 11g migration process?
Thank you.

I downloaded and installed the jdevstudio11111install.exe from the Jdeveloper download page.
While performing the migration, I received a Toplink exception (matches this thread: [TopLink exception when migrating from JDev 10.1.3.4 to 11g|http://forums.oracle.com/forums/thread.jspa?messageID=3593192&#3593192] ).
I assumed this was the bug you were referencing, was that incorrect? Did I inadvertently download the wrong version?
Studio Edition Version 11.1.1.1.0
Build JDEVADF_11.1.1.1.0_GENERIC_090615.0017.5407
Am I missing a patch? Can I move forward with the migration, or is this a show stopper?
Thank you.

Toplink....lazy fetching not working

Case description is as follows:
PollDio has a many-to-one relation with UserDio
UserDio has many-to-many relation with ReviewDio
When I try to reterive a PollDio, data from UserDio and ReviewDio are also fetched. I want data from PollDio only. To avoid the UserDio and Review entity, I used Lazy fetching.
I get the following exception when I try to run the program:
Exception [TOPLINK-60] (Oracle TopLink Essentials - 2.0 (Build b58a-rc2 (08/09/2007))): oracle.toplink.essentials.exceptions.DescriptorException Exception Description: The method [_toplink_setuser_vh] or [_toplink_getuser_vh] is not defined in the object [com.project.web122.persistence.PollDio]. Internal Exception: java.lang.NoSuchMethodException: com.project.web122.persistence.PollDio._toplink_getuser_vh() Mapping: oracle.toplink.essentials.mappings.OneToOneMapping[user] Descriptor: RelationalDescriptor(com.project.web122.persistence.PollDio --> [DatabaseTable(POLLS)])
Runtime Exceptions:
Am using the following:
toplink-essentials.jar - Implementation-Version: 2.0-b58a-rc2 (08/09/2007)
toplink-essentials-agent.jar - Implementation-Version: 2.0-b41-beta2 (03/30/2007)
The entity classes have the following entries:
@Entity(name = "PollDio")
public class PollDio implements Serializable
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name= "UID",nullable=false,referencedColumnName="ID")
public UserDio getUser()
return user;
@Entity(name = "UserDio")
public class UserDio
@ManyToMany(mappedBy = "users", fetch=FetchType.LAZY)
public Set<ReviewDio> getReviews()
return reviews;
@OneToMany(mappedBy="user", fetch=FetchType.LAZY)
public Set<PollDio> getPolls()
return polls;
@Entity
public class ReviewDio
@ManyToMany(targetEntity = UserDio.class)
@JoinTable(name = "REVIEW_USER", joinColumns = @JoinColumn(name = "REVIEW_ID", referencedColumnName = "ID"), inverseJoinColumns = @JoinColumn(name = "USER_ID", referencedColumnName = "ID"))
public Set<UserDio> getUsers()
return users;
The persistence.xml file has the following entries:
<provider>oracle.toplink.essentials.ejb.cmp3.EntityManagerFactoryProvider</provider>
<property name="toplink.logging.level" value="FINE"/>
<property name="toplink.weaving" value="true"/>
Please let me know, what rectification is required to make this work.
Thanks

Hello,
From your posting, I assume you believe this is related to the problem described in thread Toplink JPA... this should lazy load? . If you are running outside of a container, you need to list the Entity classes in the persistence.xml using the <class> tag for the provider to be able to load/weave them as apart of the persistence unit. This will also include non-entity classes such as MappedSuperclass and embedded classes.
If this isn't the issue you are running into, please provide the full stacktrace, the environment you are running in, and how you are using the agent for weaving.
Best Regards,
Chris

Exchange connections seem to get hung

For work I have to connect to two Exchange servers and then I have other accounts for personal mail. It seems that when I first open Mail.app, things seem to work pretty OK, but if I move to a new location Mail.app sits at "downloading" or "checking for new mail" forever with no updates. It almost feels as though the the connection doesn't get handed off properly from one network to the next and I wind up with a hung thread/process. Anyone else seeing this or anything like it? I also seem to experience significant heat and short battery life.
Any help would be appreciated.

Does it help if you change the download location?
Firefox may be trying to copy the file from the temp or cache folder to the finale destination folder and mat also try to copy/set the permissions of the file.
It is also possible that anti-virus software is corrupting downloaded files or is otherwise interfering with downloading files.<br />
Try to disable the real-time (live) scanning of files in your anti-virus software temporarily to see if that makes downloading work.
See:
*http://kb.mozillazine.org/Unable_to_save_or_download_files
*browser.download.manager.scanWhenDone

GW2012 PO problems after upgrade

Last weekend I finished off the upgrade of our Primary Domain and Post Offices to GW2012 (OES11) from GW8 (NW8). The MTA and 4 remote site post offices had been done prior and had mostly been working fine - segfault issue with one PO. The final 3 (and largest) post offices were done over the weekend. The MTA and these three post offices are clustered on VMWare 4.1. The remote sites are on physical servers. All are OES11 using NSS for the GroupWise volume.
Prior to the post offices being moved, the MTA had not skipped a beat and the two cluster servers behaved. Once the PO's were loaded there have constant lock ups/crashes and subsequent DB corruptions. This occurs after the POA's are loaded for the two largest PO's. Using TOP I noticed that the load average would creep up and up until the whole system had a heart attack.
I am only just learning to use Linux, so my investigation skills are fairly limited, though I was able to figure out it has something to do with the quickfinder indexing. Once I turned it off, the load average has stayed down and the post offices seem stable for the moment.
I was thinking of doing a top down rebuild and unless i'm given a very good reason why not too, i'm going to leave the quickfinder indexing off until the school holidays when I can investigate further.
I suppose what I'm asking is what more do you suggest I can do to stabilise the system? What should I do regarding the quickfinder indexing?
A few other issues we've noted that we hope someone might be able to shed some lights on
* Many staff members have been unable to access their mailbox as their password seems to have been deleted/reset.
* One post offices http monitor does not have many of the hyperlinks for settings? I.E: if I go to MTP Status, neither Send or Recieve can be clicked on to change the address
* Web Access (GW 8 Secondary domain) has started maxing out it's C/S settings and needs to be bounced. This has never been an issue until Tuesday (Monday was a public holiday - long live the Queen)
Thanks is advance for any help
Cheers
Emily

On 21.06.2012 07:36, emilym wrote:
>
> Hi Andy
>
> Thanks for the reply, sorry it's taken so long to respond, sick email,
> sick kids and sick me don't make for a fast response to anything.
>
> Many things have happened since last week so i'll try and cover it all
>
> First in answer to your questions
> *The VM's originally had 2 cores and 4Gb RAM (they now have 8Gb)
> *Novell Cluster Services is involved
> *The vNIC is listed as 'flexible' in vSphere, Yast lists it as PCnet -
> Fast 79C971, using the pcnet32 driver
> *And yes 'noatime' and 'no salvage' were set
> *The PO's are approximately 250Gb, 300Gb and 35Gb
>
> Since my first message, we've come to the conclusion that it's not the
> quickfinder indexing causing the problem, as the PO's crashed again on
> the Friday, and Saturday, and Tuesday. As of this time I have not turned
> it back on.
>
> After much work over the weekend I had thought the problem was related
> to the GW requests not being serviced quick enough by the disk, causing
> the corruptions, but that has also since been discounted (using nsscon
> /ZLSSIOStatus)
>
>
> SYMPTOMS/ISSUES/OBSERVATIONS
>
> *GroupWise will hang on a C/S thread, this causes the PO/POA to stop
> servicing new requests. Additionally the server will continue to consume
> resources (Ram) until the server becomes unresponsive.
>
> *The cluster server is unable to unload the shared disk this causes
> automatic migration to fail. Further investigation revealed that the
> eDirectory process (ndsd) was still using the resource.
>
> *Handler threads were being created with file names that would not be
> expected. From previous experience the threads file name is created as a
> function of the current unix time (epoch + seconds) converted into
> hexadecimal format, 1 witnessed hung thread had a name of 5fe0a58e which
> places it somewhere in 2020
>
> *Multiple C022 errors on user databases and msg databases, especailly
> after a crash - these settle down eventually
>
> *Multiple DF2A errors - these settle down eventually
>
> *'Possibly damaged blob in database' messages
>
> *After a first crash, the server would be powered off and on, when
> everything restarts, load quickly increases until it crashes again,
> after another power off/on and deleting the ngwcheck.db before starting
> the groupwise system, things will settle down - C022 errors show up at
> this point
>
> *Monitoring with TOP shows numerous references to gwpoa (100's), even
> though there is only three poa's running. When I query the PID numbers,
> nothing is returned besides the three running instances- this is
> confusing me.
>
> *If I try and gracefully bring down the PO's before they fail, they
> won't unload as they can't close the stuck threads. All I can do is pull
> the virtual power plug out of the wall!
>
>
> WHAT'S BEEN TRIED/DONE SO FAR
>
> *Removed all scheduled maintenance jobs for the PO's
>
> *Disabled automatic db recovery and only run the standalone GWCheck
> when I see db errors. I have since re-enabled the automatic db recovery.
> We only run GWCheck from the stand alone version if run manually
>
> *Given the VM's more RAM
>
> *Double & triple checked the DC files are the correct versions
>
> *We are in the process of installing all available patches for the OS.
> So far one server's been done, the other will be done later today. (This
> produced an interesting problem of the servers having two versions of
> NDSBase installed)
I'm not sure what you checked with the zlssiostatus, but *all* your
symptoms, from a to z sound *exactly* like how a severe I/O crippled GW
system behaves. E.G, this is exactly how a GW system "runs" (or better,
runs not), when the diks system is too slow for GW. It will slowly but
surely run into a deadlock situation that will couse precisely all your
symptoms (memory usage rises, coXX errors, inability to cleanly
dismount/deactivate pools or volumes, c/s threads maxing out, all these
sort of things.)
So, what *is* your storage behind all this?
CU,
Massimo Rosen
Novell Knowledge Partner
No emails please!
http://www.cfc-it.de

Exclamation: "This is impossible!"

We have a large site with lots of traffic.
I have reieved a number of errors in our log files
EX: Application:The request has exceeded the allowable time
limit Tag: CFSTOREDPROC
The only problem is, according to this document.
http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=tn_19438
The CFSTOREDPROC is an exception to the timeout rule.
Document snippet:
Tag exceptions:
There are certain tags that will not "obey" a timeout,
whether it's set in the ColdFusion Administrator or in code on the
CFM page. These tags are cfstoredproc,cfcontent, cfftp, cfexecute
and cfobject. Also, CFX tags fall into the category of not obeying
the ColdFusion
Has anyone else seen this issue?

quote:
Originally posted by:
MikerRoo
That technote does not mean that you will not see that error
in the logs.
It's just warning about cases when you
could see a hung thread rather than the timeout you would
otherwise expect.
Your logs mean that the CFSTOREDPROC took to long but did
eventually return.
If that is just a warning then CFMX/JRUN should be more
explicit about the so called 'exception' to the rule.

Stateless Session EJB hangs using URLConnection but WLS doesn't clean up

Hi
We have a stateless session EJB running under WLS 5.1 with service
pack 10 on Solaris.
The bean calls a remote HTTP server using the java.net.URLConnection
class and forwards the response to the EJB client. The bean is largely
working fine but some threads hang waiting on the HTTP response. Debug
statements, which are written immediately after the response has been
read and the connection has been closed, do not appear in our log for
the hung threads. The WebLogic Console displays these threads as "in
use" and a "netstat -an" displays the tcp connections as ESTABLISHED.
However, the access logs of the remote Apache server show the HTTP
connections of the threads in question completed successfully with
HTTP code 200. The Apache server is using keep-alive connections.
Some EJB threads are still waiting for something it seems.
Has anyody else experienced this when using URLConnection from
stateless session EJBs under WLS?
The second problem is why doesn't WLS time these threads out after
trans-timeout-seconds (we're using the default of 300 seconds)? The
WLS log shows no error messages relating to this problem.
I'm grateful for any info offered.
Thanks in advance
Steve

If you suspect that WLS protocol handler is at fault (and quite often it is),
one thing to try is (if you use Sun's JVM) to use Sun's HTTP protocol handler
instead of WLS (the most common symptom is when code which makes HTTP requests
works fine outside of WebLogic and you have problems getting it to work inside
WebLogic) :
replace
URL url = new URL("http://...");
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
with
URL url = new URL(null, "http://...", new sun.net.www.protocol.http.Handler());
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
You will have to edit weblogic.policy to allow your code to specify protocol
handler.
Also note that transaction timeout is only checked on method boundaries, or
when your code attempts to do something with the database - it is not going to
interrupt thread which is waiting for HTTP response.
Steve Lock <[email protected]> wrote:
Hi
Thanks for the info. The remote HTTP server's access log shows that
the requests were successfully processed. Doesn't this mean that the
connection is then closed? I know the web server is using keep-alive
connections but I thought this was transparent to the client...?
Also why doesn't WLS remove the hung threads?
Steve
"Ignacio G. Dupont" <[email protected]> wrote in message news:<[email protected]>...
We have had a problem like yours with Weblogic 6.1 SP2 on Linux
The problem is sun's implementation of the HTTP connections doesn't have a
timeout, so if the other peer doesn't close the connection your threads will
be "locked" on the connection.
We have found searching the web that the Jakarta project has a package
called Jakarta commons that implements HTTP connections with an
setSoTimeout(int timeout) method so you can open the connections with your
desired timeout. You have to download the code from the CVS as the released
version doesn't support the timedout sockets yet.
When support for the JDK 1.4 version will be announced by Bea you could use
one of its new features that will allow you to pass arguments to the JVM to
specify the maximum socket connection stablising timeout and the max
inactivity on the socket too.
Hope it helps you.
Dimitri

Agent has disabled full monitoring of this target. Only target Status is being monitored.

The following warning is displayed when going through the following:
Database instance (target type)
Oracle Database > Monitoring > All Metrics
"Agent has disabled full monitoring of this target. Only target Status is being monitored."
Triggered by the following:
Summary
Due to 5 hung threads, the target is having all collections disabled
Internal Event Name
CollectionsDisabled:HungThreads
Event Type
Target Monitoring Disruption
Category
Unclassified
Following actions executed to attempt resolution:
+ Database has been bounced, error is resolved
+ Clearstate agent
+ re-enable all collections manually: Metric and Collection Settings > collection schedule column; enable
+ Bounce agent
OMS Version: 12.1.0.4.0
Agent Version: 12.1.0.4.0
Anybody have any ideas on how to clear this error?

emctl clearstate agent TARGETNAME,TARGETTYPE
emctl status agent scheduler
emctl status agent target <Instance_name>,oracle_database
emctl status agent dbconnections
Try to colelct the particulare metric
emctl control agent runCollection TARGETNAME:TARGETTYPE COLLECTIONITEM
Finally, Observer the error message any on the log
agent_inst/sysman/log/gcagent.log

Cold Fusion MX 7 MultiServer Cluster Question

Hey all,
I've configured 2 Cold Fusion MX 7 servers running 2
instances each. I've configured a cluster between them and have the
Proxy Listener up and running.
When I access the page, everything comes up just fine... but
when I round robin between them, I'm losing my session.
I've enabled J2EE Session variables, I enabled the load
balancing of sessions in the JRun admin console on the machines.
What am I doing wrong? I know the session is being written
the machine(s) because I can see them in the directories on the
servers.

We've been having a similar issue, with occasional pages
hanging for hundreds of seconds at that same
JrppBufferedOutputStream, which SeeFusion identifies as "Writing
completed page back to the Web server".
This happens randomly throughout the day, more often when
there's more traffic, but also in the wee hours with relatively few
visitors; nothing else appears to be going on, often with the
long-running thread the only active thread.
Our setup is pretty similar to yours, CFMX7, MSSQL2000,
Win2k3 Server, but with dual Xeons instead of single-CPU. We've
applied the significant hotfixes, including the DST update to the
1.4.2_11 JVM. I'm not positive we've updated to the latest IIS
connector (as mentioned in
http://www.adobe.com/support/security/bulletins/apsb07-02.html),
but will be checking, just to be sure. If you also don't have the
newest connector installed, that would be an interesting
commonality.
We had problems in the past with CF locking up completely
during one of these hung threads, but after some tweaking to
stabilize memory usage below 80%, the lockups are rare.
So I have no answer, but will be sure to share any solution
we do find with you, and hope you'll do the same! :*)

Hung Threads (Toplink 10.1.3)

Similar Messages

Maybe you are looking for