Applications waiting on locks 100%

Hello all,
from time to time we get some errors in our bw system. The monitoring tool send the error called "Percentage of the applications that are currently waiting on locks (100%)". For this we´ve changed the parameters but nothing helps.
We´re using db6 9.5 with fixpack 4.
Regards Olaf

Hi Olaf,
Impossible to help you without more info. A few questions:
Which monitoring tool are you talking about?
Which parameters did you change?
Did you check the database snapshot or db2diag.log for lock escalations?
If this is not the case could you please collect more information with db6util ( note 327595 )?
Regards
Frank

Similar Messages

Applications waiting for Locks =2

Hello Team,
In DB01, we have problem.
Agent 1 is waiting on agent 0
Agent 0 is in UOW status = UOW waiting.
Sm21 logs shows :
SQL0911N The current transaction has been rolled back
because of a deadlock or timeout. Reason code "68".
SQLSTATE=40001
lock snapshot is below:
Database Lock Snapshot
Database name                              = BWP
Database path                              = /db2/BWP/db2bwp/NODE0000/SQL00001/
Input database alias                       = BWP
Locks held                                 = 35
Applications currently connected           = 260
Agents currently waiting on locks          = 1
Snapshot timestamp                         = 07/10/2009 05:06:24.893375
Application handle                         = 921
Application ID                             = 131.219.230.201.38271.090517111
Sequence number                            = 00005
Application name                           = disp+work
CONNECT Authorization ID                   = SAPBWP
Application handle                         = 671
Application ID                             = 131.219.230.209.38380.090517102
Sequence number                            = 00050
Application name                           = disp+work
CONNECT Authorization ID                   = SAPBWP
Application status                         = UOW Waiting
Status change time                         = 06/19/2009 11:10:13.197423
Application code page                      = 819
Locks held                                 = 0
Total wait time (ms)                       = 0
Status change time                         = 05/17/2009 04:17:17.525006
Application code page                      = 819
Locks held                                 = 0
Total wait time (ms)                       = 0
Can somebody please let me know how to solve this error.
Best Regards,
Tushar.

What we need is information from the application snapshot.
o Agent pid that is waiting and Agent pid that the first agent is waiting on
o Status of the latter (UOW Executing or UOW Waiting ...)
o Client PID => SAP WP process => determine what the status is of this process
Then, you need to determine activities running / waiting at SAP level, SQL statements running + waiting at DB2 level and determine what caused the problem.
Malte

ORA-02049: timeout: distributed transaction waiting for lock

Hi,
My name is Guneet and I'm working on an application running on BEA Weblogic Server 9.2 running on a Red Hat Linux box using Oracle 10g as the database. My problem is that recently our code started getting the following exception while updating a database table.
java.sql.SQLException: [BEA][Oracle JDBC Driver][Oracle]ORA-02049: timeout: distributed transaction waiting for lock
Application Details
* Using Stateless Session EJB
* Only one Business method in this EJB with transaction attribute set to "required"
* This method executes two select queries & one update query
* We are using JDBC to access the database.
* We have configured a Data Source & are using it to get a database connection.
* Weblogic's Oracle Driver is being used.
More details
* The application has been running well since a month.
* Two days ago,the update query failed with the above error.
* At that time, a single client was accessing the system.
* Once this problem occurs, it starts appearing frequently.
* Eventually a request to get a connection from the Data Source times out & the exception copied at the end is thrown
* At this stage the application gets stuck and all requests trying to get a connection end up with this exception.
* Fortunately, Restarting the Weblogic Server gets us out of this problem and transactions resume normally.
Now my questions are
# Why is this error happening & what does it mean?
# It looks like the second exception (unable to get a connection from ds) is an after effect of the first problem (ORA-02049) once it appears for a couple of times. Can somebody validate this?
# Though I don't understand JTA well but I don't think this application needs distributed transactions so, I'm thinking of modifying the driver type to non-XA oracle driver. Any advise/pointers/comments on this front is welcome !!!!!!!!
Thanks
Guneet Sahai
Exception Trace
Dec 27, 2006 4:47:50 PM | com.gisil.themis.db | SEVERE | Unable to load merchant DEL = 911168900164. Reason - java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'themis-ds': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException: Unexpected error during start for XAResource 'themis-ds': Transaction timed out after 29 seconds
BEA1-252DE51AC930078CA638
at weblogic.jdbc.wrapper.XA.createException(XA.java:103)
at weblogic.jdbc.jta.DataSource.start(DataSource.java:753)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1182)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1115)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:274)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:497)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:429)
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1408)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1332)
at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:440)
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at com.gisil.themis.db.impl1.DbManagerImpl.isPinValid(DbManagerImpl.java:872)
at com.gisil.themis.ejb.ThemisBean.isPinValid(ThemisBean.java:185)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl.isPinValid(Themis_aqqc4k_EOImpl.java:207)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:517)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:407)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:403)
at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:56)
at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:934)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1413)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1332)
at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:440)
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at com.gisil.themis.db.impl1.DbManagerImpl.isPinValid(DbManagerImpl.java:872)
at com.gisil.themis.ejb.ThemisBean.isPinValid(ThemisBean.java:185)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl.isPinValid(Themis_aqqc4k_EOImpl.java:207)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:517)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:407)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:403)
at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:56)
at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:934)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

guneet sahai wrote:
Hi,
My name is Guneet and I'm working on an application running on BEA Weblogic Server 9.2 running on a Red Hat Linux box using Oracle 10g as the database. My problem is that recently our code started getting the following exception while updating a database table.
java.sql.SQLException: [BEA][Oracle JDBC Driver][Oracle]ORA-02049: timeout: distributed transaction waiting for lock
Application Details
* Using Stateless Session EJB
* Only one Business method in this EJB with transaction attribute set to "required"
* This method executes two select queries & one update query
* We are using JDBC to access the database.
* We have configured a Data Source & are using it to get a database connection.
* Weblogic's Oracle Driver is being used.
More details
* The application has been running well since a month.
* Two days ago,the update query failed with the above error.
* At that time, a single client was accessing the system.
* Once this problem occurs, it starts appearing frequently.
* Eventually a request to get a connection from the Data Source times out & the exception copied at the end is thrown
* At this stage the application gets stuck and all requests trying to get a connection end up with this exception.
* Fortunately, Restarting the Weblogic Server gets us out of this problem and transactions resume normally.
Now my questions are
# Why is this error happening & what does it mean?
# It looks like the second exception (unable to get a connection from ds) is an after effect of the first problem (ORA-02049) once it appears for a couple of times. Can somebody validate this?
# Though I don't understand JTA well but I don't think this application needs distributed transactions so, I'm thinking of modifying the driver type to non-XA oracle driver. Any advise/pointers/comments on this front is welcome !!!!!!!!
Thanks
Guneet SahaiHi Guneet. If you want to debug the JTA issue, I suggest opening an official
support case. They will lead you through producing the JTA debug information.
However, I believe you are correct that the transaction you describe is
completely doable with a simple local transaction, so if you were to alter
your pool to use the non-XA driver, it would probably be faster, simpler,
and just work.
Let me know...
Joe
>
Exception Trace
Dec 27, 2006 4:47:50 PM | com.gisil.themis.db | SEVERE | Unable to load merchant DEL = 911168900164. Reason - java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'themis-ds': XAER_RMERR : A resource manager error has occured in the transaction branch
javax.transaction.xa.XAException: Unexpected error during start for XAResource 'themis-ds': Transaction timed out after 29 seconds
BEA1-252DE51AC930078CA638
at weblogic.jdbc.wrapper.XA.createException(XA.java:103)
at weblogic.jdbc.jta.DataSource.start(DataSource.java:753)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1182)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1115)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:274)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:497)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:429)
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1408)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1332)
at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:440)
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at com.gisil.themis.db.impl1.DbManagerImpl.isPinValid(DbManagerImpl.java:872)
at com.gisil.themis.ejb.ThemisBean.isPinValid(ThemisBean.java:185)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl.isPinValid(Themis_aqqc4k_EOImpl.java:207)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:517)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:407)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:403)
at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:56)
at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:934)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1413)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1332)
at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:440)
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at com.gisil.themis.db.impl1.DbManagerImpl.isPinValid(DbManagerImpl.java:872)
at com.gisil.themis.ejb.ThemisBean.isPinValid(ThemisBean.java:185)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl.isPinValid(Themis_aqqc4k_EOImpl.java:207)
at com.gisil.themis.ejb.Themis_aqqc4k_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:517)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:407)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:403)
at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:56)
at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:934)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

Thread waits and locks using Solaris 8 T1 threads

Hi,
I have got a problem relating to Webmethods running on JVM Hot Spot server VM (1.4.2_07-b05 mixed mode).
From time to time the webmethods applications seems to slow down and completely lock up with the mpstat on solaris showing:
1. context switching increases (csw)
2. spin on mutexes also show approximately 25 % increase (smtx)
The only way i could overcome this problem is to kill the process - which incidentally produces a thread dump. Incidentally i have also done a pstack on the process.
The thread dump shows among other things alots of object waits, with locks on objects in an object Pool. The main process is in a waiting state on object, locked by itself, similarly other thread deamon are also in a waiting state on objects which itself has locked. Seems rather wierd (dont understand that).
Anyway we are on Solaris 8 -using the T1 solaris thread library. My question is whether we need to move to T2 solaris thread library and use the option of -XX:-UseLWPSynchronization on JVM would help in the long run. As i am quire new to the JVM Hot Spot (excuse my lack of knowledge in this sphere), i am also posting the thread dump or parts of it here - I am reading them as dead locks but i could be wrong, would also appreciate some help on this matter to clarify what i am reading:
"IS <ENERGYAUST#wm.ip.ebxml.MSH:process>" prio=5 tid=0x00a2e570 nid=0x12c in Object.wait() [63f7e000..63f819c0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.util.pool.ObjectPool.allocate(ObjectPool.java:248)
     - locked <0x97835bf8> (a java.lang.Object)
     at com.wm.util.pool.ObjectPool.allocate(ObjectPool.java:191)
     at com.wm.app.b2b.server.ThreadManager.runTarget(ThreadManager.java:83)
     at com.wm.app.b2b.server.ThreadManager.runTarget(ThreadManager.java:62)
     at com.wm.app.b2b.server.control.ControlledThreadPool.runTarget(ControlledThreadPool.java:359)
     at wm.ip.ebxml.coder.util.getEnvelopeStream(util.java:588)
     at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:324)
     at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:322)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:113)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:635)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:584)
     at com.wm.app.tn.route.RoutingActions.invokeWrapper(RoutingActions.java:959)
     at com.wm.app.tn.route.RoutingActions.invoke(RoutingActions.java:827)
     at wm.tn.route.route(route.java:750)
     at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:324)
     at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:322)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:113)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:635)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:584)
     at wm.ip.ebxml.Receiver.processMessage(Receiver.java:588)
     at wm.ip.ebxml.Receiver.process(Receiver.java:186)
     at wm.ip.ebxml.MSH.process(MSH.java:165)
     at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:324)
     at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:322)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:113)
     at com.wm.app.b2b.server.ServiceThread.run(ServiceThread.java:82)
     - locked <0xbb1b42e0> (a com.wm.app.b2b.server.ServiceThread)
     at com.wm.util.pool.PooledThread.run(PooledThread.java:105)
     - locked <0x9ad44eb8> (a com.wm.app.b2b.server.TMPooledThread)
     at java.lang.Thread.run(Thread.java:534)
"IS <ENERGYAUST#wm.ip.ebxml.MSH:process>" prio=5 tid=0x00a2d980 nid=0x12b in Object.wait() [6407e000..640819c0]
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.util.pool.ObjectPool.allocate(ObjectPool.java:248)
     - locked <0x97835bf8> (a java.lang.Object)
     at com.wm.util.pool.ObjectPool.allocate(ObjectPool.java:191)
     at com.wm.app.b2b.server.ThreadManager.runTarget(ThreadManager.java:83)
     at com.wm.app.b2b.server.ThreadManager.runTarget(ThreadManager.java:62)
     at com.wm.app.b2b.server.control.ControlledThreadPool.runTarget(ControlledThreadPool.java:359)
     at wm.ip.ebxml.coder.util.getEnvelopeStream(util.java:588)
     at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:324)
     at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:322)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:113)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:635)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:584)
     at com.wm.app.tn.route.RoutingActions.invokeWrapper(RoutingActions.java:959)
     at com.wm.app.tn.route.RoutingActions.invoke(RoutingActions.java:827)
     at wm.tn.route.route(route.java:750)
     at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:324)
     at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:322)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:168)
     at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
     at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:581)
     at com.wm.lang.flow.FlowState.step(FlowState.java:438)
     at com.wm.lang.flow.FlowState.invoke(FlowState.java:403)
     at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:982)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:113)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:635)
     at com.wm.app.b2b.server.Service.doInvoke(Service.java:584)
     at wm.ip.ebxml.Receiver.processMessage(Receiver.java:588)
     at wm.ip.ebxml.Receiver.process(Receiver.java:186)
     at wm.ip.ebxml.MSH.process(MSH.java:165)
     at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:324)
     at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:322)
     at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:612)
     at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
     at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:226)
     at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
     at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:198)
     at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:39)
     at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:411)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:521)
     at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:369)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:246)
     at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:113)
     at com.wm.app.b2b.server.ServiceThread.run(ServiceThread.java:82)
     - locked <0xbb1b4410> (a com.wm.app.b2b.server.ServiceThread)
     at com.wm.util.pool.PooledThread.run(PooledThread.java:105)
     - locked <0x9ad44f30> (a com.wm.app.b2b.server.TMPooledThread)
     at java.lang.Thread.run(Thread.java:534)
"AuditRejectHandler" daemon prio=5 tid=0x009b23c8 nid=0x23 in Object.wait() [73681000..736819c0]
     at java.lang.Object.wait(Native Method)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:74)
     - locked <0x97800e18> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.app.store.impl.TSSimpleQueue._waitForMessage(TSSimpleQueue.java:585)
     at com.wm.app.store.impl.TSSimpleQueue.getWait(TSSimpleQueue.java:464)
     at com.wm.app.store.impl.TSSimpleQueueConsumer.getWait(TSSimpleQueueConsumer.java:37)
     at com.wm.app.audit.AuditFactory$AuditRejectHandler.run(AuditFactory.java:1646)
"AuditRetryHandler" daemon prio=5 tid=0x007bc5c8 nid=0x22 in Object.wait() [73781000..737819c0]
     at java.lang.Object.wait(Native Method)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:74)
     - locked <0x97800ed0> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.app.store.impl.TSSimpleQueue._waitForMessage(TSSimpleQueue.java:585)
     at com.wm.app.store.impl.TSSimpleQueue.getWait(TSSimpleQueue.java:464)
     at com.wm.app.store.impl.TSSimpleQueueConsumer.getWait(TSSimpleQueueConsumer.java:37)
     at com.wm.app.audit.AuditFactory$RetryHandler.run(AuditFactory.java:1495)
"AuditExceptionHandler" daemon prio=5 tid=0x009b1e30 nid=0x21 in Object.wait() [73881000..738819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x97800f90> (a com.wm.util.sync.CountingSemaphore)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:71)
     - locked <0x97800f90> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:56)
     - locked <0x97800f90> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.app.audit.AuditFactory$ExceptionHandler.run(AuditFactory.java:1410)
"AuditProducer0" daemon prio=5 tid=0x009bb4e8 nid=0x20 in Object.wait() [73a81000..73a819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x97801000> (a com.wm.util.sync.CountingSemaphore)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:71)
     - locked <0x97801000> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.app.store.impl.TSSimpleQueue._waitForMessage(TSSimpleQueue.java:585)
     at com.wm.app.store.impl.TSSimpleQueue.getWait(TSSimpleQueue.java:464)
     at com.wm.app.store.impl.TSSimpleQueueConsumer.getWait(TSSimpleQueueConsumer.java:37)
     at com.wm.app.audit.AuditFactory$AuditProducerHandler.run(AuditFactory.java:1576)
"TS-EventHandler" daemon prio=5 tid=0x005c1d20 nid=0x1f in Object.wait() [73981000..739819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x978010c8> (a com.wm.util.sync.LatchedSemaphore)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.util.sync.LatchedSemaphore.semWait(LatchedSemaphore.java:54)
     - locked <0x978010c8> (a com.wm.util.sync.LatchedSemaphore)
     at com.wm.util.sync.LatchedSemaphore.semWaitReset(LatchedSemaphore.java:101)
     - locked <0x978010c8> (a com.wm.util.sync.LatchedSemaphore)
     at com.wm.app.store.TransientStoreFactory$NotifyHandler.run(TransientStoreFactory.java:2173)
"FSData Free Cache Handler" daemon prio=5 tid=0x006eca38 nid=0x1d in Object.wait() [73b81000..73b819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x97801130> (a com.wm.util.sync.CountingSemaphore)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:71)
     - locked <0x97801130> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:56)
     - locked <0x97801130> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.driver.data.fs.FSData$FreeCacheHandler.run(FSData.java:3976)
"Thread-12" daemon prio=5 tid=0x009bc0a0 nid=0x1c in Object.wait() [73c81000..73c819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x97801140> (a com.wm.app.jdbc.JDBCConnPoolMgr$ConnectionTimeoutMgr)
     at com.wm.app.jdbc.JDBCConnPoolMgr$ConnectionTimeoutMgr.run(JDBCConnPoolMgr.java:394)
     - locked <0x97801140> (a com.wm.app.jdbc.JDBCConnPoolMgr$ConnectionTimeoutMgr)
"FSData Free Cache Handler" daemon prio=5 tid=0x006ebcf0 nid=0x1b in Object.wait() [73d81000..73d819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x978011e8> (a com.wm.util.sync.CountingSemaphore)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:71)
     - locked <0x978011e8> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.util.sync.CountingSemaphore.semWait(CountingSemaphore.java:56)
     - locked <0x978011e8> (a com.wm.util.sync.CountingSemaphore)
     at com.wm.driver.data.fs.FSData$FreeCacheHandler.run(FSData.java:3976)
"Thread-10" daemon prio=5 tid=0x00590d90 nid=0x1a in Object.wait() [73e81000..73e819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x978011f8> (a com.wm.app.repov4.server.RepoServer$RepoLockTimeoutMgr)
     at com.wm.app.repov4.server.RepoServer$RepoLockTimeoutMgr.run(RepoServer.java:1533)
     - locked <0x978011f8> (a com.wm.app.repov4.server.RepoServer$RepoLockTimeoutMgr)
"Thread-9" daemon prio=5 tid=0x000e9f48 nid=0x19 in Object.wait() [74081000..740819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x978012b0> (a com.wm.driver.queue.data.DataQueue)
     at java.lang.Object.wait(Object.java:429)
     at com.wm.driver.queue.data.DataQueue.dequeue(DataQueue.java:310)
     - locked <0x978012b0> (a com.wm.driver.queue.data.DataQueue)
     at com.wm.driver.queue.data.DataQueue.dequeueCommit(DataQueue.java:260)
     - locked <0x978012b0> (a com.wm.driver.queue.data.DataQueue)
     at com.wm.driver.queue.data.DataQueue.dequeueCommit(DataQueue.java:254)
     - locked <0x978012b0> (a com.wm.driver.queue.data.DataQueue)
     at com.wm.app.repov4.server.RepoServerBase$RepoEventManager.run(RepoServerBase.java:224)
"TxnData-Deadlock-Manager" daemon prio=5 tid=0x00799c18 nid=0x14 in Object.wait() [73f81000..73f819c0]
     at java.lang.Object.wait(Native Method)
     - waiting on <0x97801338> (a java.lang.Object)
     at com.wm.util.data.TxnData$DeadlockManager.run(TxnData.java:4945)
     - locked <0x97801338> (a java.lang.Object)
"Thread-2" daemon prio=5 tid=0x00104430 nid=0x12 in Object.wait() [74181000..741819c0]
     at java.lang.Object.wait(Native Method)
     at com.wm.util.queue.FSMTDirValuesHash$VHLockTimeoutMgr.run(FSMTDirValuesHash.java:448)
     - locked <0x978013b0> (a com.wm.util.queue.FSMTDirValuesHash$VHLockTimeoutMgr)
"Log Thread" prio=5 tid=0x002e9f10 nid=0x11 in Object.wait() [74281000..742819c0]
     at java.lang.Object.wait(Native Method)
     at com.wm.util.sync.SimpleQueue.deq(SimpleQueue.java:171)
     - locked <0x97801458> (a com.wm.util.sync.SimpleQueue)
     at com.wm.util.sync.SimpleQueue.deq(SimpleQueue.java:126)
     - locked <0x97801458> (a com.wm.util.sync.SimpleQueue)
     at com.wm.app.b2b.server.ServerLogHandler$LogThread.run(ServerLogHandler.java:274)
"Cron

Hi,
I've just found the following in another thread:
The workaround for the bugID 6221832 (java being in another path than the default /usr/j2se/bin/ for Patch Manager 2.0 on Solaris 8) is to edit the pprosetup and pprosvc scripts
from /usr/sbin/pprosvc :
JAVACMD=/usr/j2se/bin/java
edit pprosetup and pprosvc scripts file and set JAVACMD locale variable to newly java installed path and launch smpatch get command again
Thank you. I've editted both scripts and changed the JAVACMD variable to be ${JAVA_HOME}/bin/java and now everything is working.
Thanks.

Metrics "Database Time Spent Waiting (%) is ar 100 for event class "Applica

Metrics "Database Time Spent Waiting (%) is ar 100 for event class "Application".
i get this error in database. help me regarding this error.

Streams AQ: qmn coordinator idle wait     31     16     42598     1374.12     425976281     989870553     2723168908     6     Idle
Streams AQ: qmn slave idle wait     15     0     41599     2773.27     415990747     1830121438     2723168908     6     Idle
Streams AQ: waiting for messages in the queue     83     83     41478     499.73     414776290     3955192389     2723168908     6     Idle
jobq slave wait     134     131     39906     297.8     399055017     782339817     2723168908     6     Idle
Streams AQ: waiting for time management or cleanup tasks     7     4     3778     539.71     37779540     3702640206     2723168908     6     Idle
class slave wait     3     3     1499     499.54     14986144     1055154682     2723168908     6     Idle
Streams AQ: qmn coordinator waiting for slave to start     2     1     500     249.95     4999066     1565566389     1893977003     0     Other
LGWR wait for redo copy     1     0     0     0     1     4266849434     1893977003     0     Other

Ora-04021 timeout occurred while waiting to lock object

Hai All
When adding a column into a table then an error comes error
Error Message:
ORA-04021: timeout occurred while waiting to lock object .....
How it happens ...what is the solution....please help
Shiju..

Hi
Identify the session that is using this object, kill it and then perform the ALTER table... or wait until the session that is accessing this table release it.
Rgds
Adnan

Patch 9239090 - ORA-04021: timeout occurred while waiting to lock object

Hello Guys,
I need help urgently to comlete an upgrade of ebs 12.1.1 to 12.1.3.
We are upgrading ebs 11i to 12i going live this weekend. We have encountered "ORA-04021: timeout occurred while waiting to lock object" on patch 9239090 with two Workers:
In adctrl:
1 Run AutoPatch R120 AFUTLGRS.pls FAILED
2 Run AutoPatch R120 AFUTLOGS.pls FAILED
sqlplus -s APPS/***** @/u02/applfinp/apps/apps_st/appl/admin/OFPROD/out/p001invok.sql
Connected.
create or replace package wf_bes_cleanup AUTHID CURRENT_USER as
ERROR at line 1:
ORA-04021: timeout occurred while waiting to lock object
I have done everything everything imaginable but cannot get pass these 2 failed workers.
1) I have checked are there are no locks blocking on any objects:
select s1.username || '@' || s1.machine
|| ' ( SID=' || s1.sid || ' ) is blocking '
|| s2.username || '@' || s2.machine || ' ( SID=' || s2.sid || ' ) ' AS blocking_status
from v$lock l1, v$session s1, v$lock l2, v$session s2
where s1.sid=l1.sid and s2.sid=l2.sid
and l1.BLOCK=1 and l2.request > 0
and l1.id1 = l2.id1
and l2.id2 = l2.id2 ;
2) I also checked and there is space in the database.
3) From adctrl, I have tried restarting the jobs one by one but the worker 1 hangs at and eventually fails on several (12) attempts.
4) The solution on Doc ID 1291064.1 is not the problem:
SQL> select SUPPLEMENTAL_LOG_DATA_MIN, SUPPLEMENTAL_LOG_DATA_PK, SUPPLEMENTAL_LOG_DATA_UI, FORCE_LOGGING
from v$database;
SUPPLEME SUP SUP FOR
NO NO NO NO
5) Manual execution still gave ORA-04021:
sqlplus -s APPS/apps @/u02/applfinp/apps/apps_st/appl/admin/OFPROD/out/p001invok.sql
sqlplus -s APPS/apps @/u02/applfinp/apps/apps_st/appl/admin/OFPROD/out/p002invok.sql
6) The alert log has no related errors
7) I Finally, after over 6 hours of waiting, I could only get the patch to continue after by using adctrl option 8 to skip these two workers.
After proceeding, I have yet hit two more errors and now stuck:
I have hit another error with the same patch:
create or replace package body FND_LOG as
/* $Header: AFUTLOGB.pls 120 ...
AD Worker error:
ORA-04021: timeout occurred while waiting to lock object
Unable to process file in PACKAGE mode.
Not converting file for Invoker's Rights because it appears to be a package body creation script (based on the filename).
File is: /u02/applfinp/apps/apps_st/appl/fnd/12.0.0/patch/115/sql/AFUTLOGB.pls
AD Worker warning:
Product Data File
/u02/applfinp/apps/apps_st/appl/admin/xniprod.txt
does not exist for product "xni".
This product is registered in the database but the above file does not exist in APPL_TOP. The product will be ignored without error
AD Worker error:
The following ORACLE error:
ORA-04063: package body "APPS.FND_LOG" has errors
ORA-06508: PL/SQL: could not find program unit being called: "APPS.FND_LOG"
ORA-06512: at "APPS.FND_FUNCTION", line 834
ORA-06512: at "APPS.FND_MENU_ENTRIES_C_DELTRG", line 4
ORA-04088: error during execution of trigger 'APPS.FND_MENU_ENTRIES_C_DELTRG'
Can I stop and restart this patch 9239090?

Hello,
I later executed the failed jobs or scripts manually (successfully) for the patch to continue
sqlplus -s APPS/apps @/u02/applfinp/apps/apps_st/appl/admin/OFPROD/out/p001invok.sql
sqlplus -s APPS/apps @/u02/applfinp/apps/apps_st/appl/admin/OFPROD/out/p002invok.sql
The patch ended with the following :
The following Oracle Reports objects did not generate successfully:
ap reports/US APXARATE.rdf
ar reports/US ARXCTA.rdf
fa reports/US FASRSVED.rdf
ont reports/US OEXOEORD.rdf
pa reports/US PAXACMPT.rdf
pa reports/US PAXALRUN.rdf
pa reports/US PAXACRPT.rdf
pa reports/US PAXCPGAL.rdf
pa reports/US PAXPCRFD.rdf
The patch execution ended but the patch number is not record in apps.ad_bugs table:
select TO_CHAR(CREATION_DATE,'dd-Mon-yyyy, hh24:mm:ss'), BUG_NUMBER from apps.ad_bugs where bug_number in ('9239090');
no rows selected
JFI
Patches located in ad_applied_patches to which there is no a corresponding record in ad_bugs:
SQL> SELECT a.patch_name, TO_CHAR(max(a.last_update_date),'dd-Mon-yyyy, hh24:mm:ss')
2 FROM applsys.ad_applied_patches a
3 WHERE NOT EXISTS (SELECT '1'
4 FROM ad_bugs b
5 WHERE b.bug_number = a.patch_name)
6 group by a.patch_name;
PATCH_NAME TO_CHAR(MAX(A.LAST_UP
9239090 26-Oct-2013, 00:10:27
AutoConfig 25-Oct-2013, 17:10:51
merger9179588 25-Oct-2013, 11:10:30
merger_post_5903765 27-Apr-2013, 23:04:45
mergeu6678700 25-Oct-2013, 16:10:09
What could have happened that this was not recorded in ad_bugs?
How to I get this recorded in the ad_bugs table?
Thanks
Mathias

OWB9.0.4-- ORA-02049: timeout: distributed transaction waiting for lock

I'm running a simple mapping that copies all columns of data (using a filter on date for just current records) from one table in SQL Server into a staging table on my Oracle DW schema. It's using a dblink with transparent gateway for SQL Server, which works fine from SQL*plus.
The map is in default mode (bulk-failoverto-row) with bulk size and commit frequency = 1000.
The audit details show the first 1000 rows selected on the source, with an error on both the source and target tables:
Target--
ORA-02049: timeout: distributed transaction waiting for lock
Source--
ORA-01002: fetch out of sequence ORA-02063: preceding line from INTERGRATION@JXNSQL01
(INTERGRATION@JXNSQL01 is the dblink name)
Any ideas on how I can clear this up?
Thanks,
Paul

Hi,
After having upgraded to 9.0.4 (from 9.0.3) I'm running into exactly the same problems with some of my mappings. Actually I don't get any rows transferred from the mappings that fail.
Out of 7 mappings, 3 worked just liked before while the 4 others just keep on running until I cancel them and I then see the BUSY/ORA-02049 in the Audit Browser.
When comparing the mappings I see that the 3 that works all use some custom Procedures I have made.
The 4 that doesn't work are all very simple - one of them just loads all the content from a table with two columns in my source to another table with two columns in my target! Two of the other mappings that doesn't work includes some simple CASE-expressions.
Both my source and my target resides in a Oracle 9.2.0.3 database (not the same).
Regards,
Bent Madsen

Delegated Admin web application only requests first 100 accounts?

Hi,
- Sun Java System Messenger Express 6.2
- Delegated Administrator 6.3-0.09 built Sep 6, 2005
Is this true that the Delegated Admin (DA) web application only requests first 100 accounts?
Once logged in to DA web application, we only see "Retrieved Users (100)" if we want to see all users; but if we do a search on uid or username, all other users are retrieved.
One of the admin gave us the following response:
This is not a directory-related problem, but rather a matter of the design of the DA application you are using. The web-based java app only requests the first 100 accounts from the directory (presented by default as 10 pages of 10 accoints each.) , since you're supposed to be using the search facility to find accounts when you need to modify or delete them.
This is a deliberate design choice by the Sun programmers who wrote the thing, probably because the directory is capable of holding several thousand accounts and pulling them all would take quite a bit of time (not to mention memory space), so in the interest of response-time speed they limited the data pull.
I cannot modify this application's functionality. If you need a list of all user accounts in your domain, I can supply an LDIF on request, with any attributes (mail, uid, cn, etc.) that you like.
Please let us know if there is any way we can view all users (approx. 1000) from DA web application.
Thank you for your time,
GJ

Yes, the terminal commands I gave are changing permissions.
Properly written OS X apps should run under any user account, and should store any account-specific information in the each user's home folder. Some poorly written apps might only be executable by the administrator. Running the first command I have will make the app executable to all users.
Some even more poorly written apps will write user data into the application itself rather than to the user's home folder. This is a particularly bad problem with game software, which for example might write high score info into the app itself. If this is the case for your misbehaving apps, the second command I gave will make the app writable by everybody and should solve the problem.

DatabaseError: ORA-02049: timeout: distributed transaction waiting for lock

Hi all
There is script running in my database 'A' that will select some data from tables in other database through DBLINK and update some tables too. The script which 6 I am running for the past 6 monts succesfully. Yesterday I got error saying that,
DatabaseError: ORA-02049: timeout: distributed transaction waiting for lock
ORA-02063: preceding line from test
I have tested the Link and its active no issues in DBLINk
I understand the objects which I am accessing the table in locked mode,
SQL> select OBJECT_ID,SESSION_ID,PROCESS,LOCKED_MODE from v$locked_object where
SESSION_ID=121;
OBJECT_ID SESSION_ID PROCESS LOCKED_MODE
77391 121 21394 3
i
I killed the session 121 and in v$session status is Killed, but I am getting the same error repeating while running the script. And the session_id is still in locked_mode.
Could please add some light on this?
-Thanks
Antony

Hi,
I have killed the session which caused blocking and the status is KILLED for the past two days. But still the object is in locked mode
SQL> select OBJECT_ID,SESSION_ID,PROCESS,LOCKED_MODE from v$locked_object where
SESSION_ID=121;
OBJECT_ID SESSION_ID PROCESS LOCKED_MODE
77391 121 21394 3
How would I resolve this any idea
-Thanks & Cheers
Antony

ı have installed an application about lock system but the application isnt about lock systm this is an code reader. when ı see the application's pictures this is an lock application but content not same. I have got laid...what shoul I do?

I have installed an application about lock system but the application isnt about lock system this is an code reader. When ı see the application's pictures this is an lock application but content not same. I have got laid...what should I do?

semihtinas wrote:
Please make something these fraudsters
You're not addressing Apple here. This is a user to user technical support forum. You can submit feedback to Apple here:
http://www.apple.com/feedback
Other contact information can be found using the Contact Us link at the bottom right of every page.

Thread waiting to lock on an object that it has already locked

I have a thread in my thread dump where it is "waiting to lock" on an object, but it already has acquired the lock. How can this be possible?
"TP-Processor778" daemon prio=1 tid=0x230dde38 nid=0x1168 waiting for monitor entry [0x0f184000..0x0f185fa0]
at sun.net.www.protocol.file.Handler.openConnection(Handler.java:63)
- waiting to lock <0x40fbf128> (a sun.net.www.protocol.file.Handler)
at sun.net.www.protocol.file.Handler.openConnection(Handler.java:55)
- locked <0x40fbf128> (a sun.net.www.protocol.file.Handler)
at java.net.URL.openConnection(URL.java:943)
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)

If object.wait() is called, an existing lock is released so another thead can lock the object.
If you object.notify() but hold the lock in another thread, the first thread will not resume until it has required the lock.

Since installing Maverick on my 21in. 3.2ghz iMac, cannot upload to any application without it locking up, Aperature opens automatically, my printers don't work and Apple has failed to reschedule a appointment with a Tech on Case *******in 3wks

Since installing Maverick on my 21in. 3.2ghz iMac, cannot upload to any application without it locking up, Aperature opens automatically, my printers don't work and Apple has failed to reschedule a appointment with a Tech on Case ********* in 3wks
<Edited by Host>

Sorry you've decided this is a rant but the objective observer would know that all I want is a solution to at least the 3 issues mentioned. I spent the better part of 2 nights with techs that had me doing all the standard refresh items, i.e. setting the PRAM... and as time goes on my frustration grows. Last week spent days trying to find the driver software Maverick now needs and Apple is supposed to have it and doesn't. So at what point does this become a rant? I've done things the right way had mucho patience thus far and as a kicker, Apple won't let me download Mountain Lion to reinstall and rid my self of Maverick. Tell me, would you complain?

Ora-04021: timeout occured while waiting to lock object.

I have upgraded the DB From 8.1.5 to 8.1.7 then 9.0.2
but while converting to 9i it gave a warning that
"some uncompiled objects were found".
now the DB works fine with every DDL and DML operation but when I edit any precedure or function it hangs for 4 to 5 mins and then show the following error
ora-04021: timeout occured while waiting to lock object.
when I shutdown the DB and restarts it then it allowes me to edit any founction or procedure or package only for one time,then again shows the error when I try to edit second time.
what my be the reason , can any body help me pl.
thankx in advance.

If you have oracle metalink support, then, read Note:169139.1. If not, give me your email id, i will send you the note.
SJH

Is there any application ,which can lock my folders individually plz reply me

is there any application ,which can lock my folders individually plz reply me

Ann app will not help, as no 3rd party application can affect another, so no app can lock another apps files or folders.
Apps can lock their own content, and some do include that option of putting a passcode on just them and their own files, but there is no universal feature.
If you put a passcode lock on the iPhone, then no one can get to your content.

Applications waiting on locks 100%

Similar Messages

Maybe you are looking for