The thread dump has a deadlock ?

We're experiencing a problem with the OC4J server hanging when some we unknow operation . I can see it's a connection pooling problem based on a thread dump. What I can't determine is why it doesn't release a resource in the first place. I've included some code to show where it is hung.
================================================
"AJPRequestHandler-ApplicationServerThread-17" tid=0x8775468 nid=0x2f3d prio=1 waiting for monitor entry
     at oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:4502)
     - waiting to lock<0x55805618> (a oracle.jbo.JboSyncLock), locked by AJPRequestHandler-ApplicationServerThread-135
     at oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:4479)
     at oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:6818)
     at oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:3886)
     at oracle.jbo.common.ampool.ApplicationPoolImpl.doManagedCheckin(ApplicationPoolImpl.java:2131)
     at oracle.jbo.common.ampool.ApplicationPoolImpl.releaseApplicationModule(ApplicationPoolImpl.java:1211)
     at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:717)
     - locked<0x557fa248> (a oracle.jbo.common.ampool.SessionCookieImpl$SessionCookieLock)
     at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:629)
     at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:611)
     at oracle.jheadstart.model.bc4j.ApplicationModuleProvider.releaseConnection(ApplicationModuleProvider.java:477)
     at oracle.jheadstart.model.bc4j.ApplicationModuleProvider.releaseRequestScopedConnections(ApplicationModuleProvider.java:503)
     at oracle.cle.persistence.ConnectionManager.releaseConnections(ConnectionManager.java:999)
     - locked<0x50e47900> (a oracle.cle.persistence.ConnectionManager)
     at oracle.cle.persistence.ConnectionManager.releaseRequestScopedConnections(ConnectionManager.java:939)
     at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:645)
     at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
     at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
     at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
     at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
     at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
     at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
     at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
     at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
     at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
     at java.lang.Thread.run(Thread.java:534)
"AJPRequestHandler-ApplicationServerThread-135" tid=0x8064ac8 nid=0x2f3d prio=1 waiting for monitor entry
     at oracle.cle.persistence.ConnectionManager.getConnection(ConnectionManager.java:722)
     - waiting to lock<0x50e47900> (a oracle.cle.persistence.ConnectionManager), locked by AJPRequestHandler-ApplicationServerThread-17
     at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:308)
     at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:148)
     at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getConnection(DataObjectHandlerImpl.java:355)
     at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.getApplicationModule(DataObjectImpl.java:761)
     at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.doneRollback(DataObjectImpl.java:855)
     at oracle.jbo.server.DBTransactionImpl.sendEvent(DBTransactionImpl.java:5501)
     at oracle.jbo.server.DBTransactionImpl.fireTransactionStateEvent(DBTransactionImpl.java:5568)
     - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
     at oracle.jbo.server.DBTransactionImpl.rollback(DBTransactionImpl.java:2257)
     - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
     - locked<0x55805618> (a oracle.jbo.JboSyncLock)
     at oracle.jheadstart.model.bc4j.JhsDBTransactionImpl.rollback(JhsDBTransactionImpl.java:186)
     at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.doneCommit(DataObjectImpl.java:832)
     at oracle.jbo.server.DBTransactionImpl.sendEvent(DBTransactionImpl.java:5494)
     at oracle.jbo.server.DBTransactionImpl.fireTransactionStateEvent(DBTransactionImpl.java:5568)
     - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
     at oracle.jbo.server.DBTransactionImpl.commitInternal(DBTransactionImpl.java:1989)
     at oracle.jbo.server.DBTransactionImpl.commit(DBTransactionImpl.java:2046)
     - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
     - locked<0x55805618> (a oracle.jbo.JboSyncLock)
     at oracle.jheadstart.model.bc4j.JhsDBTransactionImpl.commit(JhsDBTransactionImpl.java:176)
     at oracle.jheadstart.model.bc4j.JhsApplicationModuleImpl.commitChanges(JhsApplicationModuleImpl.java:1167)
     at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.commitChanges(DataObjectHandlerImpl.java:898)
     at oracle.jheadstart.controller.struts.action.JhsAction.commitChanges(JhsAction.java:484)
     at oracle.jheadstart.controller.struts.action.JhsAction.saveChanges(JhsAction.java:409)
     at oracle.jheadstart.controller.struts.action.SaveDataObject.execute(SaveDataObject.java:467)
     at oracle.jheadstart.controller.struts.action.JhsAction.execute(JhsAction.java:245)
     at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
     at oracle.jheadstart.controller.struts.JhsRequestProcessor.processActionPerform(JhsRequestProcessor.java:351)
     at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
     at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:635)
     at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
     at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
     at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
     at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
     at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
     at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
     at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
     at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
     at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
     at java.lang.Thread.run(Thread.java:534)
"AJPRequestHandler-ApplicationServerThread-16" tid=0x8b2b510 nid=0x2f3d prio=1 waiting for monitor entry
     at oracle.cle.persistence.ConnectionManager.getConnection(ConnectionManager.java:722)
     - waiting to lock<0x50e47900> (a oracle.cle.persistence.ConnectionManager), locked by AJPRequestHandler-ApplicationServerThread-17
     at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:308)
     at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:148)
     at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getConnection(DataObjectHandlerImpl.java:355)
     at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getApplicationModule(DataObjectHandlerImpl.java:1011)
     at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getDataObjectSet(DataObjectHandlerImpl.java:511)
     at oracle.jheadstart.controller.struts.action.GetDataObjectSet.retrieveSimple(GetDataObjectSet.java:474)
     at oracle.jheadstart.controller.struts.action.GetDataObjectSet.execute(GetDataObjectSet.java:390)
     at oracle.jheadstart.controller.struts.action.JhsAction.execute(JhsAction.java:245)
     at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
     at oracle.jheadstart.controller.struts.JhsRequestProcessor.processActionPerform(JhsRequestProcessor.java:351)
     at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
     at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:635)
     at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
     at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
     at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
     at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
     at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
     at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
     at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
     at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
     at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
     at java.lang.Thread.run(Thread.java:534)
Sincerely from Ting-Rung

About the general situation describe like following, thanks you help.
Q1:Whether you could able to access the httpserver home page or any other application deployed in other OC4J container, when the server
become unresponsive.
Ans:Yes
Q2:What are the kind of applications deployed in the OC4J container ( hrmOC4J) .
Ans:The kind of applications is human resource application(Developer Tool is JDeveloper 9.0.5.2 and Jheadstart 9.0.5.19 and
oracle Application server 10.1.2.0.2)
Q3:Does server hang happen regularly or at what situation the server is becoming unresponsive ( high load, etc..)
Ans:No,It is not regularly when when server hang happen.
Q4:From the SR notes, the workaround you used is, restarting the OC4J container. ( what is the
kind of Oracle Application Server instance you have installed, please let me
know the complete version of the Oracle AS instance)
Ans:oracle Application server 10.1.2.0.2
Q5:Did you notice any out of memory issues when the server / particular OC4J container is hang
Ans:No. It is not out of memory.

Similar Messages

  • A StuckThreadMaxTime error; who can help me to analyse the thread dumps

    The StuckThreadMaxTime error freeze my system, no one can logged in.I have double checked the thread dumps, but get no help.
    The following is the thread dumps:
    <2004-12-7 ÉÏÎç05ʱ56·Ö47Ãë GMT+08:00> <Alert> <Log Management> <BEA-170017> <The log file ./myserver/myserver.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
    <2004-12-7 ÉÏÎç05ʱ56·Ö47Ãë GMT+08:00> <Alert> <Log Management> <BEA-170018> <The log file has been rotated to myserver.log00035. Log messages will continue to be logged in ./myserver/myserver.log.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '10' for queue: 'weblogic.kernel.Default' has been busy for "646" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '13' for queue: 'weblogic.kernel.Default' has been busy for "651" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default' has been busy for "650" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    Full thread dump Java HotSpot(TM) VM (1.4.2 1.4.2.03-040401-18:59-PA_RISC2.0 PA2.0 (aCC_AP) mixed mode):
    "ExecuteThread: '14' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f778 nid=71 lwp_id=39624 in Object.wait() [0x61abf000..0x61abf4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d24f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d24f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f610 nid=70 lwp_id=39623 in Object.wait() [0x61b40000..0x61b404f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d2578> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d2578> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f4a8 nid=69 lwp_id=39622 in Object.wait() [0x61bc1000..0x61bc14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d25f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d25f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f340 nid=68 lwp_id=39621 in Object.wait() [0x61c42000..0x61c424f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d2678> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.Eÿ

    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=000883f8 nid=27 lwp_id=39205 runnable [0x630ec000..0x630eb4f0]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at oracle.net.ns.Packet.receive(Unknown Source)
         at oracle.net.ns.DataPacket.receive(Unknown Source)
         at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
         at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
         at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:181)
         at oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:543)
         at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:913)
         at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:452)
         at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:984)
         at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2885)
         at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:2926)
         - locked <762d0bd8> (a oracle.jdbc.driver.T4CPreparedStatement)
         - locked <6c837e90> (a oracle.jdbc.driver.T4CConnection)
         at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:92)
         at com.huawei.oss.rms.persistence.ReadOperationPersistenceImpl.getAttributeValue(ReadOperationPersistenceImpl.java:2054)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.getAttributeValue(PersistenceOperationImpl.java:516)
         at com.huawei.oss.services.common.Utility.getNameByID(Utility.java:78)
         at com.huawei.oss.services.port.common.RMSPortUtility.setExtraToVO(RMSPortUtility.java:45)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.setExtraToVO(PhyPortConnHelper.java:2046)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.afterGet(PhyPortConnHelper.java:719)
         at com.huawei.oss.services.common.AbstractResourceHelper.getResource(AbstractResourceHelper.java:293)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.rmsGetByPort(PhyPortConnHelper.java:809)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.rmsGetAllPhyConnOfBoard(PhyPortConnHelper.java:1144)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:324)
         at com.huawei.oss.services.common.AbstractResourceHelper.directMethod(AbstractResourceHelper.java:489)
         at com.huawei.oss.services.common.AbstractResourceHelper.invokeMethod(AbstractResourceHelper.java:404)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSessionEJB.invokeMethod(ResourceMgmtSessionEJB.java:263)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl.invokeMethod(ResourceMgmtSession_1x9h7n_EOImpl.java:424)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl_WLSkel.invoke(Unknown Source)
         at weblogic.rmi.internal.activation.ActivatableServerRef.invoke(ActivatableServerRef.java:90)
         at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:420)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
         at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:415)
         at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=00088290 nid=26 lwp_id=39204 runnable [0x6316d000..0x6316c4f0]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at oracle.net.ns.Packet.receive(Unknown Source)
         at oracle.net.ns.DataPacket.receive(Unknown Source)
         at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
         at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
         at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:112)
         at oracle.jdbc.driver.T4CStatement.execute_for_rows(T4CStatement.java:474)
         at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:913)
         at oracle.jdbc.driver.T4CStatement.execute_maybe_describe(T4CStatement.java:383)
         at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:984)
         at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1124)
         - locked <76c0d1f8> (a oracle.jdbc.driver.T4CStatement)
         - locked <6f1ab788> (a oracle.jdbc.driver.T4CConnection)
         at weblogic.jdbc.wrapper.Statement.executeQuery(Statement.java:345)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.executeSQL(PersistenceOperationImpl.java:2884)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.setVersionFlagNResourceName(PersistenceOperationImpl.java:3137)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.makeAssociationValue(PersistenceOperationImpl.java:2600)
         at com.huawei.oss.rms.persistence.ReadOperationPersistenceImpl.getResourceAssociationSet(ReadOperationPersistenceImpl.java:2575)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.getResourceAssociationSet(PersistenceOperationImpl.java:603)
         at com.huawei.oss.services.port.common.RMSPortUtility.getSlotByEquip(RMSPortUtility.jÿ

  • States in the thread dump "Waiting on monitor"

     

    that is normal, it is all the execute threads waiting for requests
    Rob Baldassano wrote:
    Hi.
    I am trying to do some troubleshooting of the weblogic server,
    Currently we are using WebLogic 6.1 SP 2.
    In the thread dump we are seeing a lot of "waiting on monitor" messages for the
    individual threads.
    The application that we are running uses native threads to run the processes that
    it needs to.
    My question is this.
    Is this "Waiting for monitor" message normal?
    What could cause these messages?
    is there a way to get around them, besides recoding the application?

  • Output of the thread dump taken in unix for weblogic using kill -3 oprion

    Hi
    I am executing the kill -3 <PID> command in the unix box for weblogic server to get the thread dump.
    I am not able to find where the output of the thread dump is getting saved.Please advice.
    Regards
    Abhishek

    Hi,
    You can following the below link which shows you how to take thread dumps in *9 different ways* and where they get written also
    Topic: Taking Thread Dumps - 9 Ways
    http://middlewaremagic.com/weblogic/?p=823
    Regards,
    Ravish Mody
    http://middlewaremagic.com/weblogic/?page_id=1749
    Come, Join Us and Experience The Magic…

  • Full Thread Dump...Deadlock occured in the code

    The following code generates error saying the following description:
    ERROR Start:
    SIGBUS occured.
    Full thread dump:
    "main" (TID:0x66c9360, sys_thread_t:0x7ad4f10, taskId:0x7ae447c state:runnable name:tJmain, stack_base:0x7ae447c, stack_high:0x1080 stack_size:19696) prio=5
    "Finalizer thread" (TID:0x66c9390, sys_thread_t:0x7ad0538, taskId:0x7ad4590 state:condition variable pend name:Finalizer threa, stack_base:0x7ad4588, stack_high:0x144 stack_size:15624) prio=1
    "Thread-0" (TID:0x66dd778, sys_thread_t:0x6560460, taskId:0x65644b8 state:condition variable pend name:Thread-0, stack_base:0x65644b0, stack_high:0xb48 stack_size:15628) prio=5
    Page Fault
    Program Counter: 0x0013c6a3
    Status Register: 0x00010202
    Error Code: 0x00130000
    Task: 0x6551d20 "Thread-3"
    ERROR End
    This is the code..........
    public class abc{
         private boolean ready;
         public synchronized void toggle1() {
              ready = false;                     // de-block the other thread
              try {
              notify();                     // inform the other thread
              }catch(IllegalMonitorStateException e) {
              e.printStackTrace();
              while (!ready) {               // block until the other
                   try {                    // thread sets "ready" to
                        wait();               // "true".
                   } catch (InterruptedException e) {
                   System.out.println("Error from toggle1:InterruptedException");
                   e.printStackTrace();
                   catch(Throwable e)
                   System.out.println("Error from toggle1:Throwable");
                   e.printStackTrace();
         public synchronized void toggle0() {
              ready = true;                    // de-block the other thread
              notify();                    // inform the other thread
              while (ready) {                    // block until the other
                   try {                    // thread sets "ready" to
                        wait();               // "false".
                   } catch (InterruptedException e) {}
    Would you please put look into it and suggest a solution
    Thanks
    Message was edited by:
    Indranil.Sarkar

    Ah well if you're back on 1.2 then that's about as much error information as you will get.
    JDK 1.2 is way past end-of-life. There is nothing you can do here to fix this problem.
    Note: just because your application code was written for 1.2 doesn't mean it won't run on a later version of the VM. Try 1.3.1 if you don't want to jump up to 1.5 or 1.6 (but 1.3.1 will soon hit end-of-life too).
    Sorry.

  • Strange thread dump on deadlock

    We've been experiencing a strange deadlock during the startup of our java application. When I run jstack on the application to investigate, I see that the AWT-EventQueue is in Object.wait(), but the thread is still marked as RUNNABLE. I've included the relevent parts of the thread dump, and I'm hoping that someone can shed some light on this issue.
    "AWT-EventQueue-0" prio=6 tid=0x5f0a2400 nid=0x19e4 in Object.wait() [0x6007e000]
           java.lang.Thread.State: RUNNABLE
            at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)
              - locked <0x1fc87448> (a java.awt.Component$AWTTreeLock)
    "Thread-63-Pool-9" prio=6 tid=0x5f1a2800 nid=0x1f54 waiting for monitor entry [0x61a9f000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at java.awt.Component.setFont(Component.java:1777)
        - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
    "Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at java.awt.Component.setFont(Component.java:1777)
        - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
          ... In my search for the solution, I've noticed [this thread|http://stackoverflow.com/questions/64989/jvm-thread-dumps-containing-monitors-without-locking-threads], which mentions that accessing a static variable may be involved. This is also the case in our application. The line in getEAOTableLite in question references a static method.

    Thank you for the quick reply. I'm not sure how I missed it, but if I had read down the stack trace a little better I would have found that the issue was that the class loader for the EaoAlertManager never completed because it was blocked by the illegal call to setFont. This would cause the AWT-EventQueue thread to go into an infinite wait call on the ClassLoader.
    "Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
       java.lang.Thread.State: BLOCKED (on object monitor)
         at java.awt.Component.setFont(Component.java:1777)
         - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
         at java.awt.Container.setFont(Container.java:1554)
         at javax.swing.JComponent.setFont(JComponent.java:2723)
         at javax.swing.LookAndFeel.installColorsAndFont(LookAndFeel.java:191)
         at javax.swing.plaf.basic.BasicPanelUI.installDefaults(BasicPanelUI.java:49)
         at javax.swing.plaf.basic.BasicPanelUI.installUI(BasicPanelUI.java:39)
         at com.ge.med.ptk.laf.CuiPanelUI.installUI(CuiPanelUI.java:53)
         at javax.swing.JComponent.setUI(JComponent.java:662)
         at javax.swing.JPanel.setUI(JPanel.java:136)
         at javax.swing.JPanel.updateUI(JPanel.java:109)
         at javax.swing.JPanel.<init>(JPanel.java:69)
         at javax.swing.JPanel.<init>(JPanel.java:92)
         at javax.swing.JPanel.<init>(JPanel.java:100)
         at javax.swing.JRootPane.createGlassPane(JRootPane.java:528)
         at javax.swing.JRootPane.<init>(JRootPane.java:348)
         at javax.swing.JDialog.createRootPane(JDialog.java:611)
         at javax.swing.JDialog.dialogInit(JDialog.java:593)
         at com.ge.med.plaf.wrapper.WJDialog.dialogInit(WJDialog.java:42)
         at javax.swing.JDialog.<init>(JDialog.java:545)
         at javax.swing.JDialog.<init>(JDialog.java:515)
         at com.ge.med.plaf.wrapper.WJDialog.<init>(WJDialog.java:424)
         at com.ge.med.platinum.gui.util.PlatinumDialog.<init>(PlatinumDialog.java:138)
         at com.ge.med.platinum.gui.util.EAOAlertManager$EAOAlertDialog.<init>(EAOAlertManager.java:450)
         at com.ge.med.platinum.gui.util.EAOAlertManager.<clinit>(EAOAlertManager.java:77)
         at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)

  • What is the problem in my Thread Dump's output?

    Dear all,
    Below is my Thread Dump output. I can not figure out what and where is the problem of this Thread Dump's output. Could some one please give me some hints?
    Thanks alot
    Tu
    ----------------Thread Dump's Output-----------------
    Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode, sharing):
    "TP-Monitor" daemon prio=1 tid=0xb0b14490 nid=0xfaa in Object.wait()
    [0xb08fe000..0xb08fee40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89433f70> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
    - locked <0x89433f70> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor4" daemon prio=1 tid=0xb0b13730 nid=0xfa9 runnable
    [0xb097e000..0xb097efc0]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x89438110> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)
    at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:647)
    at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:857)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor3" daemon prio=1 tid=0xb0b0f6b8 nid=0xfa8 in
    Object.wait() [0xb09fe000..0xb09fef40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89434250> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89434250> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor2" daemon prio=1 tid=0xb0b0fec8 nid=0xfa7 in
    Object.wait() [0xb0a7e000..0xb0a7f0c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x894342e8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x894342e8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor1" daemon prio=1 tid=0xb0b14978 nid=0xfa6 in
    Object.wait() [0xb0afe000..0xb0aff040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89434380> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89434380> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Monitor" prio=1 tid=0x08609b18 nid=0xfa5 in Object.wait()
    [0xb0cc9000..0xb0cc91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378950> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
    - locked <0x89378950> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor25" daemon prio=1 tid=0x08608ce0 nid=0xfa4
    runnable [0xb0d48000..0xb0d49140]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x89010c58> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
    at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
    at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor24" daemon prio=1 tid=0x086085f8 nid=0xfa3 in
    Object.wait() [0xb0dc8000..0xb0dc8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378ab8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378ab8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor23" daemon prio=1 tid=0x08606dd0 nid=0xfa2 in
    Object.wait() [0xb0e48000..0xb0e48e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378b50> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378b50> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor22" daemon prio=1 tid=0x08605ed0 nid=0xfa1 in
    Object.wait() [0xb0ec8000..0xb0ec8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378be8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378be8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor21" daemon prio=1 tid=0x08604fd0 nid=0xfa0 in
    Object.wait() [0xb0f48000..0xb0f48f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378c80> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378c80> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor20" daemon prio=1 tid=0x086040f0 nid=0xf9f in
    Object.wait() [0xb0fc8000..0xb0fc90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378d18> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378d18> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor19" daemon prio=1 tid=0x08432538 nid=0xf9e in
    Object.wait() [0xb1048000..0xb1049040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378db0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378db0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor18" daemon prio=1 tid=0x08431638 nid=0xf9d in
    Object.wait() [0xb10c9000..0xb10c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378e48> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378e48> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor17" daemon prio=1 tid=0x08430738 nid=0xf9c in
    Object.wait() [0xb1148000..0xb1149140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378ee0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378ee0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor16" daemon prio=1 tid=0x0842f838 nid=0xf9b in
    Object.wait() [0xb11c8000..0xb11c8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378f78> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378f78> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor15" daemon prio=1 tid=0x0842e938 nid=0xf9a in
    Object.wait() [0xb1248000..0xb1248e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379010> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379010> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor14" daemon prio=1 tid=0x0842da38 nid=0xf99 in
    Object.wait() [0xb12c8000..0xb12c8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893790a8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893790a8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor13" daemon prio=1 tid=0x0842cb38 nid=0xf98 in
    Object.wait() [0xb1348000..0xb1348f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379140> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379140> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor12" daemon prio=1 tid=0x0842bc50 nid=0xf97 in
    Object.wait() [0xb13c8000..0xb13c90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893791d8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893791d8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor11" daemon prio=1 tid=0x084f75d0 nid=0xf96 in
    Object.wait() [0xb1448000..0xb1449040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379270> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379270> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor10" daemon prio=1 tid=0x084f66d0 nid=0xf95 in
    Object.wait() [0xb14c9000..0xb14c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379308> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379308> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor9" daemon prio=1 tid=0x084f57d0 nid=0xf94 in
    Object.wait() [0xb1548000..0xb1549140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893793a0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893793a0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor8" daemon prio=1 tid=0x08488970 nid=0xf93 in
    Object.wait() [0xb15c8000..0xb15c8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379438> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379438> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor7" daemon prio=1 tid=0x08487a70 nid=0xf92 in
    Object.wait() [0xb1648000..0xb1648e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893794d0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893794d0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor6" daemon prio=1 tid=0x08486b90 nid=0xf91 in
    Object.wait() [0xb16c8000..0xb16c8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379568> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379568> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor5" daemon prio=1 tid=0x084e3e30 nid=0xf90 in
    Object.wait() [0xb1748000..0xb1748f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379600> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379600> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor4" daemon prio=1 tid=0x084e2f90 nid=0xf8f in
    Object.wait() [0xb17c8000..0xb17c90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379698> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379698> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor3" daemon prio=1 tid=0x084e2158 nid=0xf8e in
    Object.wait() [0xb1848000..0xb1849040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379730> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379730> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor2" daemon prio=1 tid=0x085e74d0 nid=0xf8d in
    Object.wait() [0xb18c9000..0xb18c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893797c8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893797c8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor1" daemon prio=1 tid=0x085e6638 nid=0xf8c in
    Object.wait() [0xb1948000..0xb1949140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379860> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379860> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=1
    tid=0x083719f8 nid=0xf8b waiting on condition [0xb1a85000..0xb1a85ec0]
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1547)
    at java.lang.Thread.run(Thread.java:595)
    "Low Memory Detector" daemon prio=1 tid=0x080a44f8 nid=0xf7f runnable
    [0x00000000..0x00000000]
    "CompilerThread0" daemon prio=1 tid=0x080a2f98 nid=0xf7e waiting on
    condition [0x00000000..0xb21b2928]
    "Signal Dispatcher" daemon prio=1 tid=0x080a2068 nid=0xf7d waiting on
    condition [0x00000000..0x00000000]
    "Finalizer" daemon prio=1 tid=0x0809c540 nid=0xf7c in Object.wait()
    [0xb24b2000..0xb24b3040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=1 tid=0x0809a6b0 nid=0xf7b in
    Object.wait() [0xb2533000..0xb25331c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x88f01e60> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:474)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x88f01e60> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x0805d1c8 nid=0xf79 runnable [0xbfbd9000..0xbfbda4f8]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x8946f690> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:388)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:615)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:575)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)
    "VM Thread" prio=1 tid=0x08097b58 nid=0xf7a runnable
    "VM Periodic Task Thread" prio=1 tid=0x080a5998 nid=0xf80 waiting on condition

    Thanks alot for your reply. It is a web application
    using java and tomcat server. And the problem is that
    sometime when I click on a button or a link than the
    CPU goes to 100% and it hangs but normally it works
    smoothly without any problem.This often indicates a busy retry loop somewhere. You need to acquire the thread dump when the problem occurs, but be warned that depending on the problem it may not be possible to obtain a Java-level thread dump. In that case you need to try and take an OS level thread dump - eg using pstack on solaris/linux (and there's some win32 tool as well if I recall correctly).

  • What is the the overhead of a thread dump operation

    If I perform a thread dump every two minutes on the weblogic appserver
    is it going to have a severe impact on the performance.
    Please help.
    Thank you
    Madhuri

    Thank you very much.
    -MAdhuri
    Cameron Purdy wrote:
    You can time it by having a slightly higher priority thread measure the
    number of milliseconds between iterations in a while(true) loop. An time
    the number is >1 print it out. Run the test (in your server) and do the
    thread dump.
    Peace,
    Cameron Purdy
    Tangosol, Inc.
    http://www.tangosol.com
    +1.617.623.5782
    WebLogic Consulting Available
    "Mike Reiche" <[email protected]> wrote in message
    news:3b003ac1$[email protected]..
    No. You won't even notice a difference. It's probably
    less work than a Full GC.
    Mike
    Madhuri Mandava <[email protected]> wrote:
    If I perform a thread dump every two minutes on the weblogic appserver
    is it going to have a severe impact on the performance.
    Please help.
    Thank you
    Madhuri

  • Getting thread dump of JWS application

    Hi,
    I know this was discussed a few times before, but I still have no solution for my problem.
    I have a JWS GUI application that deadlocks sometimes (Windows 98). So I want to get the thread dump to analyze the deadlock. In Java Web Start I changed the java command from javaw.exe to java.exe, so I get a dos box. When I press CTRL-BREAK inside the dos box, I get the thread dump but because the dos box just shows a few lines I only get the last lines of the dump. Under Windows 98 I cannot resize the dos box.
    Is there a way to redirect the output of the dos box to a file? Or is there another way to get the thread dump of a JWS application?
    Thanks in advance
    Markus

    Here is 50% of the solution. To generate a thread dump programatically, you need to call JVM_DumpAllStacks through JNI. You need to link in "jvm.lib".
    JNIEXPORT void JNICALL JVM_DumpAllStacks(JNIEnv *env, jclass unused);
    JNIEXPORT void JNICALL Java_Services_generateThreadDump(JNIEnv *env, jobject thisObject) {
         // Produce a thread dump to VM standard out
         JVM_DumpAllStacks( env, NULL );
    }Here is the java class that calls the JVM_DumpAllStacks method through JNI.
    public class Services {
          * Send a message to the VM, forcing it to create a thread dump. The resulting
          * thread dump will be sent to stdout. Note that this is the VM's stdout, not
          * necessarily System.out.
         public static native void generateThreadDump();
    }When you reach the point in your code that you want to generate a thread dump, close stdout and reopen it with the thread dump destination stream. Next, call the JNI method. This will send the thread dump to stdout.
         // Replace stdout with a file output stream
         System.out.close();
         System.setOut( new PrintStream( new FileOutputStream( "stdout.log" )));
         // Generate thread dump
         Services.generateThreadDump();Why is this only 50% of the solution ? Because this works beautifully for regular applications but it does NOT work for WebStart applications. Not sure what is happening here, but so far I have not managed to get any thread dump in a file from an application that has been fired up through WebStart. Note that I CAN get a thread dump in a file if I fire up the same code from the command line.

  • How to get a thread dump, in case of an IDE freeze

    When you're working with the IDE and if it either freezes or becomes sluggish for no apparent reason, it's recommended to generate a thread dump, and report it to Studio feedback alias ([email protected]) with a description of what happened and attach the dump to it.
    A thread dump is an invaluable source of information to IDE developers when investigating deadlocks and some performance issues. It is a textual dump of all active threads and monitors of Java apps running in a Virtual Machine.
    The ways to generate a thread dump differ depending on the platform:
    * Windows systems
    Press Ctrl-Break in the command console you used to start the IDE.
    You won't be able to see the console in case you started the IDE using the default desktop icon created by the IDE installer. To be able to generate the thread dump, you must launch the IDE using a console window (invoke Command Prompt, go to {studio-install-dir}/bin and type runide.exe), or configure the desktop shortcut properties to launch the runide.exe program instead of the default runidew.exe.
    Note: You should increase the screen buffer size of the command prompt, so that it could retian the entire thread dump output. To do that, on the command prompt where IDE is running, open the properties dialog box by right clicking on the titlebar and selecting 'properties'. Then select the 'layout tab' and increase the 'screen buffer size' parameters to - width - 200, Height - 4000.
    * Unix systems
    Press Ctrl-\ in the terminal console you used to start the JSE.
    Alternatively, you can also generate a thread dump by sending the QUIT signal to the Java VM running the JSE
    kill -QUIT process_id
    where process_id is the process number of the respective java process.
    -------Here is a sample thread dump -------
    Full thread dump Java HotSpot(TM) Client VM (1.4.2_01-b06 mixed mode):
    "Text-Layout" daemon prio=2 tid=0x0886ebd8 nid=0x4e4 in Object.wait() [9e8f000..9e8fd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x11730910> (a org.netbeans.editor.view.spi.ViewLayoutQueue)
    at java.lang.Object.wait(Object.java:429)
    at org.netbeans.editor.view.spi.ViewLayoutQueue.waitForTask(ViewLayoutQueue.java:128)
    - locked <0x11730910> (a org.netbeans.editor.view.spi.ViewLayoutQueue)
    at org.netbeans.editor.view.spi.ViewLayoutQueue$LayoutThread.run(ViewLayoutQueue.java:182)
    "Compilation" daemon prio=2 tid=0x0883d878 nid=0xb1c in Object.wait() [9c8f000..9c8fd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1143dde0> (a java.util.LinkedList)
    at java.lang.Object.wait(Object.java:429)
    at org.netbeans.core.compiler.CompilationEngineImpl$CompilerThread.nextJobAndTask(CompilationEngineImpl.java:162)
    - locked <0x1143dde0> (a java.util.LinkedList)
    at org.netbeans.core.compiler.CompilationEngineImpl$CompilerThread.run(CompilationEngineImpl.java:175)
    "TimerQueue" daemon prio=5 tid=0x03575478 nid=0xb18 in Object.wait() [988f000..988fd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x10d4af60> (a javax.swing.TimerQueue)
    at javax.swing.TimerQueue.run(TimerQueue.java:231)
    - locked <0x10d4af60> (a javax.swing.TimerQueue)
    at java.lang.Thread.run(Thread.java:534)
    "AWT-EventQueue-1" prio=7 tid=0x0344fae0 nid=0xb14 in Object.wait() [3caf000..3cafd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x10d4afe8> (a java.awt.EventQueue)
    at java.lang.Object.wait(Object.java:429)
    at java.awt.EventQueue.getNextEvent(EventQueue.java:339)
    - locked <0x10d4afe8> (a java.awt.EventQueue)
    at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:162)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:145)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:137)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:100)

    You can get a thread dump (Windows only) whitout any preparation. Simply use the Stack Trace tool from this web site:
    http://tmitevski.users.mcs2.netarray.com
    It works on Windows services too.

  • What's a good way to do a thread dump into a separate file

    What is a good way to do a thread dump automatically into a separate file.
    Example. I run a script to do the thread dump, but unfortunetly, it goes into my stdout log file with the rest of my weblogic errors.
    Any ideas? I want it in a separate file when I run my script?

    Do a Google search on "Drobo S" "benchmark."  I don't have a Drobo S, only the regular Drobo.  But here's a guy who tested one on Windows:
    http://mansurovs.com/drobo-s-review-usb-3-0-2nd-generation
    This one has it a bit faster:
    http://the-gadgeteer.com/2011/12/31/drobo-s-storage-array-review/
    Do read up on a few reviews of it, and be absolutely clear that interface speed (i.e. eSATA versus Firewire versus Thunderbolt) is NOT the same as the performance of the system.  The Drobo cannot keep up with any interfaces... at least the Drobo and the Drobo S cannot.
    I am not using the FS model which is a NAS.  I am using the plain old "Drobo" which is slower than the Drobo S, but that's not to say that the Drobo S is fast, because it is not.
    The Drobo in theory is really attractive: Dead simple to manage, can mix and match drive sizes, offers you some data protection, etc.  However do note that protected storage is not, in and of itself, a backup.  You need other backups besides just the data on the Drobo.  And, because it's so slow, it's really not a great fit for photo storage.  See this review from a guy who used to think the Drobo was great for that and then appended his review:
    http://www.stuckincustoms.com/drobo-review/
    To be as clear as possible, IMO the BEST backup strategy with something like Aperture (so long as your managed Aperture library is of a manageable size, like < 800 GB), is to get a few small portable Firewire 800 drives and keep vaults on each one.  They are great because they are easy to use, to have with you, are bus powered, and you WILL offsite them.

  • Why the Process is still in deadlock when i have exhaust its error stream.

    There is the code:
    import java.io.*;
    public class ProcessExec {
      public static void main(String[] args) {
         * Test the Process created by Runtime.exec().
        Runtime rt = Runtime.getRuntime();
        String s = new String();
        if(args.length < 1) {
          System.out.println("Usage: java ProcessExec executionName");
          System.exit(-2);
        try {
          Process process = rt.exec(args[0]);
           * Buffer the error stream and exhaust it.
          System.out.println("ErrorStream data of Porcess:");
          BufferedReader br = new BufferedReader(new InputStreamReader(process.getErrorStream()));
          while( (s = br.readLine()) != null) {
            System.out.println(s);
          System.out.println("\n\n------------------------------------------------------------\n\n");
           * Buffer the input stream and exhaust it.
          br = new BufferedReader(new InputStreamReader(process.getInputStream()));
          System.out.println("InputStream data of Porcess:");
          while( (s = br.readLine()) != null) {
            System.out.println(s);
          System.out.println("Return num: " + process.waitFor());
        catch(Throwable t) {
          System.out.println(t);
    }When i run the "javac" as :
    C:\classses\lang\java ProcessExec javac
    The process run correctly and print the error stream.
    But when i run the "java" as:
    C:\classes\lang\java ProcessExec java
    The process was in deadlock?
    Why,in my code i have buffered the error stream and input stream using a BufferedReader and then exhausted them.
    And when i invoke the "javac" it ran correctly but why it couldnt run correctly when i invoked the "java" execution?

    A deadlock occurs when two locks are dependant on each other.
    This is unlikely to be the case here.
    With out your exact command line or what your output is or if you have a dealock we will need to see the thread dump which displays where the deadlock is.

  • Weblogic sp4 on linux out of memory expections - Thread dump

    Any body has idea why out of memory expections?
    Here is the thread dump.
    Full thread dump Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode):
    "Thread-33" daemon prio=1 tid=0x086175f0 nid=0x4f8e runnable [c8d57000..c8d5723c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xd48f7820> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.apache.axis.monitor.SOAPMonitorService$ServerSocketThread.run(SOAPMonitorService.java:189)
         at java.lang.Thread.run(Thread.java:534)
    "CustomerClientDispatcherThread" daemon prio=1 tid=0x08877100 nid=0x4f8e in Object.wait() [c77ff000..c77ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd45b7330> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at java.lang.Object.wait(Object.java:429)
         at com.amgen.ea.orx.queue.impl.PersistenceQueue.peek(PersistenceQueue.java:58)
         - locked <0xd45b7330> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at com.amgen.ea.orx.custclient.impl.CustomerClientDispatcher.run(CustomerClientDispatcher.java:50)
         at java.lang.Thread.run(Thread.java:534)
    "CustomerClientDispatcherThread" daemon prio=1 tid=0xcb625c28 nid=0x4f8e in Object.wait() [c7b8c000..c7b8c23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3ce7958> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at java.lang.Object.wait(Object.java:429)
         at com.amgen.ea.orx.queue.impl.PersistenceQueue.peek(PersistenceQueue.java:58)
         - locked <0xd3ce7958> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at com.amgen.ea.orx.custclient.impl.CustomerClientDispatcher.run(CustomerClientDispatcher.java:52)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-8" daemon prio=1 tid=0x08710e80 nid=0x4f8e in Object.wait() [c7cc4000..c7cc423c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd38b44c8> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at java.lang.Object.wait(Object.java:429)
         at com.amgen.ea.orx.queue.impl.PersistenceQueue.peek(PersistenceQueue.java:58)
         - locked <0xd38b44c8> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at com.amgen.ea.orx.custclient.impl.CustomerClientDispatcher.run(CustomerClientDispatcher.java:52)
         at java.lang.Thread.run(Thread.java:534)
    "ListenThread.Default" prio=1 tid=0xcc21b658 nid=0x4f8e runnable [c7c0c000..c7c0c23c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xd3811cf8> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Thread-6" prio=1 tid=0xcb3ef610 nid=0x4f8e in Object.wait() [c92a0000..c92a023c]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xd348db48> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-5" prio=1 tid=0x08c46fd8 nid=0x4f8e in Object.wait() [c9500000..c950023c]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xd34901c8> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "LDAPConnThread-0 ldap://0.0.0.0:7013" daemon prio=1 tid=0xcb67b150 nid=0x4f8e runnable [c9580000..c958023c]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <0xd3490a98> (a java.io.BufferedInputStream)
         at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
         at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
         at java.lang.Thread.run(Thread.java:534)
    "VDE Transaction Processor Thread" prio=1 tid=0xcb2fe110 nid=0x4f8e in Object.wait() [c9600000..c960023c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3490b30> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at java.lang.Object.wait(Object.java:429)
         at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:365)
         - locked <0xd3490b30> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=1 tid=0x08665ba8 nid=0x4f8e in Object.wait() [c9680000..c968023c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3490ba8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon prio=1 tid=0x0830a7e0 nid=0x4f8e in Object.wait() [c9700000..c970023c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3490c28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon prio=1 tid=0x080a0c00 nid=0x4f8e in Object.wait() [c9780000..c978023c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3490ca8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=1 tid=0xcafb3be8 nid=0x4f8e runnable [c98ff000..c98ff23c]
         at weblogic.socket.PosixSocketMuxer.poll(Native Method)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
         - locked <0xd31100e0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=1 tid=0xcaf65228 nid=0x4f8e waiting for monitor entry [c997f000..c997f23c]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
         - waiting to lock <0xd31100e0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=1 tid=0xcafb0c20 nid=0x4f8e waiting for monitor entry [c99ff000..c99ff23c]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
         - waiting to lock <0xd31100e0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "weblogic.security.SpinnerRandomSource" daemon prio=1 tid=0xcaf64f98 nid=0x4f8e in Object.wait() [c9a7f000..c9a7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31182d8> (a java.lang.Object)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.security.SpinnerRandomBitsSource.run(SpinnerRandomBitsSource.java:60)
         - locked <0xd31182d8> (a java.lang.Object)
         at java.lang.Thread.run(Thread.java:534)
    "weblogic.time.TimeEventGenerator" daemon prio=1 tid=0xcb3e1400 nid=0x4f8e in Object.wait() [c9aff000..c9aff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118348> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <0xd3118348> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb3df3a8 nid=0x4f8e in Object.wait() [c9b7f000..c9b7f23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31183b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb3de7a0 nid=0x4f8e in Object.wait() [c9bff000..c9bff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118438> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb3ddcc0 nid=0x4f8e in Object.wait() [c9c7f000..c9c7f23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31184b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb1c66f0 nid=0x4f8e in Object.wait() [c9cff000..c9cff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118538> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb1c5b58 nid=0x4f8e in Object.wait() [c9d7f000..c9d7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31185b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31185b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '39' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c4f50 nid=0x4f8e in Object.wait() [c9dff000..c9dff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118638> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '38' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c4348 nid=0x4f8e in Object.wait() [c9e7f000..c9e7f23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31186b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '37' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c3740 nid=0x4f8e in Object.wait() [c9eff000..c9eff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118738> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '36' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c2ba8 nid=0x4f8e in Object.wait() [c9f7f000..c9f7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31187b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31187b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '35' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb674078 nid=0x4f8e in Object.wait() [c9fff000..c9fff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118838> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118838> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '34' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb673470 nid=0x4f8e in Object.wait() [ca07f000..ca07f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31188b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31188b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '33' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb6728d8 nid=0x4f8e in Object.wait() [ca0ff000..ca0ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118938> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118938> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '32' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb671cd0 nid=0x4f8e in Object.wait() [ca17f000..ca17f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31189b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31189b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '31' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb6710e0 nid=0x4f8e in Object.wait() [ca1ff000..ca1ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118a38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118a38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '30' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfdb548 nid=0x4f8e in Object.wait() [ca27f000..ca27f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118ab8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118ab8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '29' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfda940 nid=0x4f8e in Object.wait() [ca2ff000..ca2ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118b38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118b38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '28' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfd9d58 nid=0x4f8e in Object.wait() [ca37f000..ca37f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118bb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118bb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '27' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfd91c0 nid=0x4f8e in Object.wait() [ca3ff000..ca3ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118c38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118c38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '26' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb119a40 nid=0x4f8e in Object.wait() [ca47f000..ca47f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118cb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118cb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '25' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb118e38 nid=0x4f8e in Object.wait() [ca4ff000..ca4ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118d38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118d38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb118230 nid=0x4f8e in Object.wait() [ca57f000..ca57f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118db8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118db8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '23' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcc207638 nid=0x4f8e in Object.wait() [ca5ff000..ca5ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118e38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118e38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcc206a30 nid=0x4f8e in Object.wait() [ca67f000..ca67f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118eb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118eb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '21' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcc205e28 nid=0x4f8e in Object.wait() [ca6ff000..ca6ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118f38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118f38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfbf1e0 nid=0x4f8e in Object.wait() [ca77f000..ca77f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118fb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118fb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfbe648 nid=0x4f8e in Object.wait() [ca7ff000..ca7ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119038> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119038> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfbda40 nid=0x4f8e in Object.wait() [ca87f000..ca87f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31190b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31190b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '17' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb62c500 nid=0x4f8e in Object.wait() [ca8ff000..ca8ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119138> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119138> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '16' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb62b8f8 nid=0x4f8e in Object.wait() [ca97f000..ca97f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31191b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31191b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb62acf0 nid=0x4f8e in Object.wait() [ca9ff000..ca9ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119238> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119238> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb61f5a8 nid=0x4f8e in Object.wait() [caa7f000..caa7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31192b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31192b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb61e9e0 nid=0x4f8e in Object.wait() [caaff000..caaff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119338> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119338> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf13af0 nid=0x4f8e in Object.wait() [cab7f000..cab7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31193b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31193b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf12ee8 nid=0x4f8e in Object.wait() [cabff000..cabff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119438> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119438> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf12350 nid=0x4f8e in Object.wait() [cac7f000..cac7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31194b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31194b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb309598 nid=0x4f8e in Object.wait() [cacff000..cacff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119538> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119538> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb3089c8 nid=0x4f8e in Object.wait() [cad7f000..cad7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31195b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31195b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf56250 nid=0x4f8e in Object.wait() [cadff000..cadff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119638> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119638> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55cb0 nid=0x4f8e in Object.wait() [cae7f000..cae7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31196b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31196b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55918 nid=0x4f8e in Object.wait() [caeff000..caeff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119738> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119738> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55510 nid=0x4f8e in Object.wait() [cb47f000..cb47f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31197b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31197b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55150 nid=0x4f8e in Object.wait() [cb4ff000..cb4ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119838> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119838> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfc7bd8 nid=0x4f8e in Object.wait() [cb57f000..cb57f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31198b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31198b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb3cd250 nid=0x4f8e in Object.wait() [cb5ff000..cb5ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119938> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119938> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb3cd0b8 nid=0x4f8e in Object.wait() [cc1fe000..cc1fe23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31199b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31199b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-1" daemon prio=1 tid=0x086d6478 nid=0x4f8e in Object.wait() [cc3e1000..cc3e123c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119a88> (a java.util.TaskQueue)
         at java.lang.Object.wait(Object.java:429)
         at java.util.TimerThread.mainLoop(Timer.java:403)
         - locked <0xd3119a88> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Signal Dispatcher" daemon prio=1 tid=0x080c4170 nid=0x4f8e waiting on condition [0..0]
    "Finalizer" daemon prio=1 tid=0x080bfac0 nid=0x4f8e in Object.wait() [cf509000..cf50923c]
         at java.lang.Object.wait(Native Method)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0xd3129c78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Refe

    <p>Can you post the stack trace. I assume these thread dumps were taken after the error occured. Can you take a few dumps just before you run out of memory.</p>
    <p>
    Hussein Badakhchani</br>
    </p>

  • 100% CPU load but no clue in thread dump, EP. Win a bottle of champagne...

    Hi,
    We are suddenly facing 100% CPU load in our EP cluster (5 x 8-way Xeon multiprocessor machines). We have a serious performance problem that is burning a lot of our time and causes a lot of stress for over 3 weeks now.
    We have taken tens of thread dumps from Application Nodes.
    - In none of the thread dumps we see the Finalizer thread
    running.
    - From the garbage collector log we see that full garbage
    collection runs occur very rarily (once an hour).
    - We use the compacting garbage collector.
    - In a lot of thread dumps we do not even see our own portal code in the stack traces of the threads!
    - In Windows Task Manager we see the jlaunch.exe processes
    consume all available CPU time.
    - Portal users see a blank portal page when the CPU load hits 100%. When the load goes down again, things return to normal.
    - There are no errors logged in any of the log files of the portal. We checked all of them. We expected a log full of errors somewhere but nothing even remotely interesting was found. Windows Event Viewer shows nothing either.
    - The amount of sockets in CLOSE_WAIT status is < 10 on every machine in the cluster.
    This forces me to conclude that something in the jlaunch.exe executable consumes the CPU time. This raises the following 3 questions:
    - What does the mystifying jlaunch.exe do besides executing java.exe ?
    - Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ? Can it be GZip compression ?
    - If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
    Our development- and support teams are desperate. All suggestions are welcome. The person that comes up with
    the solution to our problem gets a nice bottle of champagne.
    Regards,
    Chris Twigt

    Hi,
    - What does the mystifying jlaunch.exe do besides executing java.exe ?
    This is so that the startupframework can connect more easily and take control of the JVM in some situations
    - Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ?
    I assume when SAP release their JVM (in the next major release), that will be called directly.
    - If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
    Loads of thing, but there should be clues in the thread dumps.  (the reason why your code is not in those thread dumps is that your code is only active during the processing of a particular request, unless you have a service, afterwards there are no trace of it as the thread which does the processing goes back to sleep)
    I've experienced a similar situation with a an 6.40 portal, and it was then caused by the following:
    1. User A comes logs in and sees that some cache timeout has occured , therefore it issues a SQL query which does a full table scan on a table of approx. 1 GB (on of the UME tables)
    2. User B comes in just afterwards and also sees that the cache timeout has occured, and issues the same SQL query as User A
    3. User C .... and so on untill the query from User A eventually finishes
    So
    1. Check database. Any big queries running ?
    2. Check file activity, are there a lot of writing ?
    3. Check network activity, especially to the state controller which is the weakest link
    4. What about portal logs ? Any activity during the hang ?
    Also, please provide one of the thread dumps for further analysis..
    cheers
    Dagfinn

  • Waiting for monitor enty messages in thread dump

    Hi,
    I've a thread dump of a container in a Oracle Application Server 9.0.4.1 (running 1.4.2_07) right before restart because off low memory and a lot of threads in this dump are in this state:
    "AJPRequestHandler-ApplicationServerThread-1588" prio=1 tid=0x37634bf8 nid=0x1982 waiting for monitor entry [43f22000..43f2386c]
    can anyone tell me what this means? Google didn't reveal anything useful.
    Another thing I don't understand although it seems it doesn't do any harm:
    "AJPRequestHandler-ApplicationServerThread-1630" prio=1 tid=0x09011a68 nid=0x1982 in Object.wait() [40276000..4027686c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x6624e628> (a java.util.HashMap)
    at java.lang.Object.wait(Object.java:429)
    at java.net.InetAddress.checkLookupTable(InetAddress.java:1109)
    - locked <0x6624e628> (a java.util.HashMap)
    at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1048)
    at java.net.InetAddress.getLocalHost(InetAddress.java:1188)
    at com.evermind.server.http.EvermindHttpSession.encodeSession(EvermindHttpSession.java:414)
    at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:1599)
    why is this thread waiting for a lock itself has?
    Thanx for any help and hints in advance.
    Regards,
    Ingo

    Hi,
    I've a follow-up question, I have the following thread in several thread dumps:
    "AJPRequestHandler-ApplicationServerThread-717" prio=1 tid=0x4046b988 nid=0x3dd3 waiting for monitor entry [3845e000..3845e86c]
         at java.net.Inet4Address.getAddress(Inet4Address.java:287)
         at com.evermind.server.http.StandardRequestLogger.appendIP(StandardRequestLogger.java:187)
         at com.evermind.server.http.StandardRequestLogger.log(StandardRequestLogger.java:45)
         - locked <0x663b2088> (a com.evermind.server.http.StandardRequestLogger)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:906)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    What is the monitor this thread is waiting for? According to the source in Inet4Address at line 287 the method tries to create a new byte array... is there some system monitor which can't be seen in the thread dump?
    This thread holds a lock (0x663b2088) 46 other threads are waiting for...
    Thanx in advance for any hints.
    Regards,
    Ingo

Maybe you are looking for

  • Year is wrong on my ipod

    My year says 2555 BE on my ipod... how can i change that?

  • Creation of Program using ABAP Query (SQ01)

    Hi All, I have a program called "AQL4FI==========INVOICE_CHECK=" in my 4.6 server but a similar program doesnot exist in ECC. From the name, it looks like a program created using SQ01. Can anyone please help as to how to create the same program in EC

  • How do I get my contacts from a 4s iPhone onto an iPhone 3g?

    The highest operating system that the iPhone 3g is capable of is the 4.2, I think. And I had the newest operating system installed on my 4s so iTunes will not let me sync the backup of my 4s to the iPhone 3g...I need to know if there is any way aroun

  • IPhone 4 Bumper Marks on Phone?

    I have marks on the aluminium rim of my iPhone and I believe it is due to the bumper because, from day one, I have the bumper on and I've just taken it off and seen a lot of precise, small marks on the iPhone 4. I was wondering if anyone else had exp

  • How can I stop adverts with sound coming up?

    When I go to certain web sites a video with sound comes up which is very awkward for my wife in the middle of the night. Can I suppress adverts like this?