Help on this thread dump..

hey guys, im having a problem reading this thread dump. i'm just a newbie. could you help me point what's wrong in our system based on this thread dump? there were no errors in our logs, it just hangs and we're so confused as to what the issue is. here's the thread dump:
Full thread dump Java HotSpot(TM) Client VM (1.4.0_02-b02 mixed mode):
"MessageWriter_370" prio=5 tid=0x3479e0 nid=0x1df waiting on monitor [94381000..94381a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a690dd88> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a690dd88> (a java.util.ArrayList)
"MessageReader_369" prio=5 tid=0x33f888 nid=0x1de runnable [94581000..94581a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a690de38> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a690df58> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a690ddb0> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_368" prio=5 tid=0x2784c8 nid=0x1dc waiting on monitor [94881000..94881a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a68e6d90> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a68e6d90> (a java.util.ArrayList)
"MessageReader_367" prio=5 tid=0x279230 nid=0x1db waiting for monitor entry [94981000..94981a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_366" prio=5 tid=0x5c2090 nid=0x1da waiting on monitor [90e81000..90e81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a68df070> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a68df070> (a java.util.ArrayList)
"MessageReader_365" prio=5 tid=0x27da50 nid=0x1d9 runnable [90f81000..90f81a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a68cb480> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a68cb520> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a68cb408> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_364" prio=5 tid=0x5c1f00 nid=0x1d8 waiting on monitor [91281000..91281a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a68ca1a0> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a68ca1a0> (a java.util.ArrayList)
"MessageReader_363" prio=5 tid=0x753a68 nid=0x1d7 waiting for monitor entry [91381000..91381a28]
     at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
     - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
     at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
     at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
     - locked <a6b3d378> (a com.presumo.jms.message.AckHelper)
     at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
     - locked <a68ca178> (a com.presumo.jms.router.PersistentAckHandler)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_362" prio=5 tid=0x5c09a8 nid=0x1d6 waiting on monitor [91481000..91481a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a6867f58> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a6867f58> (a java.util.ArrayList)
"MessageReader_361" prio=5 tid=0x2790d8 nid=0x1d5 runnable [91581000..91581a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a6868008> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a6868128> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a6867f80> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_352" prio=5 tid=0x34f730 nid=0x1c9 waiting on monitor [94c81000..94c81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a6744968> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a6744968> (a java.util.ArrayList)
"MessageReader_351" prio=5 tid=0xd7370 nid=0x1c8 runnable [94d81000..94d81a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a6744a18> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a6744b38> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a6744990> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_350" prio=5 tid=0x34fb38 nid=0x1c7 waiting on monitor [95081000..95081a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a66ef950> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a66ef950> (a java.util.ArrayList)
"MessageReader_349" prio=5 tid=0x278620 nid=0x1c6 runnable [95181000..95181a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a66efa00> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a66efb20> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a66ef978> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_344" prio=5 tid=0x6a0370 nid=0x1c0 waiting on monitor [92481000..92481a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a66665a8> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a66665a8> (a java.util.ArrayList)
"MessageReader_343" prio=5 tid=0x33f730 nid=0x1bf waiting for monitor entry [92581000..92581a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_332" prio=5 tid=0x3a0d38 nid=0x1b0 waiting on monitor [93681000..93681a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a620e700> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a620e700> (a java.util.ArrayList)
"MessageReader_331" prio=5 tid=0x69dd18 nid=0x1af waiting for monitor entry [93781000..93781a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_318" prio=5 tid=0x39fc10 nid=0x19e waiting on monitor [95c81000..95c81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5f1f0f0> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5f1f0f0> (a java.util.ArrayList)
"MessageReader_317" prio=5 tid=0x69da68 nid=0x19d waiting on monitor [95d81000..95d81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
     at java.lang.Object.wait(Object.java:426)
     at epylon.persistence.JmsPersistenceManager.obtainLock(JmsPersistenceManager.java:144)
     - locked <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
     at com.presumo.jms.persistence.PersistentQueue.push(PersistentQueue.java:142)
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - locked <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_316" prio=5 tid=0x254c30 nid=0x19c waiting on monitor [96281000..96281a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5e9d7e0> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5e9d7e0> (a java.util.ArrayList)
"MessageReader_315" prio=5 tid=0x69d910 nid=0x19b waiting for monitor entry [96381000..96381a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_314" prio=5 tid=0x3a1bf0 nid=0x19a waiting on monitor [99081000..99081a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5e24348> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5e24348> (a java.util.ArrayList)
"MessageReader_313" prio=5 tid=0x69d7b8 nid=0x199 waiting for monitor entry [99181000..99181a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_312" prio=5 tid=0x64c608 nid=0x198 waiting on monitor [99681000..99681a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5df1a88> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5df1a88> (a java.util.ArrayList)
"MessageReader_311" prio=5 tid=0x5d00a0 nid=0x197 waiting for monitor entry [99781000..99781a28]
     at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
     - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
     at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
     at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
     - locked <a6b3d538> (a com.presumo.jms.message.AckHelper)
     at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
     - locked <a5df1a60> (a com.presumo.jms.router.PersistentAckHandler)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_310" prio=5 tid=0x3a0be0 nid=0x195 waiting on monitor [91a81000..91a81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5dd5370> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5dd5370> (a java.util.ArrayList)
"MessageReader_309" prio=5 tid=0x3a7108 nid=0x194 waiting for monitor entry [91b81000..91b81a28]
     at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:150)
     - waiting to lock <a6a87030> (a com.presumo.jms.message.AckHelper)
     at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
     - locked <a5dd5348> (a com.presumo.jms.router.PersistentAckHandler)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_308" prio=5 tid=0x3a1290 nid=0x193 waiting on monitor [91e81000..91e81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5d9c338> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5d9c338> (a java.util.ArrayList)
"MessageReader_307" prio=5 tid=0x42ce10 nid=0x192 runnable [91f81000..91f81a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a5d9c3e8> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a5d9c508> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a5d9c360> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_302" prio=5 tid=0x4beef8 nid=0x18c waiting on monitor [93a81000..93a81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5cce5c8> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5cce5c8> (a java.util.ArrayList)
"MessageReader_301" prio=5 tid=0x577eb0 nid=0x18b runnable [93b81000..93b81a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a5cce678> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a5cce798> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a5cce5f0> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_298" prio=5 tid=0x413b08 nid=0x187 waiting on monitor [93c81000..93c81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5c410d0> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5c410d0> (a java.util.ArrayList)
"MessageReader_297" prio=5 tid=0x353b58 nid=0x186 waiting for monitor entry [93d81000..93d81a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_296" prio=5 tid=0x592028 nid=0x185 waiting on monitor [94081000..94081a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5bd4cd8> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5bd4cd8> (a java.util.ArrayList)
"MessageReader_295" prio=5 tid=0x66d660 nid=0x184 waiting for monitor entry [94181000..94181a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_294" prio=5 tid=0x1fa4e0 nid=0x183 waiting on monitor [94281000..94281a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5b7e880> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5b7e880> (a java.util.ArrayList)
"MessageReader_293" prio=5 tid=0x34f9e0 nid=0x182 waiting for monitor entry [94481000..94481a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_292" prio=5 tid=0x34de60 nid=0x17f waiting on monitor [91081000..91081a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5b502a8> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5b502a8> (a java.util.ArrayList)
"MessageReader_291" prio=5 tid=0x592430 nid=0x17e runnable [91181000..91181a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a5b3cdb0> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a5b3ce50> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a5b3cd38> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_282" prio=5 tid=0x57d608 nid=0x173 waiting on monitor [92c81000..92c81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a59cd8b8> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a59cd8b8> (a java.util.ArrayList)
"MessageReader_281" prio=5 tid=0x1f5420 nid=0x172 waiting for monitor entry [92d81000..92d81a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_262" prio=5 tid=0x3a2ad8 nid=0x15b waiting on monitor [91c81000..91c81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a576a760> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a576a760> (a java.util.ArrayList)
"MessageReader_261" prio=5 tid=0x3a20a0 nid=0x15a waiting for monitor entry [91d81000..91d81a28]
     at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
     - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
     at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
     at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
     - locked <a6b39f60> (a com.presumo.jms.message.AckHelper)
     at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
     - locked <a576a738> (a com.presumo.jms.router.PersistentAckHandler)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_258" prio=5 tid=0x4129b0 nid=0x157 waiting on monitor [92a81000..92a81a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5716ab8> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5716ab8> (a java.util.ArrayList)
"MessageReader_257" prio=5 tid=0x581bc0 nid=0x156 waiting for monitor entry [92b81000..92b81a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_256" prio=5 tid=0x5873e8 nid=0x155 waiting on monitor [92081000..92081a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a56f6278> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a56f6278> (a java.util.ArrayList)
"MessageReader_255" prio=5 tid=0x587290 nid=0x154 runnable [92181000..92181a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a56e2f98> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a56e2ff0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a56e2f38> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_252" prio=5 tid=0x5806f0 nid=0x150 waiting on monitor [92881000..92881a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a567f138> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a567f138> (a java.util.ArrayList)
"MessageReader_251" prio=5 tid=0x57fd98 nid=0x14f waiting for monitor entry [92981000..92981a28]
     at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
     - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
     at com.presumo.jms.router.Router.routeMessages(Router.java:215)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_214" prio=5 tid=0x30f950 nid=0x124 waiting on monitor [95281000..95281a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a529f260> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a529f260> (a java.util.ArrayList)
"MessageReader_213" prio=5 tid=0x55faa8 nid=0x123 waiting for monitor entry [95381000..95381a28]
     at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
     - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
     at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
     at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
     - locked <a6b683a8> (a com.presumo.jms.message.AckHelper)
     at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
     - locked <a529f238> (a com.presumo.jms.router.PersistentAckHandler)
     at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
"MessageWriter_210" prio=5 tid=0x4dd0a8 nid=0x11f waiting on monitor [93081000..93081a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a52692c0> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a52692c0> (a java.util.ArrayList)
"MessageReader_209" prio=5 tid=0x4c2fb8 nid=0x11e runnable [93181000..93181a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a5256468> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a5256508> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a52563f0> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_208" prio=5 tid=0x4c6de0 nid=0x11d waiting on monitor [93481000..93481a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a5241588> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a5241588> (a java.util.ArrayList)
"MessageReader_207" prio=5 tid=0x5c2220 nid=0x11c runnable [93581000..93581a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a5241638> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a5241758> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
     - locked <a52415b0> (a java.lang.String)
     at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
"MessageWriter_188" prio=5 tid=0x642b88 nid=0x105 waiting on monitor [92281000..92281a28]
     at java.lang.Object.wait(Native Method)
     - waiting on <a50e4248> (a java.util.ArrayList)
     at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
     - locked <a50e4248> (a java.util.ArrayList)
"MessageReader_187" prio=5 tid=0x1f92e8 nid=0x104 runnable [92381000..92381a28]
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:116)
     at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     - locked <a50e42f8> (a java.lang.Object)
     at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
     at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
     - locked <a50e4418> (a com.sun.net.ssl.internal.ssl.AppInputStream)
     at ja

Here's the lengthy code :)
package com.presumo.jms.router;
import com.presumo.jms.message.JmsMessage;
import com.presumo.jms.message.SystemMessageConstants;
import com.presumo.jms.plugin.implementation.MemoryMessageQueue;
import com.presumo.jms.plugin.transport.Transport;
import com.presumo.jms.selector.JmsOperand;
import com.presumo.jms.selector.Parser;
import com.presumo.jms.JmsServer;
import com.presumo.jms.persistence.PersistentQueue;
import com.presumo.jms.resources.Resources;
import com.presumo.util.log.Logger;
import com.presumo.util.log.LoggerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.LinkedList;
import javax.jms.InvalidSelectorException;
import javax.jms.JMSException;
import epylon.bean.MessageStatusBean;
import epylon.persistence.JmsPersistenceManager;
import epylon.util.*;
import org.apache.log4j.Category;
* Class that sits between the Router and the Transport layer in
* terms of message flow. Essentially encapsulates the remote
* connection from the router, as well as handling the persistent
* message acknowledgment scheme.
* @author Dan Greff
public class RemoteSession implements RoutingTarget
/** Largest batch of messages that will be bundled together **/
protected static final int MSG_BATCH_SIZE = 2000;
/** Time that between sending batches of messages **/
protected static final long latency = 500;
/** Property name for sending filters via system messages **/
protected static final String FILTER_PROPERTY = "JMSX_FILTER";
/** Property name for sending acks via system messages **/
protected static final String ACKS_PROPERTY = "JMSX_ACKS";
/** Filter sent to this JVM from the other side of the connection
* which indicates what messages the router on the other side is
* interested in.
protected JmsOperand remoteFilter;
/** Local filter which the represents what messages the local
* router is interested in. This filter is sent to other routers.
protected JmsOperand localFilter;
/** Router this RemoteSession is connected to **/
protected Router router = null;
/** Interface which messages are sent to and received from the network **/
protected Transport transport;
/** Listener to be notified of if the connection is lost **/
protected ConnectionListener connxListener;
/** Handles acknowledgement logic for persistent messages **/
protected PersistentAckHandler persistentAckHandler;
/** Instance of inner class that reads message from the Transport **/
protected MessageReader msgReader;
/** Instance of inner class that writes message from the Transport **/
protected MessageWriter msgWriter;
/** Set unique ID assigned to the remote session by the router **/
protected int targetID;
/** Local copy of the Parser singleton **/
protected final Parser parser = Parser.getInstance();
/** Messages queued up to be send **/
protected final ArrayList outbox = new ArrayList();
     protected static int connectionLostThreadCount = 0;
     //Remote site name.
     private String remoteSite = null;
     private String siteTransportID = null;
     private JmsPersistenceManager persistentManager;
     //routeIncompletedMsgs
     protected boolean incompletedMsgsRouted = false;
     //No need to persist remote filter for EpylonRemote session
     protected boolean connectToJmsSession = false;
     static public String FALSE = "FALSE";
     static public String TRUE = "TRUE";
     private static int THREAD_COUNT = 1;
     private static Category logger = Category.getInstance( RemoteSession.class.getName() );
// Constructors //
     public RemoteSession(Router router,
Transport transport,
                              String remoteSite,
                              ConnectionListener listener) throws Exception{     
          this.router = router;
          this.transport = transport;
          this.remoteSite = remoteSite;
          this.connxListener = listener;
          siteTransportID = transport.getConnectionClientID();
          try {
               remoteFilter = parser.parseFilter("true");
          } catch (InvalidSelectorException ise) {}
          persistentAckHandler = new PersistentAckHandler(this);
          router.addTarget(this);
          persistentManager = JmsPersistenceManager.getInstance();
          start();
// Public methods //
     public void setTransport(Transport t){
// TODO:: Add error checking to make sure this is only being called
// as a result of a reconnection. DTG
          this.transport = t;
     public synchronized void start(){
if (msgReader == null) {
msgReader = new MessageReader(THREAD_COUNT);
msgReader.start();
     THREAD_COUNT ++;
if (msgWriter == null) {
msgWriter = new MessageWriter(THREAD_COUNT);
msgWriter.start();
     THREAD_COUNT ++;
public synchronized void stop()
if (msgReader != null) {
msgReader.stopReader();
msgReader = null;
if (msgWriter != null) {
msgWriter.stopWriter();
msgWriter = null;
public void close()
this.stop();
if (transport != null) {
transport.close();
transport = null;
router.removeTarget(this);
// Begin implementation of RoutingTarget interface methods
public void setTargetID(int id)
this.targetID = id;
     public JmsOperand getRoutingFilter(){
          JmsOperand retval = remoteFilter;
          return retval;
     public void setRemoteRoutingFilter(JmsOperand filter, boolean add){
          if (filter != localFilter){
               JmsMessage msg;
               String filterString;               
               try {
                    if (persistentManager == null)
                         persistentManager = JmsPersistenceManager.getInstance();
                    if(connectToJmsSession || !JmsServer.isJmsSessionInitComplete())
                         filterString = parser.unparse(filter);               
                    else                                        
                         filterString = persistentManager.prepareRemoteFilterForSite(remoteSite);
                    logger.debug("setRemoteRoutingFilter::remoteSite =>"+ remoteSite + "\n "+ filterString);
                    if (!defaultFilter(filterString.toUpperCase())){
                         msg = new JmsMessage(router.getName());
                         msg.setJMSSystemMsgType(SystemMessageConstants.REMOTE_FILTER_CHANGE_TYPE);                                             
                         msg.setStringProperty(FILTER_PROPERTY, filterString);
                         msg.setStringProperty(MessageStatusBean.ORIGINATING_SITE_NAME, JmsServer.siteName());
                         localFilter = filter;
                         if (connectToJmsSession && !JmsServer.isJmsSessionInitComplete())
                              JmsServer.addLocalFilter();
                         if ( !connectToJmsSession || !JmsServer.isAllLocalFilterProcessed())
                              queueOutboxMessage(msg);                         
               } catch (Exception e) {
                    // shouldn't happen with a in memory message queue but, dump it anyways
                    logger.error(e);
public boolean needsFilterUpdates()
return true;
* Called by the router to give a potential message to this
* routing target.
* Implementes RoutingTarget.takeMessage();
public JmsMessage takeMessage(JmsMessage msg)
throws Exception{
     logger.debug("RemoteSession:takeMessage connectToJmsSession=>"+ connectToJmsSession);
     boolean taken = false;
     if (msg.getSendingTargetID() != this.targetID && incompletedMsgsRouted) { // prevent backtracking
          if (connectToJmsSession || msg.getJMSDeliveryMode() != javax.jms.DeliveryMode.PERSISTENT ){ 
               taken = parser.evaluate(remoteFilter, msg);
          } else
               taken = msg.getAckHelper().routeMsgtoRemoteSite( remoteSite);
          if (taken) {
               if (msg.getJMSDeliveryMode() == javax.jms.DeliveryMode.PERSISTENT)
                    persistentAckHandler.handleOutgoingMsg(msg);               
               synchronized(outbox) {
                    outbox.add(msg);
                    outbox.notifyAll();
     return msg;
// End implementaiton of RoutingTarget interface methods
* Called by MessageReader.run() when messages come from the transport.
public void receiveMessages(JmsMessage [] msgs)
throws JMSException{
int length = msgs.length;
int msgsRemoved = 0;
for (int i=0; i < length; ++i) {
          logger.debug("ReceiveMessages from "+ remoteSite + msgs);
          msgs[i].setStringProperty(MessageStatusBean.ORIGINATING_SITE_NAME, remoteSite);
          boolean systemMsg = false;
          int type = msgs[i].getJMSSystemMsgType();
          switch (type) {
          case(SystemMessageConstants.REMOTE_FILTER_CHANGE_TYPE):
               handleRemoteFilterMsg(msgs[i]);
               systemMsg = true;
               break;
case(SystemMessageConstants.ACKS_MSG_TYPE):
               String acks = (String) msgs[i].getObjectProperty(ACKS_PROPERTY);
               try{
                    persistentAckHandler.handleAcks(acks);
               }catch(Exception e){
                    //TODO: Handle exception
                    logger.error(e);
               systemMsg = true;
               break;
          if (systemMsg) {
               ++msgsRemoved;
               msgs[i] = null;
          }else {               
               // Filter out duplicates
               boolean duplicate = persistentAckHandler.isDuplicate(msgs[i]);
               if (duplicate) {
                    msgs[i] = null;
                    ++msgsRemoved;
               else {
                    // Valid user message... start acknowledgment tracking
                    msgs[i].setSendingTargetID( targetID );          
                    persistentAckHandler.setOriginator(msgs[i]);
if (msgsRemoved > 0) {
          msgs = collapseMsgArray(msgs, length - msgsRemoved);
try {
          router.routeMessages(msgs);
          persistentAckHandler.handleIncomingMsgs(msgs);
} catch (Exception e) {
// TODO:: add some error handling
logger.error("An exception occurred while routing msgs: \n", e);
     public void acksAvailable(){
          synchronized(outbox) {
               outbox.notifyAll();
// Package methods //
// Private methods //
protected void queueOutboxMessage(JmsMessage msg)
synchronized(outbox) {
outbox.add(msg);
outbox.notifyAll();
     * Handle remote filter message.
     protected void handleRemoteFilterMsg(JmsMessage msg)
          throws JMSException{       
          boolean remoteFilterPersist = false;
          String filter = (String) msg.getObjectProperty(FILTER_PROPERTY);
          if (!connectToJmsSession && !JmsServer.isJmsSessionInitComplete()){
               try{
                    logger.debug("Sleep");
                    Thread.currentThread().sleep(3000);
               }catch(Exception e){
                    logger.error(e);
          if (!defaultFilter(filter.toUpperCase())){
          if (remoteSite.indexOf(JmsServer.siteName()) != -1 &&
               !JmsServer.isAllLocalFilterProcessed() &&
               !connectToJmsSession )
               JmsServer.processLocalFilter();
          if (!connectToJmsSession){
               persistRemoteFilter(remoteSite, filter);
               remoteFilterPersist = true;
          try {
               remoteFilter = parser.parseFilter(filter);
               if (remoteFilterPersist){
                    if (!incompletedMsgsRouted &&
                         !connectToJmsSession ){          
                         routeMessagesToRemoteSite();
                         incompletedMsgsRouted = true;
                    logger.debug("RemoteSession::handleRemoteFilterMsg "+ remoteFilter);
                    if (!connectToJmsSession)
                    router.recalculateFilters(true);
          } catch (javax.jms.InvalidSelectorException ex) {
               logger.error("Remote client sent invalid routing filter: " +
                              transport.getRemoteID() + " :: " + filter);
* Utility function to remove null values from the array
protected JmsMessage [] collapseMsgArray(JmsMessage [] msgs, int size)
JmsMessage [] retval = new JmsMessage[size];
for (int i=0, j=0; i < msgs.length; ++i) {
if (msgs[i] != null) {
retval[j] = msgs[i];
++j;
return retval;
* Called by the internal methods reading and sending messages
* to the transport layer when an IOException is thrown from
* the transport layer. This indicates a connection loss or
* corruption.
* TODO:: explain why this needs to spawn off as a thread
     protected void connectionLost(IOException ioe){
          //Spawn off a thread to connect remote server when connection lost.
          //This RemoteSession is accepted connection from remote client when its
          //RemoteURL is null, so there is no need to spawn off connectionLostthread
          //And close transport.
          if(getRemoteUrl() != null){
               connectionLostThreadCount++;               
               Thread t = new Thread("ConnectionLost Thread"+ connectionLostThreadCount) {
                         public void run() {
                              RemoteSession.this.close();
                              if (transport != null) {
                                   transport.close();
                                   transport = null;
                              connxListener.connectionLost(RemoteSession.this, siteTransportID);                              
               t.start();
          }     else{
               if (transport != null) {
                    transport.close();
                    transport = null;
               connxListener.connectionLost(RemoteSession.this, siteTransportID);
               //No need to close transport because its outputstream closed
     protected JmsMessage createAckMessage(){
          JmsMessage ackMessage = null;
          String acks = persistentAckHandler.getAckString();
          if (acks != null) {
               ackMessage = new JmsMessage(router.getName());
               try {
                    ackMessage.setJMSSystemMsgType(SystemMessageConstants.ACKS_MSG_TYPE);
                    ackMessage.setStringProperty(ACKS_PROPERTY, acks);
               } catch (JMSException jmsex) {}
          return ackMessage;
     // Begin Inner Class MessageReader //
     * Inner class reperesenting thread reading messages from the transport
     * implementation.
     protected class MessageReader extends Thread{
          private volatile boolean stopped = false;
          public MessageReader(int threadCount) {
               super("MessageReader_"+ threadCount );
          public void stopReader(){
               if (! stopped) stopped = true;
          public void run(){
               JmsMessage [] msgs;
               while (stopped == false) {
                    if (router.isRouterStop())
                         stopped = true;
                    try {
                         if (transport == null){                    
                         throw new IOException("Transport socket not available");                    
                         }else
                              msgs = transport.receiveMessages();                         
                         if (msgs != null && stopped == false) {
                              receiveMessages(msgs);
                              msgs = null;
                    }catch(SocketCloseException sce){
                         stopped = true;
                         connectionLost(new IOException(sce.getMessage()));
                    }catch(JMSException jmse){
                         // TODO:: handle this properly
                         logger.error(jmse);
                    }catch(NullPointerException ne){
                         logger.error(ne);
                    }catch (IOException ioe) {
                         if (! stopped) {
                              // TODO:: handle this properly
                              logger.debug(ioe);
                              stopped = true;
                              connectionLost(ioe);
     // End Inner Class MessageReader //
     // Begin Inner Class MessageWriter //
     * Inner class writing messages to the Transport
     protected class MessageWriter extends Thread{
          private volatile boolean stopped = false;
          private volatile boolean sendMessages;
          public MessageWriter(int threadCount) { super("MessageWriter_"+ threadCount); }
          public final void stopWriter()
               if (! stopped) {
                    synchronized (outbox) {
                         stopped = true;
                         outbox.notifyAll();
          public final void sendMessagesNow()
               sendMessages = true;
          public final void run()
               long nextSend = System.currentTimeMillis() + latency;
               long sleepTime = 0;
               while (!stopped) {                    
                    synchronized (outbox) {
                         sendMessages = false;                         
                         if (outbox.size() >= MSG_BATCH_SIZE) {
                              sendMessages = true;
                         } else if(outbox.size() > 0 ||
                                   persistentAckHandler.acksAvailable() ) {
                              long currentTime = System.currentTimeMillis();
                              long diff = nextSend - currentTime;
                              if (diff <= 0) {
                                   sendMessages = true;
                              else {
                                   sleepTime = diff;
                         } else {
                              sleepTime = 0;
                         if (sendMessages == false && !stopped) {
                              try {
                                   outbox.wait(sleepTime);
                              } catch (InterruptedException ie) {}
                    if (sendMessages == true) {
                         send();
                         nextSend = System.currentTimeMillis() + latency;
                         sleepTime = 0;
     private final void send(){
          logger.debug("RemoteSession:send");
          try {
               JmsMessage [] msgs = null;
               JmsMessage ackMessage = createAckMessage();
               synchronized(outbox) {
                    if (ackMessage != null) {
                         outbox.add(ackMessage);
                    msgs = new JmsMessage[outbox.size()];
                    msgs = (JmsMessage[]) outbox.toArray(msgs);
                    outbox.clear();
               if (msgs != null && msgs.length != 0 && transport != null) {
                    transport.sendMessages(msgs);
          }catch(SocketCloseException sce){
               connectionLost(new IOException(sce.getMessage()));
          } catch (IOException ioe) {
               logger.error(ioe);
               connectionLost(ioe);
     private void persistRemoteFilter( String site, String filter)
          throws JMSException{
          try{
               persistentManager.persistRemoteFilter(site, filter);
               persistentManager.endTransaction(true);
          }catch(Exception e){
               logger.error(e);
               JMSException jmsex = new JMSException("Unable to parse filter");
               jmsex.setLinkedException(e);
     * This method is called only when remotesession initially connected to
     * route messages and ack messages (stored acks only) to remote
     * site. Don't route messages to JMS core/provider when remoteSeesion
     * connectted to JmsSession and JMS core remote session.
     * @param
     private void routeMessagesToRemoteSite(){
          logger.debug("RemoteSession::routeMessagesToRemoteSite " + remoteSite);
          if (connectToJmsSession)
               return;
          boolean taken = false;
          JmsMessage[] msgs = null;
          JmsMessage msg;
          try{
               persistentManager.obtainLock();
               while (msgs == null){
                    msgs = persistentManager.getPendingDeleteMsgs();     
               int length = msgs.length;
               for (int i=0; i<length ; i++){
                    try{
                         msg = (JmsMessage)msgs[i];     
                         if (msg.getAckHelper().needStoredAckToRemoteSite(remoteSite)){
                              logger.debug("needStoredAckToRemoteSite" + remoteSite);
                              persistentAckHandler.addMsgToCheckDupsMapAndAcks(msg);
                         }else if (msg.getAckHelper().needCompletedAckToRemoteSite(remoteSite)){
                              persistentAckHandler.addMsgToSendToMapAndAcks(msg);
                         }else{
                              taken = msg.getAckHelper().routeMsgtoRemoteSite( remoteSite);
                              logger.debug("RemoteSession::routeMessagesToRemoteSite taken=> " + taken );
                              if (taken) {     
                                   if (msg.getJMSDeliveryMode() == javax.jms.DeliveryMode.PERSISTENT)
                                        persistentAckHandler.handleOutgoingMsg(msg);     
                                   while (outbox.size() >= MSG_BATCH_SIZE){
                                        msgWriter.send();
                                        logger.info("Route queued messages to remote sites. Please wait....");
                                        try{
                                             Thread.sleep(2000);
                                        }catch(Exception e){}
                                   queueOutboxMessage(msg);
                    }catch(Exception e){
                         logger.error(e);
          } finally{
               persistentManager.releaseLock();               
          logger.debug("RemoteSession::routeMessagesToRemoteSite:Exit");
// End Inner Class MessageReader //
     private String remoteUrl = null;
     public void setRemoteUrl(String url){
          remoteUrl = url;
     public String getRemoteUrl(){
          return remoteUrl;
     public String getRemoteSite(){
          return remoteSite;
     public void addSessionToRouter(){
          if (router != null)
               router.addTarget(this);
     public static boolean defaultFilter(String filter){
          return filter.indexOf(TRUE) != -1 || filter.indexOf(FALSE) != -1;

Similar Messages

  • Help to anayse Thread Dump

    hi,
    My application uses RMI and some times the application hangs-up. The following is the thread dump i got. Please help me to find out the dead lock from this dump.
    =========================================================================
    Full thread dump Java HotSpot(TM) Server VM (1.4.1 1.4.1.02-030502-15:31-PA_RISC
    2.0 PA2.0 mixed mode):
    "RMI LeaseChecker" daemon prio=8 tid=0x006eac78 nid=20 lwp_id=280805 waiting on
    condition [0x6f042000..0x6f042530]
    at java.lang.Thread.sleep(Native Method)
    at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
    at java.lang.Thread.run(Thread.java:536)
    "RMI TCP Connection(1)-192.168.174.45" daemon prio=10 tid=0x00b06570 nid=19 lwp_
    id=280803 runnable [0x6f0c3000..0x6f0c3530]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x73aea6d8> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:4
    42)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport
    .java:701)
    at java.lang.Thread.run(Thread.java:536)
    "MultiThreadedTimerMain" prio=10 tid=0x0004c660 nid=18 lwp_id=280759 in Object.w
    ait() [0x6f144000..0x6f144530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73adacb8> (a jp.co.nec.diosa.fw.util.timer.TaskQueue)
    at java.lang.Object.wait(Object.java:426)
    at jp.co.nec.diosa.fw.util.timer.MultiThreadedTimerThread.mainLoop(Multi
    ThreadedTimerThread.java:174)
    - locked <0x73adacb8> (a jp.co.nec.diosa.fw.util.timer.TaskQueue)
    at jp.co.nec.diosa.fw.util.timer.MultiThreadedTimerThread.run(MultiThrea
    dedTimerThread.java:115)
    "RMI RenewClean-[192.168.174.45:62731]" daemon prio=10 tid=0x0004c3a0 nid=16 lwp
    _id=280747 runnable [0x6f27b000..0x6f27b530]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x76400468> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:313)
    at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
    at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.jav
    a:328)
    at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:
    144)
    at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCCli
    ent.java:539)
    at java.lang.Thread.run(Thread.java:536)
    "GC Daemon" daemon prio=10 tid=0x0004c0e0 nid=14 lwp_id=280709 in Object.wait()
    [0x6f37d000..0x6f37d530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73a00588> (a sun.misc.GC$LatencyLock)
    at sun.misc.GC$Daemon.run(GC.java:100)
    - locked <0x73a00588> (a sun.misc.GC$LatencyLock)
    "RMI Reaper" prio=10 tid=0x0004bf80 nid=13 lwp_id=280708 in Object.wait() [0x762
    fe000..0x762fe530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x739ffb88> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
    - locked <0x739ffb88> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
    at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
    at java.lang.Thread.run(Thread.java:536)
    "Thread-1" daemon prio=10 tid=0x0004be20 nid=12 lwp_id=280707 in Object.wait() [
    0x7637f000..0x7637f530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x739ffe80> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:426)
    at java.util.TimerThread.mainLoop(Timer.java:403)
    - locked <0x739ffe80> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:382)
    "RMI TCP Accept-0" daemon prio=10 tid=0x0004bcc0 nid=11 lwp_id=280706 runnable [
    0x7796b000..0x7796b530]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
    - locked <0x739ffbe0> (a java.net.PlainSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:439)
    at java.net.ServerSocket.accept(ServerSocket.java:410)
    at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
    at java.lang.Thread.run(Thread.java:536)
    "Signal Dispatcher" daemon prio=10 tid=0x0004b740 nid=8 lwp_id=280652 waiting on
    condition [0x00000000..0xffffffff]
    "Finalizer" daemon prio=10 tid=0x0004b5e0 nid=5 lwp_id=280644 in Object.wait() [
    0x77b7f000..0x77b7f530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73a03960> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
    - locked <0x73a03960> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0x0004b480 nid=4 lwp_id=280643 in Object.
    wait() [0x79c08000..0x79c08530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73a039c8> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:426)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
    - locked <0x73a039c8> (a java.lang.ref.Reference$Lock)
    "main" prio=10 tid=0x0004b320 nid=1 lwp_id=280621 runnable [0x79ff1000..0x79ff19
    50]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x73ac02f8> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:
    189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at jp.co.nec.diosa.fw.ms.service.Core_Stub.configure(Unknown Source)
    at jp.co.nec.diosa.utilapp.JUtilityApplication.<init>(JUtilityApplicatio
    n.java:1083)
    at jp.co.nec.diosa.utilapp.JUtilityAppServer.getUtilityAppReference(JUti
    lityAppServer.java:185)
    at jp.co.nec.diosa.utilapp.JUtilityAppServer.<init>(JUtilityAppServer.ja
    va:114)
    at jp.co.nec.diosa.utilapp.JUtilityAppServer.main(JUtilityAppServer.java
    :216)
    "VM Thread" prio=10 tid=0x0010daa8 nid=3 lwp_id=280642 runnable
    "VM Periodic Task Thread" prio=8 tid=0x0010db28 nid=6 lwp_id=280650 waiting on c
    ondition
    "Suspend Checker Thread" prio=10 tid=0x0010dba8 nid=7 lwp_id=280651 runnable
    Full thread dump Java HotSpot(TM) Server VM (1.4.1 1.4.1.02-030502-15:31-PA_RISC
    2.0 PA2.0 mixed mode):
    "Thread-2" prio=10 tid=0x0004d0c0 nid=23 lwp_id=280801 in Object.wait() [0x6eef4
    000..0x6eef4530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73acdfc8> (a jp.co.nec.diosa.joba.appl.GuiProcessingDaem
    on)
    at java.lang.Object.wait(Object.java:426)
    at jp.co.nec.diosa.joba.appl.GuiProcessingDaemon.process(GuiProcessingDa
    emon.java:257)
    - locked <0x73acdfc8> (a jp.co.nec.diosa.joba.appl.GuiProcessingDaemon)
    at jp.co.nec.diosa.joba.appl.GuiProcessingDaemon.run(GuiProcessingDaemon
    .java:196)
    at java.lang.Thread.run(Thread.java:536)
    "RMI LeaseChecker" daemon prio=8 tid=0x0004cca0 nid=19 lwp_id=280777 waiting on
    condition [0x6f0f8000..0x6f0f8530]
    at java.lang.Thread.sleep(Native Method)
    at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
    at java.lang.Thread.run(Thread.java:536)
    "RMI TCP Connection(1)-192.168.174.45" daemon prio=10 tid=0x00235b30 nid=18 lwp_
    id=280768 runnable [0x6f179000..0x6f179530]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x73ace8c8> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:4
    42)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport
    .java:701)
    at java.lang.Thread.run(Thread.java:536)
    "RMI Reaper" prio=10 tid=0x0004cb40 nid=17 lwp_id=280758 in Object.wait() [0x6f1
    fa000..0x6f1fa530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73acea78> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
    - locked <0x73acea78> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
    at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
    at java.lang.Thread.run(Thread.java:536)
    "RMI TCP Accept-0" daemon prio=10 tid=0x0004c9e0 nid=16 lwp_id=280757 runnable [
    0x6f27b000..0x6f27b530]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
    - locked <0x73aceaf8> (a java.net.PlainSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:439)
    at java.net.ServerSocket.accept(ServerSocket.java:410)
    at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
    at java.lang.Thread.run(Thread.java:536)
    "Thread-1" daemon prio=10 tid=0x0004c880 nid=15 lwp_id=280755 in Object.wait() [
    0x6f2fc000..0x6f2fc530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73acebb0> (a java.util.TaskQueue)
    at java.lang.Object.wait(Object.java:426)
    at java.util.TimerThread.mainLoop(Timer.java:403)
    - locked <0x73acebb0> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:382)
    "GC Daemon" daemon prio=10 tid=0x0004c5c0 nid=13 lwp_id=280741 in Object.wait()
    [0x762fe000..0x762fe530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73a4d630> (a sun.misc.GC$LatencyLock)
    at sun.misc.GC$Daemon.run(GC.java:100)
    - locked <0x73a4d630> (a sun.misc.GC$LatencyLock)
    "RMI RenewClean-[192.168.174.45:62731]" daemon prio=10 tid=0x0004c460 nid=12 lwp
    _id=280740 runnable [0x7637f000..0x7637f530]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x76408468> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:313)
    at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
    at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.jav
    a:328)
    at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:
    144)
    at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCCli
    ent.java:539)
    at java.lang.Thread.run(Thread.java:536)
    "Signal Dispatcher" daemon prio=10 tid=0x0004bd80 nid=8 lwp_id=280647 waiting on
    condition [0x00000000..0xffffffff]
    "Finalizer" daemon prio=10 tid=0x0004bc20 nid=5 lwp_id=280635 in Object.wait() [
    0x77b7f000..0x77b7f530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73a4da48> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
    - locked <0x73a4da48> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0x0004bac0 nid=4 lwp_id=280634 in Object.
    wait() [0x79c08000..0x79c08530]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x73a4dab0> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:426)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
    - locked <0x73a4dab0> (a java.lang.ref.Reference$Lock)
    "main" prio=10 tid=0x0004b960 nid=1 lwp_id=280616 runnable [0x79ff1000..0x79ff19
    50]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x73ae1260> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:
    189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at jp.co.nec.diosa.fw.ms.service.Core_Stub.filterFilesForCurrentHost(Unk
    nown Source)
    at jp.co.nec.diosa.joba.appl.JobGateway.filterFilesForIncomingHost(JobGa
    teway.java:342)
    at jp.co.nec.diosa.joba.event.coreevent.SGReadingEvent.run(SGReadingEven
    t.java:322)
    at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.run(Startu
    pSGReadingEvent.java:86)
    at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.process(St
    artupSGReadingEvent.java:75)
    at jp.co.nec.diosa.joba.appl.JobMain.readSGFile(JobMain.java:1401)
    at jp.co.nec.diosa.joba.appl.JobMain.init(JobMain.java:752)
    at jp.co.nec.diosa.joba.appl.JobMain.main(JobMain.java:1014)
    "VM Thread" prio=10 tid=0x0010daa8 nid=3 lwp_id=280626 runnable
    "VM Periodic Task Thread" prio=8 tid=0x0010db28 nid=6 lwp_id=280645 waiting on c
    ondition
    "Suspend Checker Thread" prio=10 tid=0x0010dba8 nid=7 lwp_id=280646 runnable
    =========================================================================

    Hi Owen,
    Thank u for ur immediate reply on my doubt, but still my doubt not got clear.
    If u look into the locked threads, the lock is happened in RMI layer. Concider following two threads:
    -----------------------------------thread 1
    "RMI TCP Connection(1)-192.168.174.45" daemon prio=10 tid=0x00b06570 nid=19 lwp_id=280803 runnable [0x6f0c3000..0x6f0c3530]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x73aea6d8> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:442)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
    at java.lang.Thread.run(Thread.java:536)
    --------------------------------------- thread 2----------
    "main" prio=10 tid=0x0004b960 nid=1 lwp_id=280616 runnable [0x79ff1000..0x79ff1950]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <0x73ae1260> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at jp.co.nec.diosa.fw.ms.service.Core_Stub.filterFilesForCurrentHost(Unknown Source)
    at jp.co.nec.diosa.joba.appl.JobGateway.filterFilesForIncomingHost(JobGateway.java:342)
    at jp.co.nec.diosa.joba.event.coreevent.SGReadingEvent.run(SGReadingEvent.java:322)
    at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.run(StartupSGReadingEvent.java:86)
    at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.process(StartupSGReadingEvent.java:75)
    at jp.co.nec.diosa.joba.appl.JobMain.readSGFile(JobMain.java:1401)
    at jp.co.nec.diosa.joba.appl.JobMain.init(JobMain.java:752)
    at jp.co.nec.diosa.joba.appl.JobMain.main(JobMain.java:1014)
    Both the threads get locked in getting BufferedInputStread in RMI. If u can suggest some more hints that will very much helpful for me.
    Thanks in advance,
    Suresh Kumar

  • Need help interpreting jvm thread dump (linux)

    hi,
    i'm using jre1.4.2 and running AS3 Linux kernel version 2.4.
    i grep'ed for my java process id, and did a kill -3 on it to get the thread dump:
    ps -ef |grep Eatroot 3936 3845 68 16:12 pts/2 00:00:05 java EatCpu
    root 3948 30293 0 16:12 pts/3 00:00:00 grep Eat
    kill -3 3936
    thread dump:
    java EatCpuFull thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):
    "Thread-0" prio=1 tid=0x081126b8 nid=0xf60 runnable [aa7e0000..aa7e087c]
    at EatCpu.run(EatCpu.java:31)
    "Signal Dispatcher" daemon prio=1 tid=0x080a6a58 nid=0xf60 waiting on condition [0..0]
    "Finalizer" daemon prio=1 tid=0x08092ee8 nid=0xf60 in Object.wait() [aad4d000..aad4d87c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xaaed0490> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
    - locked <0xaaed0490> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=1 tid=0x08091498 nid=0xf60 in Object.wait() [aadce000..aadce87c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xaaed0380> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:429)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
    - locked <0xaaed0380> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x0805bae0 nid=0xf60 runnable [bfffc000..bfffcc98]
    at EatCpu.main(EatCpu.java:22)
    "VM Thread" prio=1 tid=0x08090238 nid=0xf60 runnable
    "VM Periodic Task Thread" prio=1 tid=0x080a9248 nid=0xf60 waiting on condition
    "Suspend Checker Thread" prio=1 tid=0x080a6020 nid=0xf60 runnable
    i read that "nid" in the thread dump is suppose to correspond to PID. when i give the -m option for the "ps" command (for all threads), i see:
    ps -efm |grep Eatroot 3936 3845 43 16:12 pts/2 00:00:04 java EatCpu
    root 3937 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3938 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3939 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3940 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3941 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3942 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3943 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3944 3936 30 16:12 pts/2 00:00:03 java EatCpu
    root 3950 30293 0 16:12 pts/3 00:00:00 grep Eat
    >
    but nid for all threads in the thead dump is the PID of my main process, 3936 (0xf60). is there a way to correlate the PIDs produced by "ps -efm" to the threads in the thread dump?
    thanks!
    -annie

    i upgraded to 1.5 version of java, and could see distinct "nids" in the thread dump after that..

  • What is the problem in my Thread Dump's output?

    Dear all,
    Below is my Thread Dump output. I can not figure out what and where is the problem of this Thread Dump's output. Could some one please give me some hints?
    Thanks alot
    Tu
    ----------------Thread Dump's Output-----------------
    Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode, sharing):
    "TP-Monitor" daemon prio=1 tid=0xb0b14490 nid=0xfaa in Object.wait()
    [0xb08fe000..0xb08fee40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89433f70> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
    - locked <0x89433f70> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor4" daemon prio=1 tid=0xb0b13730 nid=0xfa9 runnable
    [0xb097e000..0xb097efc0]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x89438110> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)
    at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:647)
    at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:857)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor3" daemon prio=1 tid=0xb0b0f6b8 nid=0xfa8 in
    Object.wait() [0xb09fe000..0xb09fef40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89434250> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89434250> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor2" daemon prio=1 tid=0xb0b0fec8 nid=0xfa7 in
    Object.wait() [0xb0a7e000..0xb0a7f0c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x894342e8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x894342e8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor1" daemon prio=1 tid=0xb0b14978 nid=0xfa6 in
    Object.wait() [0xb0afe000..0xb0aff040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89434380> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89434380> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Monitor" prio=1 tid=0x08609b18 nid=0xfa5 in Object.wait()
    [0xb0cc9000..0xb0cc91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378950> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
    - locked <0x89378950> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor25" daemon prio=1 tid=0x08608ce0 nid=0xfa4
    runnable [0xb0d48000..0xb0d49140]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x89010c58> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
    at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
    at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor24" daemon prio=1 tid=0x086085f8 nid=0xfa3 in
    Object.wait() [0xb0dc8000..0xb0dc8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378ab8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378ab8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor23" daemon prio=1 tid=0x08606dd0 nid=0xfa2 in
    Object.wait() [0xb0e48000..0xb0e48e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378b50> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378b50> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor22" daemon prio=1 tid=0x08605ed0 nid=0xfa1 in
    Object.wait() [0xb0ec8000..0xb0ec8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378be8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378be8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor21" daemon prio=1 tid=0x08604fd0 nid=0xfa0 in
    Object.wait() [0xb0f48000..0xb0f48f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378c80> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378c80> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor20" daemon prio=1 tid=0x086040f0 nid=0xf9f in
    Object.wait() [0xb0fc8000..0xb0fc90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378d18> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378d18> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor19" daemon prio=1 tid=0x08432538 nid=0xf9e in
    Object.wait() [0xb1048000..0xb1049040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378db0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378db0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor18" daemon prio=1 tid=0x08431638 nid=0xf9d in
    Object.wait() [0xb10c9000..0xb10c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378e48> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378e48> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor17" daemon prio=1 tid=0x08430738 nid=0xf9c in
    Object.wait() [0xb1148000..0xb1149140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378ee0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378ee0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor16" daemon prio=1 tid=0x0842f838 nid=0xf9b in
    Object.wait() [0xb11c8000..0xb11c8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378f78> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378f78> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor15" daemon prio=1 tid=0x0842e938 nid=0xf9a in
    Object.wait() [0xb1248000..0xb1248e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379010> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379010> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor14" daemon prio=1 tid=0x0842da38 nid=0xf99 in
    Object.wait() [0xb12c8000..0xb12c8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893790a8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893790a8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor13" daemon prio=1 tid=0x0842cb38 nid=0xf98 in
    Object.wait() [0xb1348000..0xb1348f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379140> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379140> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor12" daemon prio=1 tid=0x0842bc50 nid=0xf97 in
    Object.wait() [0xb13c8000..0xb13c90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893791d8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893791d8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor11" daemon prio=1 tid=0x084f75d0 nid=0xf96 in
    Object.wait() [0xb1448000..0xb1449040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379270> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379270> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor10" daemon prio=1 tid=0x084f66d0 nid=0xf95 in
    Object.wait() [0xb14c9000..0xb14c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379308> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379308> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor9" daemon prio=1 tid=0x084f57d0 nid=0xf94 in
    Object.wait() [0xb1548000..0xb1549140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893793a0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893793a0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor8" daemon prio=1 tid=0x08488970 nid=0xf93 in
    Object.wait() [0xb15c8000..0xb15c8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379438> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379438> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor7" daemon prio=1 tid=0x08487a70 nid=0xf92 in
    Object.wait() [0xb1648000..0xb1648e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893794d0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893794d0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor6" daemon prio=1 tid=0x08486b90 nid=0xf91 in
    Object.wait() [0xb16c8000..0xb16c8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379568> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379568> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor5" daemon prio=1 tid=0x084e3e30 nid=0xf90 in
    Object.wait() [0xb1748000..0xb1748f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379600> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379600> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor4" daemon prio=1 tid=0x084e2f90 nid=0xf8f in
    Object.wait() [0xb17c8000..0xb17c90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379698> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379698> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor3" daemon prio=1 tid=0x084e2158 nid=0xf8e in
    Object.wait() [0xb1848000..0xb1849040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379730> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379730> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor2" daemon prio=1 tid=0x085e74d0 nid=0xf8d in
    Object.wait() [0xb18c9000..0xb18c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893797c8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893797c8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor1" daemon prio=1 tid=0x085e6638 nid=0xf8c in
    Object.wait() [0xb1948000..0xb1949140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379860> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379860> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=1
    tid=0x083719f8 nid=0xf8b waiting on condition [0xb1a85000..0xb1a85ec0]
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1547)
    at java.lang.Thread.run(Thread.java:595)
    "Low Memory Detector" daemon prio=1 tid=0x080a44f8 nid=0xf7f runnable
    [0x00000000..0x00000000]
    "CompilerThread0" daemon prio=1 tid=0x080a2f98 nid=0xf7e waiting on
    condition [0x00000000..0xb21b2928]
    "Signal Dispatcher" daemon prio=1 tid=0x080a2068 nid=0xf7d waiting on
    condition [0x00000000..0x00000000]
    "Finalizer" daemon prio=1 tid=0x0809c540 nid=0xf7c in Object.wait()
    [0xb24b2000..0xb24b3040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=1 tid=0x0809a6b0 nid=0xf7b in
    Object.wait() [0xb2533000..0xb25331c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x88f01e60> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:474)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x88f01e60> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x0805d1c8 nid=0xf79 runnable [0xbfbd9000..0xbfbda4f8]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x8946f690> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:388)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:615)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:575)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)
    "VM Thread" prio=1 tid=0x08097b58 nid=0xf7a runnable
    "VM Periodic Task Thread" prio=1 tid=0x080a5998 nid=0xf80 waiting on condition

    Thanks alot for your reply. It is a web application
    using java and tomcat server. And the problem is that
    sometime when I click on a button or a link than the
    CPU goes to 100% and it hangs but normally it works
    smoothly without any problem.This often indicates a busy retry loop somewhere. You need to acquire the thread dump when the problem occurs, but be warned that depending on the problem it may not be possible to obtain a Java-level thread dump. In that case you need to try and take an OS level thread dump - eg using pstack on solaris/linux (and there's some win32 tool as well if I recall correctly).

  • Generate thread dump without sending ctrl-break on Win32

    I have a server application running as a service on Win2k (we use Service+ to do this). The output of the console is piped to a text file which we can monitor. But, I can't figure out how to generate a thread dump while running in this configuration. Does anyone know if Service+ can be configured to do this, or if it can be achieved programmatically (how does Sun spit out the thread info when it gets the request).
    Thanks,
    Larry

    If you are running a java application on a Windows server. You can press Ctrl-Break, and a thread dump will be displayed in the console. This thread dump is a list of all the threads running in the JVM, and the line of code they are each executing.
    We'd like to be able to generate this thread dump some other way than pressing Ctrl-Break, because we don't have a console, our application is running as a service using Service .
    Thanks for the reply,
    Larry0AThanks for the reply,
    Larry

  • Weblogic 8.1 sp4 on Slowwaris 2.7 - Thread dump included

    Dear experts,
    We have a time response problem within our production environment, following is our configuration :
    Oracle 8.1.7.4
    Weblogic 8.1 sp4
    Solaris Sparc 2.7
    JSP is pre-compiled, though the time response behaviour is rather weird, sometime quick and most of the time very slow, that we can not stand with ie. 5 minutes for displaying the mainpage, worse is client IE browser should be shut and restart. Below is the thread dump of our server, we will appreciate if someone can interpret it and possibly lead us to pointers how to solve our problem.
    Full thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e1848 nid=0x34 in Object.wait() [ddf81000..ddf819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0e170> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0e170> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e0b38 nid=0x33 in Object.wait() [de081000..de0819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0d9d0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0d9d0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e08f0 nid=0x32 in Object.wait() [de181000..de1819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0d230> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0d230> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00359820 nid=0x31 in Object.wait() [de281000..de2819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0ca70> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0ca70> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060d850 nid=0x30 waiting for monitor entry [de381000..de3819f0]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
         - waiting to lock <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060d0c8 nid=0x2f waiting for monitor entry [dea01000..dea019f0]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
         - waiting to lock <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060cb28 nid=0x2e runnable [deb01000..deb019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4cac810> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95)
         - locked <0xe4cac810> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
         - locked <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ListenThread.Default" prio=5 tid=0x00609978 nid=0x2d runnable [dec01000..dec019f0]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xe4cb5d80> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Thread-6" daemon prio=5 tid=0x001d2ab8 nid=0x2c in Object.wait() [df701000..df7019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6a78a80> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xe6a78a80> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d3b08 nid=0x2b in Object.wait() [dfe01000..dfe019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6a32e48> (a weblogic.jms.backend.BETimerTree)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <0xe6a32e48> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "weblogic.health.CoreHealthMonitor" daemon prio=5 tid=0x004ccf08 nid=0x2a waiting on condition [e0101000..e01019f0]
         at java.lang.Thread.sleep(Native Method)
         at weblogic.t3.srvr.CoreHealthMonitorThread.run(CoreHealthMonitorThread.java:128)
    "Thread-5" prio=5 tid=0x00387728 nid=0x29 in Object.wait() [e0201000..e02019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe66a3f78> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xe66a3f78> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "VDE Transaction Processor Thread" prio=2 tid=0x004a1138 nid=0x27 in Object.wait() [e0301000..e03019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b97d8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at java.lang.Object.wait(Object.java:429)
         at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:365)
         - locked <0xe65b97d8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003cb6b0 nid=0x26 in Object.wait() [e0401000..e04019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9858> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9858> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003caac0 nid=0x25 in Object.wait() [e0501000..e05019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b98d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b98d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003ca010 nid=0x24 in Object.wait() [e0601000..e06019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9958> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9958> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.HTTP'" daemon prio=5 tid=0x003c9560 nid=0x23 in Object.wait() [e0701000..e07019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b99d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b99d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.HTTP'" daemon prio=5 tid=0x003c9318 nid=0x22 in Object.wait() [e0801000..e08019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9a58> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9a58> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "weblogic.security.SpinnerRandomSource" daemon prio=5 tid=0x003c6900 nid=0x21 in Object.wait() [e0901000..e09019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651be08> (a java.lang.Object)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.security.SpinnerRandomBitsSource.run(SpinnerRandomBitsSource.java:60)
         - locked <0xe651be08> (a java.lang.Object)
         at java.lang.Thread.run(Thread.java:534)
    "weblogic.time.TimeEventGenerator" daemon prio=9 tid=0x003c5958 nid=0x20 runnable [e0a01000..e0a019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651be78> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <0xe651be78> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x003c3988 nid=0x1f in Object.wait() [e0b01000..e0b019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651bef0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651bef0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fb810 nid=0x1e in Object.wait() [e0c01000..e0c019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651bf70> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651bf70> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fac20 nid=0x1d waiting on condition [e0d01000..e0d019f0]
         at weblogic.platform.SunVM.threadDump0(Native Method)
         - waiting to lock <0xe4c7cdd0> (a weblogic.platform.SunVM)
         at weblogic.platform.SunVM.threadDump(SunVM.java:106)
         at weblogic.t3.srvr.T3Srvr.getThreadDump(T3Srvr.java:1184)
         at weblogic.common.internal.AdminProxy.execute(AdminProxy.java:220)
         at weblogic.t3.srvr.ClientRequest$1.run(ClientContext.java:700)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
         at weblogic.t3.srvr.ClientRequest.execute(ClientContext.java:697)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fa030 nid=0x1c in Object.wait() [e0e01000..e0e019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c070> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c070> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001f9440 nid=0x1b in Object.wait() [e0f01000..e0f019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c0f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c0f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f8850 nid=0x1a in Object.wait() [e1001000..e10019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c170> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c170> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f7c60 nid=0x19 in Object.wait() [e1101000..e11019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c1f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c1f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f7070 nid=0x18 in Object.wait() [e1201000..e12019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c270> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c270> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f6480 nid=0x17 in Object.wait() [e1301000..e13019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c2f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c2f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f5890 nid=0x16 in Object.wait() [e1401000..e14019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c370> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c370> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d67b8 nid=0x15 in Object.wait() [e1501000..e15019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c3f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c3f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d5bc8 nid=0x14 in Object.wait() [e1601000..e16019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c470> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c470> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d4fd8 nid=0x13 in Object.wait() [e1701000..e17019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c4f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c4f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d43e8 nid=0x12 in Object.wait() [e1801000..e18019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c570> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c570> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b6d70 nid=0x11 in Object.wait() [e1901000..e19019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c5f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c5f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b61e8 nid=0x10 in Object.wait() [e1a01000..e1a019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c670> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c670> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b57a8 nid=0xf in Object.wait() [e1b01000..e1b019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c6f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c6f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b4e08 nid=0xe in Object.wait() [e1c01000..e1c019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c770> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c770> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x005e9030 nid=0xd in Object.wait() [e1d01000..e1d019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c7f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c7f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x005e8e90 nid=0xc in Object.wait() [e1e01000..e1e019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c870> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c870> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-1" daemon prio=5 tid=0x00201230 nid=0xb in Object.wait() [e4981000..e49819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe637e1d0> (a java.util.TaskQueue)
         at java.lang.Object.wait(Object.java:429)
         at java.util.TimerThread.mainLoop(Timer.java:403)
         - locked <0xe637e1d0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Signal Dispatcher" daemon prio=10 tid=0x000cdbc8 nid=0x8 runnable [0..0]
    "Finalizer" daemon prio=8 tid=0x000ca7b0 nid=0x6 in Object.wait() [fc381000..fc3819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6278d78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0xe6278d78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0x000c8e50 nid=0x5 in Object.wait() [fde81000..fde819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6278de0> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:429)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
         - locked <0xe6278de0> (a java.lang.ref.Reference$Lock)
    "main" prio=5 tid=0x00038510 nid=0x1 in Object.wait() [ffbed000..ffbed7dc]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe62eeff0> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1208)
         - locked <0xe62eeff0> (a weblogic.t3.srvr.T3Srvr)
         at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:390)
         at weblogic.Server.main(Server.java:32)
    "VM Thread" prio=5 tid=0x000c8008 nid=0x4 runnable
    "VM Periodic Task Thread" prio=10 tid=0x000d0a28 nid=0xa waiting on condition
    "Suspend Checker Thread" prio=10 tid=0x000cd260 nid=0x7 runnable
    Thanks in advance,
    saiful-france

    Hi All,
    It looks like I have the similar problem: I am using Weblogic 8.1 Sp3 platform. please help.
    Full thread dump Java HotSpot(TM) Server VM (1.4.2 1.4.2.03-040401-18:59-PA_RISC2.0 PA2.0 (aCC_AP) mixed mode):
    "Login" daemon prio=10 tid=01a0a0b0 nid=17817 lwp_id=1133989 in Object.wait() [0x2da54000..0x2da544f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92588248> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=01a09f48 nid=17816 lwp_id=1133988 in Object.wait() [0x2d90f000..0x2d90f4f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92580248> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=00c40450 nid=17815 lwp_id=1133969 in Object.wait() [0x2ed85000..0x2ed854f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92446368> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=00c402e8 nid=17814 lwp_id=1133967 in Object.wait() [0x2db17000..0x2db174f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <9242e368> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "AreYouAlive" daemon prio=10 tid=077ca9b0 nid=11502 lwp_id=1080790 in Object.wait() [0x2d950000..0x2d9504f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fcb3390> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:115)
         - locked <3fcb3390> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
    "DatagramThread" daemon prio=10 tid=077ca848 nid=11501 lwp_id=1080789 runnable [0x2d991000..0x2d9914f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3fcb5458> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3fcb5488> (a java.net.DatagramPacket)
         - locked <3fcb54b0> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=077ca6e0 nid=11500 lwp_id=1080788 in Object.wait() [0x2d9d2000..0x2d9d24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fcb54d0> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3fcb54d0> (a com.inprise.vbroker.orb.GarbageCollector)
    "AreYouAlive" daemon prio=10 tid=030802e8 nid=11340 lwp_id=1078923 in Object.wait() [0x2da95000..0x2da954f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fc90508> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:115)
         - locked <3fc90508> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
    "DatagramThread" daemon prio=10 tid=03080180 nid=11339 lwp_id=1078922 runnable [0x2dad6000..0x2dad64f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3fc925d0> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3fc92600> (a java.net.DatagramPacket)
         - locked <3fc92628> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=03080018 nid=11338 lwp_id=1078921 in Object.wait() [0x2db58000..0x2db584f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fc92648> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3fc92648> (a com.inprise.vbroker.orb.GarbageCollector)
    "AreYouAlive" daemon prio=10 tid=08005a10 nid=3830 lwp_id=947497 in Object.wait() [0x2db99000..0x2db994f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.login(DSUser.java:736)
         - locked <3d4b1aa0> (a java.lang.Object)
         at com.inprise.vbroker.ds.DSUser.reconnect(DSUser.java:544)
         - locked <3d4b19b8> (a com.inprise.vbroker.ds.DSUser)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:513)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.fire(DSUser.java:95)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:117)
    "DatagramThread" daemon prio=10 tid=080058a8 nid=3829 lwp_id=947496 runnable [0x2dc1b000..0x2dc1b4f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3d4b3b20> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3d4b3b50> (a java.net.DatagramPacket)
         - locked <3d4b3b78> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=08005740 nid=3828 lwp_id=947495 in Object.wait() [0x2dbda000..0x2dbda4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3d4b3b98> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3d4b3b98> (a com.inprise.vbroker.orb.GarbageCollector)
    "ExecuteThread: '14' for queue: 'JmsDispatcher'" daemon prio=10 tid=030424a8 nid=142 lwp_id=679935 in Object.wait() [0x2dc5c000..0x2dc5c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b088> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b088> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'JmsDispatcher'" daemon prio=10 tid=03042340 nid=141 lwp_id=679934 in Object.wait() [0x2dc9d000..0x2dc9d4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b108> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b108> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'JmsDispatcher'" daemon prio=10 tid=030421d8 nid=140 lwp_id=679933 in Object.wait() [0x2dcde000..0x2dcde4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b188> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b188> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'JmsDispatcher'" daemon prio=10 tid=03042070 nid=139 lwp_id=679932 in Object.wait() [0x2dd1f000..0x2dd1f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b208> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b208> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041f08 nid=138 lwp_id=679931 in Object.wait() [0x2dd60000..0x2dd604f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b288> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b288> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041da0 nid=137 lwp_id=679930 in Object.wait() [0x2dda1000..0x2dda14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b308> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b308> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041c38 nid=136 lwp_id=679929 in Object.wait() [0x2dde2000..0x2dde24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b388> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b388> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041ad0 nid=135 lwp_id=679928 in Object.wait() [0x2de23000..0x2de234f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b408> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b408> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041968 nid=134 lwp_id=679927 in Object.wait() [0x2de64000..0x2de644f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b488> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b488> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041800 nid=133 lwp_id=679926 in Object.wait() [0x2dea5000..0x2dea54f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b508> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b508> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041698 nid=132 lwp_id=679925 in Object.wait() [0x2dee6000..0x2dee64f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b588> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b588> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041530 nid=131 lwp_id=679924 in Object.wait() [0x2df27000..0x2df274f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b608> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b608> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JmsDispatcher'" daemon prio=10 tid=030413c8 nid=130 lwp_id=679923 in Object.wait() [0x2df68000..0x2df684f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b688> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b688> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041260 nid=129 lwp_id=679922 in Object.wait() [0x2dfa9000..0x2dfa94f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b708> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b708> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JmsDispatcher'" daemon prio=10 tid=030410f8 nid=128 lwp_id=679921 in Object.wait() [0x2dfea000..0x2dfea4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b7a8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b7a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ListenThread.Default" prio=10 tid=02953558 nid=127 lwp_id=679919 runnable [0x2e02b000..0x2e02b4f0]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <3b81b8c0> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:454)
         at java.net.ServerSocket.accept(ServerSocket.java:425)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Adapter Suspend/Resume Task Handler" daemon prio=10 tid=029533f0 nid=126 lwp_id=679918 in Object.wait() [0x2e06c000..0x2e06c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b9a0> (a java.lang.String)
         at java.lang.Object.wait(Object.java:429)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:60)
         - locked <3b81b9a0> (a java.lang.String)
    "Adapter instance Auto-resume TaskHandler" daemon prio=10 tid=02953288 nid=125 lwp_id=679917 in Object.wait() [0x2e0ad000..0x2e0ad4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81ba18> (a java.lang.String)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:55)
         - locked <3b81ba18> (a java.lang.String)
    "ApplicationView Auto-resume TaskHandler" daemon prio=10 tid=000b6360 nid=124 lwp_id=679915 in Object.wait() [0x2e0ee000..0x2e0ee4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b74b988> (a java.lang.String)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:55)
         - locked <3b74b988> (a java.lang.String)
    "ApplicationView Deployment TaskHandler" daemon prio=10 tid=000b61f8 nid=123 lwp_id=679914 in Object.wait() [0x2e12f000..0x2e12f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b74b648> (a java.lang.String)
         at java.lang.Object.wait(Object.java:429)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:60)
         - locked <3b74b648> (a java.lang.String)
    "AreYouAlive" daemon prio=10 tid=000b6090 nid=122 lwp_id=679910 in Object.wait() [0x2e170000..0x2e1704f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Thread.join(Thread.java:1001)
         - locked <8e395cc8> (a java.lang.Thread)
         at java.lang.Thread.join(Thread.java:1054)
         at com.inprise.vbroker.ds.DSUser.login(DSUser.java:752)
         at com.inprise.vbroker.ds.DSUser.reconnect(DSUser.java:544)
         - locked <3b6e4558> (a com.inprise.vbroker.ds.DSUser)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:513)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.fire(DSUser.java:95)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:117)
    "DatagramThread" daemon prio=10 tid=000b5f28 nid=121 lwp_id=679909 runnable [0x2e1b1000..0x2e1b14f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3b6e6a18> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3b6e9020> (a java.net.DatagramPacket)
         - locked <3b6e69f0> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=000b5dc0 nid=120 lwp_id=679908 in Object.wait() [0x2e1f2000..0x2e1f24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b6e3338> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3b6e3338> (a com.inprise.vbroker.orb.GarbageCollector)
    "Timer" daemon prio=10 tid=000b5af0 nid=119 lwp_id=679906 in Object.wait() [0x2e233000..0x2e2334f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b65ee40> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "Timer" daemon prio=10 tid=000b5c58 nid=118 lwp_id=679905 in Object.wait() [0x2e274000..0x2e2744f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b642528> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "Timer" daemon prio=10 tid=000b5820 nid=117 lwp_id=679904 in Object.wait() [0x2e2b5000..0x2e2b54f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b639540> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca578 nid=116 lwp_id=679896 in Object.wait() [0x2e2f6000..0x2e2f64f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad75800> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca410 nid=115 lwp_id=679895 in Object.wait() [0x2e337000..0x2e3374f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3ad75328> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad75328> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca2a8 nid=114 lwp_id=679894 in Object.wait() [0x2e378000..0x2e3784f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3ad74e30> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad74e30> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_1>.ioThreadPool'" daemon prio=10 tid=000b5988 nid=113 lwp_id=679892 in Object.wait() [0x2e3b9000..0x2e3b94f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3af7c258> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-20" prio=10 tid=000b56b8 nid=112 lwp_id=679851 in Object.wait() [0x2e3fa000..0x2e3fa4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3a9b0250> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3a9b0250> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-19" prio=10 tid=000b5550 nid=111 lwp_id=679850 in Object.wait() [0x2e43b000..0x2e43b4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3a9b02f0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3a9b02f0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-18" daemon prio=10 tid=01a09c78 nid=110 lwp_id=679822 in Object.wait() [0x2e47c000..0x2e47c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39f1b930> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <39f1b930> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '11' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040f90 nid=109 lwp_id=679809 in Object.wait() [0x2e4bd000..0x2e4bd4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e53ac0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040e28 nid=108 lwp_id=679808 in Object.wait() [0x2e4fe000..0x2e4fe4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e535a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040cc0 nid=107 lwp_id=679807 in Object.wait() [0x2e53f000..0x2e53f4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e53090> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040b58 nid=106 lwp_id=679806 in Object.wait() [0x2e580000..0x2e5804f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e52b28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030409f0 nid=105 lwp_id=679805 in Object.wait() [0x2e5c1000..0x2e5c14f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e52610> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040888 nid=104 lwp_id=679804 in Object.wait() [0x2e602000..0x2e6024f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e520f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040720 nid=103 lwp_id=679803 in Object.wait() [0x2e643000..0x2e6434f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e51be0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030405b8 nid=102 lwp_id=679802 in Object.wait() [0x2e684000..0x2e6844f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e516c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040450 nid=101 lwp_id=679801 in Object.wait() [0x2e6c5000..0x2e6c54f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e511b0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e511b0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030402e8 nid=100 lwp_id=679800 in Object.wait() [0x2e706000..0x2e7064f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50c98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50c98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040180 nid=99 lwp_id=679799 in Object.wait() [0x2e747000..0x2e7474f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50780> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50780> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040018 nid=98 lwp_id=679798 in Object.wait() [0x2e788000..0x2e7884f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50268> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50268> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "SeedGenerator Thread" daemon prio=10 tid=000b53e8 nid=97 lwp_id=679782 in Object.wait() [0x2e7c9000..0x2e7c94f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39db8938> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)
         at java.lang.Object.wait(Object.java:429)
         at sun.security.provider.SeedGenerator$ThreadedSeedGenerator.run(SeedGenerator.java:334)
         - locked <39db8938> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)
         at java.lang.Thread.run(Thread.java:534)
    "Noisy Thread" daemon prio=10 tid=000b5280 nid=96 lwp_id=679781 in Object.wait() [0x2e80a000..0x2e80a4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39db89a0> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)
         at java.lang.Object.wait(Object.java:429)
         at sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread.run(SeedGenerator.java:452)
         - locked <39db89a0> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=10 tid=000b5118 nid=95 lwp_id=679741 in Object.wait() [0x2e84b000..0x2e84b4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <395c5970> (a weblogic.jms.backend.BETimerTree)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <395c5970> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "Thread-17" prio=10 tid=000b4fb0 nid=94 lwp_id=679731 in Object.wait() [0x2e88c000..0x2e88c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e8a0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e8a0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-16" prio=10 tid=000b4e48 nid=93 lwp_id=679728 in Object.wait() [0x2e8cd000..0x2e8cd4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e940> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e940> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-15" prio=10 tid=000b4ce0 nid=92 lwp_id=679727 in Object.wait() [0x2e90e000..0x2e90e4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e9e0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e9e0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-14" prio=10 tid=000b4b78 nid=91 lwp_id=679726 in Object.wait() [0x2e94f000..0x2e94f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ea80> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ea80> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-13" prio=10 tid=000b4a10 nid=90 lwp_id=679724 in Object.wait() [0x2e990000..0x2e9904f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971eb20> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971eb20> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-12" prio=10 tid=000b48a8 nid=89 lwp_id=679720 in Object.wait() [0x2e9d1000..0x2e9d14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ebc0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ebc0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-11" prio=10 tid=000b4740 nid=88 lwp_id=679719 in Object.wait() [0x2ea12000..0x2ea124f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ec60> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ec60> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-10" prio=10 tid=000b45d8 nid=87 lwp_id=679718 in Object.wait() [0x2ea53000..0x2ea534f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ed00> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ed00> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-9" prio=10 tid=000b4470 nid=86 lwp_id=679717 in Object.wait() [0x2ea94000..0x2ea944f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971eda0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971eda0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-8" prio=10 tid=000b4308 nid=85 lwp_id=679715 i

  • A StuckThreadMaxTime error; who can help me to analyse the thread dumps

    The StuckThreadMaxTime error freeze my system, no one can logged in.I have double checked the thread dumps, but get no help.
    The following is the thread dumps:
    <2004-12-7 ÉÏÎç05ʱ56·Ö47Ãë GMT+08:00> <Alert> <Log Management> <BEA-170017> <The log file ./myserver/myserver.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
    <2004-12-7 ÉÏÎç05ʱ56·Ö47Ãë GMT+08:00> <Alert> <Log Management> <BEA-170018> <The log file has been rotated to myserver.log00035. Log messages will continue to be logged in ./myserver/myserver.log.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '10' for queue: 'weblogic.kernel.Default' has been busy for "646" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '13' for queue: 'weblogic.kernel.Default' has been busy for "651" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default' has been busy for "650" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    Full thread dump Java HotSpot(TM) VM (1.4.2 1.4.2.03-040401-18:59-PA_RISC2.0 PA2.0 (aCC_AP) mixed mode):
    "ExecuteThread: '14' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f778 nid=71 lwp_id=39624 in Object.wait() [0x61abf000..0x61abf4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d24f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d24f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f610 nid=70 lwp_id=39623 in Object.wait() [0x61b40000..0x61b404f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d2578> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d2578> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f4a8 nid=69 lwp_id=39622 in Object.wait() [0x61bc1000..0x61bc14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d25f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d25f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f340 nid=68 lwp_id=39621 in Object.wait() [0x61c42000..0x61c424f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d2678> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.Eÿ

    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=000883f8 nid=27 lwp_id=39205 runnable [0x630ec000..0x630eb4f0]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at oracle.net.ns.Packet.receive(Unknown Source)
         at oracle.net.ns.DataPacket.receive(Unknown Source)
         at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
         at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
         at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:181)
         at oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:543)
         at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:913)
         at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:452)
         at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:984)
         at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2885)
         at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:2926)
         - locked <762d0bd8> (a oracle.jdbc.driver.T4CPreparedStatement)
         - locked <6c837e90> (a oracle.jdbc.driver.T4CConnection)
         at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:92)
         at com.huawei.oss.rms.persistence.ReadOperationPersistenceImpl.getAttributeValue(ReadOperationPersistenceImpl.java:2054)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.getAttributeValue(PersistenceOperationImpl.java:516)
         at com.huawei.oss.services.common.Utility.getNameByID(Utility.java:78)
         at com.huawei.oss.services.port.common.RMSPortUtility.setExtraToVO(RMSPortUtility.java:45)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.setExtraToVO(PhyPortConnHelper.java:2046)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.afterGet(PhyPortConnHelper.java:719)
         at com.huawei.oss.services.common.AbstractResourceHelper.getResource(AbstractResourceHelper.java:293)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.rmsGetByPort(PhyPortConnHelper.java:809)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.rmsGetAllPhyConnOfBoard(PhyPortConnHelper.java:1144)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:324)
         at com.huawei.oss.services.common.AbstractResourceHelper.directMethod(AbstractResourceHelper.java:489)
         at com.huawei.oss.services.common.AbstractResourceHelper.invokeMethod(AbstractResourceHelper.java:404)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSessionEJB.invokeMethod(ResourceMgmtSessionEJB.java:263)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl.invokeMethod(ResourceMgmtSession_1x9h7n_EOImpl.java:424)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl_WLSkel.invoke(Unknown Source)
         at weblogic.rmi.internal.activation.ActivatableServerRef.invoke(ActivatableServerRef.java:90)
         at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:420)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
         at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:415)
         at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=00088290 nid=26 lwp_id=39204 runnable [0x6316d000..0x6316c4f0]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at oracle.net.ns.Packet.receive(Unknown Source)
         at oracle.net.ns.DataPacket.receive(Unknown Source)
         at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
         at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
         at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:112)
         at oracle.jdbc.driver.T4CStatement.execute_for_rows(T4CStatement.java:474)
         at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:913)
         at oracle.jdbc.driver.T4CStatement.execute_maybe_describe(T4CStatement.java:383)
         at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:984)
         at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1124)
         - locked <76c0d1f8> (a oracle.jdbc.driver.T4CStatement)
         - locked <6f1ab788> (a oracle.jdbc.driver.T4CConnection)
         at weblogic.jdbc.wrapper.Statement.executeQuery(Statement.java:345)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.executeSQL(PersistenceOperationImpl.java:2884)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.setVersionFlagNResourceName(PersistenceOperationImpl.java:3137)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.makeAssociationValue(PersistenceOperationImpl.java:2600)
         at com.huawei.oss.rms.persistence.ReadOperationPersistenceImpl.getResourceAssociationSet(ReadOperationPersistenceImpl.java:2575)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.getResourceAssociationSet(PersistenceOperationImpl.java:603)
         at com.huawei.oss.services.port.common.RMSPortUtility.getSlotByEquip(RMSPortUtility.jÿ

  • Help needed with diagnosis of WebLogic server thread dump

     

    Sorry, my mistake - I was trying to search for "weblogic" before the dump file
    loaded fully.... I am not very familiar with VisiBroker and may be totally
    wrong, but my guess is it creates a thread for each connection/client. I would
    look for VisiBroker properties or settings that could limit the runaway creation
    of threads....
    Arunabh
    "Dimitri I. Rakitine" wrote:
    It is, preceded by lots and lots of CORBA (visibroker) threads. Is it
    possible that application code
    creates new ORBs more often than it should ?
    "Arunabh Hazarika" <[email protected]> wrote in message
    news:[email protected]...
    The attached file does not look like a WebLogic Server thread dump.....
    Arunabh
    Kapil Singh Pawar wrote:
    I am running a web application on WebLogic server 6.0 sp2 + rp1 (HP-UX).
    After
    I start loading the web application with more users I constantly get thejava.lang.OutOfMemoryError.
    I tried increasing the max_thread_proc kernel parameter to 2048 but thatdoes
    not seem to help much. Here is the error that I get on the WebLogicServer console.
    <Oct 7, 2002 5:00:27 PM GMT+02:00> <Error> <HTTP><[WebAppServletContext(1828590,i2)]
    Servlet failed with Exception
    java.lang.OutOfMemoryError: max threads limit being reached in VM
    at java.lang.Thread.start(Native Method)
    at netscape.ldap.LDAPConnThread.<init>(LDAPConnThread.java:112)
    atnetscape.ldap.LDAPConnection.getNewThread(LDAPConnection.java:1117)
    atnetscape.ldap.LDAPConnection.connect(LDAPConnection.java:1034)
    at netscape.ldap.LDAPConnection.connect(LDAPConnection.java:915)
    at netscape.ldap.LDAPConnection.connect(LDAPConnection.java:759)
    atcom.i2.scm.LDAP.ScmLDAPDefaultAuthenticate.authenticate(ScmLDAPDefaultAuthen
    ticate.java:129)
    at scmldap.service(scmldap.java:452)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :213)
    atweblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletCo
    ntext.java:1302)
    atweblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java
    :1676)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
    >
    I am posting the thread dump obtained at the OutOfMemoryError stage. Anyhelp
    in diagnosing the thread dump and possible fix for this error is highlyappreciated.
    For information sake, while the web application encountersOutOfMemoryError the
    example web apps and WebLogic console application work fine. So it looksto be
    a case of memory outage only for my custom application.
    Thanks
    Kapil Singh Pawar
    Name: thread_dump.txt
    thread_dump.txt Type: Plain Text (text/plain)
    Encoding: base64--
    Dimitri

  • Help with Thread dump analisys

    We have problems with our application which runs on bea8.1 sp1.
    Problem is that bea runs out from Threads (StuckThreads).
    thread dums look same for all threads but i can's see what is wrong.
    Here are dumps from first and last threads :
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x7f3628
    nid=0xd runnable [a6280000..a6281994
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c3197aa0> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
    "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x6dd548
    nid=0x24 runnable [a3d80000..a3d819
    94]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c30c5010> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

    Rob Woollen <[email protected]> wrote:
    Can you post the full thread dump?
    It looks like you have some code making an RMI call over the network.
    Is it calling another server, or perhaps a loopback call into the same
    server. The latter (opening a socket to your own process) is a good
    way
    to get a deadlock.
    -- Rob
    dara wrote:
    We have problems with our application which runs on bea8.1 sp1.
    Problem is that bea runs out from Threads (StuckThreads).
    thread dums look same for all threads but i can's see what is wrong.
    Here are dumps from first and last threads :
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5tid=0x7f3628
    nid=0xd runnable [a6280000..a6281994
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c3197aa0> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
    "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=5tid=0x6dd548
    nid=0x24 runnable [a3d80000..a3d819
    94]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c30c5010> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
    Thanks Rob,
    sorry for delay but I was on vacation.
    You are right about RMI, our app. consist from FE (front end) and BE (back end),
    comunication between them is RMI.
    We had some problems with BE but now we have new version and it looks ok.
    Anyway thanks for help, if I will have again some problems then it will be
    Help with thread analisys 2 ;-)

  • HELP!!! Why do my thread dump have so many locked monitor

    In recent weeks, My customer's weblogic server become more busy.Each time I downloaded
    the thread dump when CPU is busy than 90%,I found a lot of lock in the thread
    dump! Especially the class named "weblogic.kernel.ExecuteThread" . It's the kernel
    class of weblogic, Why were most of them locked?
    Full thread dump:
    "process forker" daemon prio=1 tid=0x95b4c8e0 nid=0x3084 waiting on monitor [0x977ff000..0x977ff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53d26420> (a java.lang.Object)
         at java.lang.UNIXProcessForker.run(Native Method)
         at java.lang.Thread.run(Thread.java:479)
    "process reaper" daemon prio=1 tid=0x95376758 nid=0x3083 runnable [0x96eff000..0x96eff890]
         at java.lang.UNIXProcessReaper.run(Native Method)
         at java.lang.Thread.run(Thread.java:479)
    "HighPriority TimeEventGenerator" daemon prio=1 tid=0x825bf18 nid=0x25a1 waiting
    on monitor [0x95dff000..0x95dff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53a3bd10> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
         - locked <53a3bd10> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:141)
         at java.lang.Thread.run(Thread.java:479)
    "ListenThread" prio=1 tid=0x94d00630 nid=0x25a0 runnable [0x95fff000..0x95fff890]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
         - locked <538fa238> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:238)
         at java.net.ServerSocket.accept(ServerSocket.java:217)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:325)
    "NBExecuteThread-1" daemon prio=1 tid=0x8c0dd1d0 nid=0x259c waiting on monitor
    [0x94521000..0x94521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a9650> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a9650> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "NBExecuteThread-0" daemon prio=1 tid=0x8c0dc7d0 nid=0x259b waiting on monitor
    [0x94321000..0x94321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a9060> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a9060> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    //"ExecuteThread-49" daemon prio=1 tid=0x8c034660 nid=0x259a runnable [0x94121000..0x94121890]
         at java.net.PlainDatagramSocketImpl.peek(Native Method)
         - locked <537acf60> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:349)
         - locked <486f9980> (a java.net.DatagramPacket)
         - locked <537acf40> (a java.net.MulticastSocket)
         at weblogic.cluster.TDMSocketIPM.receive(TDMSocketIPM.java:86)
         at weblogic.cluster.TMSocket.execute(TMSocket.java:249)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-48" daemon prio=1 tid=0x8c0c3d18 nid=0x2599 waiting for monitor
    entry [0x93f21000..0x93f21890]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - waiting to lock <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-47" daemon prio=1 tid=0x8c0c32d0 nid=0x2598 runnable [0x93d21000..0x93d21890]
         at weblogic.socket.PosixSocketMuxer.poll(Native Method)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - locked <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-46" daemon prio=1 tid=0x8c0bf410 nid=0x2597 waiting for monitor
    entry [0x93b21000..0x93b21890]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - waiting to lock <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-45" daemon prio=1 tid=0x8c0be9c8 nid=0x2596 waiting on monitor
    [0x93921000..0x93921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6df8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6df8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-44" daemon prio=1 tid=0x8c0bdab0 nid=0x2595 waiting on monitor
    [0x93721000..0x93721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6828> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6828> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-43" daemon prio=1 tid=0x8c0bd068 nid=0x2594 waiting on monitor
    [0x93521000..0x93521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6258> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6258> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-42" daemon prio=1 tid=0x8c0c0ce0 nid=0x2593 waiting on monitor
    [0x93321000..0x93321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a5c88> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a5c88> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-41" daemon prio=1 tid=0x8c0c0298 nid=0x2592 waiting on monitor
    [0x93121000..0x93121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a56b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a56b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-40" daemon prio=1 tid=0x8c0a21e0 nid=0x2591 waiting on monitor
    [0x92f21000..0x92f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a50e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a50e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-39" daemon prio=1 tid=0x8c0d4520 nid=0x2590 waiting on monitor
    [0x92d21000..0x92d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a4b18> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a4b18> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-38" daemon prio=1 tid=0x8c0d43d0 nid=0x258f waiting on monitor
    [0x92b21000..0x92b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a4548> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a4548> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-37" daemon prio=1 tid=0x8c0d3988 nid=0x258e waiting on monitor
    [0x92921000..0x92921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a3f78> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a3f78> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-36" daemon prio=1 tid=0x8c0ded20 nid=0x258d waiting on monitor
    [0x92721000..0x92721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a39a8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a39a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-35" daemon prio=1 tid=0x8c0de2d8 nid=0x258c waiting on monitor
    [0x92521000..0x92521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a33d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a33d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-34" daemon prio=1 tid=0x8c0cd528 nid=0x258b waiting on monitor
    [0x92321000..0x92321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2e08> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2e08> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-33" daemon prio=1 tid=0x8c0ccae0 nid=0x258a waiting on monitor
    [0x92121000..0x92121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2838> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2838> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-32" daemon prio=1 tid=0x8c075360 nid=0x2589 waiting on monitor
    [0x91f21000..0x91f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2268> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2268> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-31" daemon prio=1 tid=0x8c0837d0 nid=0x2588 waiting on monitor
    [0x91d21000..0x91d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a1c98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a1c98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-30" daemon prio=1 tid=0x8c0a5628 nid=0x2587 waiting on monitor
    [0x91b21000..0x91b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a16c8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a16c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-29" daemon prio=1 tid=0x8c07aaf8 nid=0x2586 waiting on monitor
    [0x91921000..0x91921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a10f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a10f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-28" daemon prio=1 tid=0x8c0c23d0 nid=0x2585 waiting on monitor
    [0x91721000..0x91721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a0b28> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a0b28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-27" daemon prio=1 tid=0x8c0c1988 nid=0x2584 waiting on monitor
    [0x91521000..0x91521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a0558> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a0558> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-26" daemon prio=1 tid=0x8c07ca00 nid=0x2583 waiting on monitor
    [0x91321000..0x91321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ff88> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ff88> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-25" daemon prio=1 tid=0x8c079e70 nid=0x2582 waiting on monitor
    [0x91121000..0x91121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379f9b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379f9b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-24" daemon prio=1 tid=0x8c078038 nid=0x2581 waiting on monitor
    [0x90f21000..0x90f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379f3e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379f3e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-23" daemon prio=1 tid=0x8c0a3298 nid=0x2580 waiting on monitor
    [0x90d21000..0x90d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ee18> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ee18> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-22" daemon prio=1 tid=0x8c0e30d8 nid=0x257f waiting on monitor
    [0x90b21000..0x90b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379e848> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379e848> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-21" daemon prio=1 tid=0x8c0c5948 nid=0x257e waiting on monitor
    [0x90921000..0x90921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379e278> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379e278> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-20" daemon prio=1 tid=0x8c0d0470 nid=0x257d waiting on monitor
    [0x90721000..0x90721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379dca8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379dca8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-19" daemon prio=1 tid=0x8c080798 nid=0x257c waiting on monitor
    [0x90521000..0x90521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379d6d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379d6d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-18" daemon prio=1 tid=0x8c080648 nid=0x257b waiting on monitor
    [0x90321000..0x90321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379d108> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379d108> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-17" daemon prio=1 tid=0x8c032380 nid=0x257a runnable [0x90121000..0x90121890]
         at java.lang.String.concat(String.java:1553)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:122)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:772)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-16" daemon prio=1 tid=0x8c0df9a8 nid=0x2579 waiting on monitor
    [0x8ff21000..0x8ff21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379c568> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379c568> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-15" daemon prio=1 tid=0x8c0a6288 nid=0x2578 waiting on monitor
    [0x8fd21000..0x8fd21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379bf98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379bf98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-14" daemon prio=1 tid=0x8c0dbd60 nid=0x2577 waiting on monitor
    [0x8fb21000..0x8fb21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379b9c8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379b9c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-13" daemon prio=1 tid=0x8c0c8498 nid=0x2576 waiting on monitor
    [0x8f921000..0x8f921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379b3f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379b3f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-12" daemon prio=1 tid=0x8c0a6a28 nid=0x2575 waiting on monitor
    [0x8f721000..0x8f721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ae28> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ae28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-11" daemon prio=1 tid=0x8c07d348 nid=0x2574 runnable [0x8f521000..0x8f521890]
         at java.lang.System.arraycopy(Native Method)
         at java.lang.String.getChars(String.java:557)
         at java.lang.String.concat(String.java:1554)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:251)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doGet(ClaimServlet.java:77)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-10" daemon prio=1 tid=0x8c0369b0 nid=0x2571 waiting on monitor
    [0x8f321000..0x8f321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379a288> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379a288> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-9" daemon prio=1 tid=0x8c079898 nid=0x256f waiting on monitor [0x8f121000..0x8f121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53799cb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53799cb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-8" daemon prio=1 tid=0x8c035260 nid=0x256e waiting on monitor [0x8ef21000..0x8ef21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537996e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537996e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-7" daemon prio=1 tid=0x8c034510 nid=0x256d waiting on monitor [0x8ed21000..0x8ed21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53799118> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53799118> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-6" daemon prio=1 tid=0x8c081130 nid=0x256c waiting on monitor [0x8eb21000..0x8eb21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53798b48> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53798b48> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-5" daemon prio=1 tid=0x8c0c7108 nid=0x256b waiting on monitor [0x8e921000..0x8e921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53798578> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53798578> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-4" daemon prio=1 tid=0x8c0a4388 nid=0x256a waiting on monitor [0x8e721000..0x8e721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53797fa8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53797fa8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-3" daemon prio=1 tid=0x8c0367b8 nid=0x2569 waiting on monitor [0x8e521000..0x8e521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537979d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537979d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-2" daemon prio=1 tid=0x8c036290 nid=0x2568 waiting on monitor [0x8e321000..0x8e321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53797408> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53797408> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-1" daemon prio=1 tid=0x8c0a4ee0 nid=0x2567 waiting on monitor [0x8e121000..0x8e121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53796e38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53796e38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-0" daemon prio=1 tid=0x8c0a4550 nid=0x2566 runnable [0x8df21000..0x8df21890]
         at java.lang.String.concat(String.java:1553)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.end.EndCaseServlet.service(EndCaseServlet.java:56)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "TimeEventGenerator" daemon prio=1 tid=0x8c076008 nid=0x2565 waiting on monitor
    [0x8dd21000..0x8dd21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53795330> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
         - locked <53795330> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:141)
         at java.lang.Thread.run(Thread.java:479)
    "SpinnerRandomSource" daemon prio=1 tid=0x8c0b34a0 nid=0x2563 waiting on monitor
    [0x8db21000..0x8db21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53793040> (a weblogic.security.SpinnerThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:104)
         - locked <53793040> (a weblogic.security.SpinnerThread)
         at weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:121)
    "Signal Dispatcher" daemon prio=1 tid=0x8c0077e8 nid=0x2561 waiting on monitor
    [0..0]
    "Finalizer" daemon prio=1 tid=0x8c000540 nid=0x255e waiting on monitor [0x8c2ff000..0x8c2ff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:103)
         - locked <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:157)
    "Reference Handler" daemon prio=1 tid=0x80851f0 nid=0x255d waiting on monitor
    [0x8bf8c000..0x8bf8c890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53720208> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:415)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:105)
         - locked <53720208> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x804ead0 nid=0x2543 waiting on monitor [0xbfffc000..0xbfffd00c]
         at java.lang.Object.wait(Native Method)
         - waiting on <53743c20> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1958)
         - locked <53743c20> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.reflect.Method.invoke(Native Method)
         at weblogic.Server.startServerDynamically(Server.java:148)
         at weblogic.Server.main(Server.java:97)
         at weblogic.Server.main(Server.java:58)
    "VM Thread" prio=1 tid=0x8082110 nid=0x255c runnable
    "VM Periodic Task Thread" prio=1 tid=0x8c0064f8 nid=0x255f waiting on monitor
    "Suspend Checker Thread" prio=1 tid=0x8c006e30 nid=0x2560 runnable

    These are execute threads waiting for requests to come in.
    BTW, you probably want to look into using StringBuffer instead of
    strings concatenation in com.ebao.carcase.bean.query.Query.generatorXML.
    "min.lu" <[email protected]> wrote in message
    news:[email protected]...
    >
    In recent weeks, My customer's weblogic server become more busy.Each timeI downloaded
    the thread dump when CPU is busy than 90%,I found a lot of lock in thethread
    dump! Especially the class named "weblogic.kernel.ExecuteThread" . It'sthe kernel
    class of weblogic, Why were most of them locked?
    Full thread dump:
    "process forker" daemon prio=1 tid=0x95b4c8e0 nid=0x3084 waiting onmonitor [0x977ff000..0x977ff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53d26420> (a java.lang.Object)
    at java.lang.UNIXProcessForker.run(Native Method)
    at java.lang.Thread.run(Thread.java:479)
    "process reaper" daemon prio=1 tid=0x95376758 nid=0x3083 runnable[0x96eff000..0x96eff890]
    at java.lang.UNIXProcessReaper.run(Native Method)
    at java.lang.Thread.run(Thread.java:479)
    "HighPriority TimeEventGenerator" daemon prio=1 tid=0x825bf18 nid=0x25a1waiting
    on monitor [0x95dff000..0x95dff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53a3bd10> (a weblogic.time.common.internal.TimeTable)
    at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
    - locked <53a3bd10> (a weblogic.time.common.internal.TimeTable)
    atweblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java
    :141)
    at java.lang.Thread.run(Thread.java:479)
    "ListenThread" prio=1 tid=0x94d00630 nid=0x25a0 runnable[0x95fff000..0x95fff890]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
    - locked <538fa238> (a java.net.PlainSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:238)
    at java.net.ServerSocket.accept(ServerSocket.java:217)
    atweblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
    at weblogic.t3.srvr.ListenThread.run(ListenThread.java:325)
    "NBExecuteThread-1" daemon prio=1 tid=0x8c0dd1d0 nid=0x259c waiting onmonitor
    [0x94521000..0x94521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a9650> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a9650> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "NBExecuteThread-0" daemon prio=1 tid=0x8c0dc7d0 nid=0x259b waiting onmonitor
    [0x94321000..0x94321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a9060> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a9060> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    //"ExecuteThread-49" daemon prio=1 tid=0x8c034660 nid=0x259a runnable[0x94121000..0x94121890]
    at java.net.PlainDatagramSocketImpl.peek(Native Method)
    - locked <537acf60> (a java.net.PlainDatagramSocketImpl)
    at java.net.DatagramSocket.receive(DatagramSocket.java:349)
    - locked <486f9980> (a java.net.DatagramPacket)
    - locked <537acf40> (a java.net.MulticastSocket)
    at weblogic.cluster.TDMSocketIPM.receive(TDMSocketIPM.java:86)
    at weblogic.cluster.TMSocket.execute(TMSocket.java:249)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-48" daemon prio=1 tid=0x8c0c3d18 nid=0x2599 waiting formonitor
    entry [0x93f21000..0x93f21890]
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - waiting to lock <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-47" daemon prio=1 tid=0x8c0c32d0 nid=0x2598 runnable[0x93d21000..0x93d21890]
    at weblogic.socket.PosixSocketMuxer.poll(Native Method)
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - locked <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-46" daemon prio=1 tid=0x8c0bf410 nid=0x2597 waiting formonitor
    entry [0x93b21000..0x93b21890]
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - waiting to lock <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-45" daemon prio=1 tid=0x8c0be9c8 nid=0x2596 waiting onmonitor
    [0x93921000..0x93921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6df8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6df8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-44" daemon prio=1 tid=0x8c0bdab0 nid=0x2595 waiting onmonitor
    [0x93721000..0x93721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6828> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6828> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-43" daemon prio=1 tid=0x8c0bd068 nid=0x2594 waiting onmonitor
    [0x93521000..0x93521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6258> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6258> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-42" daemon prio=1 tid=0x8c0c0ce0 nid=0x2593 waiting onmonitor
    [0x93321000..0x93321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a5c88> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a5c88> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-41" daemon prio=1 tid=0x8c0c0298 nid=0x2592 waiting onmonitor
    [0x93121000..0x93121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a56b8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a56b8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-40" daemon prio=1 tid=0x8c0a21e0 nid=0x2591 waiting onmonitor
    [0x92f21000..0x92f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a50e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a50e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-39" daemon prio=1 tid=0x8c0d4520 nid=0x2590 waiting onmonitor
    [0x92d21000..0x92d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a4b18> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a4b18> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-38" daemon prio=1 tid=0x8c0d43d0 nid=0x258f waiting onmonitor
    [0x92b21000..0x92b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a4548> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a4548> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-37" daemon prio=1 tid=0x8c0d3988 nid=0x258e waiting onmonitor
    [0x92921000..0x92921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a3f78> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a3f78> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-36" daemon prio=1 tid=0x8c0ded20 nid=0x258d waiting onmonitor
    [0x92721000..0x92721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a39a8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a39a8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-35" daemon prio=1 tid=0x8c0de2d8 nid=0x258c waiting onmonitor
    [0x92521000..0x92521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a33d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a33d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-34" daemon prio=1 tid=0x8c0cd528 nid=0x258b waiting onmonitor
    [0x92321000..0x92321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2e08> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2e08> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-33" daemon prio=1 tid=0x8c0ccae0 nid=0x258a waiting onmonitor
    [0x92121000..0x92121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2838> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2838> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-32" daemon prio=1 tid=0x8c075360 nid=0x2589 waiting onmonitor
    [0x91f21000..0x91f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2268> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2268> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-31" daemon prio=1 tid=0x8c0837d0 nid=0x2588 waiting onmonitor
    [0x91d21000..0x91d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a1c98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a1c98> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-30" daemon prio=1 tid=0x8c0a5628 nid=0x2587 waiting onmonitor
    [0x91b21000..0x91b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a16c8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a16c8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-29" daemon prio=1 tid=0x8c07aaf8 nid=0x2586 waiting onmonitor
    [0x91921000..0x91921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a10f8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a10f8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-28" daemon prio=1 tid=0x8c0c23d0 nid=0x2585 waiting onmonitor
    [0x91721000..0x91721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a0b28> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a0b28> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-27" daemon prio=1 tid=0x8c0c1988 nid=0x2584 waiting onmonitor
    [0x91521000..0x91521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a0558> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a0558> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-26" daemon prio=1 tid=0x8c07ca00 nid=0x2583 waiting onmonitor
    [0x91321000..0x91321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ff88> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ff88> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-25" daemon prio=1 tid=0x8c079e70 nid=0x2582 waiting onmonitor
    [0x91121000..0x91121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379f9b8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379f9b8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-24" daemon prio=1 tid=0x8c078038 nid=0x2581 waiting onmonitor
    [0x90f21000..0x90f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379f3e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379f3e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-23" daemon prio=1 tid=0x8c0a3298 nid=0x2580 waiting onmonitor
    [0x90d21000..0x90d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ee18> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ee18> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-22" daemon prio=1 tid=0x8c0e30d8 nid=0x257f waiting onmonitor
    [0x90b21000..0x90b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379e848> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379e848> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-21" daemon prio=1 tid=0x8c0c5948 nid=0x257e waiting onmonitor
    [0x90921000..0x90921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379e278> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379e278> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-20" daemon prio=1 tid=0x8c0d0470 nid=0x257d waiting onmonitor
    [0x90721000..0x90721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379dca8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379dca8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-19" daemon prio=1 tid=0x8c080798 nid=0x257c waiting onmonitor
    [0x90521000..0x90521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379d6d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379d6d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-18" daemon prio=1 tid=0x8c080648 nid=0x257b waiting onmonitor
    [0x90321000..0x90321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379d108> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379d108> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-17" daemon prio=1 tid=0x8c032380 nid=0x257a runnable[0x90121000..0x90121890]
    at java.lang.String.concat(String.java:1553)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:122)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:772)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-16" daemon prio=1 tid=0x8c0df9a8 nid=0x2579 waiting onmonitor
    [0x8ff21000..0x8ff21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379c568> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379c568> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-15" daemon prio=1 tid=0x8c0a6288 nid=0x2578 waiting onmonitor
    [0x8fd21000..0x8fd21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379bf98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379bf98> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-14" daemon prio=1 tid=0x8c0dbd60 nid=0x2577 waiting onmonitor
    [0x8fb21000..0x8fb21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379b9c8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379b9c8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-13" daemon prio=1 tid=0x8c0c8498 nid=0x2576 waiting onmonitor
    [0x8f921000..0x8f921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379b3f8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379b3f8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-12" daemon prio=1 tid=0x8c0a6a28 nid=0x2575 waiting onmonitor
    [0x8f721000..0x8f721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ae28> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ae28> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-11" daemon prio=1 tid=0x8c07d348 nid=0x2574 runnable[0x8f521000..0x8f521890]
    at java.lang.System.arraycopy(Native Method)
    at java.lang.String.getChars(String.java:557)
    at java.lang.String.concat(String.java:1554)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:251)
    at com.ebao.carcase.servlet.claim.ClaimServlet.doGet(ClaimServlet.java:77)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-10" daemon prio=1 tid=0x8c0369b0 nid=0x2571 waiting onmonitor
    [0x8f321000..0x8f321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379a288> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379a288> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-9" daemon prio=1 tid=0x8c079898 nid=0x256f waiting onmonitor [0x8f121000..0x8f121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53799cb8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53799cb8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-8" daemon prio=1 tid=0x8c035260 nid=0x256e waiting onmonitor [0x8ef21000..0x8ef21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537996e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537996e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-7" daemon prio=1 tid=0x8c034510 nid=0x256d waiting onmonitor [0x8ed21000..0x8ed21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53799118> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53799118> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-6" daemon prio=1 tid=0x8c081130 nid=0x256c waiting onmonitor [0x8eb21000..0x8eb21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53798b48> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53798b48> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-5" daemon prio=1 tid=0x8c0c7108 nid=0x256b waiting onmonitor [0x8e921000..0x8e921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53798578> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53798578> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-4" daemon prio=1 tid=0x8c0a4388 nid=0x256a waiting onmonitor [0x8e721000..0x8e721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53797fa8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53797fa8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-3" daemon prio=1 tid=0x8c0367b8 nid=0x2569 waiting onmonitor [0x8e521000..0x8e521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537979d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537979d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-2" daemon prio=1 tid=0x8c036290 nid=0x2568 waiting onmonitor [0x8e321000..0x8e321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53797408> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53797408> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-1" daemon prio=1 tid=0x8c0a4ee0 nid=0x2567 waiting onmonitor [0x8e121000..0x8e121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53796e38> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53796e38> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-0" daemon prio=1 tid=0x8c0a4550 nid=0x2566 runnable[0x8df21000..0x8df21890]
    at java.lang.String.concat(String.java:1553)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.end.EndCaseServlet.service(EndCaseServlet.java:56)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "TimeEventGenerator" daemon prio=1 tid=0x8c076008 nid=0x2565 waiting onmonitor
    [0x8dd21000..0x8dd21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53795330> (a weblogic.time.common.internal.TimeTable)
    at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
    - locked <53795330> (a weblogic.time.common.internal.TimeTable)
    atweblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java
    :141)
    at java.lang.Thread.run(Thread.java:479)
    "SpinnerRandomSource" daemon prio=1 tid=0x8c0b34a0 nid=0x2563 waiting onmonitor
    [0x8db21000..0x8db21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53793040> (a weblogic.security.SpinnerThread)
    at java.lang.Object.wait(Object.java:415)
    atweblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:10
    4)
    - locked <53793040> (a weblogic.security.SpinnerThread)
    at weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:121)
    "Signal Dispatcher" daemon prio=1 tid=0x8c0077e8 nid=0x2561 waiting onmonitor
    [0..0]
    "Finalizer" daemon prio=1 tid=0x8c000540 nid=0x255e waiting on monitor[0x8c2ff000..0x8c2ff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:103)
    - locked <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:157)
    "Reference Handler" daemon prio=1 tid=0x80851f0 nid=0x255d waiting onmonitor
    [0x8bf8c000..0x8bf8c890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53720208> (a java.l

  • Query quote help-Please ignore this thread

    Dear all,
    Please ignore this thread.
    Thanks for understanding
    Kai
    Edited by: KaiS on Nov 12, 2009 9:33 AM

    try
    and A.subno =''' || subno || ''' ;' from usr_inf where rownum <300;
    those are two single quotes ('), not double quotes.

  • Thread Dump issue with LD_ASSUME_KERNEL=2.4.1

    Hi ,
    When I take Thread Dump using 'jstack <PID>' in JDK 1.5 it givss me "sun.jvm.hotspot.debugger.DebuggerException" in the dump nothing more
    I set LD_ASSUME_KERNEL=2.4.1 in my server to avaoid some other issue ( JVM crash some times)
    Surprisingly, I can not stop my server ( my java process ) after that using our Shutdow scripts , Ctrl C or even "kill -9 <PID>",
    I have to restart the machine or manually release uncleaned resources that my server occupied and restart the server
    This happens Redhar 9 as well as in Linux ES.
    Anybody faced similar problem?
    Any help or information regarding this is highly apprecialted
    Vasu
    Thread Dump Output:
    Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Meth
    od)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:34)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(Li
    nuxDebuggerLocal.java:431)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(Linux
    DebuggerLocal.java:109)
    sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_re
    gs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(L
    inuxDebuggerLocal.java:134)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebugge
    rLocal.java:437)
    at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:48)
    at sun.jvm.hotspot.runtime.linux_x86.LinuxX86JavaThreadPDAccess.getCurrentFrameGuess(LinuxX86
    JavaThreadPDAccess.java:75)
    at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:252)
    at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:211)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:42)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:58)
    Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Meth
    od)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:34)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(Li
    nuxDebuggerLocal.java:431)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(Linux
    DebuggerLocal.java:109)
    sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_re
    gs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(L
    inuxDebuggerLocal.java:134)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebugge
    rLocal.java:437)

    @brain0
    I've downloaded the glic-2.3.6 sources from gnu, so I could build it from those. I'm however reluctant to do this because I really don't want to break my install.
    I do agree with you on the NPTL statement, but pvs relies on allegro, which relies on LinuxThreads. Allegro is not being ported to new versions of glibc, so that approach is unfortunately not viable.
    @iphitus
    I wasn't very specific - it's allegro as in a lisp environment.
    I think I'll try and install an old version of arch on wmware instead. Is there anywhere you can check out glibc version numbering on old arch install isos (ie. do I need arch-0.[1-9].iso)? And anywhere you can download the old isos (tried filewatcher, but a lot of the older sites seem broken)?
    Thanks for the replies,
    Mads
    PS. I noticed that you recommended slackware for old kernels in another thread. I'm however in a bit different situation as I need old versions of glibc. Furthermore I would prefer sticking to arch, but was wondering whether there were any specific reasons for not doing that.

  • Waiting for monitor enty messages in thread dump

    Hi,
    I've a thread dump of a container in a Oracle Application Server 9.0.4.1 (running 1.4.2_07) right before restart because off low memory and a lot of threads in this dump are in this state:
    "AJPRequestHandler-ApplicationServerThread-1588" prio=1 tid=0x37634bf8 nid=0x1982 waiting for monitor entry [43f22000..43f2386c]
    can anyone tell me what this means? Google didn't reveal anything useful.
    Another thing I don't understand although it seems it doesn't do any harm:
    "AJPRequestHandler-ApplicationServerThread-1630" prio=1 tid=0x09011a68 nid=0x1982 in Object.wait() [40276000..4027686c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x6624e628> (a java.util.HashMap)
    at java.lang.Object.wait(Object.java:429)
    at java.net.InetAddress.checkLookupTable(InetAddress.java:1109)
    - locked <0x6624e628> (a java.util.HashMap)
    at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1048)
    at java.net.InetAddress.getLocalHost(InetAddress.java:1188)
    at com.evermind.server.http.EvermindHttpSession.encodeSession(EvermindHttpSession.java:414)
    at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:1599)
    why is this thread waiting for a lock itself has?
    Thanx for any help and hints in advance.
    Regards,
    Ingo

    Hi,
    I've a follow-up question, I have the following thread in several thread dumps:
    "AJPRequestHandler-ApplicationServerThread-717" prio=1 tid=0x4046b988 nid=0x3dd3 waiting for monitor entry [3845e000..3845e86c]
         at java.net.Inet4Address.getAddress(Inet4Address.java:287)
         at com.evermind.server.http.StandardRequestLogger.appendIP(StandardRequestLogger.java:187)
         at com.evermind.server.http.StandardRequestLogger.log(StandardRequestLogger.java:45)
         - locked <0x663b2088> (a com.evermind.server.http.StandardRequestLogger)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:906)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    What is the monitor this thread is waiting for? According to the source in Inet4Address at line 287 the method tries to create a new byte array... is there some system monitor which can't be seen in the thread dump?
    This thread holds a lock (0x663b2088) 46 other threads are waiting for...
    Thanx in advance for any hints.
    Regards,
    Ingo

  • JVM falls down often. Why? (I have thread dump)

    Good day!
    Sometimes (about each two days) our production server falls down. But we can't understand why.
    There is no one record in logs and stdout.
    When server has falled down two days ago, I make full stack dump (kill -3). This is the link to stackdump:
    http://users.g.com.ua/~vitaly_b/work/stackdump2.zip
    I have writed tool that analyzes stackdump.
    I've seen that there is 5 locks that are in waiting state from more than one another threads. This locks are in waiting state for some minutes.
    One of this lock (0x00002b58afb89630) is never locked in stack dump, but 85 threads waits for it. How can it be???
    Another 5 locks are locked with threads, that makes primitive operations for very long time. For example this thread now in java.util.HashMap.newKeyIterator, but 89 other threads are waiting for lock <0x00002b58afb85b1> for few minutes:
    "http-8080-Processor1533" daemon prio=1 tid=0x00002aaab2566160 nid=0x6acb runnable
    [0x000000004c410000..0x000000004c412c20]
         at java.util.HashMap.newKeyIterator(HashMap.java:889)
         at java.util.HashMap$KeySet.iterator(HashMap.java:921)
         at java.util.HashSet.iterator(HashSet.java:154)
         at net.sf.ehcache.event.RegisteredEventListeners.notifyElementExpiry(RegisteredEventListeners.java:116)
         at net.sf.ehcache.Cache.remove(Cache.java:1017)
         at net.sf.ehcache.Cache.searchInMemoryStore(Cache.java:829)
         at net.sf.ehcache.Cache.get(Cache.java:654)
         - locked <0x00002b58afb85b18> (a net.sf.ehcache.Cache)
         at org.hibernate.cache.EhCache.get(EhCache.java:75)
         at org.hibernate.cache.StandardQueryCache.get(StandardQueryCache.java:96)
         at org.hibernate.loader.Loader.getResultFromQueryCache(Loader.java:2086)
         at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
         at java.lang.Thread.run(Thread.java:595)Other locks are waits in same state.
    Stack dump analysis with my tool:
    waiting to lock <xxx>::
    <0x00002b58afb8bd18>: 1 times.
    <0x00002b58afb9aa50>: 1 times.
    <0x00002b58afb9c138>: 2 times.
    <0x00002b58afb8b690>: 1 times.
    <0x00002b597ae1ecf8>: 1 times.
    <0x00002b58afb85b18>: 89 times.
    <0x00002b58afb88548>: 1 times.
    <0x00002b58afb96110>: 1 times.
    <0x00002b58afb89630>: 85 times.
    <0x00002b58afb8e5a0>: 1 times.
    <0x00002b58afb98090>: 1 times.
    <0x00002b58afbacd28>: 1 times.
    <0x00002b58afb99380>: 1 times.
    <0x00002b58afb9a0f8>: 1 times.
    <0x00002b58afb90d48>: 1 times.
    <0x00002b58a34127a8>: 4 times.
    <0x00002b58b09c5138>: 29 times.
    <0x00002b58afb91d18>: 1 times.
    waiting on <xxx>::
    <0x00002b58afbb0370>: 1 times.
    <0x00002b58adae0ff8>: 1 times.
    <0x00002b58adae0f48>: 1 times.
    <0x00002b58adae0d38>: 1 times.
    <0x00002b58afbb0518>: 1 times.
    <0x00002b58adae0de8>: 1 times.
    <0x00002b58af9a7ba0>: 1 times.
    <0x00002b58adae0e98>: 1 times.
    <0x00002b58afbaf898>: 1 times.
    <0x00002b58b0930400>: 1 times.
    <0x00002b58afbaff00>: 1 times.My system configuration:
    jdk 1.5.0.11
    tomcat 5.5.20
    SuSe Linux 10.1
    How can I solve this problem? Help me please!
    Thanks!

    Good day!
    Sometimes (about each two days) our production server falls down. But we can't understand why.
    There is no one record in logs and stdout.
    When server has falled down two days ago, I make full stack dump (kill -3). This is the link to stackdump:
    http://users.g.com.ua/~vitaly_b/work/stackdump2.zip
    I have writed tool that analyzes stackdump.
    I've seen that there is 5 locks that are in waiting state from more than one another threads. This locks are in waiting state for some minutes.
    One of this lock (0x00002b58afb89630) is never locked in stack dump, but 85 threads waits for it. How can it be???
    Another 5 locks are locked with threads, that makes primitive operations for very long time. For example this thread now in java.util.HashMap.newKeyIterator, but 89 other threads are waiting for lock <0x00002b58afb85b1> for few minutes:
    "http-8080-Processor1533" daemon prio=1 tid=0x00002aaab2566160 nid=0x6acb runnable
    [0x000000004c410000..0x000000004c412c20]
         at java.util.HashMap.newKeyIterator(HashMap.java:889)
         at java.util.HashMap$KeySet.iterator(HashMap.java:921)
         at java.util.HashSet.iterator(HashSet.java:154)
         at net.sf.ehcache.event.RegisteredEventListeners.notifyElementExpiry(RegisteredEventListeners.java:116)
         at net.sf.ehcache.Cache.remove(Cache.java:1017)
         at net.sf.ehcache.Cache.searchInMemoryStore(Cache.java:829)
         at net.sf.ehcache.Cache.get(Cache.java:654)
         - locked <0x00002b58afb85b18> (a net.sf.ehcache.Cache)
         at org.hibernate.cache.EhCache.get(EhCache.java:75)
         at org.hibernate.cache.StandardQueryCache.get(StandardQueryCache.java:96)
         at org.hibernate.loader.Loader.getResultFromQueryCache(Loader.java:2086)
         at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
         at java.lang.Thread.run(Thread.java:595)Other locks are waits in same state.
    Stack dump analysis with my tool:
    waiting to lock <xxx>::
    <0x00002b58afb8bd18>: 1 times.
    <0x00002b58afb9aa50>: 1 times.
    <0x00002b58afb9c138>: 2 times.
    <0x00002b58afb8b690>: 1 times.
    <0x00002b597ae1ecf8>: 1 times.
    <0x00002b58afb85b18>: 89 times.
    <0x00002b58afb88548>: 1 times.
    <0x00002b58afb96110>: 1 times.
    <0x00002b58afb89630>: 85 times.
    <0x00002b58afb8e5a0>: 1 times.
    <0x00002b58afb98090>: 1 times.
    <0x00002b58afbacd28>: 1 times.
    <0x00002b58afb99380>: 1 times.
    <0x00002b58afb9a0f8>: 1 times.
    <0x00002b58afb90d48>: 1 times.
    <0x00002b58a34127a8>: 4 times.
    <0x00002b58b09c5138>: 29 times.
    <0x00002b58afb91d18>: 1 times.
    waiting on <xxx>::
    <0x00002b58afbb0370>: 1 times.
    <0x00002b58adae0ff8>: 1 times.
    <0x00002b58adae0f48>: 1 times.
    <0x00002b58adae0d38>: 1 times.
    <0x00002b58afbb0518>: 1 times.
    <0x00002b58adae0de8>: 1 times.
    <0x00002b58af9a7ba0>: 1 times.
    <0x00002b58adae0e98>: 1 times.
    <0x00002b58afbaf898>: 1 times.
    <0x00002b58b0930400>: 1 times.
    <0x00002b58afbaff00>: 1 times.My system configuration:
    jdk 1.5.0.11
    tomcat 5.5.20
    SuSe Linux 10.1
    How can I solve this problem? Help me please!
    Thanks!

  • Program hangs, can't get thread dump

    Hello
    I have a Java program which sometimes works OK untill the very end, but sometimes just hangs in a random moment and doesn't react to Ctrl+C or Ctrl+Break. Process doesn't use CPU, so I guess it isn't something like infinite loop (I actually cannot think of places in my program where this kind of loop may occur). I tried to get thread dump to see if there are any deadlocks, but Ctrl+Break doesn't work and jstack doesn't work also (I've read that there is -F option, but it isn't available on Windows). Also I tried to connect to a process with Java Visual VM - doesn't help too (just says 'Connecting...' forever).
    What else can I do to find out what's going on in my programm and fix it?
    I tried to run it with JDK 1.6.0_07 and 1.6.0_16. I'm using Windows Vista Enterprise SP1, 64-bit.
    Thanks in advance, Yulia.
    Edited by: Yulia_Dubinina on Oct 26, 2009 11:58 PM
    Edited by: Yulia_Dubinina on Oct 27, 2009 12:13 AM

    Yulia_Dubinina wrote:
    Yesterday I tried to start my program, connect with Java Visual VM to it and see how threads visualisation will look like in a moment when program hangs. My programm died after about 50 min and all info which I was able to see was about this period of time (e.g. a timeline diagram stopped changing and the last time mark was 49:23), but Java Visual VM was still connected to a process. I tried to press Thread Dump button, it worked, but there was nothing 'criminal' on the dump it provided. So I guess maybe it's not actual thread dump, it's something that Java Visual VM was able to get before process hanged.
    I don't have much experience in resolving this kind of problems, so any advices will be great appreciated.
    Edited by: Yulia_Dubinina on Oct 27, 2009 2:14 PMTurn on verbose GC and see if you got lots and lots of full GC when the freeze happens. A full GC can take several minutes if you have a bad configuration (where e.g. memory has been paged out to disk)

Maybe you are looking for