Help to anayse Thread Dump

hi,
My application uses RMI and some times the application hangs-up. The following is the thread dump i got. Please help me to find out the dead lock from this dump.
=========================================================================
Full thread dump Java HotSpot(TM) Server VM (1.4.1 1.4.1.02-030502-15:31-PA_RISC
2.0 PA2.0 mixed mode):
"RMI LeaseChecker" daemon prio=8 tid=0x006eac78 nid=20 lwp_id=280805 waiting on
condition [0x6f042000..0x6f042530]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
at java.lang.Thread.run(Thread.java:536)
"RMI TCP Connection(1)-192.168.174.45" daemon prio=10 tid=0x00b06570 nid=19 lwp_
id=280803 runnable [0x6f0c3000..0x6f0c3530]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x73aea6d8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:4
42)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport
.java:701)
at java.lang.Thread.run(Thread.java:536)
"MultiThreadedTimerMain" prio=10 tid=0x0004c660 nid=18 lwp_id=280759 in Object.w
ait() [0x6f144000..0x6f144530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73adacb8> (a jp.co.nec.diosa.fw.util.timer.TaskQueue)
at java.lang.Object.wait(Object.java:426)
at jp.co.nec.diosa.fw.util.timer.MultiThreadedTimerThread.mainLoop(Multi
ThreadedTimerThread.java:174)
- locked <0x73adacb8> (a jp.co.nec.diosa.fw.util.timer.TaskQueue)
at jp.co.nec.diosa.fw.util.timer.MultiThreadedTimerThread.run(MultiThrea
dedTimerThread.java:115)
"RMI RenewClean-[192.168.174.45:62731]" daemon prio=10 tid=0x0004c3a0 nid=16 lwp
_id=280747 runnable [0x6f27b000..0x6f27b530]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x76400468> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readByte(DataInputStream.java:276)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:313)
at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.jav
a:328)
at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:
144)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCCli
ent.java:539)
at java.lang.Thread.run(Thread.java:536)
"GC Daemon" daemon prio=10 tid=0x0004c0e0 nid=14 lwp_id=280709 in Object.wait()
[0x6f37d000..0x6f37d530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73a00588> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x73a00588> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=10 tid=0x0004bf80 nid=13 lwp_id=280708 in Object.wait() [0x762
fe000..0x762fe530]
at java.lang.Object.wait(Native Method)
- waiting on <0x739ffb88> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x739ffb88> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
at java.lang.Thread.run(Thread.java:536)
"Thread-1" daemon prio=10 tid=0x0004be20 nid=12 lwp_id=280707 in Object.wait() [
0x7637f000..0x7637f530]
at java.lang.Object.wait(Native Method)
- waiting on <0x739ffe80> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:426)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x739ffe80> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"RMI TCP Accept-0" daemon prio=10 tid=0x0004bcc0 nid=11 lwp_id=280706 runnable [
0x7796b000..0x7796b530]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x739ffbe0> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:439)
at java.net.ServerSocket.accept(ServerSocket.java:410)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:536)
"Signal Dispatcher" daemon prio=10 tid=0x0004b740 nid=8 lwp_id=280652 waiting on
condition [0x00000000..0xffffffff]
"Finalizer" daemon prio=10 tid=0x0004b5e0 nid=5 lwp_id=280644 in Object.wait() [
0x77b7f000..0x77b7f530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73a03960> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x73a03960> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0004b480 nid=4 lwp_id=280643 in Object.
wait() [0x79c08000..0x79c08530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73a039c8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:426)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
- locked <0x73a039c8> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0004b320 nid=1 lwp_id=280621 runnable [0x79ff1000..0x79ff19
50]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x73ac02f8> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readByte(DataInputStream.java:276)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:
189)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
at jp.co.nec.diosa.fw.ms.service.Core_Stub.configure(Unknown Source)
at jp.co.nec.diosa.utilapp.JUtilityApplication.<init>(JUtilityApplicatio
n.java:1083)
at jp.co.nec.diosa.utilapp.JUtilityAppServer.getUtilityAppReference(JUti
lityAppServer.java:185)
at jp.co.nec.diosa.utilapp.JUtilityAppServer.<init>(JUtilityAppServer.ja
va:114)
at jp.co.nec.diosa.utilapp.JUtilityAppServer.main(JUtilityAppServer.java
:216)
"VM Thread" prio=10 tid=0x0010daa8 nid=3 lwp_id=280642 runnable
"VM Periodic Task Thread" prio=8 tid=0x0010db28 nid=6 lwp_id=280650 waiting on c
ondition
"Suspend Checker Thread" prio=10 tid=0x0010dba8 nid=7 lwp_id=280651 runnable
Full thread dump Java HotSpot(TM) Server VM (1.4.1 1.4.1.02-030502-15:31-PA_RISC
2.0 PA2.0 mixed mode):
"Thread-2" prio=10 tid=0x0004d0c0 nid=23 lwp_id=280801 in Object.wait() [0x6eef4
000..0x6eef4530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73acdfc8> (a jp.co.nec.diosa.joba.appl.GuiProcessingDaem
on)
at java.lang.Object.wait(Object.java:426)
at jp.co.nec.diosa.joba.appl.GuiProcessingDaemon.process(GuiProcessingDa
emon.java:257)
- locked <0x73acdfc8> (a jp.co.nec.diosa.joba.appl.GuiProcessingDaemon)
at jp.co.nec.diosa.joba.appl.GuiProcessingDaemon.run(GuiProcessingDaemon
.java:196)
at java.lang.Thread.run(Thread.java:536)
"RMI LeaseChecker" daemon prio=8 tid=0x0004cca0 nid=19 lwp_id=280777 waiting on
condition [0x6f0f8000..0x6f0f8530]
at java.lang.Thread.sleep(Native Method)
at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
at java.lang.Thread.run(Thread.java:536)
"RMI TCP Connection(1)-192.168.174.45" daemon prio=10 tid=0x00235b30 nid=18 lwp_
id=280768 runnable [0x6f179000..0x6f179530]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x73ace8c8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:4
42)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport
.java:701)
at java.lang.Thread.run(Thread.java:536)
"RMI Reaper" prio=10 tid=0x0004cb40 nid=17 lwp_id=280758 in Object.wait() [0x6f1
fa000..0x6f1fa530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73acea78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x73acea78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
at java.lang.Thread.run(Thread.java:536)
"RMI TCP Accept-0" daemon prio=10 tid=0x0004c9e0 nid=16 lwp_id=280757 runnable [
0x6f27b000..0x6f27b530]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0x73aceaf8> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:439)
at java.net.ServerSocket.accept(ServerSocket.java:410)
at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
at java.lang.Thread.run(Thread.java:536)
"Thread-1" daemon prio=10 tid=0x0004c880 nid=15 lwp_id=280755 in Object.wait() [
0x6f2fc000..0x6f2fc530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73acebb0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:426)
at java.util.TimerThread.mainLoop(Timer.java:403)
- locked <0x73acebb0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:382)
"GC Daemon" daemon prio=10 tid=0x0004c5c0 nid=13 lwp_id=280741 in Object.wait()
[0x762fe000..0x762fe530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73a4d630> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x73a4d630> (a sun.misc.GC$LatencyLock)
"RMI RenewClean-[192.168.174.45:62731]" daemon prio=10 tid=0x0004c460 nid=12 lwp
_id=280740 runnable [0x7637f000..0x7637f530]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x76408468> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readByte(DataInputStream.java:276)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:313)
at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.jav
a:328)
at sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:
144)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCCli
ent.java:539)
at java.lang.Thread.run(Thread.java:536)
"Signal Dispatcher" daemon prio=10 tid=0x0004bd80 nid=8 lwp_id=280647 waiting on
condition [0x00000000..0xffffffff]
"Finalizer" daemon prio=10 tid=0x0004bc20 nid=5 lwp_id=280635 in Object.wait() [
0x77b7f000..0x77b7f530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73a4da48> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x73a4da48> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0004bac0 nid=4 lwp_id=280634 in Object.
wait() [0x79c08000..0x79c08530]
at java.lang.Object.wait(Native Method)
- waiting on <0x73a4dab0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:426)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
- locked <0x73a4dab0> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0004b960 nid=1 lwp_id=280616 runnable [0x79ff1000..0x79ff19
50]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x73ae1260> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readByte(DataInputStream.java:276)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:
189)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
at jp.co.nec.diosa.fw.ms.service.Core_Stub.filterFilesForCurrentHost(Unk
nown Source)
at jp.co.nec.diosa.joba.appl.JobGateway.filterFilesForIncomingHost(JobGa
teway.java:342)
at jp.co.nec.diosa.joba.event.coreevent.SGReadingEvent.run(SGReadingEven
t.java:322)
at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.run(Startu
pSGReadingEvent.java:86)
at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.process(St
artupSGReadingEvent.java:75)
at jp.co.nec.diosa.joba.appl.JobMain.readSGFile(JobMain.java:1401)
at jp.co.nec.diosa.joba.appl.JobMain.init(JobMain.java:752)
at jp.co.nec.diosa.joba.appl.JobMain.main(JobMain.java:1014)
"VM Thread" prio=10 tid=0x0010daa8 nid=3 lwp_id=280626 runnable
"VM Periodic Task Thread" prio=8 tid=0x0010db28 nid=6 lwp_id=280645 waiting on c
ondition
"Suspend Checker Thread" prio=10 tid=0x0010dba8 nid=7 lwp_id=280646 runnable
=========================================================================

Hi Owen,
Thank u for ur immediate reply on my doubt, but still my doubt not got clear.
If u look into the locked threads, the lock is happened in RMI layer. Concider following two threads:
-----------------------------------thread 1
"RMI TCP Connection(1)-192.168.174.45" daemon prio=10 tid=0x00b06570 nid=19 lwp_id=280803 runnable [0x6f0c3000..0x6f0c3530]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x73aea6d8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:442)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:536)
--------------------------------------- thread 2----------
"main" prio=10 tid=0x0004b960 nid=1 lwp_id=280616 runnable [0x79ff1000..0x79ff1950]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
- locked <0x73ae1260> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readByte(DataInputStream.java:276)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
at jp.co.nec.diosa.fw.ms.service.Core_Stub.filterFilesForCurrentHost(Unknown Source)
at jp.co.nec.diosa.joba.appl.JobGateway.filterFilesForIncomingHost(JobGateway.java:342)
at jp.co.nec.diosa.joba.event.coreevent.SGReadingEvent.run(SGReadingEvent.java:322)
at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.run(StartupSGReadingEvent.java:86)
at jp.co.nec.diosa.joba.event.coreevent.StartupSGReadingEvent.process(StartupSGReadingEvent.java:75)
at jp.co.nec.diosa.joba.appl.JobMain.readSGFile(JobMain.java:1401)
at jp.co.nec.diosa.joba.appl.JobMain.init(JobMain.java:752)
at jp.co.nec.diosa.joba.appl.JobMain.main(JobMain.java:1014)
Both the threads get locked in getting BufferedInputStread in RMI. If u can suggest some more hints that will very much helpful for me.
Thanks in advance,
Suresh Kumar

Similar Messages

  • Help on this thread dump..

    hey guys, im having a problem reading this thread dump. i'm just a newbie. could you help me point what's wrong in our system based on this thread dump? there were no errors in our logs, it just hangs and we're so confused as to what the issue is. here's the thread dump:
    Full thread dump Java HotSpot(TM) Client VM (1.4.0_02-b02 mixed mode):
    "MessageWriter_370" prio=5 tid=0x3479e0 nid=0x1df waiting on monitor [94381000..94381a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a690dd88> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a690dd88> (a java.util.ArrayList)
    "MessageReader_369" prio=5 tid=0x33f888 nid=0x1de runnable [94581000..94581a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a690de38> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a690df58> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a690ddb0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_368" prio=5 tid=0x2784c8 nid=0x1dc waiting on monitor [94881000..94881a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a68e6d90> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a68e6d90> (a java.util.ArrayList)
    "MessageReader_367" prio=5 tid=0x279230 nid=0x1db waiting for monitor entry [94981000..94981a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_366" prio=5 tid=0x5c2090 nid=0x1da waiting on monitor [90e81000..90e81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a68df070> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a68df070> (a java.util.ArrayList)
    "MessageReader_365" prio=5 tid=0x27da50 nid=0x1d9 runnable [90f81000..90f81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a68cb480> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a68cb520> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a68cb408> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_364" prio=5 tid=0x5c1f00 nid=0x1d8 waiting on monitor [91281000..91281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a68ca1a0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a68ca1a0> (a java.util.ArrayList)
    "MessageReader_363" prio=5 tid=0x753a68 nid=0x1d7 waiting for monitor entry [91381000..91381a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b3d378> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a68ca178> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_362" prio=5 tid=0x5c09a8 nid=0x1d6 waiting on monitor [91481000..91481a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a6867f58> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a6867f58> (a java.util.ArrayList)
    "MessageReader_361" prio=5 tid=0x2790d8 nid=0x1d5 runnable [91581000..91581a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a6868008> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a6868128> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a6867f80> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_352" prio=5 tid=0x34f730 nid=0x1c9 waiting on monitor [94c81000..94c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a6744968> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a6744968> (a java.util.ArrayList)
    "MessageReader_351" prio=5 tid=0xd7370 nid=0x1c8 runnable [94d81000..94d81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a6744a18> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a6744b38> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a6744990> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_350" prio=5 tid=0x34fb38 nid=0x1c7 waiting on monitor [95081000..95081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a66ef950> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a66ef950> (a java.util.ArrayList)
    "MessageReader_349" prio=5 tid=0x278620 nid=0x1c6 runnable [95181000..95181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a66efa00> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a66efb20> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a66ef978> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_344" prio=5 tid=0x6a0370 nid=0x1c0 waiting on monitor [92481000..92481a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a66665a8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a66665a8> (a java.util.ArrayList)
    "MessageReader_343" prio=5 tid=0x33f730 nid=0x1bf waiting for monitor entry [92581000..92581a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_332" prio=5 tid=0x3a0d38 nid=0x1b0 waiting on monitor [93681000..93681a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a620e700> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a620e700> (a java.util.ArrayList)
    "MessageReader_331" prio=5 tid=0x69dd18 nid=0x1af waiting for monitor entry [93781000..93781a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_318" prio=5 tid=0x39fc10 nid=0x19e waiting on monitor [95c81000..95c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5f1f0f0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5f1f0f0> (a java.util.ArrayList)
    "MessageReader_317" prio=5 tid=0x69da68 nid=0x19d waiting on monitor [95d81000..95d81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at java.lang.Object.wait(Object.java:426)
         at epylon.persistence.JmsPersistenceManager.obtainLock(JmsPersistenceManager.java:144)
         - locked <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at com.presumo.jms.persistence.PersistentQueue.push(PersistentQueue.java:142)
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - locked <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_316" prio=5 tid=0x254c30 nid=0x19c waiting on monitor [96281000..96281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5e9d7e0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5e9d7e0> (a java.util.ArrayList)
    "MessageReader_315" prio=5 tid=0x69d910 nid=0x19b waiting for monitor entry [96381000..96381a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_314" prio=5 tid=0x3a1bf0 nid=0x19a waiting on monitor [99081000..99081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5e24348> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5e24348> (a java.util.ArrayList)
    "MessageReader_313" prio=5 tid=0x69d7b8 nid=0x199 waiting for monitor entry [99181000..99181a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_312" prio=5 tid=0x64c608 nid=0x198 waiting on monitor [99681000..99681a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5df1a88> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5df1a88> (a java.util.ArrayList)
    "MessageReader_311" prio=5 tid=0x5d00a0 nid=0x197 waiting for monitor entry [99781000..99781a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b3d538> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a5df1a60> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_310" prio=5 tid=0x3a0be0 nid=0x195 waiting on monitor [91a81000..91a81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5dd5370> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5dd5370> (a java.util.ArrayList)
    "MessageReader_309" prio=5 tid=0x3a7108 nid=0x194 waiting for monitor entry [91b81000..91b81a28]
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:150)
         - waiting to lock <a6a87030> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a5dd5348> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_308" prio=5 tid=0x3a1290 nid=0x193 waiting on monitor [91e81000..91e81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5d9c338> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5d9c338> (a java.util.ArrayList)
    "MessageReader_307" prio=5 tid=0x42ce10 nid=0x192 runnable [91f81000..91f81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5d9c3e8> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5d9c508> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a5d9c360> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_302" prio=5 tid=0x4beef8 nid=0x18c waiting on monitor [93a81000..93a81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5cce5c8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5cce5c8> (a java.util.ArrayList)
    "MessageReader_301" prio=5 tid=0x577eb0 nid=0x18b runnable [93b81000..93b81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5cce678> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5cce798> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a5cce5f0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_298" prio=5 tid=0x413b08 nid=0x187 waiting on monitor [93c81000..93c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5c410d0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5c410d0> (a java.util.ArrayList)
    "MessageReader_297" prio=5 tid=0x353b58 nid=0x186 waiting for monitor entry [93d81000..93d81a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_296" prio=5 tid=0x592028 nid=0x185 waiting on monitor [94081000..94081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5bd4cd8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5bd4cd8> (a java.util.ArrayList)
    "MessageReader_295" prio=5 tid=0x66d660 nid=0x184 waiting for monitor entry [94181000..94181a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_294" prio=5 tid=0x1fa4e0 nid=0x183 waiting on monitor [94281000..94281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5b7e880> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5b7e880> (a java.util.ArrayList)
    "MessageReader_293" prio=5 tid=0x34f9e0 nid=0x182 waiting for monitor entry [94481000..94481a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_292" prio=5 tid=0x34de60 nid=0x17f waiting on monitor [91081000..91081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5b502a8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5b502a8> (a java.util.ArrayList)
    "MessageReader_291" prio=5 tid=0x592430 nid=0x17e runnable [91181000..91181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5b3cdb0> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5b3ce50> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a5b3cd38> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_282" prio=5 tid=0x57d608 nid=0x173 waiting on monitor [92c81000..92c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a59cd8b8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a59cd8b8> (a java.util.ArrayList)
    "MessageReader_281" prio=5 tid=0x1f5420 nid=0x172 waiting for monitor entry [92d81000..92d81a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_262" prio=5 tid=0x3a2ad8 nid=0x15b waiting on monitor [91c81000..91c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a576a760> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a576a760> (a java.util.ArrayList)
    "MessageReader_261" prio=5 tid=0x3a20a0 nid=0x15a waiting for monitor entry [91d81000..91d81a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b39f60> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a576a738> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_258" prio=5 tid=0x4129b0 nid=0x157 waiting on monitor [92a81000..92a81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5716ab8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5716ab8> (a java.util.ArrayList)
    "MessageReader_257" prio=5 tid=0x581bc0 nid=0x156 waiting for monitor entry [92b81000..92b81a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_256" prio=5 tid=0x5873e8 nid=0x155 waiting on monitor [92081000..92081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a56f6278> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a56f6278> (a java.util.ArrayList)
    "MessageReader_255" prio=5 tid=0x587290 nid=0x154 runnable [92181000..92181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a56e2f98> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a56e2ff0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a56e2f38> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_252" prio=5 tid=0x5806f0 nid=0x150 waiting on monitor [92881000..92881a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a567f138> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a567f138> (a java.util.ArrayList)
    "MessageReader_251" prio=5 tid=0x57fd98 nid=0x14f waiting for monitor entry [92981000..92981a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_214" prio=5 tid=0x30f950 nid=0x124 waiting on monitor [95281000..95281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a529f260> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a529f260> (a java.util.ArrayList)
    "MessageReader_213" prio=5 tid=0x55faa8 nid=0x123 waiting for monitor entry [95381000..95381a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b683a8> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a529f238> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_210" prio=5 tid=0x4dd0a8 nid=0x11f waiting on monitor [93081000..93081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a52692c0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a52692c0> (a java.util.ArrayList)
    "MessageReader_209" prio=5 tid=0x4c2fb8 nid=0x11e runnable [93181000..93181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5256468> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5256508> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a52563f0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_208" prio=5 tid=0x4c6de0 nid=0x11d waiting on monitor [93481000..93481a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5241588> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5241588> (a java.util.ArrayList)
    "MessageReader_207" prio=5 tid=0x5c2220 nid=0x11c runnable [93581000..93581a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5241638> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5241758> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a52415b0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_188" prio=5 tid=0x642b88 nid=0x105 waiting on monitor [92281000..92281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a50e4248> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a50e4248> (a java.util.ArrayList)
    "MessageReader_187" prio=5 tid=0x1f92e8 nid=0x104 runnable [92381000..92381a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a50e42f8> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a50e4418> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at ja

    Here's the lengthy code :)
    package com.presumo.jms.router;
    import com.presumo.jms.message.JmsMessage;
    import com.presumo.jms.message.SystemMessageConstants;
    import com.presumo.jms.plugin.implementation.MemoryMessageQueue;
    import com.presumo.jms.plugin.transport.Transport;
    import com.presumo.jms.selector.JmsOperand;
    import com.presumo.jms.selector.Parser;
    import com.presumo.jms.JmsServer;
    import com.presumo.jms.persistence.PersistentQueue;
    import com.presumo.jms.resources.Resources;
    import com.presumo.util.log.Logger;
    import com.presumo.util.log.LoggerFactory;
    import java.io.IOException;
    import java.util.ArrayList;
    import java.util.LinkedList;
    import javax.jms.InvalidSelectorException;
    import javax.jms.JMSException;
    import epylon.bean.MessageStatusBean;
    import epylon.persistence.JmsPersistenceManager;
    import epylon.util.*;
    import org.apache.log4j.Category;
    * Class that sits between the Router and the Transport layer in
    * terms of message flow. Essentially encapsulates the remote
    * connection from the router, as well as handling the persistent
    * message acknowledgment scheme.
    * @author Dan Greff
    public class RemoteSession implements RoutingTarget
    /** Largest batch of messages that will be bundled together **/
    protected static final int MSG_BATCH_SIZE = 2000;
    /** Time that between sending batches of messages **/
    protected static final long latency = 500;
    /** Property name for sending filters via system messages **/
    protected static final String FILTER_PROPERTY = "JMSX_FILTER";
    /** Property name for sending acks via system messages **/
    protected static final String ACKS_PROPERTY = "JMSX_ACKS";
    /** Filter sent to this JVM from the other side of the connection
    * which indicates what messages the router on the other side is
    * interested in.
    protected JmsOperand remoteFilter;
    /** Local filter which the represents what messages the local
    * router is interested in. This filter is sent to other routers.
    protected JmsOperand localFilter;
    /** Router this RemoteSession is connected to **/
    protected Router router = null;
    /** Interface which messages are sent to and received from the network **/
    protected Transport transport;
    /** Listener to be notified of if the connection is lost **/
    protected ConnectionListener connxListener;
    /** Handles acknowledgement logic for persistent messages **/
    protected PersistentAckHandler persistentAckHandler;
    /** Instance of inner class that reads message from the Transport **/
    protected MessageReader msgReader;
    /** Instance of inner class that writes message from the Transport **/
    protected MessageWriter msgWriter;
    /** Set unique ID assigned to the remote session by the router **/
    protected int targetID;
    /** Local copy of the Parser singleton **/
    protected final Parser parser = Parser.getInstance();
    /** Messages queued up to be send **/
    protected final ArrayList outbox = new ArrayList();
         protected static int connectionLostThreadCount = 0;
         //Remote site name.
         private String remoteSite = null;
         private String siteTransportID = null;
         private JmsPersistenceManager persistentManager;
         //routeIncompletedMsgs
         protected boolean incompletedMsgsRouted = false;
         //No need to persist remote filter for EpylonRemote session
         protected boolean connectToJmsSession = false;
         static public String FALSE = "FALSE";
         static public String TRUE = "TRUE";
         private static int THREAD_COUNT = 1;
         private static Category logger = Category.getInstance( RemoteSession.class.getName() );
    // Constructors //
         public RemoteSession(Router router,
    Transport transport,
                                  String remoteSite,
                                  ConnectionListener listener) throws Exception{     
              this.router = router;
              this.transport = transport;
              this.remoteSite = remoteSite;
              this.connxListener = listener;
              siteTransportID = transport.getConnectionClientID();
              try {
                   remoteFilter = parser.parseFilter("true");
              } catch (InvalidSelectorException ise) {}
              persistentAckHandler = new PersistentAckHandler(this);
              router.addTarget(this);
              persistentManager = JmsPersistenceManager.getInstance();
              start();
    // Public methods //
         public void setTransport(Transport t){
    // TODO:: Add error checking to make sure this is only being called
    // as a result of a reconnection. DTG
              this.transport = t;
         public synchronized void start(){
    if (msgReader == null) {
    msgReader = new MessageReader(THREAD_COUNT);
    msgReader.start();
         THREAD_COUNT ++;
    if (msgWriter == null) {
    msgWriter = new MessageWriter(THREAD_COUNT);
    msgWriter.start();
         THREAD_COUNT ++;
    public synchronized void stop()
    if (msgReader != null) {
    msgReader.stopReader();
    msgReader = null;
    if (msgWriter != null) {
    msgWriter.stopWriter();
    msgWriter = null;
    public void close()
    this.stop();
    if (transport != null) {
    transport.close();
    transport = null;
    router.removeTarget(this);
    // Begin implementation of RoutingTarget interface methods
    public void setTargetID(int id)
    this.targetID = id;
         public JmsOperand getRoutingFilter(){
              JmsOperand retval = remoteFilter;
              return retval;
         public void setRemoteRoutingFilter(JmsOperand filter, boolean add){
              if (filter != localFilter){
                   JmsMessage msg;
                   String filterString;               
                   try {
                        if (persistentManager == null)
                             persistentManager = JmsPersistenceManager.getInstance();
                        if(connectToJmsSession || !JmsServer.isJmsSessionInitComplete())
                             filterString = parser.unparse(filter);               
                        else                                        
                             filterString = persistentManager.prepareRemoteFilterForSite(remoteSite);
                        logger.debug("setRemoteRoutingFilter::remoteSite =>"+ remoteSite + "\n "+ filterString);
                        if (!defaultFilter(filterString.toUpperCase())){
                             msg = new JmsMessage(router.getName());
                             msg.setJMSSystemMsgType(SystemMessageConstants.REMOTE_FILTER_CHANGE_TYPE);                                             
                             msg.setStringProperty(FILTER_PROPERTY, filterString);
                             msg.setStringProperty(MessageStatusBean.ORIGINATING_SITE_NAME, JmsServer.siteName());
                             localFilter = filter;
                             if (connectToJmsSession && !JmsServer.isJmsSessionInitComplete())
                                  JmsServer.addLocalFilter();
                             if ( !connectToJmsSession || !JmsServer.isAllLocalFilterProcessed())
                                  queueOutboxMessage(msg);                         
                   } catch (Exception e) {
                        // shouldn't happen with a in memory message queue but, dump it anyways
                        logger.error(e);
    public boolean needsFilterUpdates()
    return true;
    * Called by the router to give a potential message to this
    * routing target.
    * Implementes RoutingTarget.takeMessage();
    public JmsMessage takeMessage(JmsMessage msg)
    throws Exception{
         logger.debug("RemoteSession:takeMessage connectToJmsSession=>"+ connectToJmsSession);
         boolean taken = false;
         if (msg.getSendingTargetID() != this.targetID && incompletedMsgsRouted) { // prevent backtracking
              if (connectToJmsSession || msg.getJMSDeliveryMode() != javax.jms.DeliveryMode.PERSISTENT ){ 
                   taken = parser.evaluate(remoteFilter, msg);
              } else
                   taken = msg.getAckHelper().routeMsgtoRemoteSite( remoteSite);
              if (taken) {
                   if (msg.getJMSDeliveryMode() == javax.jms.DeliveryMode.PERSISTENT)
                        persistentAckHandler.handleOutgoingMsg(msg);               
                   synchronized(outbox) {
                        outbox.add(msg);
                        outbox.notifyAll();
         return msg;
    // End implementaiton of RoutingTarget interface methods
    * Called by MessageReader.run() when messages come from the transport.
    public void receiveMessages(JmsMessage [] msgs)
    throws JMSException{
    int length = msgs.length;
    int msgsRemoved = 0;
    for (int i=0; i < length; ++i) {
              logger.debug("ReceiveMessages from "+ remoteSite + msgs);
              msgs[i].setStringProperty(MessageStatusBean.ORIGINATING_SITE_NAME, remoteSite);
              boolean systemMsg = false;
              int type = msgs[i].getJMSSystemMsgType();
              switch (type) {
              case(SystemMessageConstants.REMOTE_FILTER_CHANGE_TYPE):
                   handleRemoteFilterMsg(msgs[i]);
                   systemMsg = true;
                   break;
    case(SystemMessageConstants.ACKS_MSG_TYPE):
                   String acks = (String) msgs[i].getObjectProperty(ACKS_PROPERTY);
                   try{
                        persistentAckHandler.handleAcks(acks);
                   }catch(Exception e){
                        //TODO: Handle exception
                        logger.error(e);
                   systemMsg = true;
                   break;
              if (systemMsg) {
                   ++msgsRemoved;
                   msgs[i] = null;
              }else {               
                   // Filter out duplicates
                   boolean duplicate = persistentAckHandler.isDuplicate(msgs[i]);
                   if (duplicate) {
                        msgs[i] = null;
                        ++msgsRemoved;
                   else {
                        // Valid user message... start acknowledgment tracking
                        msgs[i].setSendingTargetID( targetID );          
                        persistentAckHandler.setOriginator(msgs[i]);
    if (msgsRemoved > 0) {
              msgs = collapseMsgArray(msgs, length - msgsRemoved);
    try {
              router.routeMessages(msgs);
              persistentAckHandler.handleIncomingMsgs(msgs);
    } catch (Exception e) {
    // TODO:: add some error handling
    logger.error("An exception occurred while routing msgs: \n", e);
         public void acksAvailable(){
              synchronized(outbox) {
                   outbox.notifyAll();
    // Package methods //
    // Private methods //
    protected void queueOutboxMessage(JmsMessage msg)
    synchronized(outbox) {
    outbox.add(msg);
    outbox.notifyAll();
         * Handle remote filter message.
         protected void handleRemoteFilterMsg(JmsMessage msg)
              throws JMSException{       
              boolean remoteFilterPersist = false;
              String filter = (String) msg.getObjectProperty(FILTER_PROPERTY);
              if (!connectToJmsSession && !JmsServer.isJmsSessionInitComplete()){
                   try{
                        logger.debug("Sleep");
                        Thread.currentThread().sleep(3000);
                   }catch(Exception e){
                        logger.error(e);
              if (!defaultFilter(filter.toUpperCase())){
              if (remoteSite.indexOf(JmsServer.siteName()) != -1 &&
                   !JmsServer.isAllLocalFilterProcessed() &&
                   !connectToJmsSession )
                   JmsServer.processLocalFilter();
              if (!connectToJmsSession){
                   persistRemoteFilter(remoteSite, filter);
                   remoteFilterPersist = true;
              try {
                   remoteFilter = parser.parseFilter(filter);
                   if (remoteFilterPersist){
                        if (!incompletedMsgsRouted &&
                             !connectToJmsSession ){          
                             routeMessagesToRemoteSite();
                             incompletedMsgsRouted = true;
                        logger.debug("RemoteSession::handleRemoteFilterMsg "+ remoteFilter);
                        if (!connectToJmsSession)
                        router.recalculateFilters(true);
              } catch (javax.jms.InvalidSelectorException ex) {
                   logger.error("Remote client sent invalid routing filter: " +
                                  transport.getRemoteID() + " :: " + filter);
    * Utility function to remove null values from the array
    protected JmsMessage [] collapseMsgArray(JmsMessage [] msgs, int size)
    JmsMessage [] retval = new JmsMessage[size];
    for (int i=0, j=0; i < msgs.length; ++i) {
    if (msgs[i] != null) {
    retval[j] = msgs[i];
    ++j;
    return retval;
    * Called by the internal methods reading and sending messages
    * to the transport layer when an IOException is thrown from
    * the transport layer. This indicates a connection loss or
    * corruption.
    * TODO:: explain why this needs to spawn off as a thread
         protected void connectionLost(IOException ioe){
              //Spawn off a thread to connect remote server when connection lost.
              //This RemoteSession is accepted connection from remote client when its
              //RemoteURL is null, so there is no need to spawn off connectionLostthread
              //And close transport.
              if(getRemoteUrl() != null){
                   connectionLostThreadCount++;               
                   Thread t = new Thread("ConnectionLost Thread"+ connectionLostThreadCount) {
                             public void run() {
                                  RemoteSession.this.close();
                                  if (transport != null) {
                                       transport.close();
                                       transport = null;
                                  connxListener.connectionLost(RemoteSession.this, siteTransportID);                              
                   t.start();
              }     else{
                   if (transport != null) {
                        transport.close();
                        transport = null;
                   connxListener.connectionLost(RemoteSession.this, siteTransportID);
                   //No need to close transport because its outputstream closed
         protected JmsMessage createAckMessage(){
              JmsMessage ackMessage = null;
              String acks = persistentAckHandler.getAckString();
              if (acks != null) {
                   ackMessage = new JmsMessage(router.getName());
                   try {
                        ackMessage.setJMSSystemMsgType(SystemMessageConstants.ACKS_MSG_TYPE);
                        ackMessage.setStringProperty(ACKS_PROPERTY, acks);
                   } catch (JMSException jmsex) {}
              return ackMessage;
         // Begin Inner Class MessageReader //
         * Inner class reperesenting thread reading messages from the transport
         * implementation.
         protected class MessageReader extends Thread{
              private volatile boolean stopped = false;
              public MessageReader(int threadCount) {
                   super("MessageReader_"+ threadCount );
              public void stopReader(){
                   if (! stopped) stopped = true;
              public void run(){
                   JmsMessage [] msgs;
                   while (stopped == false) {
                        if (router.isRouterStop())
                             stopped = true;
                        try {
                             if (transport == null){                    
                             throw new IOException("Transport socket not available");                    
                             }else
                                  msgs = transport.receiveMessages();                         
                             if (msgs != null && stopped == false) {
                                  receiveMessages(msgs);
                                  msgs = null;
                        }catch(SocketCloseException sce){
                             stopped = true;
                             connectionLost(new IOException(sce.getMessage()));
                        }catch(JMSException jmse){
                             // TODO:: handle this properly
                             logger.error(jmse);
                        }catch(NullPointerException ne){
                             logger.error(ne);
                        }catch (IOException ioe) {
                             if (! stopped) {
                                  // TODO:: handle this properly
                                  logger.debug(ioe);
                                  stopped = true;
                                  connectionLost(ioe);
         // End Inner Class MessageReader //
         // Begin Inner Class MessageWriter //
         * Inner class writing messages to the Transport
         protected class MessageWriter extends Thread{
              private volatile boolean stopped = false;
              private volatile boolean sendMessages;
              public MessageWriter(int threadCount) { super("MessageWriter_"+ threadCount); }
              public final void stopWriter()
                   if (! stopped) {
                        synchronized (outbox) {
                             stopped = true;
                             outbox.notifyAll();
              public final void sendMessagesNow()
                   sendMessages = true;
              public final void run()
                   long nextSend = System.currentTimeMillis() + latency;
                   long sleepTime = 0;
                   while (!stopped) {                    
                        synchronized (outbox) {
                             sendMessages = false;                         
                             if (outbox.size() >= MSG_BATCH_SIZE) {
                                  sendMessages = true;
                             } else if(outbox.size() > 0 ||
                                       persistentAckHandler.acksAvailable() ) {
                                  long currentTime = System.currentTimeMillis();
                                  long diff = nextSend - currentTime;
                                  if (diff <= 0) {
                                       sendMessages = true;
                                  else {
                                       sleepTime = diff;
                             } else {
                                  sleepTime = 0;
                             if (sendMessages == false && !stopped) {
                                  try {
                                       outbox.wait(sleepTime);
                                  } catch (InterruptedException ie) {}
                        if (sendMessages == true) {
                             send();
                             nextSend = System.currentTimeMillis() + latency;
                             sleepTime = 0;
         private final void send(){
              logger.debug("RemoteSession:send");
              try {
                   JmsMessage [] msgs = null;
                   JmsMessage ackMessage = createAckMessage();
                   synchronized(outbox) {
                        if (ackMessage != null) {
                             outbox.add(ackMessage);
                        msgs = new JmsMessage[outbox.size()];
                        msgs = (JmsMessage[]) outbox.toArray(msgs);
                        outbox.clear();
                   if (msgs != null && msgs.length != 0 && transport != null) {
                        transport.sendMessages(msgs);
              }catch(SocketCloseException sce){
                   connectionLost(new IOException(sce.getMessage()));
              } catch (IOException ioe) {
                   logger.error(ioe);
                   connectionLost(ioe);
         private void persistRemoteFilter( String site, String filter)
              throws JMSException{
              try{
                   persistentManager.persistRemoteFilter(site, filter);
                   persistentManager.endTransaction(true);
              }catch(Exception e){
                   logger.error(e);
                   JMSException jmsex = new JMSException("Unable to parse filter");
                   jmsex.setLinkedException(e);
         * This method is called only when remotesession initially connected to
         * route messages and ack messages (stored acks only) to remote
         * site. Don't route messages to JMS core/provider when remoteSeesion
         * connectted to JmsSession and JMS core remote session.
         * @param
         private void routeMessagesToRemoteSite(){
              logger.debug("RemoteSession::routeMessagesToRemoteSite " + remoteSite);
              if (connectToJmsSession)
                   return;
              boolean taken = false;
              JmsMessage[] msgs = null;
              JmsMessage msg;
              try{
                   persistentManager.obtainLock();
                   while (msgs == null){
                        msgs = persistentManager.getPendingDeleteMsgs();     
                   int length = msgs.length;
                   for (int i=0; i<length ; i++){
                        try{
                             msg = (JmsMessage)msgs[i];     
                             if (msg.getAckHelper().needStoredAckToRemoteSite(remoteSite)){
                                  logger.debug("needStoredAckToRemoteSite" + remoteSite);
                                  persistentAckHandler.addMsgToCheckDupsMapAndAcks(msg);
                             }else if (msg.getAckHelper().needCompletedAckToRemoteSite(remoteSite)){
                                  persistentAckHandler.addMsgToSendToMapAndAcks(msg);
                             }else{
                                  taken = msg.getAckHelper().routeMsgtoRemoteSite( remoteSite);
                                  logger.debug("RemoteSession::routeMessagesToRemoteSite taken=> " + taken );
                                  if (taken) {     
                                       if (msg.getJMSDeliveryMode() == javax.jms.DeliveryMode.PERSISTENT)
                                            persistentAckHandler.handleOutgoingMsg(msg);     
                                       while (outbox.size() >= MSG_BATCH_SIZE){
                                            msgWriter.send();
                                            logger.info("Route queued messages to remote sites. Please wait....");
                                            try{
                                                 Thread.sleep(2000);
                                            }catch(Exception e){}
                                       queueOutboxMessage(msg);
                        }catch(Exception e){
                             logger.error(e);
              } finally{
                   persistentManager.releaseLock();               
              logger.debug("RemoteSession::routeMessagesToRemoteSite:Exit");
    // End Inner Class MessageReader //
         private String remoteUrl = null;
         public void setRemoteUrl(String url){
              remoteUrl = url;
         public String getRemoteUrl(){
              return remoteUrl;
         public String getRemoteSite(){
              return remoteSite;
         public void addSessionToRouter(){
              if (router != null)
                   router.addTarget(this);
         public static boolean defaultFilter(String filter){
              return filter.indexOf(TRUE) != -1 || filter.indexOf(FALSE) != -1;

  • Need help interpreting jvm thread dump (linux)

    hi,
    i'm using jre1.4.2 and running AS3 Linux kernel version 2.4.
    i grep'ed for my java process id, and did a kill -3 on it to get the thread dump:
    ps -ef |grep Eatroot 3936 3845 68 16:12 pts/2 00:00:05 java EatCpu
    root 3948 30293 0 16:12 pts/3 00:00:00 grep Eat
    kill -3 3936
    thread dump:
    java EatCpuFull thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):
    "Thread-0" prio=1 tid=0x081126b8 nid=0xf60 runnable [aa7e0000..aa7e087c]
    at EatCpu.run(EatCpu.java:31)
    "Signal Dispatcher" daemon prio=1 tid=0x080a6a58 nid=0xf60 waiting on condition [0..0]
    "Finalizer" daemon prio=1 tid=0x08092ee8 nid=0xf60 in Object.wait() [aad4d000..aad4d87c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xaaed0490> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
    - locked <0xaaed0490> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=1 tid=0x08091498 nid=0xf60 in Object.wait() [aadce000..aadce87c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xaaed0380> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:429)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
    - locked <0xaaed0380> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x0805bae0 nid=0xf60 runnable [bfffc000..bfffcc98]
    at EatCpu.main(EatCpu.java:22)
    "VM Thread" prio=1 tid=0x08090238 nid=0xf60 runnable
    "VM Periodic Task Thread" prio=1 tid=0x080a9248 nid=0xf60 waiting on condition
    "Suspend Checker Thread" prio=1 tid=0x080a6020 nid=0xf60 runnable
    i read that "nid" in the thread dump is suppose to correspond to PID. when i give the -m option for the "ps" command (for all threads), i see:
    ps -efm |grep Eatroot 3936 3845 43 16:12 pts/2 00:00:04 java EatCpu
    root 3937 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3938 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3939 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3940 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3941 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3942 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3943 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3944 3936 30 16:12 pts/2 00:00:03 java EatCpu
    root 3950 30293 0 16:12 pts/3 00:00:00 grep Eat
    >
    but nid for all threads in the thead dump is the PID of my main process, 3936 (0xf60). is there a way to correlate the PIDs produced by "ps -efm" to the threads in the thread dump?
    thanks!
    -annie

    i upgraded to 1.5 version of java, and could see distinct "nids" in the thread dump after that..

  • Weblogic 8.1 sp4 on Slowwaris 2.7 - Thread dump included

    Dear experts,
    We have a time response problem within our production environment, following is our configuration :
    Oracle 8.1.7.4
    Weblogic 8.1 sp4
    Solaris Sparc 2.7
    JSP is pre-compiled, though the time response behaviour is rather weird, sometime quick and most of the time very slow, that we can not stand with ie. 5 minutes for displaying the mainpage, worse is client IE browser should be shut and restart. Below is the thread dump of our server, we will appreciate if someone can interpret it and possibly lead us to pointers how to solve our problem.
    Full thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e1848 nid=0x34 in Object.wait() [ddf81000..ddf819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0e170> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0e170> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e0b38 nid=0x33 in Object.wait() [de081000..de0819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0d9d0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0d9d0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e08f0 nid=0x32 in Object.wait() [de181000..de1819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0d230> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0d230> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00359820 nid=0x31 in Object.wait() [de281000..de2819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0ca70> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0ca70> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060d850 nid=0x30 waiting for monitor entry [de381000..de3819f0]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
         - waiting to lock <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060d0c8 nid=0x2f waiting for monitor entry [dea01000..dea019f0]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
         - waiting to lock <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060cb28 nid=0x2e runnable [deb01000..deb019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4cac810> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95)
         - locked <0xe4cac810> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
         - locked <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ListenThread.Default" prio=5 tid=0x00609978 nid=0x2d runnable [dec01000..dec019f0]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xe4cb5d80> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Thread-6" daemon prio=5 tid=0x001d2ab8 nid=0x2c in Object.wait() [df701000..df7019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6a78a80> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xe6a78a80> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d3b08 nid=0x2b in Object.wait() [dfe01000..dfe019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6a32e48> (a weblogic.jms.backend.BETimerTree)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <0xe6a32e48> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "weblogic.health.CoreHealthMonitor" daemon prio=5 tid=0x004ccf08 nid=0x2a waiting on condition [e0101000..e01019f0]
         at java.lang.Thread.sleep(Native Method)
         at weblogic.t3.srvr.CoreHealthMonitorThread.run(CoreHealthMonitorThread.java:128)
    "Thread-5" prio=5 tid=0x00387728 nid=0x29 in Object.wait() [e0201000..e02019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe66a3f78> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xe66a3f78> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "VDE Transaction Processor Thread" prio=2 tid=0x004a1138 nid=0x27 in Object.wait() [e0301000..e03019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b97d8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at java.lang.Object.wait(Object.java:429)
         at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:365)
         - locked <0xe65b97d8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003cb6b0 nid=0x26 in Object.wait() [e0401000..e04019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9858> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9858> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003caac0 nid=0x25 in Object.wait() [e0501000..e05019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b98d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b98d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003ca010 nid=0x24 in Object.wait() [e0601000..e06019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9958> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9958> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.HTTP'" daemon prio=5 tid=0x003c9560 nid=0x23 in Object.wait() [e0701000..e07019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b99d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b99d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.HTTP'" daemon prio=5 tid=0x003c9318 nid=0x22 in Object.wait() [e0801000..e08019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9a58> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9a58> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "weblogic.security.SpinnerRandomSource" daemon prio=5 tid=0x003c6900 nid=0x21 in Object.wait() [e0901000..e09019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651be08> (a java.lang.Object)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.security.SpinnerRandomBitsSource.run(SpinnerRandomBitsSource.java:60)
         - locked <0xe651be08> (a java.lang.Object)
         at java.lang.Thread.run(Thread.java:534)
    "weblogic.time.TimeEventGenerator" daemon prio=9 tid=0x003c5958 nid=0x20 runnable [e0a01000..e0a019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651be78> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <0xe651be78> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x003c3988 nid=0x1f in Object.wait() [e0b01000..e0b019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651bef0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651bef0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fb810 nid=0x1e in Object.wait() [e0c01000..e0c019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651bf70> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651bf70> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fac20 nid=0x1d waiting on condition [e0d01000..e0d019f0]
         at weblogic.platform.SunVM.threadDump0(Native Method)
         - waiting to lock <0xe4c7cdd0> (a weblogic.platform.SunVM)
         at weblogic.platform.SunVM.threadDump(SunVM.java:106)
         at weblogic.t3.srvr.T3Srvr.getThreadDump(T3Srvr.java:1184)
         at weblogic.common.internal.AdminProxy.execute(AdminProxy.java:220)
         at weblogic.t3.srvr.ClientRequest$1.run(ClientContext.java:700)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
         at weblogic.t3.srvr.ClientRequest.execute(ClientContext.java:697)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fa030 nid=0x1c in Object.wait() [e0e01000..e0e019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c070> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c070> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001f9440 nid=0x1b in Object.wait() [e0f01000..e0f019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c0f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c0f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f8850 nid=0x1a in Object.wait() [e1001000..e10019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c170> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c170> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f7c60 nid=0x19 in Object.wait() [e1101000..e11019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c1f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c1f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f7070 nid=0x18 in Object.wait() [e1201000..e12019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c270> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c270> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f6480 nid=0x17 in Object.wait() [e1301000..e13019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c2f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c2f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f5890 nid=0x16 in Object.wait() [e1401000..e14019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c370> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c370> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d67b8 nid=0x15 in Object.wait() [e1501000..e15019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c3f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c3f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d5bc8 nid=0x14 in Object.wait() [e1601000..e16019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c470> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c470> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d4fd8 nid=0x13 in Object.wait() [e1701000..e17019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c4f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c4f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d43e8 nid=0x12 in Object.wait() [e1801000..e18019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c570> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c570> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b6d70 nid=0x11 in Object.wait() [e1901000..e19019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c5f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c5f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b61e8 nid=0x10 in Object.wait() [e1a01000..e1a019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c670> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c670> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b57a8 nid=0xf in Object.wait() [e1b01000..e1b019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c6f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c6f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b4e08 nid=0xe in Object.wait() [e1c01000..e1c019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c770> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c770> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x005e9030 nid=0xd in Object.wait() [e1d01000..e1d019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c7f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c7f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x005e8e90 nid=0xc in Object.wait() [e1e01000..e1e019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c870> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c870> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-1" daemon prio=5 tid=0x00201230 nid=0xb in Object.wait() [e4981000..e49819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe637e1d0> (a java.util.TaskQueue)
         at java.lang.Object.wait(Object.java:429)
         at java.util.TimerThread.mainLoop(Timer.java:403)
         - locked <0xe637e1d0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Signal Dispatcher" daemon prio=10 tid=0x000cdbc8 nid=0x8 runnable [0..0]
    "Finalizer" daemon prio=8 tid=0x000ca7b0 nid=0x6 in Object.wait() [fc381000..fc3819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6278d78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0xe6278d78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0x000c8e50 nid=0x5 in Object.wait() [fde81000..fde819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6278de0> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:429)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
         - locked <0xe6278de0> (a java.lang.ref.Reference$Lock)
    "main" prio=5 tid=0x00038510 nid=0x1 in Object.wait() [ffbed000..ffbed7dc]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe62eeff0> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1208)
         - locked <0xe62eeff0> (a weblogic.t3.srvr.T3Srvr)
         at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:390)
         at weblogic.Server.main(Server.java:32)
    "VM Thread" prio=5 tid=0x000c8008 nid=0x4 runnable
    "VM Periodic Task Thread" prio=10 tid=0x000d0a28 nid=0xa waiting on condition
    "Suspend Checker Thread" prio=10 tid=0x000cd260 nid=0x7 runnable
    Thanks in advance,
    saiful-france

    Hi All,
    It looks like I have the similar problem: I am using Weblogic 8.1 Sp3 platform. please help.
    Full thread dump Java HotSpot(TM) Server VM (1.4.2 1.4.2.03-040401-18:59-PA_RISC2.0 PA2.0 (aCC_AP) mixed mode):
    "Login" daemon prio=10 tid=01a0a0b0 nid=17817 lwp_id=1133989 in Object.wait() [0x2da54000..0x2da544f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92588248> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=01a09f48 nid=17816 lwp_id=1133988 in Object.wait() [0x2d90f000..0x2d90f4f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92580248> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=00c40450 nid=17815 lwp_id=1133969 in Object.wait() [0x2ed85000..0x2ed854f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92446368> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=00c402e8 nid=17814 lwp_id=1133967 in Object.wait() [0x2db17000..0x2db174f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <9242e368> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "AreYouAlive" daemon prio=10 tid=077ca9b0 nid=11502 lwp_id=1080790 in Object.wait() [0x2d950000..0x2d9504f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fcb3390> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:115)
         - locked <3fcb3390> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
    "DatagramThread" daemon prio=10 tid=077ca848 nid=11501 lwp_id=1080789 runnable [0x2d991000..0x2d9914f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3fcb5458> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3fcb5488> (a java.net.DatagramPacket)
         - locked <3fcb54b0> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=077ca6e0 nid=11500 lwp_id=1080788 in Object.wait() [0x2d9d2000..0x2d9d24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fcb54d0> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3fcb54d0> (a com.inprise.vbroker.orb.GarbageCollector)
    "AreYouAlive" daemon prio=10 tid=030802e8 nid=11340 lwp_id=1078923 in Object.wait() [0x2da95000..0x2da954f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fc90508> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:115)
         - locked <3fc90508> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
    "DatagramThread" daemon prio=10 tid=03080180 nid=11339 lwp_id=1078922 runnable [0x2dad6000..0x2dad64f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3fc925d0> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3fc92600> (a java.net.DatagramPacket)
         - locked <3fc92628> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=03080018 nid=11338 lwp_id=1078921 in Object.wait() [0x2db58000..0x2db584f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fc92648> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3fc92648> (a com.inprise.vbroker.orb.GarbageCollector)
    "AreYouAlive" daemon prio=10 tid=08005a10 nid=3830 lwp_id=947497 in Object.wait() [0x2db99000..0x2db994f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.login(DSUser.java:736)
         - locked <3d4b1aa0> (a java.lang.Object)
         at com.inprise.vbroker.ds.DSUser.reconnect(DSUser.java:544)
         - locked <3d4b19b8> (a com.inprise.vbroker.ds.DSUser)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:513)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.fire(DSUser.java:95)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:117)
    "DatagramThread" daemon prio=10 tid=080058a8 nid=3829 lwp_id=947496 runnable [0x2dc1b000..0x2dc1b4f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3d4b3b20> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3d4b3b50> (a java.net.DatagramPacket)
         - locked <3d4b3b78> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=08005740 nid=3828 lwp_id=947495 in Object.wait() [0x2dbda000..0x2dbda4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3d4b3b98> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3d4b3b98> (a com.inprise.vbroker.orb.GarbageCollector)
    "ExecuteThread: '14' for queue: 'JmsDispatcher'" daemon prio=10 tid=030424a8 nid=142 lwp_id=679935 in Object.wait() [0x2dc5c000..0x2dc5c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b088> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b088> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'JmsDispatcher'" daemon prio=10 tid=03042340 nid=141 lwp_id=679934 in Object.wait() [0x2dc9d000..0x2dc9d4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b108> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b108> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'JmsDispatcher'" daemon prio=10 tid=030421d8 nid=140 lwp_id=679933 in Object.wait() [0x2dcde000..0x2dcde4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b188> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b188> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'JmsDispatcher'" daemon prio=10 tid=03042070 nid=139 lwp_id=679932 in Object.wait() [0x2dd1f000..0x2dd1f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b208> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b208> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041f08 nid=138 lwp_id=679931 in Object.wait() [0x2dd60000..0x2dd604f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b288> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b288> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041da0 nid=137 lwp_id=679930 in Object.wait() [0x2dda1000..0x2dda14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b308> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b308> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041c38 nid=136 lwp_id=679929 in Object.wait() [0x2dde2000..0x2dde24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b388> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b388> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041ad0 nid=135 lwp_id=679928 in Object.wait() [0x2de23000..0x2de234f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b408> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b408> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041968 nid=134 lwp_id=679927 in Object.wait() [0x2de64000..0x2de644f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b488> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b488> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041800 nid=133 lwp_id=679926 in Object.wait() [0x2dea5000..0x2dea54f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b508> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b508> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041698 nid=132 lwp_id=679925 in Object.wait() [0x2dee6000..0x2dee64f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b588> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b588> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041530 nid=131 lwp_id=679924 in Object.wait() [0x2df27000..0x2df274f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b608> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b608> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JmsDispatcher'" daemon prio=10 tid=030413c8 nid=130 lwp_id=679923 in Object.wait() [0x2df68000..0x2df684f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b688> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b688> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041260 nid=129 lwp_id=679922 in Object.wait() [0x2dfa9000..0x2dfa94f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b708> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b708> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JmsDispatcher'" daemon prio=10 tid=030410f8 nid=128 lwp_id=679921 in Object.wait() [0x2dfea000..0x2dfea4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b7a8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b7a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ListenThread.Default" prio=10 tid=02953558 nid=127 lwp_id=679919 runnable [0x2e02b000..0x2e02b4f0]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <3b81b8c0> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:454)
         at java.net.ServerSocket.accept(ServerSocket.java:425)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Adapter Suspend/Resume Task Handler" daemon prio=10 tid=029533f0 nid=126 lwp_id=679918 in Object.wait() [0x2e06c000..0x2e06c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b9a0> (a java.lang.String)
         at java.lang.Object.wait(Object.java:429)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:60)
         - locked <3b81b9a0> (a java.lang.String)
    "Adapter instance Auto-resume TaskHandler" daemon prio=10 tid=02953288 nid=125 lwp_id=679917 in Object.wait() [0x2e0ad000..0x2e0ad4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81ba18> (a java.lang.String)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:55)
         - locked <3b81ba18> (a java.lang.String)
    "ApplicationView Auto-resume TaskHandler" daemon prio=10 tid=000b6360 nid=124 lwp_id=679915 in Object.wait() [0x2e0ee000..0x2e0ee4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b74b988> (a java.lang.String)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:55)
         - locked <3b74b988> (a java.lang.String)
    "ApplicationView Deployment TaskHandler" daemon prio=10 tid=000b61f8 nid=123 lwp_id=679914 in Object.wait() [0x2e12f000..0x2e12f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b74b648> (a java.lang.String)
         at java.lang.Object.wait(Object.java:429)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:60)
         - locked <3b74b648> (a java.lang.String)
    "AreYouAlive" daemon prio=10 tid=000b6090 nid=122 lwp_id=679910 in Object.wait() [0x2e170000..0x2e1704f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Thread.join(Thread.java:1001)
         - locked <8e395cc8> (a java.lang.Thread)
         at java.lang.Thread.join(Thread.java:1054)
         at com.inprise.vbroker.ds.DSUser.login(DSUser.java:752)
         at com.inprise.vbroker.ds.DSUser.reconnect(DSUser.java:544)
         - locked <3b6e4558> (a com.inprise.vbroker.ds.DSUser)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:513)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.fire(DSUser.java:95)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:117)
    "DatagramThread" daemon prio=10 tid=000b5f28 nid=121 lwp_id=679909 runnable [0x2e1b1000..0x2e1b14f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3b6e6a18> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3b6e9020> (a java.net.DatagramPacket)
         - locked <3b6e69f0> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=000b5dc0 nid=120 lwp_id=679908 in Object.wait() [0x2e1f2000..0x2e1f24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b6e3338> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3b6e3338> (a com.inprise.vbroker.orb.GarbageCollector)
    "Timer" daemon prio=10 tid=000b5af0 nid=119 lwp_id=679906 in Object.wait() [0x2e233000..0x2e2334f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b65ee40> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "Timer" daemon prio=10 tid=000b5c58 nid=118 lwp_id=679905 in Object.wait() [0x2e274000..0x2e2744f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b642528> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "Timer" daemon prio=10 tid=000b5820 nid=117 lwp_id=679904 in Object.wait() [0x2e2b5000..0x2e2b54f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b639540> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca578 nid=116 lwp_id=679896 in Object.wait() [0x2e2f6000..0x2e2f64f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad75800> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca410 nid=115 lwp_id=679895 in Object.wait() [0x2e337000..0x2e3374f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3ad75328> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad75328> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca2a8 nid=114 lwp_id=679894 in Object.wait() [0x2e378000..0x2e3784f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3ad74e30> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad74e30> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_1>.ioThreadPool'" daemon prio=10 tid=000b5988 nid=113 lwp_id=679892 in Object.wait() [0x2e3b9000..0x2e3b94f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3af7c258> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-20" prio=10 tid=000b56b8 nid=112 lwp_id=679851 in Object.wait() [0x2e3fa000..0x2e3fa4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3a9b0250> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3a9b0250> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-19" prio=10 tid=000b5550 nid=111 lwp_id=679850 in Object.wait() [0x2e43b000..0x2e43b4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3a9b02f0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3a9b02f0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-18" daemon prio=10 tid=01a09c78 nid=110 lwp_id=679822 in Object.wait() [0x2e47c000..0x2e47c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39f1b930> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <39f1b930> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '11' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040f90 nid=109 lwp_id=679809 in Object.wait() [0x2e4bd000..0x2e4bd4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e53ac0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040e28 nid=108 lwp_id=679808 in Object.wait() [0x2e4fe000..0x2e4fe4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e535a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040cc0 nid=107 lwp_id=679807 in Object.wait() [0x2e53f000..0x2e53f4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e53090> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040b58 nid=106 lwp_id=679806 in Object.wait() [0x2e580000..0x2e5804f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e52b28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030409f0 nid=105 lwp_id=679805 in Object.wait() [0x2e5c1000..0x2e5c14f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e52610> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040888 nid=104 lwp_id=679804 in Object.wait() [0x2e602000..0x2e6024f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e520f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040720 nid=103 lwp_id=679803 in Object.wait() [0x2e643000..0x2e6434f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e51be0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030405b8 nid=102 lwp_id=679802 in Object.wait() [0x2e684000..0x2e6844f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e516c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040450 nid=101 lwp_id=679801 in Object.wait() [0x2e6c5000..0x2e6c54f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e511b0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e511b0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030402e8 nid=100 lwp_id=679800 in Object.wait() [0x2e706000..0x2e7064f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50c98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50c98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040180 nid=99 lwp_id=679799 in Object.wait() [0x2e747000..0x2e7474f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50780> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50780> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040018 nid=98 lwp_id=679798 in Object.wait() [0x2e788000..0x2e7884f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50268> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50268> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "SeedGenerator Thread" daemon prio=10 tid=000b53e8 nid=97 lwp_id=679782 in Object.wait() [0x2e7c9000..0x2e7c94f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39db8938> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)
         at java.lang.Object.wait(Object.java:429)
         at sun.security.provider.SeedGenerator$ThreadedSeedGenerator.run(SeedGenerator.java:334)
         - locked <39db8938> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)
         at java.lang.Thread.run(Thread.java:534)
    "Noisy Thread" daemon prio=10 tid=000b5280 nid=96 lwp_id=679781 in Object.wait() [0x2e80a000..0x2e80a4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39db89a0> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)
         at java.lang.Object.wait(Object.java:429)
         at sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread.run(SeedGenerator.java:452)
         - locked <39db89a0> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=10 tid=000b5118 nid=95 lwp_id=679741 in Object.wait() [0x2e84b000..0x2e84b4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <395c5970> (a weblogic.jms.backend.BETimerTree)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <395c5970> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "Thread-17" prio=10 tid=000b4fb0 nid=94 lwp_id=679731 in Object.wait() [0x2e88c000..0x2e88c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e8a0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e8a0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-16" prio=10 tid=000b4e48 nid=93 lwp_id=679728 in Object.wait() [0x2e8cd000..0x2e8cd4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e940> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e940> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-15" prio=10 tid=000b4ce0 nid=92 lwp_id=679727 in Object.wait() [0x2e90e000..0x2e90e4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e9e0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e9e0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-14" prio=10 tid=000b4b78 nid=91 lwp_id=679726 in Object.wait() [0x2e94f000..0x2e94f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ea80> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ea80> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-13" prio=10 tid=000b4a10 nid=90 lwp_id=679724 in Object.wait() [0x2e990000..0x2e9904f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971eb20> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971eb20> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-12" prio=10 tid=000b48a8 nid=89 lwp_id=679720 in Object.wait() [0x2e9d1000..0x2e9d14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ebc0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ebc0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-11" prio=10 tid=000b4740 nid=88 lwp_id=679719 in Object.wait() [0x2ea12000..0x2ea124f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ec60> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ec60> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-10" prio=10 tid=000b45d8 nid=87 lwp_id=679718 in Object.wait() [0x2ea53000..0x2ea534f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ed00> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ed00> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-9" prio=10 tid=000b4470 nid=86 lwp_id=679717 in Object.wait() [0x2ea94000..0x2ea944f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971eda0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971eda0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-8" prio=10 tid=000b4308 nid=85 lwp_id=679715 i

  • A StuckThreadMaxTime error; who can help me to analyse the thread dumps

    The StuckThreadMaxTime error freeze my system, no one can logged in.I have double checked the thread dumps, but get no help.
    The following is the thread dumps:
    <2004-12-7 ÉÏÎç05ʱ56·Ö47Ãë GMT+08:00> <Alert> <Log Management> <BEA-170017> <The log file ./myserver/myserver.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
    <2004-12-7 ÉÏÎç05ʱ56·Ö47Ãë GMT+08:00> <Alert> <Log Management> <BEA-170018> <The log file has been rotated to myserver.log00035. Log messages will continue to be logged in ./myserver/myserver.log.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '10' for queue: 'weblogic.kernel.Default' has been busy for "646" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '13' for queue: 'weblogic.kernel.Default' has been busy for "651" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    <2004-12-7 ÉÏÎç10ʱ09·Ö32Ãë GMT+08:00> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '14' for queue: 'weblogic.kernel.Default' has been busy for "650" seconds working on the request "com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
    Full thread dump Java HotSpot(TM) VM (1.4.2 1.4.2.03-040401-18:59-PA_RISC2.0 PA2.0 (aCC_AP) mixed mode):
    "ExecuteThread: '14' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f778 nid=71 lwp_id=39624 in Object.wait() [0x61abf000..0x61abf4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d24f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d24f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f610 nid=70 lwp_id=39623 in Object.wait() [0x61b40000..0x61b404f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d2578> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d2578> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f4a8 nid=69 lwp_id=39622 in Object.wait() [0x61bc1000..0x61bc14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d25f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <6d1d25f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'JmsDispatcher'" daemon prio=5 tid=01c3f340 nid=68 lwp_id=39621 in Object.wait() [0x61c42000..0x61c424f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <6d1d2678> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.Eÿ

    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=000883f8 nid=27 lwp_id=39205 runnable [0x630ec000..0x630eb4f0]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at oracle.net.ns.Packet.receive(Unknown Source)
         at oracle.net.ns.DataPacket.receive(Unknown Source)
         at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
         at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
         at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:181)
         at oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:543)
         at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:913)
         at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:452)
         at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:984)
         at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2885)
         at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:2926)
         - locked <762d0bd8> (a oracle.jdbc.driver.T4CPreparedStatement)
         - locked <6c837e90> (a oracle.jdbc.driver.T4CConnection)
         at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:92)
         at com.huawei.oss.rms.persistence.ReadOperationPersistenceImpl.getAttributeValue(ReadOperationPersistenceImpl.java:2054)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.getAttributeValue(PersistenceOperationImpl.java:516)
         at com.huawei.oss.services.common.Utility.getNameByID(Utility.java:78)
         at com.huawei.oss.services.port.common.RMSPortUtility.setExtraToVO(RMSPortUtility.java:45)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.setExtraToVO(PhyPortConnHelper.java:2046)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.afterGet(PhyPortConnHelper.java:719)
         at com.huawei.oss.services.common.AbstractResourceHelper.getResource(AbstractResourceHelper.java:293)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.rmsGetByPort(PhyPortConnHelper.java:809)
         at com.huawei.oss.services.port.server.PhyPortConnHelper.rmsGetAllPhyConnOfBoard(PhyPortConnHelper.java:1144)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:324)
         at com.huawei.oss.services.common.AbstractResourceHelper.directMethod(AbstractResourceHelper.java:489)
         at com.huawei.oss.services.common.AbstractResourceHelper.invokeMethod(AbstractResourceHelper.java:404)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSessionEJB.invokeMethod(ResourceMgmtSessionEJB.java:263)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl.invokeMethod(ResourceMgmtSession_1x9h7n_EOImpl.java:424)
         at com.huawei.oss.rms.resourceaccess.ResourceMgmtSession_1x9h7n_EOImpl_WLSkel.invoke(Unknown Source)
         at weblogic.rmi.internal.activation.ActivatableServerRef.invoke(ActivatableServerRef.java:90)
         at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:420)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:144)
         at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:415)
         at weblogic.rmi.internal.BasicExecuteRequest.execute(BasicExecuteRequest.java:30)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=00088290 nid=26 lwp_id=39204 runnable [0x6316d000..0x6316c4f0]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at oracle.net.ns.Packet.receive(Unknown Source)
         at oracle.net.ns.DataPacket.receive(Unknown Source)
         at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.net.ns.NetInputStream.read(Unknown Source)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:971)
         at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:941)
         at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:432)
         at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:112)
         at oracle.jdbc.driver.T4CStatement.execute_for_rows(T4CStatement.java:474)
         at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:913)
         at oracle.jdbc.driver.T4CStatement.execute_maybe_describe(T4CStatement.java:383)
         at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:984)
         at oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1124)
         - locked <76c0d1f8> (a oracle.jdbc.driver.T4CStatement)
         - locked <6f1ab788> (a oracle.jdbc.driver.T4CConnection)
         at weblogic.jdbc.wrapper.Statement.executeQuery(Statement.java:345)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.executeSQL(PersistenceOperationImpl.java:2884)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.setVersionFlagNResourceName(PersistenceOperationImpl.java:3137)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.makeAssociationValue(PersistenceOperationImpl.java:2600)
         at com.huawei.oss.rms.persistence.ReadOperationPersistenceImpl.getResourceAssociationSet(ReadOperationPersistenceImpl.java:2575)
         at com.huawei.oss.rms.persistence.PersistenceOperationImpl.getResourceAssociationSet(PersistenceOperationImpl.java:603)
         at com.huawei.oss.services.port.common.RMSPortUtility.getSlotByEquip(RMSPortUtility.jÿ

  • Help needed with diagnosis of WebLogic server thread dump

     

    Sorry, my mistake - I was trying to search for "weblogic" before the dump file
    loaded fully.... I am not very familiar with VisiBroker and may be totally
    wrong, but my guess is it creates a thread for each connection/client. I would
    look for VisiBroker properties or settings that could limit the runaway creation
    of threads....
    Arunabh
    "Dimitri I. Rakitine" wrote:
    It is, preceded by lots and lots of CORBA (visibroker) threads. Is it
    possible that application code
    creates new ORBs more often than it should ?
    "Arunabh Hazarika" <[email protected]> wrote in message
    news:[email protected]...
    The attached file does not look like a WebLogic Server thread dump.....
    Arunabh
    Kapil Singh Pawar wrote:
    I am running a web application on WebLogic server 6.0 sp2 + rp1 (HP-UX).
    After
    I start loading the web application with more users I constantly get thejava.lang.OutOfMemoryError.
    I tried increasing the max_thread_proc kernel parameter to 2048 but thatdoes
    not seem to help much. Here is the error that I get on the WebLogicServer console.
    <Oct 7, 2002 5:00:27 PM GMT+02:00> <Error> <HTTP><[WebAppServletContext(1828590,i2)]
    Servlet failed with Exception
    java.lang.OutOfMemoryError: max threads limit being reached in VM
    at java.lang.Thread.start(Native Method)
    at netscape.ldap.LDAPConnThread.<init>(LDAPConnThread.java:112)
    atnetscape.ldap.LDAPConnection.getNewThread(LDAPConnection.java:1117)
    atnetscape.ldap.LDAPConnection.connect(LDAPConnection.java:1034)
    at netscape.ldap.LDAPConnection.connect(LDAPConnection.java:915)
    at netscape.ldap.LDAPConnection.connect(LDAPConnection.java:759)
    atcom.i2.scm.LDAP.ScmLDAPDefaultAuthenticate.authenticate(ScmLDAPDefaultAuthen
    ticate.java:129)
    at scmldap.service(scmldap.java:452)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :213)
    atweblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletCo
    ntext.java:1302)
    atweblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java
    :1676)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:137)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)
    >
    I am posting the thread dump obtained at the OutOfMemoryError stage. Anyhelp
    in diagnosing the thread dump and possible fix for this error is highlyappreciated.
    For information sake, while the web application encountersOutOfMemoryError the
    example web apps and WebLogic console application work fine. So it looksto be
    a case of memory outage only for my custom application.
    Thanks
    Kapil Singh Pawar
    Name: thread_dump.txt
    thread_dump.txt Type: Plain Text (text/plain)
    Encoding: base64--
    Dimitri

  • Help with Thread dump analisys

    We have problems with our application which runs on bea8.1 sp1.
    Problem is that bea runs out from Threads (StuckThreads).
    thread dums look same for all threads but i can's see what is wrong.
    Here are dumps from first and last threads :
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x7f3628
    nid=0xd runnable [a6280000..a6281994
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c3197aa0> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
    "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x6dd548
    nid=0x24 runnable [a3d80000..a3d819
    94]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c30c5010> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

    Rob Woollen <[email protected]> wrote:
    Can you post the full thread dump?
    It looks like you have some code making an RMI call over the network.
    Is it calling another server, or perhaps a loopback call into the same
    server. The latter (opening a socket to your own process) is a good
    way
    to get a deadlock.
    -- Rob
    dara wrote:
    We have problems with our application which runs on bea8.1 sp1.
    Problem is that bea runs out from Threads (StuckThreads).
    thread dums look same for all threads but i can's see what is wrong.
    Here are dumps from first and last threads :
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5tid=0x7f3628
    nid=0xd runnable [a6280000..a6281994
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c3197aa0> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
    "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=5tid=0x6dd548
    nid=0x24 runnable [a3d80000..a3d819
    94]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
    - locked <c30c5010> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readByte(DataInputStream.java:276)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
    at com.hutchison3g.core.product.homebase.services.HomebaseFactoryImpl_Stub.getIdentity(Unknown
    Source)
    at com.hutchison3g.core.product.homebase.client.HomebaseClient.getIdentity(HomebaseClient.java:207)
    at com.hutchison3g.core.product.homebase.client.util.HomebaseProxy.getIdentity(HomebaseProxy.java:78)
    at com.hutchison3g.core.product.homebase.client.util.UserUtil.getIdentity(UserUtil.java:132)
    at com.hutchison3g.core.product.homebase.servlets.TalonUniServlet.doGet(TalonUniServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at com.hutchison3g.core.product.homebase.client.servlet.core.HomebaseCommonServlet.service(HomebaseCommonServlet.
    java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
    at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1053)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:387)
    at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:305)
    at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6310)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
    at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
    at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3622)
    at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2569)
    at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)
    Thanks Rob,
    sorry for delay but I was on vacation.
    You are right about RMI, our app. consist from FE (front end) and BE (back end),
    comunication between them is RMI.
    We had some problems with BE but now we have new version and it looks ok.
    Anyway thanks for help, if I will have again some problems then it will be
    Help with thread analisys 2 ;-)

  • HELP!!! Why do my thread dump have so many locked monitor

    In recent weeks, My customer's weblogic server become more busy.Each time I downloaded
    the thread dump when CPU is busy than 90%,I found a lot of lock in the thread
    dump! Especially the class named "weblogic.kernel.ExecuteThread" . It's the kernel
    class of weblogic, Why were most of them locked?
    Full thread dump:
    "process forker" daemon prio=1 tid=0x95b4c8e0 nid=0x3084 waiting on monitor [0x977ff000..0x977ff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53d26420> (a java.lang.Object)
         at java.lang.UNIXProcessForker.run(Native Method)
         at java.lang.Thread.run(Thread.java:479)
    "process reaper" daemon prio=1 tid=0x95376758 nid=0x3083 runnable [0x96eff000..0x96eff890]
         at java.lang.UNIXProcessReaper.run(Native Method)
         at java.lang.Thread.run(Thread.java:479)
    "HighPriority TimeEventGenerator" daemon prio=1 tid=0x825bf18 nid=0x25a1 waiting
    on monitor [0x95dff000..0x95dff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53a3bd10> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
         - locked <53a3bd10> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:141)
         at java.lang.Thread.run(Thread.java:479)
    "ListenThread" prio=1 tid=0x94d00630 nid=0x25a0 runnable [0x95fff000..0x95fff890]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
         - locked <538fa238> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:238)
         at java.net.ServerSocket.accept(ServerSocket.java:217)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:325)
    "NBExecuteThread-1" daemon prio=1 tid=0x8c0dd1d0 nid=0x259c waiting on monitor
    [0x94521000..0x94521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a9650> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a9650> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "NBExecuteThread-0" daemon prio=1 tid=0x8c0dc7d0 nid=0x259b waiting on monitor
    [0x94321000..0x94321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a9060> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a9060> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    //"ExecuteThread-49" daemon prio=1 tid=0x8c034660 nid=0x259a runnable [0x94121000..0x94121890]
         at java.net.PlainDatagramSocketImpl.peek(Native Method)
         - locked <537acf60> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:349)
         - locked <486f9980> (a java.net.DatagramPacket)
         - locked <537acf40> (a java.net.MulticastSocket)
         at weblogic.cluster.TDMSocketIPM.receive(TDMSocketIPM.java:86)
         at weblogic.cluster.TMSocket.execute(TMSocket.java:249)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-48" daemon prio=1 tid=0x8c0c3d18 nid=0x2599 waiting for monitor
    entry [0x93f21000..0x93f21890]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - waiting to lock <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-47" daemon prio=1 tid=0x8c0c32d0 nid=0x2598 runnable [0x93d21000..0x93d21890]
         at weblogic.socket.PosixSocketMuxer.poll(Native Method)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - locked <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-46" daemon prio=1 tid=0x8c0bf410 nid=0x2597 waiting for monitor
    entry [0x93b21000..0x93b21890]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - waiting to lock <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-45" daemon prio=1 tid=0x8c0be9c8 nid=0x2596 waiting on monitor
    [0x93921000..0x93921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6df8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6df8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-44" daemon prio=1 tid=0x8c0bdab0 nid=0x2595 waiting on monitor
    [0x93721000..0x93721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6828> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6828> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-43" daemon prio=1 tid=0x8c0bd068 nid=0x2594 waiting on monitor
    [0x93521000..0x93521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6258> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6258> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-42" daemon prio=1 tid=0x8c0c0ce0 nid=0x2593 waiting on monitor
    [0x93321000..0x93321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a5c88> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a5c88> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-41" daemon prio=1 tid=0x8c0c0298 nid=0x2592 waiting on monitor
    [0x93121000..0x93121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a56b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a56b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-40" daemon prio=1 tid=0x8c0a21e0 nid=0x2591 waiting on monitor
    [0x92f21000..0x92f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a50e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a50e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-39" daemon prio=1 tid=0x8c0d4520 nid=0x2590 waiting on monitor
    [0x92d21000..0x92d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a4b18> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a4b18> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-38" daemon prio=1 tid=0x8c0d43d0 nid=0x258f waiting on monitor
    [0x92b21000..0x92b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a4548> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a4548> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-37" daemon prio=1 tid=0x8c0d3988 nid=0x258e waiting on monitor
    [0x92921000..0x92921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a3f78> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a3f78> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-36" daemon prio=1 tid=0x8c0ded20 nid=0x258d waiting on monitor
    [0x92721000..0x92721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a39a8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a39a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-35" daemon prio=1 tid=0x8c0de2d8 nid=0x258c waiting on monitor
    [0x92521000..0x92521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a33d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a33d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-34" daemon prio=1 tid=0x8c0cd528 nid=0x258b waiting on monitor
    [0x92321000..0x92321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2e08> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2e08> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-33" daemon prio=1 tid=0x8c0ccae0 nid=0x258a waiting on monitor
    [0x92121000..0x92121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2838> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2838> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-32" daemon prio=1 tid=0x8c075360 nid=0x2589 waiting on monitor
    [0x91f21000..0x91f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2268> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2268> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-31" daemon prio=1 tid=0x8c0837d0 nid=0x2588 waiting on monitor
    [0x91d21000..0x91d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a1c98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a1c98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-30" daemon prio=1 tid=0x8c0a5628 nid=0x2587 waiting on monitor
    [0x91b21000..0x91b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a16c8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a16c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-29" daemon prio=1 tid=0x8c07aaf8 nid=0x2586 waiting on monitor
    [0x91921000..0x91921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a10f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a10f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-28" daemon prio=1 tid=0x8c0c23d0 nid=0x2585 waiting on monitor
    [0x91721000..0x91721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a0b28> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a0b28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-27" daemon prio=1 tid=0x8c0c1988 nid=0x2584 waiting on monitor
    [0x91521000..0x91521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a0558> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a0558> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-26" daemon prio=1 tid=0x8c07ca00 nid=0x2583 waiting on monitor
    [0x91321000..0x91321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ff88> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ff88> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-25" daemon prio=1 tid=0x8c079e70 nid=0x2582 waiting on monitor
    [0x91121000..0x91121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379f9b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379f9b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-24" daemon prio=1 tid=0x8c078038 nid=0x2581 waiting on monitor
    [0x90f21000..0x90f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379f3e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379f3e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-23" daemon prio=1 tid=0x8c0a3298 nid=0x2580 waiting on monitor
    [0x90d21000..0x90d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ee18> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ee18> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-22" daemon prio=1 tid=0x8c0e30d8 nid=0x257f waiting on monitor
    [0x90b21000..0x90b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379e848> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379e848> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-21" daemon prio=1 tid=0x8c0c5948 nid=0x257e waiting on monitor
    [0x90921000..0x90921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379e278> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379e278> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-20" daemon prio=1 tid=0x8c0d0470 nid=0x257d waiting on monitor
    [0x90721000..0x90721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379dca8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379dca8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-19" daemon prio=1 tid=0x8c080798 nid=0x257c waiting on monitor
    [0x90521000..0x90521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379d6d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379d6d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-18" daemon prio=1 tid=0x8c080648 nid=0x257b waiting on monitor
    [0x90321000..0x90321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379d108> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379d108> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-17" daemon prio=1 tid=0x8c032380 nid=0x257a runnable [0x90121000..0x90121890]
         at java.lang.String.concat(String.java:1553)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:122)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:772)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-16" daemon prio=1 tid=0x8c0df9a8 nid=0x2579 waiting on monitor
    [0x8ff21000..0x8ff21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379c568> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379c568> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-15" daemon prio=1 tid=0x8c0a6288 nid=0x2578 waiting on monitor
    [0x8fd21000..0x8fd21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379bf98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379bf98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-14" daemon prio=1 tid=0x8c0dbd60 nid=0x2577 waiting on monitor
    [0x8fb21000..0x8fb21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379b9c8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379b9c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-13" daemon prio=1 tid=0x8c0c8498 nid=0x2576 waiting on monitor
    [0x8f921000..0x8f921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379b3f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379b3f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-12" daemon prio=1 tid=0x8c0a6a28 nid=0x2575 waiting on monitor
    [0x8f721000..0x8f721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ae28> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ae28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-11" daemon prio=1 tid=0x8c07d348 nid=0x2574 runnable [0x8f521000..0x8f521890]
         at java.lang.System.arraycopy(Native Method)
         at java.lang.String.getChars(String.java:557)
         at java.lang.String.concat(String.java:1554)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:251)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doGet(ClaimServlet.java:77)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-10" daemon prio=1 tid=0x8c0369b0 nid=0x2571 waiting on monitor
    [0x8f321000..0x8f321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379a288> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379a288> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-9" daemon prio=1 tid=0x8c079898 nid=0x256f waiting on monitor [0x8f121000..0x8f121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53799cb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53799cb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-8" daemon prio=1 tid=0x8c035260 nid=0x256e waiting on monitor [0x8ef21000..0x8ef21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537996e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537996e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-7" daemon prio=1 tid=0x8c034510 nid=0x256d waiting on monitor [0x8ed21000..0x8ed21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53799118> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53799118> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-6" daemon prio=1 tid=0x8c081130 nid=0x256c waiting on monitor [0x8eb21000..0x8eb21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53798b48> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53798b48> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-5" daemon prio=1 tid=0x8c0c7108 nid=0x256b waiting on monitor [0x8e921000..0x8e921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53798578> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53798578> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-4" daemon prio=1 tid=0x8c0a4388 nid=0x256a waiting on monitor [0x8e721000..0x8e721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53797fa8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53797fa8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-3" daemon prio=1 tid=0x8c0367b8 nid=0x2569 waiting on monitor [0x8e521000..0x8e521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537979d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537979d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-2" daemon prio=1 tid=0x8c036290 nid=0x2568 waiting on monitor [0x8e321000..0x8e321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53797408> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53797408> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-1" daemon prio=1 tid=0x8c0a4ee0 nid=0x2567 waiting on monitor [0x8e121000..0x8e121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53796e38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53796e38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-0" daemon prio=1 tid=0x8c0a4550 nid=0x2566 runnable [0x8df21000..0x8df21890]
         at java.lang.String.concat(String.java:1553)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.end.EndCaseServlet.service(EndCaseServlet.java:56)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "TimeEventGenerator" daemon prio=1 tid=0x8c076008 nid=0x2565 waiting on monitor
    [0x8dd21000..0x8dd21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53795330> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
         - locked <53795330> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:141)
         at java.lang.Thread.run(Thread.java:479)
    "SpinnerRandomSource" daemon prio=1 tid=0x8c0b34a0 nid=0x2563 waiting on monitor
    [0x8db21000..0x8db21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53793040> (a weblogic.security.SpinnerThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:104)
         - locked <53793040> (a weblogic.security.SpinnerThread)
         at weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:121)
    "Signal Dispatcher" daemon prio=1 tid=0x8c0077e8 nid=0x2561 waiting on monitor
    [0..0]
    "Finalizer" daemon prio=1 tid=0x8c000540 nid=0x255e waiting on monitor [0x8c2ff000..0x8c2ff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:103)
         - locked <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:157)
    "Reference Handler" daemon prio=1 tid=0x80851f0 nid=0x255d waiting on monitor
    [0x8bf8c000..0x8bf8c890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53720208> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:415)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:105)
         - locked <53720208> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x804ead0 nid=0x2543 waiting on monitor [0xbfffc000..0xbfffd00c]
         at java.lang.Object.wait(Native Method)
         - waiting on <53743c20> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1958)
         - locked <53743c20> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.reflect.Method.invoke(Native Method)
         at weblogic.Server.startServerDynamically(Server.java:148)
         at weblogic.Server.main(Server.java:97)
         at weblogic.Server.main(Server.java:58)
    "VM Thread" prio=1 tid=0x8082110 nid=0x255c runnable
    "VM Periodic Task Thread" prio=1 tid=0x8c0064f8 nid=0x255f waiting on monitor
    "Suspend Checker Thread" prio=1 tid=0x8c006e30 nid=0x2560 runnable

    These are execute threads waiting for requests to come in.
    BTW, you probably want to look into using StringBuffer instead of
    strings concatenation in com.ebao.carcase.bean.query.Query.generatorXML.
    "min.lu" <[email protected]> wrote in message
    news:[email protected]...
    >
    In recent weeks, My customer's weblogic server become more busy.Each timeI downloaded
    the thread dump when CPU is busy than 90%,I found a lot of lock in thethread
    dump! Especially the class named "weblogic.kernel.ExecuteThread" . It'sthe kernel
    class of weblogic, Why were most of them locked?
    Full thread dump:
    "process forker" daemon prio=1 tid=0x95b4c8e0 nid=0x3084 waiting onmonitor [0x977ff000..0x977ff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53d26420> (a java.lang.Object)
    at java.lang.UNIXProcessForker.run(Native Method)
    at java.lang.Thread.run(Thread.java:479)
    "process reaper" daemon prio=1 tid=0x95376758 nid=0x3083 runnable[0x96eff000..0x96eff890]
    at java.lang.UNIXProcessReaper.run(Native Method)
    at java.lang.Thread.run(Thread.java:479)
    "HighPriority TimeEventGenerator" daemon prio=1 tid=0x825bf18 nid=0x25a1waiting
    on monitor [0x95dff000..0x95dff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53a3bd10> (a weblogic.time.common.internal.TimeTable)
    at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
    - locked <53a3bd10> (a weblogic.time.common.internal.TimeTable)
    atweblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java
    :141)
    at java.lang.Thread.run(Thread.java:479)
    "ListenThread" prio=1 tid=0x94d00630 nid=0x25a0 runnable[0x95fff000..0x95fff890]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
    - locked <538fa238> (a java.net.PlainSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:238)
    at java.net.ServerSocket.accept(ServerSocket.java:217)
    atweblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
    at weblogic.t3.srvr.ListenThread.run(ListenThread.java:325)
    "NBExecuteThread-1" daemon prio=1 tid=0x8c0dd1d0 nid=0x259c waiting onmonitor
    [0x94521000..0x94521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a9650> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a9650> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "NBExecuteThread-0" daemon prio=1 tid=0x8c0dc7d0 nid=0x259b waiting onmonitor
    [0x94321000..0x94321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a9060> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a9060> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    //"ExecuteThread-49" daemon prio=1 tid=0x8c034660 nid=0x259a runnable[0x94121000..0x94121890]
    at java.net.PlainDatagramSocketImpl.peek(Native Method)
    - locked <537acf60> (a java.net.PlainDatagramSocketImpl)
    at java.net.DatagramSocket.receive(DatagramSocket.java:349)
    - locked <486f9980> (a java.net.DatagramPacket)
    - locked <537acf40> (a java.net.MulticastSocket)
    at weblogic.cluster.TDMSocketIPM.receive(TDMSocketIPM.java:86)
    at weblogic.cluster.TMSocket.execute(TMSocket.java:249)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-48" daemon prio=1 tid=0x8c0c3d18 nid=0x2599 waiting formonitor
    entry [0x93f21000..0x93f21890]
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - waiting to lock <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-47" daemon prio=1 tid=0x8c0c32d0 nid=0x2598 runnable[0x93d21000..0x93d21890]
    at weblogic.socket.PosixSocketMuxer.poll(Native Method)
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - locked <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-46" daemon prio=1 tid=0x8c0bf410 nid=0x2597 waiting formonitor
    entry [0x93b21000..0x93b21890]
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - waiting to lock <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-45" daemon prio=1 tid=0x8c0be9c8 nid=0x2596 waiting onmonitor
    [0x93921000..0x93921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6df8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6df8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-44" daemon prio=1 tid=0x8c0bdab0 nid=0x2595 waiting onmonitor
    [0x93721000..0x93721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6828> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6828> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-43" daemon prio=1 tid=0x8c0bd068 nid=0x2594 waiting onmonitor
    [0x93521000..0x93521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6258> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6258> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-42" daemon prio=1 tid=0x8c0c0ce0 nid=0x2593 waiting onmonitor
    [0x93321000..0x93321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a5c88> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a5c88> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-41" daemon prio=1 tid=0x8c0c0298 nid=0x2592 waiting onmonitor
    [0x93121000..0x93121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a56b8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a56b8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-40" daemon prio=1 tid=0x8c0a21e0 nid=0x2591 waiting onmonitor
    [0x92f21000..0x92f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a50e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a50e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-39" daemon prio=1 tid=0x8c0d4520 nid=0x2590 waiting onmonitor
    [0x92d21000..0x92d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a4b18> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a4b18> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-38" daemon prio=1 tid=0x8c0d43d0 nid=0x258f waiting onmonitor
    [0x92b21000..0x92b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a4548> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a4548> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-37" daemon prio=1 tid=0x8c0d3988 nid=0x258e waiting onmonitor
    [0x92921000..0x92921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a3f78> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a3f78> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-36" daemon prio=1 tid=0x8c0ded20 nid=0x258d waiting onmonitor
    [0x92721000..0x92721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a39a8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a39a8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-35" daemon prio=1 tid=0x8c0de2d8 nid=0x258c waiting onmonitor
    [0x92521000..0x92521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a33d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a33d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-34" daemon prio=1 tid=0x8c0cd528 nid=0x258b waiting onmonitor
    [0x92321000..0x92321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2e08> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2e08> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-33" daemon prio=1 tid=0x8c0ccae0 nid=0x258a waiting onmonitor
    [0x92121000..0x92121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2838> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2838> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-32" daemon prio=1 tid=0x8c075360 nid=0x2589 waiting onmonitor
    [0x91f21000..0x91f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2268> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2268> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-31" daemon prio=1 tid=0x8c0837d0 nid=0x2588 waiting onmonitor
    [0x91d21000..0x91d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a1c98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a1c98> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-30" daemon prio=1 tid=0x8c0a5628 nid=0x2587 waiting onmonitor
    [0x91b21000..0x91b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a16c8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a16c8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-29" daemon prio=1 tid=0x8c07aaf8 nid=0x2586 waiting onmonitor
    [0x91921000..0x91921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a10f8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a10f8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-28" daemon prio=1 tid=0x8c0c23d0 nid=0x2585 waiting onmonitor
    [0x91721000..0x91721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a0b28> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a0b28> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-27" daemon prio=1 tid=0x8c0c1988 nid=0x2584 waiting onmonitor
    [0x91521000..0x91521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a0558> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a0558> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-26" daemon prio=1 tid=0x8c07ca00 nid=0x2583 waiting onmonitor
    [0x91321000..0x91321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ff88> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ff88> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-25" daemon prio=1 tid=0x8c079e70 nid=0x2582 waiting onmonitor
    [0x91121000..0x91121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379f9b8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379f9b8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-24" daemon prio=1 tid=0x8c078038 nid=0x2581 waiting onmonitor
    [0x90f21000..0x90f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379f3e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379f3e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-23" daemon prio=1 tid=0x8c0a3298 nid=0x2580 waiting onmonitor
    [0x90d21000..0x90d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ee18> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ee18> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-22" daemon prio=1 tid=0x8c0e30d8 nid=0x257f waiting onmonitor
    [0x90b21000..0x90b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379e848> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379e848> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-21" daemon prio=1 tid=0x8c0c5948 nid=0x257e waiting onmonitor
    [0x90921000..0x90921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379e278> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379e278> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-20" daemon prio=1 tid=0x8c0d0470 nid=0x257d waiting onmonitor
    [0x90721000..0x90721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379dca8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379dca8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-19" daemon prio=1 tid=0x8c080798 nid=0x257c waiting onmonitor
    [0x90521000..0x90521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379d6d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379d6d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-18" daemon prio=1 tid=0x8c080648 nid=0x257b waiting onmonitor
    [0x90321000..0x90321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379d108> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379d108> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-17" daemon prio=1 tid=0x8c032380 nid=0x257a runnable[0x90121000..0x90121890]
    at java.lang.String.concat(String.java:1553)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:122)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:772)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-16" daemon prio=1 tid=0x8c0df9a8 nid=0x2579 waiting onmonitor
    [0x8ff21000..0x8ff21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379c568> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379c568> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-15" daemon prio=1 tid=0x8c0a6288 nid=0x2578 waiting onmonitor
    [0x8fd21000..0x8fd21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379bf98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379bf98> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-14" daemon prio=1 tid=0x8c0dbd60 nid=0x2577 waiting onmonitor
    [0x8fb21000..0x8fb21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379b9c8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379b9c8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-13" daemon prio=1 tid=0x8c0c8498 nid=0x2576 waiting onmonitor
    [0x8f921000..0x8f921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379b3f8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379b3f8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-12" daemon prio=1 tid=0x8c0a6a28 nid=0x2575 waiting onmonitor
    [0x8f721000..0x8f721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ae28> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ae28> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-11" daemon prio=1 tid=0x8c07d348 nid=0x2574 runnable[0x8f521000..0x8f521890]
    at java.lang.System.arraycopy(Native Method)
    at java.lang.String.getChars(String.java:557)
    at java.lang.String.concat(String.java:1554)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:251)
    at com.ebao.carcase.servlet.claim.ClaimServlet.doGet(ClaimServlet.java:77)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-10" daemon prio=1 tid=0x8c0369b0 nid=0x2571 waiting onmonitor
    [0x8f321000..0x8f321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379a288> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379a288> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-9" daemon prio=1 tid=0x8c079898 nid=0x256f waiting onmonitor [0x8f121000..0x8f121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53799cb8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53799cb8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-8" daemon prio=1 tid=0x8c035260 nid=0x256e waiting onmonitor [0x8ef21000..0x8ef21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537996e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537996e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-7" daemon prio=1 tid=0x8c034510 nid=0x256d waiting onmonitor [0x8ed21000..0x8ed21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53799118> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53799118> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-6" daemon prio=1 tid=0x8c081130 nid=0x256c waiting onmonitor [0x8eb21000..0x8eb21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53798b48> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53798b48> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-5" daemon prio=1 tid=0x8c0c7108 nid=0x256b waiting onmonitor [0x8e921000..0x8e921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53798578> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53798578> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-4" daemon prio=1 tid=0x8c0a4388 nid=0x256a waiting onmonitor [0x8e721000..0x8e721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53797fa8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53797fa8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-3" daemon prio=1 tid=0x8c0367b8 nid=0x2569 waiting onmonitor [0x8e521000..0x8e521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537979d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537979d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-2" daemon prio=1 tid=0x8c036290 nid=0x2568 waiting onmonitor [0x8e321000..0x8e321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53797408> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53797408> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-1" daemon prio=1 tid=0x8c0a4ee0 nid=0x2567 waiting onmonitor [0x8e121000..0x8e121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53796e38> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53796e38> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-0" daemon prio=1 tid=0x8c0a4550 nid=0x2566 runnable[0x8df21000..0x8df21890]
    at java.lang.String.concat(String.java:1553)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.end.EndCaseServlet.service(EndCaseServlet.java:56)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "TimeEventGenerator" daemon prio=1 tid=0x8c076008 nid=0x2565 waiting onmonitor
    [0x8dd21000..0x8dd21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53795330> (a weblogic.time.common.internal.TimeTable)
    at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
    - locked <53795330> (a weblogic.time.common.internal.TimeTable)
    atweblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java
    :141)
    at java.lang.Thread.run(Thread.java:479)
    "SpinnerRandomSource" daemon prio=1 tid=0x8c0b34a0 nid=0x2563 waiting onmonitor
    [0x8db21000..0x8db21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53793040> (a weblogic.security.SpinnerThread)
    at java.lang.Object.wait(Object.java:415)
    atweblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:10
    4)
    - locked <53793040> (a weblogic.security.SpinnerThread)
    at weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:121)
    "Signal Dispatcher" daemon prio=1 tid=0x8c0077e8 nid=0x2561 waiting onmonitor
    [0..0]
    "Finalizer" daemon prio=1 tid=0x8c000540 nid=0x255e waiting on monitor[0x8c2ff000..0x8c2ff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:103)
    - locked <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:157)
    "Reference Handler" daemon prio=1 tid=0x80851f0 nid=0x255d waiting onmonitor
    [0x8bf8c000..0x8bf8c890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53720208> (a java.l

  • Large Memory Usage and Frozen [thread dump attached]

    Hi,
    SQL Developer (Version 2.1.1.64, WinXP Pro, Java 1.6, Oracle 10g R2) always uses at least 550MB of memory, if left open, for more than a day, it invariably goes up to 700-800MB.
    If I close all windows and connections, virtually no memory is reclaimed.
    I clicked + beside a view to see the columns, sqldeveloper froze for a few minutes, below is the thread dump.
    2010-05-21 11:31:58
    Full thread dump Java HotSpot(TM) Client VM (14.2-b01 mixed mode):
    "CONNECTION_ALIVE" prio=6 tid=0x3891c000 nid=0x1d58 waiting for monitor entry [0x3a55f000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at oracle.jdbc.driver.T4CConnection.doPingDatabase(T4CConnection.java:3398)
    - waiting to lock <0x09ad05f8> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.PhysicalConnection.pingDatabase(PhysicalConnection.java:7074)
    at oracle.javatools.db.ora.BaseOracleDatabase.isConnectionAlive(BaseOracleDatabase.java:165)
    at oracle.javatools.db.AbstractDatabase$1.run(AbstractDatabase.java:316)
    at java.lang.Thread.run(Thread.java:619)
    "Loading Children" prio=6 tid=0x3891bc00 nid=0x12c4 waiting on condition [0x39d5f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x03eab570> (a java.util.concurrent.FutureTask$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$IdeTaskTracker.run(RaptorTaskManager.java:532)
    at java.lang.Thread.run(Thread.java:619)
    "pool-2-thread-57" prio=6 tid=0x3891b400 nid=0x2094 waiting on condition [0x35b5f000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x09b819f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:841)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1160)
    at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:416)
    at oracle.dbtools.raptor.utils.Connections.lock(Connections.java:1334)
    at oracle.dbtools.raptor.utils.Connections.lock(Connections.java:1319)
    at oracle.dbtools.raptor.utils.Connections.lock(Connections.java:1306)
    at oracle.dbtools.raptor.navigator.xml.AbstractItemInstance.lockConnection(AbstractItemInstance.java:52)
    at oracle.dbtools.raptor.navigator.xml.XmlNodeInstance.listChildren(XmlNodeInstance.java:55)
    at oracle.dbtools.raptor.navigator.ObjectNodeFilter$ObjectNodeFilterLoadTask.doWorkImpl(ObjectNodeFilter.java:78)
    at oracle.dbtools.raptor.navigator.DeferredLoadingFilter$FilterLoadTask.doWork(DeferredLoadingFilter.java:124)
    at oracle.dbtools.raptor.navigator.DeferredLoadingFilter$FilterLoadTask.doWork(DeferredLoadingFilter.java:113)
    at oracle.dbtools.raptor.backgroundTask.RaptorTask.call(RaptorTask.java:193)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$RaptorFutureTask.run(RaptorTaskManager.java:492)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
    "pool-2-thread-55" prio=6 tid=0x38d68400 nid=0xd74 runnable [0x3a15f000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.ns.Packet.receive(Packet.java:239)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:92)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:172)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1023)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:999)
    at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:584)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194)
    at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:785)
    at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:860)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1186)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3381)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3425)
    - locked <0x09ad05f8> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1490)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel.openResultSet(ResultSetTableModel.java:412)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel.fetchNext(ResultSetTableModel.java:169)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel$4.doWork(ResultSetTableModel.java:556)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel$4.doWork(ResultSetTableModel.java:540)
    at oracle.dbtools.raptor.backgroundTask.RaptorTask.call(RaptorTask.java:193)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$RaptorFutureTask.run(RaptorTaskManager.java:492)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
    "AWT-EventQueue-0" prio=6 tid=0x354fac00 nid=0x18d4 waiting for monitor entry [0x3a45f000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at oracle.jdbc.driver.PhysicalConnection.getMetaData(PhysicalConnection.java:3891)
    - waiting to lock <0x09ad05f8> (a oracle.jdbc.driver.T4CConnection)
    at oracle.dbtools.raptor.plscope.Query.getToolTipText(Query.java:58)
    at oracle.dbtools.raptor.phighlight.TooltipPlugin$1.getToolTipText(TooltipPlugin.java:29)
    at oracle.ide.ceditor.CodeEditor.getToolTipText(CodeEditor.java:1734)
    - locked <0x17245f98> (a java.util.ArrayList)
    at oracle.javatools.editor.BasicEditorPane.getToolTipText(BasicEditorPane.java:1530)
    at javax.swing.ToolTipManager$insideTimerAction.actionPerformed(ToolTipManager.java:658)
    at javax.swing.Timer.fireActionPerformed(Timer.java:271)
    at javax.swing.Timer$DoPostEvent.run(Timer.java:201)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
    "Background Parser" prio=6 tid=0x38a43400 nid=0x23c4 waiting on condition [0x39c5f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "SwingWorker-pool-5-thread-1142" prio=6 tid=0x38868800 nid=0x1a7c waiting on condition [0x3820f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x0e77bc10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)
    "Background Parser" prio=6 tid=0x38a76c00 nid=0x1d18 waiting on condition [0x3a35f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "Background Parser" prio=6 tid=0x387e2800 nid=0xd00 waiting on condition [0x3a25f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "Background Parser" prio=6 tid=0x38752000 nid=0x2794 waiting on condition [0x39e5f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "Swing-Shell" daemon prio=6 tid=0x367f0000 nid=0xd7c waiting on condition [0x3a85f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x1069d0d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at sun.awt.shell.Win32ShellFolderManager2$ComInvoker$3.run(Win32ShellFolderManager2.java:458)
    at java.lang.Thread.run(Thread.java:619)
    "Persistence Auto Flusher" daemon prio=6 tid=0x388a5c00 nid=0x1950 in Object.wait() [0x37e0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x0e892d48> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "Thread-76" prio=6 tid=0x387aec00 nid=0x259c runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "WeakDataReference polling" prio=2 tid=0x38752800 nid=0xe94 in Object.wait() [0x3a05f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x0b6ba260> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at oracle.ide.util.WeakDataReference$Cleaner.run(WeakDataReference.java:88)
    at java.lang.Thread.run(Thread.java:619)
    "Image Animator 3" daemon prio=4 tid=0x3861b400 nid=0x2330 waiting on condition [0x3978f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at sun.awt.image.GifFrame.dispose(GifImageDecoder.java:653)
    at sun.awt.image.GifImageDecoder.produceImage(GifImageDecoder.java:230)
    at sun.awt.image.InputStreamImageSource.doFetch(InputStreamImageSource.java:246)
    at sun.awt.image.ImageFetcher.fetchloop(ImageFetcher.java:172)
    at sun.awt.image.ImageFetcher.run(ImageFetcher.java:136)
    "Background Parser" prio=6 tid=0x38348800 nid=0x978 waiting on condition [0x3988f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "TextBufferScavenger" prio=6 tid=0x38324800 nid=0x1af0 in Object.wait() [0x3968f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x09bb9158> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at oracle.ide.model.TextNode$FacadeBufferReference$PollingThread.run(TextNode.java:1949)
    "pool-4-thread-1" prio=6 tid=0x36960400 nid=0x1a84 waiting on condition [0x35a5f000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x08150a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)
    "IconOverlayTracker Timer" prio=6 tid=0x38520c00 nid=0x1b48 in Object.wait() [0x3950f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x0795f290> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "Timer queue for AWT thread" daemon prio=6 tid=0x369b4000 nid=0x1f20 in Object.wait() [0x3940f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x077e1358> (a java.lang.Object)
    at java.lang.Object.wait(Object.java:485)
    at ice.util.awt.TimedAWTExecutor.nextElem(TimedAWTExecutor.java:108)
    - locked <0x077e1358> (a java.lang.Object)
    at ice.util.awt.TimedAWTExecutor.runScheduler(TimedAWTExecutor.java:130)
    at ice.util.awt.TimedAWTExecutor$1.run(TimedAWTExecutor.java:19)
    "WaitCursor-Timer" prio=6 tid=0x367c3c00 nid=0x1010 in Object.wait() [0x37b0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x077e1400> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "Native Directory Watcher" prio=2 tid=0x368d4000 nid=0x2334 runnable [0x3830f000]
    java.lang.Thread.State: RUNNABLE
    at oracle.ide.natives.NativeHandler.enterWatcherThread(Native Method)
    at oracle.ide.natives.NativeHandler$2.run(NativeHandler.java:252)
    at java.lang.Thread.run(Thread.java:619)
    "BaseTreeExplorer.NodeOpeningExecutor" prio=6 tid=0x3684b000 nid=0x8b0 waiting on condition [0x3810f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x06950af0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)
    "Scheduler" daemon prio=6 tid=0x36846400 nid=0x14bc in Object.wait() [0x37f0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at oracle.dbtools.raptor.backgroundTask.TaskLinkedList.takeNextTask(TaskLinkedList.java:47)
    - locked <0x06950c70> (a oracle.dbtools.raptor.backgroundTask.TaskLinkedList)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$SchedulerThread.run(RaptorTaskManager.java:422)
    "TimerQueue" daemon prio=6 tid=0x36832800 nid=0x88c in Object.wait() [0x37d0f000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at javax.swing.TimerQueue.postExpiredTimers(TimerQueue.java:218)
    - locked <0x06950da8> (a javax.swing.TimerQueue)
    at javax.swing.TimerQueue.run(TimerQueue.java:234)
    - locked <0x06950da8> (a javax.swing.TimerQueue)
    at java.lang.Thread.run(Thread.java:619)
    "ChangeSetService" prio=2 tid=0x367fd400 nid=0x2754 in Object.wait() [0x37c0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x06892cc0> (a oracle.jdevimpl.vcs.changeset.ChangeSetService)
    at java.lang.Object.wait(Object.java:485)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService.awaitEvents(ChangeSetService.java:178)
    - locked <0x06892cc0> (a oracle.jdevimpl.vcs.changeset.ChangeSetService)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService.eventLoop(ChangeSetService.java:199)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService.access$200(ChangeSetService.java:56)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService$2.run(ChangeSetService.java:138)
    at java.lang.Thread.run(Thread.java:619)
    "TimedCache-Timer" daemon prio=6 tid=0x35d52c00 nid=0x2448 in Object.wait() [0x3624f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x062ab118> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "JarIndex Timer" daemon prio=6 tid=0x35556400 nid=0x2488 in Object.wait() [0x35d4f000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x062206e0> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "AWT-Windows" daemon prio=6 tid=0x354dd000 nid=0x1d0c runnable [0x3595f000]
    java.lang.Thread.State: RUNNABLE
    at sun.awt.windows.WToolkit.eventLoop(Native Method)
    at sun.awt.windows.WToolkit.run(WToolkit.java:291)
    at java.lang.Thread.run(Thread.java:619)
    "AWT-Shutdown" prio=6 tid=0x354a1800 nid=0x268c in Object.wait() [0x3585f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:259)
    - locked <0x062208d8> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:619)
    "Java2D Disposer" daemon prio=10 tid=0x354b7400 nid=0x5f4 in Object.wait() [0x3575f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x06220968> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at sun.java2d.Disposer.run(Disposer.java:125)
    at java.lang.Thread.run(Thread.java:619)
    "Low Memory Detector" daemon prio=6 tid=0x00ee1000 nid=0x2664 runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "CompilerThread0" daemon prio=10 tid=0x00edc000 nid=0x1f6c waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "Attach Listener" daemon prio=10 tid=0x00eda800 nid=0x206c runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "Signal Dispatcher" daemon prio=10 tid=0x00ed9400 nid=0xf64 waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "Finalizer" daemon prio=8 tid=0x00ec6c00 nid=0x2348 in Object.wait() [0x34e7f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x061b0298> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0x00ec5800 nid=0x2484 in Object.wait() [0x34d7f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x061b0320> (a java.lang.ref.Reference$Lock)
    "main" prio=6 tid=0x009f8000 nid=0x19e0 waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "VM Thread" prio=10 tid=0x00ec1800 nid=0x22e0 runnable
    "VM Periodic Task Thread" prio=10 tid=0x00eec000 nid=0x2604 waiting on condition
    JNI global references: 5312
    Heap
    def new generation total 45376K, used 28499K [0x03080000, 0x061b0000, 0x061b0000)
    eden space 40384K, 67% used [0x03080000, 0x04af44c0, 0x057f0000)
    from space 4992K, 28% used [0x05cd0000, 0x05e30850, 0x061b0000)
    to space 4992K, 0% used [0x057f0000, 0x057f0000, 0x05cd0000)
    tenured generation total 604992K, used 364537K [0x061b0000, 0x2b080000, 0x2b080000)
    the space 604992K, 60% used [0x061b0000, 0x1c5ae548, 0x1c5ae600, 0x2b080000)
    compacting perm gen total 57344K, used 57260K [0x2b080000, 0x2e880000, 0x33080000)
    the space 57344K, 99% used [0x2b080000, 0x2e86b348, 0x2e86b400, 0x2e880000)
    No shared spaces configured.
    Edited by: mdaly on May 21, 2010 11:42 AM

    The cause of hang could be figured out from the thread dump.
    I have fixed that in development code. It was happening from PL/SQL editor trying to create tooltip which locked up the connection.
    I am not sure what would lead to memory bloat even if the tool is not in use. There could be memory leak from some component. I am investigating that.
    Suggest you to keep ObjectViewers, reports & PL/SQL editor closed when not in use.
    Have Sql Array Fetch Size preference to 50 if it's higher.
    Do not have auto-refresh for Report editor.
    Keep Sql History, Find Db Object, Snippets collapsed.
    See if that helps.
    -Raghu

  • Thread Dump issue with LD_ASSUME_KERNEL=2.4.1

    Hi ,
    When I take Thread Dump using 'jstack <PID>' in JDK 1.5 it givss me "sun.jvm.hotspot.debugger.DebuggerException" in the dump nothing more
    I set LD_ASSUME_KERNEL=2.4.1 in my server to avaoid some other issue ( JVM crash some times)
    Surprisingly, I can not stop my server ( my java process ) after that using our Shutdow scripts , Ctrl C or even "kill -9 <PID>",
    I have to restart the machine or manually release uncleaned resources that my server occupied and restart the server
    This happens Redhar 9 as well as in Linux ES.
    Anybody faced similar problem?
    Any help or information regarding this is highly apprecialted
    Vasu
    Thread Dump Output:
    Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Meth
    od)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:34)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(Li
    nuxDebuggerLocal.java:431)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(Linux
    DebuggerLocal.java:109)
    sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_re
    gs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(L
    inuxDebuggerLocal.java:134)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebugge
    rLocal.java:437)
    at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:48)
    at sun.jvm.hotspot.runtime.linux_x86.LinuxX86JavaThreadPDAccess.getCurrentFrameGuess(LinuxX86
    JavaThreadPDAccess.java:75)
    at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:252)
    at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:211)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:42)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:58)
    Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Meth
    od)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:34)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(Li
    nuxDebuggerLocal.java:431)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(Linux
    DebuggerLocal.java:109)
    sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_re
    gs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(L
    inuxDebuggerLocal.java:134)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebugge
    rLocal.java:437)

    @brain0
    I've downloaded the glic-2.3.6 sources from gnu, so I could build it from those. I'm however reluctant to do this because I really don't want to break my install.
    I do agree with you on the NPTL statement, but pvs relies on allegro, which relies on LinuxThreads. Allegro is not being ported to new versions of glibc, so that approach is unfortunately not viable.
    @iphitus
    I wasn't very specific - it's allegro as in a lisp environment.
    I think I'll try and install an old version of arch on wmware instead. Is there anywhere you can check out glibc version numbering on old arch install isos (ie. do I need arch-0.[1-9].iso)? And anywhere you can download the old isos (tried filewatcher, but a lot of the older sites seem broken)?
    Thanks for the replies,
    Mads
    PS. I noticed that you recommended slackware for old kernels in another thread. I'm however in a bit different situation as I need old versions of glibc. Furthermore I would prefer sticking to arch, but was wondering whether there were any specific reasons for not doing that.

  • Help on stuck thread

    Hi, From my server log I noticed that there are some stuck threads. After looked at the thread dump thread 18 is blocking. MainPortalServlet.java is extends PortalServlet and line 96 is super.service(request,response); Please help me to analyse why it is blocking. Thanks.
    "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - waiting to lock <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
         at java.util.HashMap.getEntry(HashMap.java:361)
         at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
         at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
         at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
         at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
         at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
         - locked <0x78ccf6c8> (a portlets.Portlet1Controller)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
         at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
         at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
         at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
         at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
         at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
         at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
         at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
         at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
         at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
         at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
         at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
         at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:116)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
    "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
         at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
         at java.util.HashMap.resize(HashMap.java:452)
         at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
         at java.util.HashMap.put(HashMap.java:392)
         at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
         - locked <0x8d3a3f18> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
         at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
         at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
         at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
         at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
         at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
         at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
         at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
         at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
         at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
         at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
         at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
         at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
         at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
         at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

    Hi user9503787,
    Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?
    Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:
    Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
    Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation
    Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.
    We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).
    Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.
    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/

  • Waiting for monitor enty messages in thread dump

    Hi,
    I've a thread dump of a container in a Oracle Application Server 9.0.4.1 (running 1.4.2_07) right before restart because off low memory and a lot of threads in this dump are in this state:
    "AJPRequestHandler-ApplicationServerThread-1588" prio=1 tid=0x37634bf8 nid=0x1982 waiting for monitor entry [43f22000..43f2386c]
    can anyone tell me what this means? Google didn't reveal anything useful.
    Another thing I don't understand although it seems it doesn't do any harm:
    "AJPRequestHandler-ApplicationServerThread-1630" prio=1 tid=0x09011a68 nid=0x1982 in Object.wait() [40276000..4027686c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x6624e628> (a java.util.HashMap)
    at java.lang.Object.wait(Object.java:429)
    at java.net.InetAddress.checkLookupTable(InetAddress.java:1109)
    - locked <0x6624e628> (a java.util.HashMap)
    at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1048)
    at java.net.InetAddress.getLocalHost(InetAddress.java:1188)
    at com.evermind.server.http.EvermindHttpSession.encodeSession(EvermindHttpSession.java:414)
    at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:1599)
    why is this thread waiting for a lock itself has?
    Thanx for any help and hints in advance.
    Regards,
    Ingo

    Hi,
    I've a follow-up question, I have the following thread in several thread dumps:
    "AJPRequestHandler-ApplicationServerThread-717" prio=1 tid=0x4046b988 nid=0x3dd3 waiting for monitor entry [3845e000..3845e86c]
         at java.net.Inet4Address.getAddress(Inet4Address.java:287)
         at com.evermind.server.http.StandardRequestLogger.appendIP(StandardRequestLogger.java:187)
         at com.evermind.server.http.StandardRequestLogger.log(StandardRequestLogger.java:45)
         - locked <0x663b2088> (a com.evermind.server.http.StandardRequestLogger)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:906)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    What is the monitor this thread is waiting for? According to the source in Inet4Address at line 287 the method tries to create a new byte array... is there some system monitor which can't be seen in the thread dump?
    This thread holds a lock (0x663b2088) 46 other threads are waiting for...
    Thanx in advance for any hints.
    Regards,
    Ingo

  • Collect Thread Dump using Script and direct to separate log file

    Dear Legends,
    I request you to provide me some useful guide lines to collect Thread Dump using script in linux and redirect the output to a separate log file, not the STDOUT file. If it is redirecting to STDOUT then how to extract only the thread stack to a separate log file.
    i used
    ps -ef | grep java
    kill -3 <pid> >> ss1_td.log
    but it is not providing the thread stack to the log file.
    Anykind of help would be appreciated much.
    Regards,
    Karthik

    Hi Ram,
    Really a great and useful script for me at the right time. It solved my requirement and i made some changes to my requirement and the script it working fine to collect the thread dump and even i have checked the thread dump with SAMURAI works fine.
    Helpful site also refer sites like these which contains more valuable things for weblogic. We will meet in the forums again.
    Note: also please update several links available in the site like UNIX scripts, WLST, Python etc.
    Regards,
    Karthik

  • The thread dump has a deadlock ?

    We're experiencing a problem with the OC4J server hanging when some we unknow operation . I can see it's a connection pooling problem based on a thread dump. What I can't determine is why it doesn't release a resource in the first place. I've included some code to show where it is hung.
    ================================================
    "AJPRequestHandler-ApplicationServerThread-17" tid=0x8775468 nid=0x2f3d prio=1 waiting for monitor entry
         at oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:4502)
         - waiting to lock<0x55805618> (a oracle.jbo.JboSyncLock), locked by AJPRequestHandler-ApplicationServerThread-135
         at oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:4479)
         at oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:6818)
         at oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:3886)
         at oracle.jbo.common.ampool.ApplicationPoolImpl.doManagedCheckin(ApplicationPoolImpl.java:2131)
         at oracle.jbo.common.ampool.ApplicationPoolImpl.releaseApplicationModule(ApplicationPoolImpl.java:1211)
         at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:717)
         - locked<0x557fa248> (a oracle.jbo.common.ampool.SessionCookieImpl$SessionCookieLock)
         at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:629)
         at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:611)
         at oracle.jheadstart.model.bc4j.ApplicationModuleProvider.releaseConnection(ApplicationModuleProvider.java:477)
         at oracle.jheadstart.model.bc4j.ApplicationModuleProvider.releaseRequestScopedConnections(ApplicationModuleProvider.java:503)
         at oracle.cle.persistence.ConnectionManager.releaseConnections(ConnectionManager.java:999)
         - locked<0x50e47900> (a oracle.cle.persistence.ConnectionManager)
         at oracle.cle.persistence.ConnectionManager.releaseRequestScopedConnections(ConnectionManager.java:939)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:645)
         at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
         at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
         at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
         at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    "AJPRequestHandler-ApplicationServerThread-135" tid=0x8064ac8 nid=0x2f3d prio=1 waiting for monitor entry
         at oracle.cle.persistence.ConnectionManager.getConnection(ConnectionManager.java:722)
         - waiting to lock<0x50e47900> (a oracle.cle.persistence.ConnectionManager), locked by AJPRequestHandler-ApplicationServerThread-17
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:308)
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:148)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getConnection(DataObjectHandlerImpl.java:355)
         at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.getApplicationModule(DataObjectImpl.java:761)
         at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.doneRollback(DataObjectImpl.java:855)
         at oracle.jbo.server.DBTransactionImpl.sendEvent(DBTransactionImpl.java:5501)
         at oracle.jbo.server.DBTransactionImpl.fireTransactionStateEvent(DBTransactionImpl.java:5568)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         at oracle.jbo.server.DBTransactionImpl.rollback(DBTransactionImpl.java:2257)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         - locked<0x55805618> (a oracle.jbo.JboSyncLock)
         at oracle.jheadstart.model.bc4j.JhsDBTransactionImpl.rollback(JhsDBTransactionImpl.java:186)
         at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.doneCommit(DataObjectImpl.java:832)
         at oracle.jbo.server.DBTransactionImpl.sendEvent(DBTransactionImpl.java:5494)
         at oracle.jbo.server.DBTransactionImpl.fireTransactionStateEvent(DBTransactionImpl.java:5568)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         at oracle.jbo.server.DBTransactionImpl.commitInternal(DBTransactionImpl.java:1989)
         at oracle.jbo.server.DBTransactionImpl.commit(DBTransactionImpl.java:2046)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         - locked<0x55805618> (a oracle.jbo.JboSyncLock)
         at oracle.jheadstart.model.bc4j.JhsDBTransactionImpl.commit(JhsDBTransactionImpl.java:176)
         at oracle.jheadstart.model.bc4j.JhsApplicationModuleImpl.commitChanges(JhsApplicationModuleImpl.java:1167)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.commitChanges(DataObjectHandlerImpl.java:898)
         at oracle.jheadstart.controller.struts.action.JhsAction.commitChanges(JhsAction.java:484)
         at oracle.jheadstart.controller.struts.action.JhsAction.saveChanges(JhsAction.java:409)
         at oracle.jheadstart.controller.struts.action.SaveDataObject.execute(SaveDataObject.java:467)
         at oracle.jheadstart.controller.struts.action.JhsAction.execute(JhsAction.java:245)
         at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.processActionPerform(JhsRequestProcessor.java:351)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:635)
         at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
         at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
         at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
         at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    "AJPRequestHandler-ApplicationServerThread-16" tid=0x8b2b510 nid=0x2f3d prio=1 waiting for monitor entry
         at oracle.cle.persistence.ConnectionManager.getConnection(ConnectionManager.java:722)
         - waiting to lock<0x50e47900> (a oracle.cle.persistence.ConnectionManager), locked by AJPRequestHandler-ApplicationServerThread-17
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:308)
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:148)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getConnection(DataObjectHandlerImpl.java:355)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getApplicationModule(DataObjectHandlerImpl.java:1011)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getDataObjectSet(DataObjectHandlerImpl.java:511)
         at oracle.jheadstart.controller.struts.action.GetDataObjectSet.retrieveSimple(GetDataObjectSet.java:474)
         at oracle.jheadstart.controller.struts.action.GetDataObjectSet.execute(GetDataObjectSet.java:390)
         at oracle.jheadstart.controller.struts.action.JhsAction.execute(JhsAction.java:245)
         at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.processActionPerform(JhsRequestProcessor.java:351)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:635)
         at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
         at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
         at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    Sincerely from Ting-Rung

    About the general situation describe like following, thanks you help.
    Q1:Whether you could able to access the httpserver home page or any other application deployed in other OC4J container, when the server
    become unresponsive.
    Ans:Yes
    Q2:What are the kind of applications deployed in the OC4J container ( hrmOC4J) .
    Ans:The kind of applications is human resource application(Developer Tool is JDeveloper 9.0.5.2 and Jheadstart 9.0.5.19 and
    oracle Application server 10.1.2.0.2)
    Q3:Does server hang happen regularly or at what situation the server is becoming unresponsive ( high load, etc..)
    Ans:No,It is not regularly when when server hang happen.
    Q4:From the SR notes, the workaround you used is, restarting the OC4J container. ( what is the
    kind of Oracle Application Server instance you have installed, please let me
    know the complete version of the Oracle AS instance)
    Ans:oracle Application server 10.1.2.0.2
    Q5:Did you notice any out of memory issues when the server / particular OC4J container is hang
    Ans:No. It is not out of memory.

  • Program hangs, can't get thread dump

    Hello
    I have a Java program which sometimes works OK untill the very end, but sometimes just hangs in a random moment and doesn't react to Ctrl+C or Ctrl+Break. Process doesn't use CPU, so I guess it isn't something like infinite loop (I actually cannot think of places in my program where this kind of loop may occur). I tried to get thread dump to see if there are any deadlocks, but Ctrl+Break doesn't work and jstack doesn't work also (I've read that there is -F option, but it isn't available on Windows). Also I tried to connect to a process with Java Visual VM - doesn't help too (just says 'Connecting...' forever).
    What else can I do to find out what's going on in my programm and fix it?
    I tried to run it with JDK 1.6.0_07 and 1.6.0_16. I'm using Windows Vista Enterprise SP1, 64-bit.
    Thanks in advance, Yulia.
    Edited by: Yulia_Dubinina on Oct 26, 2009 11:58 PM
    Edited by: Yulia_Dubinina on Oct 27, 2009 12:13 AM

    Yulia_Dubinina wrote:
    Yesterday I tried to start my program, connect with Java Visual VM to it and see how threads visualisation will look like in a moment when program hangs. My programm died after about 50 min and all info which I was able to see was about this period of time (e.g. a timeline diagram stopped changing and the last time mark was 49:23), but Java Visual VM was still connected to a process. I tried to press Thread Dump button, it worked, but there was nothing 'criminal' on the dump it provided. So I guess maybe it's not actual thread dump, it's something that Java Visual VM was able to get before process hanged.
    I don't have much experience in resolving this kind of problems, so any advices will be great appreciated.
    Edited by: Yulia_Dubinina on Oct 27, 2009 2:14 PMTurn on verbose GC and see if you got lots and lots of full GC when the freeze happens. A full GC can take several minutes if you have a bad configuration (where e.g. memory has been paged out to disk)

Maybe you are looking for

  • ITunes will not recognize my iPod Touch 4 so I am not able to restore

    I am trying to restore my iPod Touch 4th Generation so I can get rid of the jailbreak but whenever I plug in my device, the computer recognizes it but not iTunes. I have managed to get it to recognize it a few times which is when I try to restore it,

  • Com.solarmetric.kodo.util.ImplHelper.getPersistentClasses(ImplHelper.java:46)

    This trace uses almost 10% of my CPU time. It looks like static data collection. Is it being called much more often than it should, or am I calling something more often than I should? (After this, I am down to the DBMS being the bottleneck and a fact

  • CC to CC 2014 - Sync Not Working

    Reposting since my post from yesterday was deleted somehow... In Photoshop CC (14.2.1) I ensure my setting are uploaded... - In Preferences > Sync Settings it shows I'm signed in and my last sync was a few minutes ago. - All the checkboxes are checke

  • CMDlet get-recipient export as CSV removes Displayname Attribute

    Dear Forum, i´ve found an "interesting" behavior of the get-recipient CMDLet. We operate a 2 node DAG and use this CMDLet for Statistics and Chargeback - currently it runs as follows get-recipient -RecipientTypeDetails Usermailbox -resultsize unlimit

  • Disable autoRecovery in CS6

    How can I completely disable autorecovery? So far it has caused me more problems than simply saving my own backups frequently. I can't find any way of doing this.