Why is thread nid missing in thread dump

Hello,
If I create a thread dump locally on my machine I get thread IDs like this
>2015-02-09 10:30:53
>Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode):
>"RMI TCP Connection(2)-10.10.10.103" daemon prio=6 tid=0x000000001047a000 nid=0x1004 runnable [0x000000001584e000]
However, one of our customers has an environment where he gets this when he generated  a thread dump
> 2015-02-03 11:09:09
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode):
> "Thread-65526" - Thread t@139145
What you see is that for the latter dump the nid is missing from which we can obtain a relation to the OS thread.
Can you tell me how this can happen? A lot googling didn't gave me a thread dump like this either.
Best regards
Michael

I recall that post.
You were announcing or advertising some web page of yours.
The link you had wasn't to some web page
but when clicked a download installatin of some software started.
A moderator probably moved the forum post out of wherever you posted it
to a location where the site admins could inspect it.

Similar Messages

  • HELP!!! Why do my thread dump have so many locked monitor

    In recent weeks, My customer's weblogic server become more busy.Each time I downloaded
    the thread dump when CPU is busy than 90%,I found a lot of lock in the thread
    dump! Especially the class named "weblogic.kernel.ExecuteThread" . It's the kernel
    class of weblogic, Why were most of them locked?
    Full thread dump:
    "process forker" daemon prio=1 tid=0x95b4c8e0 nid=0x3084 waiting on monitor [0x977ff000..0x977ff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53d26420> (a java.lang.Object)
         at java.lang.UNIXProcessForker.run(Native Method)
         at java.lang.Thread.run(Thread.java:479)
    "process reaper" daemon prio=1 tid=0x95376758 nid=0x3083 runnable [0x96eff000..0x96eff890]
         at java.lang.UNIXProcessReaper.run(Native Method)
         at java.lang.Thread.run(Thread.java:479)
    "HighPriority TimeEventGenerator" daemon prio=1 tid=0x825bf18 nid=0x25a1 waiting
    on monitor [0x95dff000..0x95dff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53a3bd10> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
         - locked <53a3bd10> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:141)
         at java.lang.Thread.run(Thread.java:479)
    "ListenThread" prio=1 tid=0x94d00630 nid=0x25a0 runnable [0x95fff000..0x95fff890]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
         - locked <538fa238> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:238)
         at java.net.ServerSocket.accept(ServerSocket.java:217)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:325)
    "NBExecuteThread-1" daemon prio=1 tid=0x8c0dd1d0 nid=0x259c waiting on monitor
    [0x94521000..0x94521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a9650> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a9650> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "NBExecuteThread-0" daemon prio=1 tid=0x8c0dc7d0 nid=0x259b waiting on monitor
    [0x94321000..0x94321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a9060> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a9060> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    //"ExecuteThread-49" daemon prio=1 tid=0x8c034660 nid=0x259a runnable [0x94121000..0x94121890]
         at java.net.PlainDatagramSocketImpl.peek(Native Method)
         - locked <537acf60> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:349)
         - locked <486f9980> (a java.net.DatagramPacket)
         - locked <537acf40> (a java.net.MulticastSocket)
         at weblogic.cluster.TDMSocketIPM.receive(TDMSocketIPM.java:86)
         at weblogic.cluster.TMSocket.execute(TMSocket.java:249)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-48" daemon prio=1 tid=0x8c0c3d18 nid=0x2599 waiting for monitor
    entry [0x93f21000..0x93f21890]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - waiting to lock <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-47" daemon prio=1 tid=0x8c0c32d0 nid=0x2598 runnable [0x93d21000..0x93d21890]
         at weblogic.socket.PosixSocketMuxer.poll(Native Method)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - locked <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-46" daemon prio=1 tid=0x8c0bf410 nid=0x2597 waiting for monitor
    entry [0x93b21000..0x93b21890]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
         - waiting to lock <538fa5c8> (a java.lang.Object)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-45" daemon prio=1 tid=0x8c0be9c8 nid=0x2596 waiting on monitor
    [0x93921000..0x93921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6df8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6df8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-44" daemon prio=1 tid=0x8c0bdab0 nid=0x2595 waiting on monitor
    [0x93721000..0x93721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6828> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6828> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-43" daemon prio=1 tid=0x8c0bd068 nid=0x2594 waiting on monitor
    [0x93521000..0x93521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a6258> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a6258> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-42" daemon prio=1 tid=0x8c0c0ce0 nid=0x2593 waiting on monitor
    [0x93321000..0x93321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a5c88> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a5c88> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-41" daemon prio=1 tid=0x8c0c0298 nid=0x2592 waiting on monitor
    [0x93121000..0x93121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a56b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a56b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-40" daemon prio=1 tid=0x8c0a21e0 nid=0x2591 waiting on monitor
    [0x92f21000..0x92f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a50e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a50e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-39" daemon prio=1 tid=0x8c0d4520 nid=0x2590 waiting on monitor
    [0x92d21000..0x92d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a4b18> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a4b18> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-38" daemon prio=1 tid=0x8c0d43d0 nid=0x258f waiting on monitor
    [0x92b21000..0x92b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a4548> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a4548> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-37" daemon prio=1 tid=0x8c0d3988 nid=0x258e waiting on monitor
    [0x92921000..0x92921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a3f78> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a3f78> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-36" daemon prio=1 tid=0x8c0ded20 nid=0x258d waiting on monitor
    [0x92721000..0x92721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a39a8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a39a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-35" daemon prio=1 tid=0x8c0de2d8 nid=0x258c waiting on monitor
    [0x92521000..0x92521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a33d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a33d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-34" daemon prio=1 tid=0x8c0cd528 nid=0x258b waiting on monitor
    [0x92321000..0x92321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2e08> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2e08> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-33" daemon prio=1 tid=0x8c0ccae0 nid=0x258a waiting on monitor
    [0x92121000..0x92121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2838> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2838> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-32" daemon prio=1 tid=0x8c075360 nid=0x2589 waiting on monitor
    [0x91f21000..0x91f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a2268> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a2268> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-31" daemon prio=1 tid=0x8c0837d0 nid=0x2588 waiting on monitor
    [0x91d21000..0x91d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a1c98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a1c98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-30" daemon prio=1 tid=0x8c0a5628 nid=0x2587 waiting on monitor
    [0x91b21000..0x91b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a16c8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a16c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-29" daemon prio=1 tid=0x8c07aaf8 nid=0x2586 waiting on monitor
    [0x91921000..0x91921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a10f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a10f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-28" daemon prio=1 tid=0x8c0c23d0 nid=0x2585 waiting on monitor
    [0x91721000..0x91721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a0b28> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a0b28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-27" daemon prio=1 tid=0x8c0c1988 nid=0x2584 waiting on monitor
    [0x91521000..0x91521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537a0558> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537a0558> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-26" daemon prio=1 tid=0x8c07ca00 nid=0x2583 waiting on monitor
    [0x91321000..0x91321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ff88> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ff88> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-25" daemon prio=1 tid=0x8c079e70 nid=0x2582 waiting on monitor
    [0x91121000..0x91121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379f9b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379f9b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-24" daemon prio=1 tid=0x8c078038 nid=0x2581 waiting on monitor
    [0x90f21000..0x90f21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379f3e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379f3e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-23" daemon prio=1 tid=0x8c0a3298 nid=0x2580 waiting on monitor
    [0x90d21000..0x90d21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ee18> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ee18> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-22" daemon prio=1 tid=0x8c0e30d8 nid=0x257f waiting on monitor
    [0x90b21000..0x90b21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379e848> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379e848> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-21" daemon prio=1 tid=0x8c0c5948 nid=0x257e waiting on monitor
    [0x90921000..0x90921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379e278> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379e278> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-20" daemon prio=1 tid=0x8c0d0470 nid=0x257d waiting on monitor
    [0x90721000..0x90721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379dca8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379dca8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-19" daemon prio=1 tid=0x8c080798 nid=0x257c waiting on monitor
    [0x90521000..0x90521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379d6d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379d6d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-18" daemon prio=1 tid=0x8c080648 nid=0x257b waiting on monitor
    [0x90321000..0x90321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379d108> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379d108> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-17" daemon prio=1 tid=0x8c032380 nid=0x257a runnable [0x90121000..0x90121890]
         at java.lang.String.concat(String.java:1553)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:122)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:772)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-16" daemon prio=1 tid=0x8c0df9a8 nid=0x2579 waiting on monitor
    [0x8ff21000..0x8ff21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379c568> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379c568> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-15" daemon prio=1 tid=0x8c0a6288 nid=0x2578 waiting on monitor
    [0x8fd21000..0x8fd21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379bf98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379bf98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-14" daemon prio=1 tid=0x8c0dbd60 nid=0x2577 waiting on monitor
    [0x8fb21000..0x8fb21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379b9c8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379b9c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-13" daemon prio=1 tid=0x8c0c8498 nid=0x2576 waiting on monitor
    [0x8f921000..0x8f921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379b3f8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379b3f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-12" daemon prio=1 tid=0x8c0a6a28 nid=0x2575 waiting on monitor
    [0x8f721000..0x8f721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379ae28> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379ae28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-11" daemon prio=1 tid=0x8c07d348 nid=0x2574 runnable [0x8f521000..0x8f521890]
         at java.lang.System.arraycopy(Native Method)
         at java.lang.String.getChars(String.java:557)
         at java.lang.String.concat(String.java:1554)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:251)
         at com.ebao.carcase.servlet.claim.ClaimServlet.doGet(ClaimServlet.java:77)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-10" daemon prio=1 tid=0x8c0369b0 nid=0x2571 waiting on monitor
    [0x8f321000..0x8f321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <5379a288> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <5379a288> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-9" daemon prio=1 tid=0x8c079898 nid=0x256f waiting on monitor [0x8f121000..0x8f121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53799cb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53799cb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-8" daemon prio=1 tid=0x8c035260 nid=0x256e waiting on monitor [0x8ef21000..0x8ef21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537996e8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537996e8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-7" daemon prio=1 tid=0x8c034510 nid=0x256d waiting on monitor [0x8ed21000..0x8ed21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53799118> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53799118> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-6" daemon prio=1 tid=0x8c081130 nid=0x256c waiting on monitor [0x8eb21000..0x8eb21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53798b48> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53798b48> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-5" daemon prio=1 tid=0x8c0c7108 nid=0x256b waiting on monitor [0x8e921000..0x8e921890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53798578> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53798578> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-4" daemon prio=1 tid=0x8c0a4388 nid=0x256a waiting on monitor [0x8e721000..0x8e721890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53797fa8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53797fa8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-3" daemon prio=1 tid=0x8c0367b8 nid=0x2569 waiting on monitor [0x8e521000..0x8e521890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537979d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <537979d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-2" daemon prio=1 tid=0x8c036290 nid=0x2568 waiting on monitor [0x8e321000..0x8e321890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53797408> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53797408> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-1" daemon prio=1 tid=0x8c0a4ee0 nid=0x2567 waiting on monitor [0x8e121000..0x8e121890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53796e38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
         - locked <53796e38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-0" daemon prio=1 tid=0x8c0a4550 nid=0x2566 runnable [0x8df21000..0x8df21890]
         at java.lang.String.concat(String.java:1553)
         at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
         at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
         at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
         at com.ebao.carcase.servlet.end.EndCaseServlet.service(EndCaseServlet.java:56)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
         at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:120)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:945)
         at weblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImpl.java:909)
         at weblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContextManager.java:269)
         at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
         at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "TimeEventGenerator" daemon prio=1 tid=0x8c076008 nid=0x2565 waiting on monitor
    [0x8dd21000..0x8dd21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53795330> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
         - locked <53795330> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:141)
         at java.lang.Thread.run(Thread.java:479)
    "SpinnerRandomSource" daemon prio=1 tid=0x8c0b34a0 nid=0x2563 waiting on monitor
    [0x8db21000..0x8db21890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53793040> (a weblogic.security.SpinnerThread)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:104)
         - locked <53793040> (a weblogic.security.SpinnerThread)
         at weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:121)
    "Signal Dispatcher" daemon prio=1 tid=0x8c0077e8 nid=0x2561 waiting on monitor
    [0..0]
    "Finalizer" daemon prio=1 tid=0x8c000540 nid=0x255e waiting on monitor [0x8c2ff000..0x8c2ff890]
         at java.lang.Object.wait(Native Method)
         - waiting on <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:103)
         - locked <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:157)
    "Reference Handler" daemon prio=1 tid=0x80851f0 nid=0x255d waiting on monitor
    [0x8bf8c000..0x8bf8c890]
         at java.lang.Object.wait(Native Method)
         - waiting on <53720208> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:415)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:105)
         - locked <53720208> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x804ead0 nid=0x2543 waiting on monitor [0xbfffc000..0xbfffd00c]
         at java.lang.Object.wait(Native Method)
         - waiting on <53743c20> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.Object.wait(Object.java:415)
         at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1958)
         - locked <53743c20> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.reflect.Method.invoke(Native Method)
         at weblogic.Server.startServerDynamically(Server.java:148)
         at weblogic.Server.main(Server.java:97)
         at weblogic.Server.main(Server.java:58)
    "VM Thread" prio=1 tid=0x8082110 nid=0x255c runnable
    "VM Periodic Task Thread" prio=1 tid=0x8c0064f8 nid=0x255f waiting on monitor
    "Suspend Checker Thread" prio=1 tid=0x8c006e30 nid=0x2560 runnable

    These are execute threads waiting for requests to come in.
    BTW, you probably want to look into using StringBuffer instead of
    strings concatenation in com.ebao.carcase.bean.query.Query.generatorXML.
    "min.lu" <[email protected]> wrote in message
    news:[email protected]...
    >
    In recent weeks, My customer's weblogic server become more busy.Each timeI downloaded
    the thread dump when CPU is busy than 90%,I found a lot of lock in thethread
    dump! Especially the class named "weblogic.kernel.ExecuteThread" . It'sthe kernel
    class of weblogic, Why were most of them locked?
    Full thread dump:
    "process forker" daemon prio=1 tid=0x95b4c8e0 nid=0x3084 waiting onmonitor [0x977ff000..0x977ff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53d26420> (a java.lang.Object)
    at java.lang.UNIXProcessForker.run(Native Method)
    at java.lang.Thread.run(Thread.java:479)
    "process reaper" daemon prio=1 tid=0x95376758 nid=0x3083 runnable[0x96eff000..0x96eff890]
    at java.lang.UNIXProcessReaper.run(Native Method)
    at java.lang.Thread.run(Thread.java:479)
    "HighPriority TimeEventGenerator" daemon prio=1 tid=0x825bf18 nid=0x25a1waiting
    on monitor [0x95dff000..0x95dff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53a3bd10> (a weblogic.time.common.internal.TimeTable)
    at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
    - locked <53a3bd10> (a weblogic.time.common.internal.TimeTable)
    atweblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java
    :141)
    at java.lang.Thread.run(Thread.java:479)
    "ListenThread" prio=1 tid=0x94d00630 nid=0x25a0 runnable[0x95fff000..0x95fff890]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:463)
    - locked <538fa238> (a java.net.PlainSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:238)
    at java.net.ServerSocket.accept(ServerSocket.java:217)
    atweblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
    at weblogic.t3.srvr.ListenThread.run(ListenThread.java:325)
    "NBExecuteThread-1" daemon prio=1 tid=0x8c0dd1d0 nid=0x259c waiting onmonitor
    [0x94521000..0x94521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a9650> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a9650> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "NBExecuteThread-0" daemon prio=1 tid=0x8c0dc7d0 nid=0x259b waiting onmonitor
    [0x94321000..0x94321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a9060> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a9060> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    //"ExecuteThread-49" daemon prio=1 tid=0x8c034660 nid=0x259a runnable[0x94121000..0x94121890]
    at java.net.PlainDatagramSocketImpl.peek(Native Method)
    - locked <537acf60> (a java.net.PlainDatagramSocketImpl)
    at java.net.DatagramSocket.receive(DatagramSocket.java:349)
    - locked <486f9980> (a java.net.DatagramPacket)
    - locked <537acf40> (a java.net.MulticastSocket)
    at weblogic.cluster.TDMSocketIPM.receive(TDMSocketIPM.java:86)
    at weblogic.cluster.TMSocket.execute(TMSocket.java:249)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-48" daemon prio=1 tid=0x8c0c3d18 nid=0x2599 waiting formonitor
    entry [0x93f21000..0x93f21890]
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - waiting to lock <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-47" daemon prio=1 tid=0x8c0c32d0 nid=0x2598 runnable[0x93d21000..0x93d21890]
    at weblogic.socket.PosixSocketMuxer.poll(Native Method)
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - locked <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-46" daemon prio=1 tid=0x8c0bf410 nid=0x2597 waiting formonitor
    entry [0x93b21000..0x93b21890]
    atweblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:290)
    - waiting to lock <538fa5c8> (a java.lang.Object)
    atweblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-45" daemon prio=1 tid=0x8c0be9c8 nid=0x2596 waiting onmonitor
    [0x93921000..0x93921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6df8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6df8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-44" daemon prio=1 tid=0x8c0bdab0 nid=0x2595 waiting onmonitor
    [0x93721000..0x93721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6828> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6828> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-43" daemon prio=1 tid=0x8c0bd068 nid=0x2594 waiting onmonitor
    [0x93521000..0x93521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a6258> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a6258> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-42" daemon prio=1 tid=0x8c0c0ce0 nid=0x2593 waiting onmonitor
    [0x93321000..0x93321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a5c88> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a5c88> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-41" daemon prio=1 tid=0x8c0c0298 nid=0x2592 waiting onmonitor
    [0x93121000..0x93121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a56b8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a56b8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-40" daemon prio=1 tid=0x8c0a21e0 nid=0x2591 waiting onmonitor
    [0x92f21000..0x92f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a50e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a50e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-39" daemon prio=1 tid=0x8c0d4520 nid=0x2590 waiting onmonitor
    [0x92d21000..0x92d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a4b18> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a4b18> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-38" daemon prio=1 tid=0x8c0d43d0 nid=0x258f waiting onmonitor
    [0x92b21000..0x92b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a4548> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a4548> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-37" daemon prio=1 tid=0x8c0d3988 nid=0x258e waiting onmonitor
    [0x92921000..0x92921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a3f78> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a3f78> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-36" daemon prio=1 tid=0x8c0ded20 nid=0x258d waiting onmonitor
    [0x92721000..0x92721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a39a8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a39a8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-35" daemon prio=1 tid=0x8c0de2d8 nid=0x258c waiting onmonitor
    [0x92521000..0x92521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a33d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a33d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-34" daemon prio=1 tid=0x8c0cd528 nid=0x258b waiting onmonitor
    [0x92321000..0x92321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2e08> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2e08> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-33" daemon prio=1 tid=0x8c0ccae0 nid=0x258a waiting onmonitor
    [0x92121000..0x92121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2838> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2838> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-32" daemon prio=1 tid=0x8c075360 nid=0x2589 waiting onmonitor
    [0x91f21000..0x91f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a2268> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a2268> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-31" daemon prio=1 tid=0x8c0837d0 nid=0x2588 waiting onmonitor
    [0x91d21000..0x91d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a1c98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a1c98> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-30" daemon prio=1 tid=0x8c0a5628 nid=0x2587 waiting onmonitor
    [0x91b21000..0x91b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a16c8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a16c8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-29" daemon prio=1 tid=0x8c07aaf8 nid=0x2586 waiting onmonitor
    [0x91921000..0x91921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a10f8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a10f8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-28" daemon prio=1 tid=0x8c0c23d0 nid=0x2585 waiting onmonitor
    [0x91721000..0x91721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a0b28> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a0b28> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-27" daemon prio=1 tid=0x8c0c1988 nid=0x2584 waiting onmonitor
    [0x91521000..0x91521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537a0558> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537a0558> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-26" daemon prio=1 tid=0x8c07ca00 nid=0x2583 waiting onmonitor
    [0x91321000..0x91321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ff88> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ff88> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-25" daemon prio=1 tid=0x8c079e70 nid=0x2582 waiting onmonitor
    [0x91121000..0x91121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379f9b8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379f9b8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-24" daemon prio=1 tid=0x8c078038 nid=0x2581 waiting onmonitor
    [0x90f21000..0x90f21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379f3e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379f3e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-23" daemon prio=1 tid=0x8c0a3298 nid=0x2580 waiting onmonitor
    [0x90d21000..0x90d21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ee18> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ee18> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-22" daemon prio=1 tid=0x8c0e30d8 nid=0x257f waiting onmonitor
    [0x90b21000..0x90b21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379e848> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379e848> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-21" daemon prio=1 tid=0x8c0c5948 nid=0x257e waiting onmonitor
    [0x90921000..0x90921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379e278> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379e278> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-20" daemon prio=1 tid=0x8c0d0470 nid=0x257d waiting onmonitor
    [0x90721000..0x90721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379dca8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379dca8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-19" daemon prio=1 tid=0x8c080798 nid=0x257c waiting onmonitor
    [0x90521000..0x90521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379d6d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379d6d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-18" daemon prio=1 tid=0x8c080648 nid=0x257b waiting onmonitor
    [0x90321000..0x90321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379d108> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379d108> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-17" daemon prio=1 tid=0x8c032380 nid=0x257a runnable[0x90121000..0x90121890]
    at java.lang.String.concat(String.java:1553)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:122)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:772)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-16" daemon prio=1 tid=0x8c0df9a8 nid=0x2579 waiting onmonitor
    [0x8ff21000..0x8ff21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379c568> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379c568> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-15" daemon prio=1 tid=0x8c0a6288 nid=0x2578 waiting onmonitor
    [0x8fd21000..0x8fd21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379bf98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379bf98> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-14" daemon prio=1 tid=0x8c0dbd60 nid=0x2577 waiting onmonitor
    [0x8fb21000..0x8fb21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379b9c8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379b9c8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-13" daemon prio=1 tid=0x8c0c8498 nid=0x2576 waiting onmonitor
    [0x8f921000..0x8f921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379b3f8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379b3f8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-12" daemon prio=1 tid=0x8c0a6a28 nid=0x2575 waiting onmonitor
    [0x8f721000..0x8f721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379ae28> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379ae28> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-11" daemon prio=1 tid=0x8c07d348 nid=0x2574 runnable[0x8f521000..0x8f521890]
    at java.lang.System.arraycopy(Native Method)
    at java.lang.String.getChars(String.java:557)
    at java.lang.String.concat(String.java:1554)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.claim.ClaimServlet.doPost(ClaimServlet.java:251)
    at com.ebao.carcase.servlet.claim.ClaimServlet.doGet(ClaimServlet.java:77)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "ExecuteThread-10" daemon prio=1 tid=0x8c0369b0 nid=0x2571 waiting onmonitor
    [0x8f321000..0x8f321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <5379a288> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <5379a288> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-9" daemon prio=1 tid=0x8c079898 nid=0x256f waiting onmonitor [0x8f121000..0x8f121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53799cb8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53799cb8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-8" daemon prio=1 tid=0x8c035260 nid=0x256e waiting onmonitor [0x8ef21000..0x8ef21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537996e8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537996e8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-7" daemon prio=1 tid=0x8c034510 nid=0x256d waiting onmonitor [0x8ed21000..0x8ed21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53799118> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53799118> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-6" daemon prio=1 tid=0x8c081130 nid=0x256c waiting onmonitor [0x8eb21000..0x8eb21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53798b48> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53798b48> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-5" daemon prio=1 tid=0x8c0c7108 nid=0x256b waiting onmonitor [0x8e921000..0x8e921890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53798578> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53798578> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-4" daemon prio=1 tid=0x8c0a4388 nid=0x256a waiting onmonitor [0x8e721000..0x8e721890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53797fa8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53797fa8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-3" daemon prio=1 tid=0x8c0367b8 nid=0x2569 waiting onmonitor [0x8e521000..0x8e521890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537979d8> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <537979d8> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-2" daemon prio=1 tid=0x8c036290 nid=0x2568 waiting onmonitor [0x8e321000..0x8e321890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53797408> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53797408> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-1" daemon prio=1 tid=0x8c0a4ee0 nid=0x2567 waiting onmonitor [0x8e121000..0x8e121890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53796e38> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:415)
    at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:100)
    - locked <53796e38> (a weblogic.kernel.ExecuteThread)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:127)
    "ExecuteThread-0" daemon prio=1 tid=0x8c0a4550 nid=0x2566 runnable[0x8df21000..0x8df21890]
    at java.lang.String.concat(String.java:1553)
    at com.ebao.carcase.bean.query.Query.generatorXML(Query.java:239)
    at com.ebao.carcase.bean.query.Query.generatorXMLByList(Query.java:158)
    at com.ebao.carcase.servlet.common.CommonBean.query(CommonBean.java:652)
    atcom.ebao.carcase.servlet.end.EndCaseServlet.service(EndCaseServlet.java:56)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    at weblogic.servlet.ServletServlet.service(ServletServlet.java:46)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
    atweblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java
    :120)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:945)
    atweblogic.servlet.internal.ServletContextImpl.invokeServlet(ServletContextImp
    l.java:909)
    atweblogic.servlet.internal.ServletContextManager.invokeServlet(ServletContext
    Manager.java:269)
    atweblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
    at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
    at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)
    "TimeEventGenerator" daemon prio=1 tid=0x8c076008 nid=0x2565 waiting onmonitor
    [0x8dd21000..0x8dd21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53795330> (a weblogic.time.common.internal.TimeTable)
    at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:279)
    - locked <53795330> (a weblogic.time.common.internal.TimeTable)
    atweblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java
    :141)
    at java.lang.Thread.run(Thread.java:479)
    "SpinnerRandomSource" daemon prio=1 tid=0x8c0b34a0 nid=0x2563 waiting onmonitor
    [0x8db21000..0x8db21890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53793040> (a weblogic.security.SpinnerThread)
    at java.lang.Object.wait(Object.java:415)
    atweblogic.security.SpinnerThread.stopSpinning(SpinnerRandomBitsSource.java:10
    4)
    - locked <53793040> (a weblogic.security.SpinnerThread)
    at weblogic.security.SpinnerThread.run(SpinnerRandomBitsSource.java:121)
    "Signal Dispatcher" daemon prio=1 tid=0x8c0077e8 nid=0x2561 waiting onmonitor
    [0..0]
    "Finalizer" daemon prio=1 tid=0x8c000540 nid=0x255e waiting on monitor[0x8c2ff000..0x8c2ff890]
    at java.lang.Object.wait(Native Method)
    - waiting on <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:103)
    - locked <537202f8> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:157)
    "Reference Handler" daemon prio=1 tid=0x80851f0 nid=0x255d waiting onmonitor
    [0x8bf8c000..0x8bf8c890]
    at java.lang.Object.wait(Native Method)
    - waiting on <53720208> (a java.l

  • Weblogic sp4 on linux out of memory expections - Thread dump

    Any body has idea why out of memory expections?
    Here is the thread dump.
    Full thread dump Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode):
    "Thread-33" daemon prio=1 tid=0x086175f0 nid=0x4f8e runnable [c8d57000..c8d5723c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xd48f7820> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.apache.axis.monitor.SOAPMonitorService$ServerSocketThread.run(SOAPMonitorService.java:189)
         at java.lang.Thread.run(Thread.java:534)
    "CustomerClientDispatcherThread" daemon prio=1 tid=0x08877100 nid=0x4f8e in Object.wait() [c77ff000..c77ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd45b7330> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at java.lang.Object.wait(Object.java:429)
         at com.amgen.ea.orx.queue.impl.PersistenceQueue.peek(PersistenceQueue.java:58)
         - locked <0xd45b7330> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at com.amgen.ea.orx.custclient.impl.CustomerClientDispatcher.run(CustomerClientDispatcher.java:50)
         at java.lang.Thread.run(Thread.java:534)
    "CustomerClientDispatcherThread" daemon prio=1 tid=0xcb625c28 nid=0x4f8e in Object.wait() [c7b8c000..c7b8c23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3ce7958> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at java.lang.Object.wait(Object.java:429)
         at com.amgen.ea.orx.queue.impl.PersistenceQueue.peek(PersistenceQueue.java:58)
         - locked <0xd3ce7958> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at com.amgen.ea.orx.custclient.impl.CustomerClientDispatcher.run(CustomerClientDispatcher.java:52)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-8" daemon prio=1 tid=0x08710e80 nid=0x4f8e in Object.wait() [c7cc4000..c7cc423c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd38b44c8> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at java.lang.Object.wait(Object.java:429)
         at com.amgen.ea.orx.queue.impl.PersistenceQueue.peek(PersistenceQueue.java:58)
         - locked <0xd38b44c8> (a com.amgen.ea.orx.queue.impl.PersistenceQueue)
         at com.amgen.ea.orx.custclient.impl.CustomerClientDispatcher.run(CustomerClientDispatcher.java:52)
         at java.lang.Thread.run(Thread.java:534)
    "ListenThread.Default" prio=1 tid=0xcc21b658 nid=0x4f8e runnable [c7c0c000..c7c0c23c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xd3811cf8> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Thread-6" prio=1 tid=0xcb3ef610 nid=0x4f8e in Object.wait() [c92a0000..c92a023c]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xd348db48> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-5" prio=1 tid=0x08c46fd8 nid=0x4f8e in Object.wait() [c9500000..c950023c]
         at java.lang.Object.wait(Native Method)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xd34901c8> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "LDAPConnThread-0 ldap://0.0.0.0:7013" daemon prio=1 tid=0xcb67b150 nid=0x4f8e runnable [c9580000..c958023c]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <0xd3490a98> (a java.io.BufferedInputStream)
         at netscape.ldap.ber.stream.BERElement.getElement(BERElement.java:101)
         at netscape.ldap.LDAPConnThread.run(LDAPConnThread.java:538)
         at java.lang.Thread.run(Thread.java:534)
    "VDE Transaction Processor Thread" prio=1 tid=0xcb2fe110 nid=0x4f8e in Object.wait() [c9600000..c960023c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3490b30> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at java.lang.Object.wait(Object.java:429)
         at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:365)
         - locked <0xd3490b30> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=1 tid=0x08665ba8 nid=0x4f8e in Object.wait() [c9680000..c968023c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3490ba8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon prio=1 tid=0x0830a7e0 nid=0x4f8e in Object.wait() [c9700000..c970023c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3490c28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon prio=1 tid=0x080a0c00 nid=0x4f8e in Object.wait() [c9780000..c978023c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3490ca8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=1 tid=0xcafb3be8 nid=0x4f8e runnable [c98ff000..c98ff23c]
         at weblogic.socket.PosixSocketMuxer.poll(Native Method)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
         - locked <0xd31100e0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=1 tid=0xcaf65228 nid=0x4f8e waiting for monitor entry [c997f000..c997f23c]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
         - waiting to lock <0xd31100e0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=1 tid=0xcafb0c20 nid=0x4f8e waiting for monitor entry [c99ff000..c99ff23c]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
         - waiting to lock <0xd31100e0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "weblogic.security.SpinnerRandomSource" daemon prio=1 tid=0xcaf64f98 nid=0x4f8e in Object.wait() [c9a7f000..c9a7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31182d8> (a java.lang.Object)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.security.SpinnerRandomBitsSource.run(SpinnerRandomBitsSource.java:60)
         - locked <0xd31182d8> (a java.lang.Object)
         at java.lang.Thread.run(Thread.java:534)
    "weblogic.time.TimeEventGenerator" daemon prio=1 tid=0xcb3e1400 nid=0x4f8e in Object.wait() [c9aff000..c9aff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118348> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <0xd3118348> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb3df3a8 nid=0x4f8e in Object.wait() [c9b7f000..c9b7f23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31183b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb3de7a0 nid=0x4f8e in Object.wait() [c9bff000..c9bff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118438> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb3ddcc0 nid=0x4f8e in Object.wait() [c9c7f000..c9c7f23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31184b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb1c66f0 nid=0x4f8e in Object.wait() [c9cff000..c9cff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118538> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon prio=1 tid=0xcb1c5b58 nid=0x4f8e in Object.wait() [c9d7f000..c9d7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31185b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31185b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '39' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c4f50 nid=0x4f8e in Object.wait() [c9dff000..c9dff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118638> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '38' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c4348 nid=0x4f8e in Object.wait() [c9e7f000..c9e7f23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31186b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '37' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c3740 nid=0x4f8e in Object.wait() [c9eff000..c9eff23c]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118738> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '36' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb1c2ba8 nid=0x4f8e in Object.wait() [c9f7f000..c9f7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31187b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31187b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '35' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb674078 nid=0x4f8e in Object.wait() [c9fff000..c9fff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118838> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118838> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '34' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb673470 nid=0x4f8e in Object.wait() [ca07f000..ca07f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31188b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31188b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '33' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb6728d8 nid=0x4f8e in Object.wait() [ca0ff000..ca0ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118938> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118938> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '32' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb671cd0 nid=0x4f8e in Object.wait() [ca17f000..ca17f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31189b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31189b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '31' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb6710e0 nid=0x4f8e in Object.wait() [ca1ff000..ca1ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118a38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118a38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '30' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfdb548 nid=0x4f8e in Object.wait() [ca27f000..ca27f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118ab8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118ab8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '29' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfda940 nid=0x4f8e in Object.wait() [ca2ff000..ca2ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118b38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118b38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '28' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfd9d58 nid=0x4f8e in Object.wait() [ca37f000..ca37f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118bb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118bb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '27' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfd91c0 nid=0x4f8e in Object.wait() [ca3ff000..ca3ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118c38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118c38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '26' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb119a40 nid=0x4f8e in Object.wait() [ca47f000..ca47f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118cb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118cb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '25' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb118e38 nid=0x4f8e in Object.wait() [ca4ff000..ca4ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118d38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118d38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb118230 nid=0x4f8e in Object.wait() [ca57f000..ca57f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118db8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118db8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '23' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcc207638 nid=0x4f8e in Object.wait() [ca5ff000..ca5ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118e38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118e38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcc206a30 nid=0x4f8e in Object.wait() [ca67f000..ca67f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118eb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118eb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '21' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcc205e28 nid=0x4f8e in Object.wait() [ca6ff000..ca6ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118f38> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118f38> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfbf1e0 nid=0x4f8e in Object.wait() [ca77f000..ca77f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3118fb8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3118fb8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfbe648 nid=0x4f8e in Object.wait() [ca7ff000..ca7ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119038> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119038> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfbda40 nid=0x4f8e in Object.wait() [ca87f000..ca87f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31190b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31190b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '17' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb62c500 nid=0x4f8e in Object.wait() [ca8ff000..ca8ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119138> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119138> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '16' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb62b8f8 nid=0x4f8e in Object.wait() [ca97f000..ca97f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31191b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31191b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb62acf0 nid=0x4f8e in Object.wait() [ca9ff000..ca9ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119238> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119238> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb61f5a8 nid=0x4f8e in Object.wait() [caa7f000..caa7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31192b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31192b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb61e9e0 nid=0x4f8e in Object.wait() [caaff000..caaff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119338> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119338> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf13af0 nid=0x4f8e in Object.wait() [cab7f000..cab7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31193b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31193b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf12ee8 nid=0x4f8e in Object.wait() [cabff000..cabff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119438> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119438> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf12350 nid=0x4f8e in Object.wait() [cac7f000..cac7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31194b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31194b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb309598 nid=0x4f8e in Object.wait() [cacff000..cacff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119538> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119538> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb3089c8 nid=0x4f8e in Object.wait() [cad7f000..cad7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31195b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31195b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf56250 nid=0x4f8e in Object.wait() [cadff000..cadff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119638> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119638> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55cb0 nid=0x4f8e in Object.wait() [cae7f000..cae7f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31196b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31196b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55918 nid=0x4f8e in Object.wait() [caeff000..caeff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119738> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119738> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55510 nid=0x4f8e in Object.wait() [cb47f000..cb47f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31197b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31197b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcaf55150 nid=0x4f8e in Object.wait() [cb4ff000..cb4ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119838> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119838> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcbfc7bd8 nid=0x4f8e in Object.wait() [cb57f000..cb57f23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31198b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31198b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb3cd250 nid=0x4f8e in Object.wait() [cb5ff000..cb5ff23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119938> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd3119938> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon prio=1 tid=0xcb3cd0b8 nid=0x4f8e in Object.wait() [cc1fe000..cc1fe23c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd31199b8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xd31199b8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-1" daemon prio=1 tid=0x086d6478 nid=0x4f8e in Object.wait() [cc3e1000..cc3e123c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xd3119a88> (a java.util.TaskQueue)
         at java.lang.Object.wait(Object.java:429)
         at java.util.TimerThread.mainLoop(Timer.java:403)
         - locked <0xd3119a88> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Signal Dispatcher" daemon prio=1 tid=0x080c4170 nid=0x4f8e waiting on condition [0..0]
    "Finalizer" daemon prio=1 tid=0x080bfac0 nid=0x4f8e in Object.wait() [cf509000..cf50923c]
         at java.lang.Object.wait(Native Method)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0xd3129c78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Refe

    <p>Can you post the stack trace. I assume these thread dumps were taken after the error occured. Can you take a few dumps just before you run out of memory.</p>
    <p>
    Hussein Badakhchani</br>
    </p>

  • Full thread dump Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode):

    Hi All,
    I am getting below error message when i am starting JBoss
    Anyone has any idea, why it is happening?
    Full thread dump Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode):
    "RMI ConnectionExpiration-[127.0.0.1:33030]" daemon prio=1 tid=0x2a375e90 nid=0xcf9 waiting on condition [2820d000..2820d86c]
         at java.lang.Thread.sleep(Native Method)
         at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:447)
         at java.lang.Thread.run(Thread.java:534)
    "RMI TCP Connection(1681)-127.0.0.1" daemon prio=1 tid=0x2cc3e008 nid=0xcf9 runnable [27b7b000..27b7b86c]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <0x2ed63640> (a java.io.BufferedInputStream)
         at java.io.FilterInputStream.read(FilterInputStream.java:66)
         at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:442)
         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
         at java.lang.Thread.run(Thread.java:534)
    "RMI RenewClean-[127.0.0.1:33030]" daemon prio=1 tid=0x2ca76ad0 nid=0xcf9 in Object.wait() [29dcd000..29dcd86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x2ed61c50> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0x2ed61c50> (a java.lang.ref.ReferenceQueue$Lock)
         at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
         at java.lang.Thread.run(Thread.java:534)
    "JNP Server" prio=1 tid=0x082dd9c0 nid=0xcf9 runnable [27afa000..27afa86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d145770> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.jnp.server.Main.run(Main.java:257)
         at java.lang.Thread.run(Thread.java:534)
    "RMI TCP Connection(1680)-127.0.0.1" daemon prio=1 tid=0x2ca55d80 nid=0xcf9 runnable [27f88000..27f8886c]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <0x2ecd1e40> (a java.io.BufferedInputStream)
         at java.io.FilterInputStream.read(FilterInputStream.java:66)
         at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:442)
         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
         at java.lang.Thread.run(Thread.java:534)
    "RMI ConnectionExpiration-[127.0.0.1:33030]" daemon prio=1 tid=0x2df71458 nid=0xcf9 waiting on condition [29e4e000..29e4e86c]
         at java.lang.Thread.sleep(Native Method)
         at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:447)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-36" daemon prio=1 tid=0x08541f10 nid=0xcf9 runnable [279f8000..279f886c]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
         - locked <0x3d8dbc38> (a java.io.BufferedInputStream)
         at com.sun.jndi.ldap.Connection.run(Connection.java:780)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-35" daemon prio=1 tid=0x0811fd00 nid=0xcf9 runnable [27a79000..27a7986c]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
         - locked <0x3d8d7d50> (a java.io.BufferedInputStream)
         at com.sun.jndi.ldap.Connection.run(Connection.java:780)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-34" daemon prio=1 tid=0x08bc3588 nid=0xcf9 in Object.wait() [27e86000..27e8686c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d8d6050> (a org.jboss.mx.util.ThreadPool$Worker)
         at java.lang.Object.wait(Object.java:429)
         at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
         - locked <0x3d8d6050> (a org.jboss.mx.util.ThreadPool$Worker)
    "Thread-33" daemon prio=1 tid=0x08bc47b8 nid=0xcf9 in Object.wait() [27f07000..27f0786c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d8d4780> (a org.jboss.mx.util.ThreadPool$Worker)
         at java.lang.Object.wait(Object.java:429)
         at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
         - locked <0x3d8d4780> (a org.jboss.mx.util.ThreadPool$Worker)
    "Thread-32" daemon prio=1 tid=0x08407010 nid=0xcf9 in Object.wait() [2d2af000..2d2af86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d895fa8> (a org.jboss.mx.util.ThreadPool$Worker)
         at java.lang.Object.wait(Object.java:429)
         at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:243)
         - locked <0x3d895fa8> (a org.jboss.mx.util.ThreadPool$Worker)
    "Thread-31" prio=1 tid=0x084065d0 nid=0xcf9 in Object.wait() [27bfc000..27bfc86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d895bd8> (a java.lang.Object)
         at java.lang.Object.wait(Object.java:429)
         at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:731)
         - locked <0x3d895bd8> (a java.lang.Object)
    "Thread-30" prio=1 tid=0x082ef440 nid=0xcf9 in Object.wait() [27c7d000..27c7d86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d852290> (a org.jboss.mx.util.RunnableScheduler)
         at org.jboss.mx.util.RunnableScheduler.waitOutstanding(RunnableScheduler.java:188)
         - locked <0x3d852290> (a org.jboss.mx.util.RunnableScheduler)
         at org.jboss.mx.util.RunnableScheduler.run(RunnableScheduler.java:93)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-29" prio=1 tid=0x085357c0 nid=0xcf9 in Object.wait() [27cfe000..27cfe86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d8502e8> (a org.jboss.mx.util.RunnableScheduler)
         at org.jboss.mx.util.RunnableScheduler.waitOutstanding(RunnableScheduler.java:188)
         - locked <0x3d8502e8> (a org.jboss.mx.util.RunnableScheduler)
         at org.jboss.mx.util.RunnableScheduler.run(RunnableScheduler.java:93)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-28" prio=1 tid=0x08fc8d58 nid=0xcf9 in Object.wait() [27d7f000..27d7f86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d84d6b0> (a org.jboss.mx.util.RunnableScheduler)
         at org.jboss.mx.util.RunnableScheduler.waitOutstanding(RunnableScheduler.java:188)
         - locked <0x3d84d6b0> (a org.jboss.mx.util.RunnableScheduler)
         at org.jboss.mx.util.RunnableScheduler.run(RunnableScheduler.java:93)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-27" daemon prio=1 tid=0x08a3c5f0 nid=0xcf9 runnable [27e05000..27e0586c]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:129)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
         - locked <0x3d8494b8> (a java.io.BufferedInputStream)
         at com.sun.jndi.ldap.Connection.run(Connection.java:780)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-26" daemon prio=1 tid=0x08829d80 nid=0xcf9 in Object.wait() [2bf3a000..2bf3a86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d83f190> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0x3d83f190> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "RMI LeaseChecker" daemon prio=1 tid=0x08d475b0 nid=0xcf9 waiting on condition [29bff000..29bff86c]
         at java.lang.Thread.sleep(Native Method)
         at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:307)
         at java.lang.Thread.run(Thread.java:534)
    "SessionScavenger" daemon prio=1 tid=0x08b8c420 nid=0xcf9 waiting on condition [2a1ff000..2a1ff86c]
         at java.lang.Thread.sleep(Native Method)
         at org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:428)
    "SessionScavenger" daemon prio=1 tid=0x082f2760 nid=0xcf9 waiting on condition [2ab72000..2ab7286c]
         at java.lang.Thread.sleep(Native Method)
         at org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:428)
    "UILServerILService Accept Thread" prio=1 tid=0x2cc3e9c0 nid=0xcf9 runnable [2abfb000..2abfb86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d63d830> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "OIL2 Worker Server" prio=1 tid=0x2dfe0df0 nid=0xcf9 runnable [2ac7c000..2ac7c86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d63d958> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.jboss.mq.il.oil2.OIL2ServerILService.run(OIL2ServerILService.java:366)
         at java.lang.Thread.run(Thread.java:534)
    "UIL Worker" prio=1 tid=0x2ccfca90 nid=0xcf9 runnable [2acfd000..2acfd86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d63da18> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.jboss.mq.il.uil.UILServerILService.run(UILServerILService.java:184)
         at java.lang.Thread.run(Thread.java:534)
    "OIL Worker Server" prio=1 tid=0x2bab0a70 nid=0xcf9 runnable [2ad7e000..2ad7e86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d63dad8> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.jboss.mq.il.oil.OILServerILService.run(OILServerILService.java:471)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-25" prio=1 tid=0x2cc9ef88 nid=0xcf9 in Object.wait() [2adff000..2adff86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d63db88> (a java.util.LinkedList)
         at java.lang.Object.wait(Object.java:429)
         at org.jboss.resource.connectionmanager.PoolFiller$1.run(PoolFiller.java:75)
         - locked <0x3d63db88> (a java.util.LinkedList)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-24" prio=1 tid=0x2dfe0268 nid=0xcf9 in Object.wait() [2afc4000..2afc486c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d52f320> (a java.util.ArrayList)
         at org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:66)
         - locked <0x3d52f320> (a java.util.ArrayList)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-23" daemon prio=1 tid=0x2dfce680 nid=0xcf9 in Object.wait() [2b045000..2b04586c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d52f390> (a org.jboss.util.timeout.TimeoutFactory)
         at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:503)
         - locked <0x3d52f390> (a org.jboss.util.timeout.TimeoutFactory)
         at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:27)
         at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:542)
    "Thread-22" prio=1 tid=0x2ccef150 nid=0xcf9 runnable [2b0c6000..2b0c686c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d52f400> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:216)
         at java.lang.Thread.run(Thread.java:534)
    "JBossMQ Cache Reference Softner" daemon prio=1 tid=0x2dfe1a28 nid=0xcf9 in Object.wait() [2b147000..2b14786c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d411a00> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0x3d411a00> (a java.lang.ref.ReferenceQueue$Lock)
         at org.jboss.mq.server.MessageCache.run(MessageCache.java:233)
         at java.lang.Thread.run(Thread.java:534)
    "RMI TCP Accept-0" daemon prio=1 tid=0x2c388ff0 nid=0xcf9 runnable [2b1ef000..2b1ef86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d2c0f90> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
         at java.lang.Thread.run(Thread.java:534)
    "SessionScavenger" daemon prio=1 tid=0x2c3864e0 nid=0xcf9 waiting on condition [2b270000..2b27086c]
         at java.lang.Thread.sleep(Native Method)
         at org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:428)
    "PoolThread-4" prio=1 tid=0x2c384e10 nid=0xcf9 in Object.wait() [2b2f1000..2b2f186c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c10c8> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c10c8> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-3" prio=1 tid=0x2ba1d4b0 nid=0xcf9 in Object.wait() [2b372000..2b37286c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1120> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1120> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-2" prio=1 tid=0x2ba1c960 nid=0xcf9 in Object.wait() [2b3f3000..2b3f386c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1178> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1178> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-1" prio=1 tid=0x2ba1be10 nid=0xcf9 in Object.wait() [2b474000..2b47486c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c11d0> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c11d0> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-0" prio=1 tid=0x2ba1b2c0 nid=0xcf9 in Object.wait() [2b4f5000..2b4f586c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1228> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1228> (a org.mortbay.util.ThreadPool$PoolThread)
    "Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8009]" prio=1 tid=0x2ba1af70 nid=0xcf9 runnable [2b576000..2b57686c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d2c12d8> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:346)
         at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:507)
    "PoolThread-9" prio=1 tid=0x2ba1a420 nid=0xcf9 in Object.wait() [2b5f7000..2b5f786c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c13a8> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c13a8> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-8" prio=1 tid=0x2ba198d0 nid=0xcf9 in Object.wait() [2b678000..2b67886c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1400> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1400> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-7" prio=1 tid=0x2c3a4ad8 nid=0xcf9 in Object.wait() [2b6f9000..2b6f986c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1458> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1458> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-6" prio=1 tid=0x2c3a4788 nid=0xcf9 in Object.wait() [2b77a000..2b77a86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c14b0> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c14b0> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-5" prio=1 tid=0x2c3a3c58 nid=0xcf9 in Object.wait() [2b7fb000..2b7fb86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1508> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1508> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-4" prio=1 tid=0x2ba8bfe8 nid=0xcf9 in Object.wait() [2b87c000..2b87c86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1560> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1560> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-3" prio=1 tid=0x2ba78040 nid=0xcf9 in Object.wait() [2b8fd000..2b8fd86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c15b8> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c15b8> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-2" prio=1 tid=0x2ba69c18 nid=0xcf9 in Object.wait() [2b97e000..2b97e86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1610> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1610> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-1" prio=1 tid=0x2ba69a08 nid=0xcf9 in Object.wait() [2b9ff000..2b9ff86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c1668> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c1668> (a org.mortbay.util.ThreadPool$PoolThread)
    "PoolThread-0" prio=1 tid=0x2ba7ebc8 nid=0xcf9 in Object.wait() [2bb8e000..2bb8e86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d2c16c0> (a org.mortbay.util.ThreadPool$PoolThread)
         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:450)
         - locked <0x3d2c16c0> (a org.mortbay.util.ThreadPool$PoolThread)
    "Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8020]" prio=1 tid=0x2ba7ea18 nid=0xcf9 runnable [2bc0f000..2bc0f86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d2c1770> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:346)
         at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:507)
    "SessionScavenger" daemon prio=1 tid=0x2ba8bb18 nid=0xcf9 waiting on condition [2bd90000..2bd9086c]
         at java.lang.Thread.sleep(Native Method)
         at org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:428)
    "Thread-4" daemon prio=1 tid=0x2ba8c6f0 nid=0xcf9 in Object.wait() [2beb9000..2beb986c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d23d438> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0x3d23d438> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "GC Daemon" daemon prio=1 tid=0x2c32cd20 nid=0xcf9 in Object.wait() [2bfbb000..2bfbb86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d0b4b28> (a sun.misc.GC$LatencyLock)
         at sun.misc.GC$Daemon.run(GC.java:100)
         - locked <0x3d0b4b28> (a sun.misc.GC$LatencyLock)
    "RMI Reaper" prio=1 tid=0x2c325c88 nid=0xcf9 in Object.wait() [2c03c000..2c03c86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d0b4078> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0x3d0b4078> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-3" daemon prio=1 tid=0x2c33be48 nid=0xcf9 in Object.wait() [2c0bd000..2c0bd86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d0b4378> (a java.util.TaskQueue)
         at java.lang.Object.wait(Object.java:429)
         at java.util.TimerThread.mainLoop(Timer.java:403)
         - locked <0x3d0b4378> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "RMI TCP Accept-0" daemon prio=1 tid=0x2c33b8e0 nid=0xcf9 runnable [2c13e000..2c13e86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d0b40d8> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
         at java.lang.Thread.run(Thread.java:534)
    "Thread-2" daemon prio=1 tid=0x2c340060 nid=0xcf9 runnable [2c1bf000..2c1bf86c]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0x3d0b1730> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at org.jboss.web.WebServer.run(WebServer.java:258)
         at org.jboss.web.ThreadPool$Worker.run(ThreadPool.java:163)
    "ScannerThread" daemon prio=1 tid=0x2c3403d8 nid=0xcf9 waiting on condition [2cdc7000..2cdc786c]
         at java.lang.Thread.sleep(Native Method)
         at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:217)
         at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:190)
    "Thread-0" daemon prio=1 tid=0x2c327b00 nid=0xcf9 in Object.wait() [2c240000..2c24086c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d05d2d0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0x3d05d2d0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "DestroyJavaVM" prio=1 tid=0x2df28218 nid=0xcf9 waiting on condition [0..bfffccc4]
    "Signal Dispatcher" daemon prio=1 tid=0x080c33f8 nid=0xcf9 waiting on condition [0..0]
    "Finalizer" daemon prio=1 tid=0x080bed60 nid=0xcf9 in Object.wait() [2e3fc000..2e3fc86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d0b5930> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0x3d0b5930> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=1 tid=0x080be9d0 nid=0xcf9 in Object.wait() [2e47d000..2e47d86c]
         at java.lang.Object.wait(Native Method)
         - waiting on <0x3d0b52f8> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:429)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
         - locked <0x3d0b52f8> (a java.lang.ref.Reference$Lock)

    have you checked bugs.sun.com to see if this has been reported?
    And if not, file one and attach your dump for them to look into it.

  • Waiting for monitor enty messages in thread dump

    Hi,
    I've a thread dump of a container in a Oracle Application Server 9.0.4.1 (running 1.4.2_07) right before restart because off low memory and a lot of threads in this dump are in this state:
    "AJPRequestHandler-ApplicationServerThread-1588" prio=1 tid=0x37634bf8 nid=0x1982 waiting for monitor entry [43f22000..43f2386c]
    can anyone tell me what this means? Google didn't reveal anything useful.
    Another thing I don't understand although it seems it doesn't do any harm:
    "AJPRequestHandler-ApplicationServerThread-1630" prio=1 tid=0x09011a68 nid=0x1982 in Object.wait() [40276000..4027686c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x6624e628> (a java.util.HashMap)
    at java.lang.Object.wait(Object.java:429)
    at java.net.InetAddress.checkLookupTable(InetAddress.java:1109)
    - locked <0x6624e628> (a java.util.HashMap)
    at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1048)
    at java.net.InetAddress.getLocalHost(InetAddress.java:1188)
    at com.evermind.server.http.EvermindHttpSession.encodeSession(EvermindHttpSession.java:414)
    at com.evermind.server.http.HttpApplication.createSession(HttpApplication.java:1599)
    why is this thread waiting for a lock itself has?
    Thanx for any help and hints in advance.
    Regards,
    Ingo

    Hi,
    I've a follow-up question, I have the following thread in several thread dumps:
    "AJPRequestHandler-ApplicationServerThread-717" prio=1 tid=0x4046b988 nid=0x3dd3 waiting for monitor entry [3845e000..3845e86c]
         at java.net.Inet4Address.getAddress(Inet4Address.java:287)
         at com.evermind.server.http.StandardRequestLogger.appendIP(StandardRequestLogger.java:187)
         at com.evermind.server.http.StandardRequestLogger.log(StandardRequestLogger.java:45)
         - locked <0x663b2088> (a com.evermind.server.http.StandardRequestLogger)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:906)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    What is the monitor this thread is waiting for? According to the source in Inet4Address at line 287 the method tries to create a new byte array... is there some system monitor which can't be seen in the thread dump?
    This thread holds a lock (0x663b2088) 46 other threads are waiting for...
    Thanx in advance for any hints.
    Regards,
    Ingo

  • The thread dump has a deadlock ?

    We're experiencing a problem with the OC4J server hanging when some we unknow operation . I can see it's a connection pooling problem based on a thread dump. What I can't determine is why it doesn't release a resource in the first place. I've included some code to show where it is hung.
    ================================================
    "AJPRequestHandler-ApplicationServerThread-17" tid=0x8775468 nid=0x2f3d prio=1 waiting for monitor entry
         at oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:4502)
         - waiting to lock<0x55805618> (a oracle.jbo.JboSyncLock), locked by AJPRequestHandler-ApplicationServerThread-135
         at oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:4479)
         at oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:6818)
         at oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:3886)
         at oracle.jbo.common.ampool.ApplicationPoolImpl.doManagedCheckin(ApplicationPoolImpl.java:2131)
         at oracle.jbo.common.ampool.ApplicationPoolImpl.releaseApplicationModule(ApplicationPoolImpl.java:1211)
         at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:717)
         - locked<0x557fa248> (a oracle.jbo.common.ampool.SessionCookieImpl$SessionCookieLock)
         at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:629)
         at oracle.jbo.common.ampool.SessionCookieImpl.releaseApplicationModule(SessionCookieImpl.java:611)
         at oracle.jheadstart.model.bc4j.ApplicationModuleProvider.releaseConnection(ApplicationModuleProvider.java:477)
         at oracle.jheadstart.model.bc4j.ApplicationModuleProvider.releaseRequestScopedConnections(ApplicationModuleProvider.java:503)
         at oracle.cle.persistence.ConnectionManager.releaseConnections(ConnectionManager.java:999)
         - locked<0x50e47900> (a oracle.cle.persistence.ConnectionManager)
         at oracle.cle.persistence.ConnectionManager.releaseRequestScopedConnections(ConnectionManager.java:939)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:645)
         at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
         at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
         at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
         at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    "AJPRequestHandler-ApplicationServerThread-135" tid=0x8064ac8 nid=0x2f3d prio=1 waiting for monitor entry
         at oracle.cle.persistence.ConnectionManager.getConnection(ConnectionManager.java:722)
         - waiting to lock<0x50e47900> (a oracle.cle.persistence.ConnectionManager), locked by AJPRequestHandler-ApplicationServerThread-17
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:308)
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:148)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getConnection(DataObjectHandlerImpl.java:355)
         at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.getApplicationModule(DataObjectImpl.java:761)
         at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.doneRollback(DataObjectImpl.java:855)
         at oracle.jbo.server.DBTransactionImpl.sendEvent(DBTransactionImpl.java:5501)
         at oracle.jbo.server.DBTransactionImpl.fireTransactionStateEvent(DBTransactionImpl.java:5568)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         at oracle.jbo.server.DBTransactionImpl.rollback(DBTransactionImpl.java:2257)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         - locked<0x55805618> (a oracle.jbo.JboSyncLock)
         at oracle.jheadstart.model.bc4j.JhsDBTransactionImpl.rollback(JhsDBTransactionImpl.java:186)
         at oracle.jheadstart.model.bc4j.handler.DataObjectImpl.doneCommit(DataObjectImpl.java:832)
         at oracle.jbo.server.DBTransactionImpl.sendEvent(DBTransactionImpl.java:5494)
         at oracle.jbo.server.DBTransactionImpl.fireTransactionStateEvent(DBTransactionImpl.java:5568)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         at oracle.jbo.server.DBTransactionImpl.commitInternal(DBTransactionImpl.java:1989)
         at oracle.jbo.server.DBTransactionImpl.commit(DBTransactionImpl.java:2046)
         - locked<0x55805620> (a oracle.jheadstart.model.bc4j.JhsDBTransactionImpl)
         - locked<0x55805618> (a oracle.jbo.JboSyncLock)
         at oracle.jheadstart.model.bc4j.JhsDBTransactionImpl.commit(JhsDBTransactionImpl.java:176)
         at oracle.jheadstart.model.bc4j.JhsApplicationModuleImpl.commitChanges(JhsApplicationModuleImpl.java:1167)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.commitChanges(DataObjectHandlerImpl.java:898)
         at oracle.jheadstart.controller.struts.action.JhsAction.commitChanges(JhsAction.java:484)
         at oracle.jheadstart.controller.struts.action.JhsAction.saveChanges(JhsAction.java:409)
         at oracle.jheadstart.controller.struts.action.SaveDataObject.execute(SaveDataObject.java:467)
         at oracle.jheadstart.controller.struts.action.JhsAction.execute(JhsAction.java:245)
         at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.processActionPerform(JhsRequestProcessor.java:351)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:635)
         at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
         at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
         at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
         at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    "AJPRequestHandler-ApplicationServerThread-16" tid=0x8b2b510 nid=0x2f3d prio=1 waiting for monitor entry
         at oracle.cle.persistence.ConnectionManager.getConnection(ConnectionManager.java:722)
         - waiting to lock<0x50e47900> (a oracle.cle.persistence.ConnectionManager), locked by AJPRequestHandler-ApplicationServerThread-17
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:308)
         at oracle.cle.persistence.DeploymentConnector.getConnection(DeploymentConnector.java:148)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getConnection(DataObjectHandlerImpl.java:355)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getApplicationModule(DataObjectHandlerImpl.java:1011)
         at oracle.jheadstart.model.bc4j.handler.DataObjectHandlerImpl.getDataObjectSet(DataObjectHandlerImpl.java:511)
         at oracle.jheadstart.controller.struts.action.GetDataObjectSet.retrieveSimple(GetDataObjectSet.java:474)
         at oracle.jheadstart.controller.struts.action.GetDataObjectSet.execute(GetDataObjectSet.java:390)
         at oracle.jheadstart.controller.struts.action.JhsAction.execute(JhsAction.java:245)
         at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.processActionPerform(JhsRequestProcessor.java:351)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
         at oracle.jheadstart.controller.struts.JhsRequestProcessor.process(JhsRequestProcessor.java:635)
         at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
         at oracle.jheadstart.controller.struts.JhsActionServlet.process(JhsActionServlet.java:139)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
         at oracle.jheadstart.controller.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:174)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:649)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:322)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:790)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:208)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:125)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    Sincerely from Ting-Rung

    About the general situation describe like following, thanks you help.
    Q1:Whether you could able to access the httpserver home page or any other application deployed in other OC4J container, when the server
    become unresponsive.
    Ans:Yes
    Q2:What are the kind of applications deployed in the OC4J container ( hrmOC4J) .
    Ans:The kind of applications is human resource application(Developer Tool is JDeveloper 9.0.5.2 and Jheadstart 9.0.5.19 and
    oracle Application server 10.1.2.0.2)
    Q3:Does server hang happen regularly or at what situation the server is becoming unresponsive ( high load, etc..)
    Ans:No,It is not regularly when when server hang happen.
    Q4:From the SR notes, the workaround you used is, restarting the OC4J container. ( what is the
    kind of Oracle Application Server instance you have installed, please let me
    know the complete version of the Oracle AS instance)
    Ans:oracle Application server 10.1.2.0.2
    Q5:Did you notice any out of memory issues when the server / particular OC4J container is hang
    Ans:No. It is not out of memory.

  • Strange thread dump on deadlock

    We've been experiencing a strange deadlock during the startup of our java application. When I run jstack on the application to investigate, I see that the AWT-EventQueue is in Object.wait(), but the thread is still marked as RUNNABLE. I've included the relevent parts of the thread dump, and I'm hoping that someone can shed some light on this issue.
    "AWT-EventQueue-0" prio=6 tid=0x5f0a2400 nid=0x19e4 in Object.wait() [0x6007e000]
           java.lang.Thread.State: RUNNABLE
            at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)
              - locked <0x1fc87448> (a java.awt.Component$AWTTreeLock)
    "Thread-63-Pool-9" prio=6 tid=0x5f1a2800 nid=0x1f54 waiting for monitor entry [0x61a9f000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at java.awt.Component.setFont(Component.java:1777)
        - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
    "Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at java.awt.Component.setFont(Component.java:1777)
        - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
          ... In my search for the solution, I've noticed [this thread|http://stackoverflow.com/questions/64989/jvm-thread-dumps-containing-monitors-without-locking-threads], which mentions that accessing a static variable may be involved. This is also the case in our application. The line in getEAOTableLite in question references a static method.

    Thank you for the quick reply. I'm not sure how I missed it, but if I had read down the stack trace a little better I would have found that the issue was that the class loader for the EaoAlertManager never completed because it was blocked by the illegal call to setFont. This would cause the AWT-EventQueue thread to go into an infinite wait call on the ClassLoader.
    "Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
       java.lang.Thread.State: BLOCKED (on object monitor)
         at java.awt.Component.setFont(Component.java:1777)
         - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
         at java.awt.Container.setFont(Container.java:1554)
         at javax.swing.JComponent.setFont(JComponent.java:2723)
         at javax.swing.LookAndFeel.installColorsAndFont(LookAndFeel.java:191)
         at javax.swing.plaf.basic.BasicPanelUI.installDefaults(BasicPanelUI.java:49)
         at javax.swing.plaf.basic.BasicPanelUI.installUI(BasicPanelUI.java:39)
         at com.ge.med.ptk.laf.CuiPanelUI.installUI(CuiPanelUI.java:53)
         at javax.swing.JComponent.setUI(JComponent.java:662)
         at javax.swing.JPanel.setUI(JPanel.java:136)
         at javax.swing.JPanel.updateUI(JPanel.java:109)
         at javax.swing.JPanel.<init>(JPanel.java:69)
         at javax.swing.JPanel.<init>(JPanel.java:92)
         at javax.swing.JPanel.<init>(JPanel.java:100)
         at javax.swing.JRootPane.createGlassPane(JRootPane.java:528)
         at javax.swing.JRootPane.<init>(JRootPane.java:348)
         at javax.swing.JDialog.createRootPane(JDialog.java:611)
         at javax.swing.JDialog.dialogInit(JDialog.java:593)
         at com.ge.med.plaf.wrapper.WJDialog.dialogInit(WJDialog.java:42)
         at javax.swing.JDialog.<init>(JDialog.java:545)
         at javax.swing.JDialog.<init>(JDialog.java:515)
         at com.ge.med.plaf.wrapper.WJDialog.<init>(WJDialog.java:424)
         at com.ge.med.platinum.gui.util.PlatinumDialog.<init>(PlatinumDialog.java:138)
         at com.ge.med.platinum.gui.util.EAOAlertManager$EAOAlertDialog.<init>(EAOAlertManager.java:450)
         at com.ge.med.platinum.gui.util.EAOAlertManager.<clinit>(EAOAlertManager.java:77)
         at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)

  • Help on this thread dump..

    hey guys, im having a problem reading this thread dump. i'm just a newbie. could you help me point what's wrong in our system based on this thread dump? there were no errors in our logs, it just hangs and we're so confused as to what the issue is. here's the thread dump:
    Full thread dump Java HotSpot(TM) Client VM (1.4.0_02-b02 mixed mode):
    "MessageWriter_370" prio=5 tid=0x3479e0 nid=0x1df waiting on monitor [94381000..94381a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a690dd88> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a690dd88> (a java.util.ArrayList)
    "MessageReader_369" prio=5 tid=0x33f888 nid=0x1de runnable [94581000..94581a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a690de38> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a690df58> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a690ddb0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_368" prio=5 tid=0x2784c8 nid=0x1dc waiting on monitor [94881000..94881a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a68e6d90> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a68e6d90> (a java.util.ArrayList)
    "MessageReader_367" prio=5 tid=0x279230 nid=0x1db waiting for monitor entry [94981000..94981a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_366" prio=5 tid=0x5c2090 nid=0x1da waiting on monitor [90e81000..90e81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a68df070> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a68df070> (a java.util.ArrayList)
    "MessageReader_365" prio=5 tid=0x27da50 nid=0x1d9 runnable [90f81000..90f81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a68cb480> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a68cb520> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a68cb408> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_364" prio=5 tid=0x5c1f00 nid=0x1d8 waiting on monitor [91281000..91281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a68ca1a0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a68ca1a0> (a java.util.ArrayList)
    "MessageReader_363" prio=5 tid=0x753a68 nid=0x1d7 waiting for monitor entry [91381000..91381a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b3d378> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a68ca178> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_362" prio=5 tid=0x5c09a8 nid=0x1d6 waiting on monitor [91481000..91481a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a6867f58> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a6867f58> (a java.util.ArrayList)
    "MessageReader_361" prio=5 tid=0x2790d8 nid=0x1d5 runnable [91581000..91581a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a6868008> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a6868128> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a6867f80> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_352" prio=5 tid=0x34f730 nid=0x1c9 waiting on monitor [94c81000..94c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a6744968> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a6744968> (a java.util.ArrayList)
    "MessageReader_351" prio=5 tid=0xd7370 nid=0x1c8 runnable [94d81000..94d81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a6744a18> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a6744b38> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a6744990> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_350" prio=5 tid=0x34fb38 nid=0x1c7 waiting on monitor [95081000..95081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a66ef950> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a66ef950> (a java.util.ArrayList)
    "MessageReader_349" prio=5 tid=0x278620 nid=0x1c6 runnable [95181000..95181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a66efa00> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a66efb20> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a66ef978> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_344" prio=5 tid=0x6a0370 nid=0x1c0 waiting on monitor [92481000..92481a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a66665a8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a66665a8> (a java.util.ArrayList)
    "MessageReader_343" prio=5 tid=0x33f730 nid=0x1bf waiting for monitor entry [92581000..92581a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_332" prio=5 tid=0x3a0d38 nid=0x1b0 waiting on monitor [93681000..93681a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a620e700> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a620e700> (a java.util.ArrayList)
    "MessageReader_331" prio=5 tid=0x69dd18 nid=0x1af waiting for monitor entry [93781000..93781a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_318" prio=5 tid=0x39fc10 nid=0x19e waiting on monitor [95c81000..95c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5f1f0f0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5f1f0f0> (a java.util.ArrayList)
    "MessageReader_317" prio=5 tid=0x69da68 nid=0x19d waiting on monitor [95d81000..95d81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at java.lang.Object.wait(Object.java:426)
         at epylon.persistence.JmsPersistenceManager.obtainLock(JmsPersistenceManager.java:144)
         - locked <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at com.presumo.jms.persistence.PersistentQueue.push(PersistentQueue.java:142)
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - locked <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_316" prio=5 tid=0x254c30 nid=0x19c waiting on monitor [96281000..96281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5e9d7e0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5e9d7e0> (a java.util.ArrayList)
    "MessageReader_315" prio=5 tid=0x69d910 nid=0x19b waiting for monitor entry [96381000..96381a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_314" prio=5 tid=0x3a1bf0 nid=0x19a waiting on monitor [99081000..99081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5e24348> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5e24348> (a java.util.ArrayList)
    "MessageReader_313" prio=5 tid=0x69d7b8 nid=0x199 waiting for monitor entry [99181000..99181a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_312" prio=5 tid=0x64c608 nid=0x198 waiting on monitor [99681000..99681a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5df1a88> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5df1a88> (a java.util.ArrayList)
    "MessageReader_311" prio=5 tid=0x5d00a0 nid=0x197 waiting for monitor entry [99781000..99781a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b3d538> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a5df1a60> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_310" prio=5 tid=0x3a0be0 nid=0x195 waiting on monitor [91a81000..91a81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5dd5370> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5dd5370> (a java.util.ArrayList)
    "MessageReader_309" prio=5 tid=0x3a7108 nid=0x194 waiting for monitor entry [91b81000..91b81a28]
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:150)
         - waiting to lock <a6a87030> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a5dd5348> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_308" prio=5 tid=0x3a1290 nid=0x193 waiting on monitor [91e81000..91e81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5d9c338> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5d9c338> (a java.util.ArrayList)
    "MessageReader_307" prio=5 tid=0x42ce10 nid=0x192 runnable [91f81000..91f81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5d9c3e8> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5d9c508> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a5d9c360> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_302" prio=5 tid=0x4beef8 nid=0x18c waiting on monitor [93a81000..93a81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5cce5c8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5cce5c8> (a java.util.ArrayList)
    "MessageReader_301" prio=5 tid=0x577eb0 nid=0x18b runnable [93b81000..93b81a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5cce678> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5cce798> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a5cce5f0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_298" prio=5 tid=0x413b08 nid=0x187 waiting on monitor [93c81000..93c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5c410d0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5c410d0> (a java.util.ArrayList)
    "MessageReader_297" prio=5 tid=0x353b58 nid=0x186 waiting for monitor entry [93d81000..93d81a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_296" prio=5 tid=0x592028 nid=0x185 waiting on monitor [94081000..94081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5bd4cd8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5bd4cd8> (a java.util.ArrayList)
    "MessageReader_295" prio=5 tid=0x66d660 nid=0x184 waiting for monitor entry [94181000..94181a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_294" prio=5 tid=0x1fa4e0 nid=0x183 waiting on monitor [94281000..94281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5b7e880> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5b7e880> (a java.util.ArrayList)
    "MessageReader_293" prio=5 tid=0x34f9e0 nid=0x182 waiting for monitor entry [94481000..94481a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_292" prio=5 tid=0x34de60 nid=0x17f waiting on monitor [91081000..91081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5b502a8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5b502a8> (a java.util.ArrayList)
    "MessageReader_291" prio=5 tid=0x592430 nid=0x17e runnable [91181000..91181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5b3cdb0> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5b3ce50> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a5b3cd38> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_282" prio=5 tid=0x57d608 nid=0x173 waiting on monitor [92c81000..92c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a59cd8b8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a59cd8b8> (a java.util.ArrayList)
    "MessageReader_281" prio=5 tid=0x1f5420 nid=0x172 waiting for monitor entry [92d81000..92d81a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_262" prio=5 tid=0x3a2ad8 nid=0x15b waiting on monitor [91c81000..91c81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a576a760> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a576a760> (a java.util.ArrayList)
    "MessageReader_261" prio=5 tid=0x3a20a0 nid=0x15a waiting for monitor entry [91d81000..91d81a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b39f60> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a576a738> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_258" prio=5 tid=0x4129b0 nid=0x157 waiting on monitor [92a81000..92a81a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5716ab8> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5716ab8> (a java.util.ArrayList)
    "MessageReader_257" prio=5 tid=0x581bc0 nid=0x156 waiting for monitor entry [92b81000..92b81a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_256" prio=5 tid=0x5873e8 nid=0x155 waiting on monitor [92081000..92081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a56f6278> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a56f6278> (a java.util.ArrayList)
    "MessageReader_255" prio=5 tid=0x587290 nid=0x154 runnable [92181000..92181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a56e2f98> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a56e2ff0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a56e2f38> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_252" prio=5 tid=0x5806f0 nid=0x150 waiting on monitor [92881000..92881a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a567f138> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a567f138> (a java.util.ArrayList)
    "MessageReader_251" prio=5 tid=0x57fd98 nid=0x14f waiting for monitor entry [92981000..92981a28]
         at com.presumo.jms.router.RouterAdapter.queueMessages(RouterAdapter.java:290)
         - waiting to lock <a3a38bb8> (a com.presumo.jms.persistence.PersistentQueue)
         at com.presumo.jms.router.Router.routeMessages(Router.java:215)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:353)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_214" prio=5 tid=0x30f950 nid=0x124 waiting on monitor [95281000..95281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a529f260> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a529f260> (a java.util.ArrayList)
    "MessageReader_213" prio=5 tid=0x55faa8 nid=0x123 waiting for monitor entry [95381000..95381a28]
         at epylon.persistence.JmsPersistenceManager.persist(JmsPersistenceManager.java:171)
         - waiting to lock <a3a38d28> (a epylon.persistence.JmsPersistenceManager)
         at epylon.persistence.JmsPersistenceManager.processMsgStatus(JmsPersistenceManager.java:337)
         at com.presumo.jms.message.AckHelper.completedAck(AckHelper.java:158)
         - locked <a6b683a8> (a com.presumo.jms.message.AckHelper)
         at com.presumo.jms.router.PersistentAckHandler.handleAcks(PersistentAckHandler.java:247)
         - locked <a529f238> (a com.presumo.jms.router.PersistentAckHandler)
         at com.presumo.jms.router.RemoteSession.receiveMessages(RemoteSession.java:319)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:561)
    "MessageWriter_210" prio=5 tid=0x4dd0a8 nid=0x11f waiting on monitor [93081000..93081a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a52692c0> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a52692c0> (a java.util.ArrayList)
    "MessageReader_209" prio=5 tid=0x4c2fb8 nid=0x11e runnable [93181000..93181a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5256468> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5256508> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a52563f0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_208" prio=5 tid=0x4c6de0 nid=0x11d waiting on monitor [93481000..93481a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a5241588> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a5241588> (a java.util.ArrayList)
    "MessageReader_207" prio=5 tid=0x5c2220 nid=0x11c runnable [93581000..93581a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a5241638> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a5241758> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at java.io.DataInputStream.readInt(DataInputStream.java:392)
         at com.presumo.jms.plugin.implementation.transport.tcp.TransportImpl.receiveMessages(TransportImpl.java:191)
         - locked <a52415b0> (a java.lang.String)
         at com.presumo.jms.router.RemoteSession$MessageReader.run(RemoteSession.java:559)
    "MessageWriter_188" prio=5 tid=0x642b88 nid=0x105 waiting on monitor [92281000..92281a28]
         at java.lang.Object.wait(Native Method)
         - waiting on <a50e4248> (a java.util.ArrayList)
         at com.presumo.jms.router.RemoteSession$MessageWriter.run(RemoteSession.java:641)
         - locked <a50e4248> (a java.util.ArrayList)
    "MessageReader_187" prio=5 tid=0x1f92e8 nid=0x104 runnable [92381000..92381a28]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA6275)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         - locked <a50e42f8> (a java.lang.Object)
         at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
         at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA6275)
         - locked <a50e4418> (a com.sun.net.ssl.internal.ssl.AppInputStream)
         at ja

    Here's the lengthy code :)
    package com.presumo.jms.router;
    import com.presumo.jms.message.JmsMessage;
    import com.presumo.jms.message.SystemMessageConstants;
    import com.presumo.jms.plugin.implementation.MemoryMessageQueue;
    import com.presumo.jms.plugin.transport.Transport;
    import com.presumo.jms.selector.JmsOperand;
    import com.presumo.jms.selector.Parser;
    import com.presumo.jms.JmsServer;
    import com.presumo.jms.persistence.PersistentQueue;
    import com.presumo.jms.resources.Resources;
    import com.presumo.util.log.Logger;
    import com.presumo.util.log.LoggerFactory;
    import java.io.IOException;
    import java.util.ArrayList;
    import java.util.LinkedList;
    import javax.jms.InvalidSelectorException;
    import javax.jms.JMSException;
    import epylon.bean.MessageStatusBean;
    import epylon.persistence.JmsPersistenceManager;
    import epylon.util.*;
    import org.apache.log4j.Category;
    * Class that sits between the Router and the Transport layer in
    * terms of message flow. Essentially encapsulates the remote
    * connection from the router, as well as handling the persistent
    * message acknowledgment scheme.
    * @author Dan Greff
    public class RemoteSession implements RoutingTarget
    /** Largest batch of messages that will be bundled together **/
    protected static final int MSG_BATCH_SIZE = 2000;
    /** Time that between sending batches of messages **/
    protected static final long latency = 500;
    /** Property name for sending filters via system messages **/
    protected static final String FILTER_PROPERTY = "JMSX_FILTER";
    /** Property name for sending acks via system messages **/
    protected static final String ACKS_PROPERTY = "JMSX_ACKS";
    /** Filter sent to this JVM from the other side of the connection
    * which indicates what messages the router on the other side is
    * interested in.
    protected JmsOperand remoteFilter;
    /** Local filter which the represents what messages the local
    * router is interested in. This filter is sent to other routers.
    protected JmsOperand localFilter;
    /** Router this RemoteSession is connected to **/
    protected Router router = null;
    /** Interface which messages are sent to and received from the network **/
    protected Transport transport;
    /** Listener to be notified of if the connection is lost **/
    protected ConnectionListener connxListener;
    /** Handles acknowledgement logic for persistent messages **/
    protected PersistentAckHandler persistentAckHandler;
    /** Instance of inner class that reads message from the Transport **/
    protected MessageReader msgReader;
    /** Instance of inner class that writes message from the Transport **/
    protected MessageWriter msgWriter;
    /** Set unique ID assigned to the remote session by the router **/
    protected int targetID;
    /** Local copy of the Parser singleton **/
    protected final Parser parser = Parser.getInstance();
    /** Messages queued up to be send **/
    protected final ArrayList outbox = new ArrayList();
         protected static int connectionLostThreadCount = 0;
         //Remote site name.
         private String remoteSite = null;
         private String siteTransportID = null;
         private JmsPersistenceManager persistentManager;
         //routeIncompletedMsgs
         protected boolean incompletedMsgsRouted = false;
         //No need to persist remote filter for EpylonRemote session
         protected boolean connectToJmsSession = false;
         static public String FALSE = "FALSE";
         static public String TRUE = "TRUE";
         private static int THREAD_COUNT = 1;
         private static Category logger = Category.getInstance( RemoteSession.class.getName() );
    // Constructors //
         public RemoteSession(Router router,
    Transport transport,
                                  String remoteSite,
                                  ConnectionListener listener) throws Exception{     
              this.router = router;
              this.transport = transport;
              this.remoteSite = remoteSite;
              this.connxListener = listener;
              siteTransportID = transport.getConnectionClientID();
              try {
                   remoteFilter = parser.parseFilter("true");
              } catch (InvalidSelectorException ise) {}
              persistentAckHandler = new PersistentAckHandler(this);
              router.addTarget(this);
              persistentManager = JmsPersistenceManager.getInstance();
              start();
    // Public methods //
         public void setTransport(Transport t){
    // TODO:: Add error checking to make sure this is only being called
    // as a result of a reconnection. DTG
              this.transport = t;
         public synchronized void start(){
    if (msgReader == null) {
    msgReader = new MessageReader(THREAD_COUNT);
    msgReader.start();
         THREAD_COUNT ++;
    if (msgWriter == null) {
    msgWriter = new MessageWriter(THREAD_COUNT);
    msgWriter.start();
         THREAD_COUNT ++;
    public synchronized void stop()
    if (msgReader != null) {
    msgReader.stopReader();
    msgReader = null;
    if (msgWriter != null) {
    msgWriter.stopWriter();
    msgWriter = null;
    public void close()
    this.stop();
    if (transport != null) {
    transport.close();
    transport = null;
    router.removeTarget(this);
    // Begin implementation of RoutingTarget interface methods
    public void setTargetID(int id)
    this.targetID = id;
         public JmsOperand getRoutingFilter(){
              JmsOperand retval = remoteFilter;
              return retval;
         public void setRemoteRoutingFilter(JmsOperand filter, boolean add){
              if (filter != localFilter){
                   JmsMessage msg;
                   String filterString;               
                   try {
                        if (persistentManager == null)
                             persistentManager = JmsPersistenceManager.getInstance();
                        if(connectToJmsSession || !JmsServer.isJmsSessionInitComplete())
                             filterString = parser.unparse(filter);               
                        else                                        
                             filterString = persistentManager.prepareRemoteFilterForSite(remoteSite);
                        logger.debug("setRemoteRoutingFilter::remoteSite =>"+ remoteSite + "\n "+ filterString);
                        if (!defaultFilter(filterString.toUpperCase())){
                             msg = new JmsMessage(router.getName());
                             msg.setJMSSystemMsgType(SystemMessageConstants.REMOTE_FILTER_CHANGE_TYPE);                                             
                             msg.setStringProperty(FILTER_PROPERTY, filterString);
                             msg.setStringProperty(MessageStatusBean.ORIGINATING_SITE_NAME, JmsServer.siteName());
                             localFilter = filter;
                             if (connectToJmsSession && !JmsServer.isJmsSessionInitComplete())
                                  JmsServer.addLocalFilter();
                             if ( !connectToJmsSession || !JmsServer.isAllLocalFilterProcessed())
                                  queueOutboxMessage(msg);                         
                   } catch (Exception e) {
                        // shouldn't happen with a in memory message queue but, dump it anyways
                        logger.error(e);
    public boolean needsFilterUpdates()
    return true;
    * Called by the router to give a potential message to this
    * routing target.
    * Implementes RoutingTarget.takeMessage();
    public JmsMessage takeMessage(JmsMessage msg)
    throws Exception{
         logger.debug("RemoteSession:takeMessage connectToJmsSession=>"+ connectToJmsSession);
         boolean taken = false;
         if (msg.getSendingTargetID() != this.targetID && incompletedMsgsRouted) { // prevent backtracking
              if (connectToJmsSession || msg.getJMSDeliveryMode() != javax.jms.DeliveryMode.PERSISTENT ){ 
                   taken = parser.evaluate(remoteFilter, msg);
              } else
                   taken = msg.getAckHelper().routeMsgtoRemoteSite( remoteSite);
              if (taken) {
                   if (msg.getJMSDeliveryMode() == javax.jms.DeliveryMode.PERSISTENT)
                        persistentAckHandler.handleOutgoingMsg(msg);               
                   synchronized(outbox) {
                        outbox.add(msg);
                        outbox.notifyAll();
         return msg;
    // End implementaiton of RoutingTarget interface methods
    * Called by MessageReader.run() when messages come from the transport.
    public void receiveMessages(JmsMessage [] msgs)
    throws JMSException{
    int length = msgs.length;
    int msgsRemoved = 0;
    for (int i=0; i < length; ++i) {
              logger.debug("ReceiveMessages from "+ remoteSite + msgs);
              msgs[i].setStringProperty(MessageStatusBean.ORIGINATING_SITE_NAME, remoteSite);
              boolean systemMsg = false;
              int type = msgs[i].getJMSSystemMsgType();
              switch (type) {
              case(SystemMessageConstants.REMOTE_FILTER_CHANGE_TYPE):
                   handleRemoteFilterMsg(msgs[i]);
                   systemMsg = true;
                   break;
    case(SystemMessageConstants.ACKS_MSG_TYPE):
                   String acks = (String) msgs[i].getObjectProperty(ACKS_PROPERTY);
                   try{
                        persistentAckHandler.handleAcks(acks);
                   }catch(Exception e){
                        //TODO: Handle exception
                        logger.error(e);
                   systemMsg = true;
                   break;
              if (systemMsg) {
                   ++msgsRemoved;
                   msgs[i] = null;
              }else {               
                   // Filter out duplicates
                   boolean duplicate = persistentAckHandler.isDuplicate(msgs[i]);
                   if (duplicate) {
                        msgs[i] = null;
                        ++msgsRemoved;
                   else {
                        // Valid user message... start acknowledgment tracking
                        msgs[i].setSendingTargetID( targetID );          
                        persistentAckHandler.setOriginator(msgs[i]);
    if (msgsRemoved > 0) {
              msgs = collapseMsgArray(msgs, length - msgsRemoved);
    try {
              router.routeMessages(msgs);
              persistentAckHandler.handleIncomingMsgs(msgs);
    } catch (Exception e) {
    // TODO:: add some error handling
    logger.error("An exception occurred while routing msgs: \n", e);
         public void acksAvailable(){
              synchronized(outbox) {
                   outbox.notifyAll();
    // Package methods //
    // Private methods //
    protected void queueOutboxMessage(JmsMessage msg)
    synchronized(outbox) {
    outbox.add(msg);
    outbox.notifyAll();
         * Handle remote filter message.
         protected void handleRemoteFilterMsg(JmsMessage msg)
              throws JMSException{       
              boolean remoteFilterPersist = false;
              String filter = (String) msg.getObjectProperty(FILTER_PROPERTY);
              if (!connectToJmsSession && !JmsServer.isJmsSessionInitComplete()){
                   try{
                        logger.debug("Sleep");
                        Thread.currentThread().sleep(3000);
                   }catch(Exception e){
                        logger.error(e);
              if (!defaultFilter(filter.toUpperCase())){
              if (remoteSite.indexOf(JmsServer.siteName()) != -1 &&
                   !JmsServer.isAllLocalFilterProcessed() &&
                   !connectToJmsSession )
                   JmsServer.processLocalFilter();
              if (!connectToJmsSession){
                   persistRemoteFilter(remoteSite, filter);
                   remoteFilterPersist = true;
              try {
                   remoteFilter = parser.parseFilter(filter);
                   if (remoteFilterPersist){
                        if (!incompletedMsgsRouted &&
                             !connectToJmsSession ){          
                             routeMessagesToRemoteSite();
                             incompletedMsgsRouted = true;
                        logger.debug("RemoteSession::handleRemoteFilterMsg "+ remoteFilter);
                        if (!connectToJmsSession)
                        router.recalculateFilters(true);
              } catch (javax.jms.InvalidSelectorException ex) {
                   logger.error("Remote client sent invalid routing filter: " +
                                  transport.getRemoteID() + " :: " + filter);
    * Utility function to remove null values from the array
    protected JmsMessage [] collapseMsgArray(JmsMessage [] msgs, int size)
    JmsMessage [] retval = new JmsMessage[size];
    for (int i=0, j=0; i < msgs.length; ++i) {
    if (msgs[i] != null) {
    retval[j] = msgs[i];
    ++j;
    return retval;
    * Called by the internal methods reading and sending messages
    * to the transport layer when an IOException is thrown from
    * the transport layer. This indicates a connection loss or
    * corruption.
    * TODO:: explain why this needs to spawn off as a thread
         protected void connectionLost(IOException ioe){
              //Spawn off a thread to connect remote server when connection lost.
              //This RemoteSession is accepted connection from remote client when its
              //RemoteURL is null, so there is no need to spawn off connectionLostthread
              //And close transport.
              if(getRemoteUrl() != null){
                   connectionLostThreadCount++;               
                   Thread t = new Thread("ConnectionLost Thread"+ connectionLostThreadCount) {
                             public void run() {
                                  RemoteSession.this.close();
                                  if (transport != null) {
                                       transport.close();
                                       transport = null;
                                  connxListener.connectionLost(RemoteSession.this, siteTransportID);                              
                   t.start();
              }     else{
                   if (transport != null) {
                        transport.close();
                        transport = null;
                   connxListener.connectionLost(RemoteSession.this, siteTransportID);
                   //No need to close transport because its outputstream closed
         protected JmsMessage createAckMessage(){
              JmsMessage ackMessage = null;
              String acks = persistentAckHandler.getAckString();
              if (acks != null) {
                   ackMessage = new JmsMessage(router.getName());
                   try {
                        ackMessage.setJMSSystemMsgType(SystemMessageConstants.ACKS_MSG_TYPE);
                        ackMessage.setStringProperty(ACKS_PROPERTY, acks);
                   } catch (JMSException jmsex) {}
              return ackMessage;
         // Begin Inner Class MessageReader //
         * Inner class reperesenting thread reading messages from the transport
         * implementation.
         protected class MessageReader extends Thread{
              private volatile boolean stopped = false;
              public MessageReader(int threadCount) {
                   super("MessageReader_"+ threadCount );
              public void stopReader(){
                   if (! stopped) stopped = true;
              public void run(){
                   JmsMessage [] msgs;
                   while (stopped == false) {
                        if (router.isRouterStop())
                             stopped = true;
                        try {
                             if (transport == null){                    
                             throw new IOException("Transport socket not available");                    
                             }else
                                  msgs = transport.receiveMessages();                         
                             if (msgs != null && stopped == false) {
                                  receiveMessages(msgs);
                                  msgs = null;
                        }catch(SocketCloseException sce){
                             stopped = true;
                             connectionLost(new IOException(sce.getMessage()));
                        }catch(JMSException jmse){
                             // TODO:: handle this properly
                             logger.error(jmse);
                        }catch(NullPointerException ne){
                             logger.error(ne);
                        }catch (IOException ioe) {
                             if (! stopped) {
                                  // TODO:: handle this properly
                                  logger.debug(ioe);
                                  stopped = true;
                                  connectionLost(ioe);
         // End Inner Class MessageReader //
         // Begin Inner Class MessageWriter //
         * Inner class writing messages to the Transport
         protected class MessageWriter extends Thread{
              private volatile boolean stopped = false;
              private volatile boolean sendMessages;
              public MessageWriter(int threadCount) { super("MessageWriter_"+ threadCount); }
              public final void stopWriter()
                   if (! stopped) {
                        synchronized (outbox) {
                             stopped = true;
                             outbox.notifyAll();
              public final void sendMessagesNow()
                   sendMessages = true;
              public final void run()
                   long nextSend = System.currentTimeMillis() + latency;
                   long sleepTime = 0;
                   while (!stopped) {                    
                        synchronized (outbox) {
                             sendMessages = false;                         
                             if (outbox.size() >= MSG_BATCH_SIZE) {
                                  sendMessages = true;
                             } else if(outbox.size() > 0 ||
                                       persistentAckHandler.acksAvailable() ) {
                                  long currentTime = System.currentTimeMillis();
                                  long diff = nextSend - currentTime;
                                  if (diff <= 0) {
                                       sendMessages = true;
                                  else {
                                       sleepTime = diff;
                             } else {
                                  sleepTime = 0;
                             if (sendMessages == false && !stopped) {
                                  try {
                                       outbox.wait(sleepTime);
                                  } catch (InterruptedException ie) {}
                        if (sendMessages == true) {
                             send();
                             nextSend = System.currentTimeMillis() + latency;
                             sleepTime = 0;
         private final void send(){
              logger.debug("RemoteSession:send");
              try {
                   JmsMessage [] msgs = null;
                   JmsMessage ackMessage = createAckMessage();
                   synchronized(outbox) {
                        if (ackMessage != null) {
                             outbox.add(ackMessage);
                        msgs = new JmsMessage[outbox.size()];
                        msgs = (JmsMessage[]) outbox.toArray(msgs);
                        outbox.clear();
                   if (msgs != null && msgs.length != 0 && transport != null) {
                        transport.sendMessages(msgs);
              }catch(SocketCloseException sce){
                   connectionLost(new IOException(sce.getMessage()));
              } catch (IOException ioe) {
                   logger.error(ioe);
                   connectionLost(ioe);
         private void persistRemoteFilter( String site, String filter)
              throws JMSException{
              try{
                   persistentManager.persistRemoteFilter(site, filter);
                   persistentManager.endTransaction(true);
              }catch(Exception e){
                   logger.error(e);
                   JMSException jmsex = new JMSException("Unable to parse filter");
                   jmsex.setLinkedException(e);
         * This method is called only when remotesession initially connected to
         * route messages and ack messages (stored acks only) to remote
         * site. Don't route messages to JMS core/provider when remoteSeesion
         * connectted to JmsSession and JMS core remote session.
         * @param
         private void routeMessagesToRemoteSite(){
              logger.debug("RemoteSession::routeMessagesToRemoteSite " + remoteSite);
              if (connectToJmsSession)
                   return;
              boolean taken = false;
              JmsMessage[] msgs = null;
              JmsMessage msg;
              try{
                   persistentManager.obtainLock();
                   while (msgs == null){
                        msgs = persistentManager.getPendingDeleteMsgs();     
                   int length = msgs.length;
                   for (int i=0; i<length ; i++){
                        try{
                             msg = (JmsMessage)msgs[i];     
                             if (msg.getAckHelper().needStoredAckToRemoteSite(remoteSite)){
                                  logger.debug("needStoredAckToRemoteSite" + remoteSite);
                                  persistentAckHandler.addMsgToCheckDupsMapAndAcks(msg);
                             }else if (msg.getAckHelper().needCompletedAckToRemoteSite(remoteSite)){
                                  persistentAckHandler.addMsgToSendToMapAndAcks(msg);
                             }else{
                                  taken = msg.getAckHelper().routeMsgtoRemoteSite( remoteSite);
                                  logger.debug("RemoteSession::routeMessagesToRemoteSite taken=> " + taken );
                                  if (taken) {     
                                       if (msg.getJMSDeliveryMode() == javax.jms.DeliveryMode.PERSISTENT)
                                            persistentAckHandler.handleOutgoingMsg(msg);     
                                       while (outbox.size() >= MSG_BATCH_SIZE){
                                            msgWriter.send();
                                            logger.info("Route queued messages to remote sites. Please wait....");
                                            try{
                                                 Thread.sleep(2000);
                                            }catch(Exception e){}
                                       queueOutboxMessage(msg);
                        }catch(Exception e){
                             logger.error(e);
              } finally{
                   persistentManager.releaseLock();               
              logger.debug("RemoteSession::routeMessagesToRemoteSite:Exit");
    // End Inner Class MessageReader //
         private String remoteUrl = null;
         public void setRemoteUrl(String url){
              remoteUrl = url;
         public String getRemoteUrl(){
              return remoteUrl;
         public String getRemoteSite(){
              return remoteSite;
         public void addSessionToRouter(){
              if (router != null)
                   router.addTarget(this);
         public static boolean defaultFilter(String filter){
              return filter.indexOf(TRUE) != -1 || filter.indexOf(FALSE) != -1;

  • Large Memory Usage and Frozen [thread dump attached]

    Hi,
    SQL Developer (Version 2.1.1.64, WinXP Pro, Java 1.6, Oracle 10g R2) always uses at least 550MB of memory, if left open, for more than a day, it invariably goes up to 700-800MB.
    If I close all windows and connections, virtually no memory is reclaimed.
    I clicked + beside a view to see the columns, sqldeveloper froze for a few minutes, below is the thread dump.
    2010-05-21 11:31:58
    Full thread dump Java HotSpot(TM) Client VM (14.2-b01 mixed mode):
    "CONNECTION_ALIVE" prio=6 tid=0x3891c000 nid=0x1d58 waiting for monitor entry [0x3a55f000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at oracle.jdbc.driver.T4CConnection.doPingDatabase(T4CConnection.java:3398)
    - waiting to lock <0x09ad05f8> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.PhysicalConnection.pingDatabase(PhysicalConnection.java:7074)
    at oracle.javatools.db.ora.BaseOracleDatabase.isConnectionAlive(BaseOracleDatabase.java:165)
    at oracle.javatools.db.AbstractDatabase$1.run(AbstractDatabase.java:316)
    at java.lang.Thread.run(Thread.java:619)
    "Loading Children" prio=6 tid=0x3891bc00 nid=0x12c4 waiting on condition [0x39d5f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x03eab570> (a java.util.concurrent.FutureTask$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$IdeTaskTracker.run(RaptorTaskManager.java:532)
    at java.lang.Thread.run(Thread.java:619)
    "pool-2-thread-57" prio=6 tid=0x3891b400 nid=0x2094 waiting on condition [0x35b5f000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x09b819f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:841)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1160)
    at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:416)
    at oracle.dbtools.raptor.utils.Connections.lock(Connections.java:1334)
    at oracle.dbtools.raptor.utils.Connections.lock(Connections.java:1319)
    at oracle.dbtools.raptor.utils.Connections.lock(Connections.java:1306)
    at oracle.dbtools.raptor.navigator.xml.AbstractItemInstance.lockConnection(AbstractItemInstance.java:52)
    at oracle.dbtools.raptor.navigator.xml.XmlNodeInstance.listChildren(XmlNodeInstance.java:55)
    at oracle.dbtools.raptor.navigator.ObjectNodeFilter$ObjectNodeFilterLoadTask.doWorkImpl(ObjectNodeFilter.java:78)
    at oracle.dbtools.raptor.navigator.DeferredLoadingFilter$FilterLoadTask.doWork(DeferredLoadingFilter.java:124)
    at oracle.dbtools.raptor.navigator.DeferredLoadingFilter$FilterLoadTask.doWork(DeferredLoadingFilter.java:113)
    at oracle.dbtools.raptor.backgroundTask.RaptorTask.call(RaptorTask.java:193)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$RaptorFutureTask.run(RaptorTaskManager.java:492)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
    "pool-2-thread-55" prio=6 tid=0x38d68400 nid=0xd74 runnable [0x3a15f000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.ns.Packet.receive(Packet.java:239)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:92)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:172)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1023)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:999)
    at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:584)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194)
    at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:785)
    at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:860)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1186)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3381)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3425)
    - locked <0x09ad05f8> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1490)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel.openResultSet(ResultSetTableModel.java:412)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel.fetchNext(ResultSetTableModel.java:169)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel$4.doWork(ResultSetTableModel.java:556)
    at oracle.dbtools.raptor.controls.grid.ResultSetTableModel$4.doWork(ResultSetTableModel.java:540)
    at oracle.dbtools.raptor.backgroundTask.RaptorTask.call(RaptorTask.java:193)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$RaptorFutureTask.run(RaptorTaskManager.java:492)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
    "AWT-EventQueue-0" prio=6 tid=0x354fac00 nid=0x18d4 waiting for monitor entry [0x3a45f000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at oracle.jdbc.driver.PhysicalConnection.getMetaData(PhysicalConnection.java:3891)
    - waiting to lock <0x09ad05f8> (a oracle.jdbc.driver.T4CConnection)
    at oracle.dbtools.raptor.plscope.Query.getToolTipText(Query.java:58)
    at oracle.dbtools.raptor.phighlight.TooltipPlugin$1.getToolTipText(TooltipPlugin.java:29)
    at oracle.ide.ceditor.CodeEditor.getToolTipText(CodeEditor.java:1734)
    - locked <0x17245f98> (a java.util.ArrayList)
    at oracle.javatools.editor.BasicEditorPane.getToolTipText(BasicEditorPane.java:1530)
    at javax.swing.ToolTipManager$insideTimerAction.actionPerformed(ToolTipManager.java:658)
    at javax.swing.Timer.fireActionPerformed(Timer.java:271)
    at javax.swing.Timer$DoPostEvent.run(Timer.java:201)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
    "Background Parser" prio=6 tid=0x38a43400 nid=0x23c4 waiting on condition [0x39c5f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "SwingWorker-pool-5-thread-1142" prio=6 tid=0x38868800 nid=0x1a7c waiting on condition [0x3820f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x0e77bc10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)
    "Background Parser" prio=6 tid=0x38a76c00 nid=0x1d18 waiting on condition [0x3a35f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "Background Parser" prio=6 tid=0x387e2800 nid=0xd00 waiting on condition [0x3a25f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "Background Parser" prio=6 tid=0x38752000 nid=0x2794 waiting on condition [0x39e5f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "Swing-Shell" daemon prio=6 tid=0x367f0000 nid=0xd7c waiting on condition [0x3a85f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x1069d0d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at sun.awt.shell.Win32ShellFolderManager2$ComInvoker$3.run(Win32ShellFolderManager2.java:458)
    at java.lang.Thread.run(Thread.java:619)
    "Persistence Auto Flusher" daemon prio=6 tid=0x388a5c00 nid=0x1950 in Object.wait() [0x37e0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x0e892d48> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "Thread-76" prio=6 tid=0x387aec00 nid=0x259c runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "WeakDataReference polling" prio=2 tid=0x38752800 nid=0xe94 in Object.wait() [0x3a05f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x0b6ba260> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at oracle.ide.util.WeakDataReference$Cleaner.run(WeakDataReference.java:88)
    at java.lang.Thread.run(Thread.java:619)
    "Image Animator 3" daemon prio=4 tid=0x3861b400 nid=0x2330 waiting on condition [0x3978f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at sun.awt.image.GifFrame.dispose(GifImageDecoder.java:653)
    at sun.awt.image.GifImageDecoder.produceImage(GifImageDecoder.java:230)
    at sun.awt.image.InputStreamImageSource.doFetch(InputStreamImageSource.java:246)
    at sun.awt.image.ImageFetcher.fetchloop(ImageFetcher.java:172)
    at sun.awt.image.ImageFetcher.run(ImageFetcher.java:136)
    "Background Parser" prio=6 tid=0x38348800 nid=0x978 waiting on condition [0x3988f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at oracle.dbtools.raptor.plsql.BackgroundParser$1.construct(BackgroundParser.java:112)
    at oracle.dbtools.raptor.utils.NamedSwingWorker$2.run(NamedSwingWorker.java:115)
    at java.lang.Thread.run(Thread.java:619)
    "TextBufferScavenger" prio=6 tid=0x38324800 nid=0x1af0 in Object.wait() [0x3968f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x09bb9158> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at oracle.ide.model.TextNode$FacadeBufferReference$PollingThread.run(TextNode.java:1949)
    "pool-4-thread-1" prio=6 tid=0x36960400 nid=0x1a84 waiting on condition [0x35a5f000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x08150a78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)
    "IconOverlayTracker Timer" prio=6 tid=0x38520c00 nid=0x1b48 in Object.wait() [0x3950f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x0795f290> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "Timer queue for AWT thread" daemon prio=6 tid=0x369b4000 nid=0x1f20 in Object.wait() [0x3940f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x077e1358> (a java.lang.Object)
    at java.lang.Object.wait(Object.java:485)
    at ice.util.awt.TimedAWTExecutor.nextElem(TimedAWTExecutor.java:108)
    - locked <0x077e1358> (a java.lang.Object)
    at ice.util.awt.TimedAWTExecutor.runScheduler(TimedAWTExecutor.java:130)
    at ice.util.awt.TimedAWTExecutor$1.run(TimedAWTExecutor.java:19)
    "WaitCursor-Timer" prio=6 tid=0x367c3c00 nid=0x1010 in Object.wait() [0x37b0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x077e1400> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "Native Directory Watcher" prio=2 tid=0x368d4000 nid=0x2334 runnable [0x3830f000]
    java.lang.Thread.State: RUNNABLE
    at oracle.ide.natives.NativeHandler.enterWatcherThread(Native Method)
    at oracle.ide.natives.NativeHandler$2.run(NativeHandler.java:252)
    at java.lang.Thread.run(Thread.java:619)
    "BaseTreeExplorer.NodeOpeningExecutor" prio=6 tid=0x3684b000 nid=0x8b0 waiting on condition [0x3810f000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <0x06950af0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
    at java.lang.Thread.run(Thread.java:619)
    "Scheduler" daemon prio=6 tid=0x36846400 nid=0x14bc in Object.wait() [0x37f0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at oracle.dbtools.raptor.backgroundTask.TaskLinkedList.takeNextTask(TaskLinkedList.java:47)
    - locked <0x06950c70> (a oracle.dbtools.raptor.backgroundTask.TaskLinkedList)
    at oracle.dbtools.raptor.backgroundTask.RaptorTaskManager$SchedulerThread.run(RaptorTaskManager.java:422)
    "TimerQueue" daemon prio=6 tid=0x36832800 nid=0x88c in Object.wait() [0x37d0f000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at javax.swing.TimerQueue.postExpiredTimers(TimerQueue.java:218)
    - locked <0x06950da8> (a javax.swing.TimerQueue)
    at javax.swing.TimerQueue.run(TimerQueue.java:234)
    - locked <0x06950da8> (a javax.swing.TimerQueue)
    at java.lang.Thread.run(Thread.java:619)
    "ChangeSetService" prio=2 tid=0x367fd400 nid=0x2754 in Object.wait() [0x37c0f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x06892cc0> (a oracle.jdevimpl.vcs.changeset.ChangeSetService)
    at java.lang.Object.wait(Object.java:485)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService.awaitEvents(ChangeSetService.java:178)
    - locked <0x06892cc0> (a oracle.jdevimpl.vcs.changeset.ChangeSetService)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService.eventLoop(ChangeSetService.java:199)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService.access$200(ChangeSetService.java:56)
    at oracle.jdevimpl.vcs.changeset.ChangeSetService$2.run(ChangeSetService.java:138)
    at java.lang.Thread.run(Thread.java:619)
    "TimedCache-Timer" daemon prio=6 tid=0x35d52c00 nid=0x2448 in Object.wait() [0x3624f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.util.TimerThread.mainLoop(Timer.java:483)
    - locked <0x062ab118> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "JarIndex Timer" daemon prio=6 tid=0x35556400 nid=0x2488 in Object.wait() [0x35d4f000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x062206e0> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)
    "AWT-Windows" daemon prio=6 tid=0x354dd000 nid=0x1d0c runnable [0x3595f000]
    java.lang.Thread.State: RUNNABLE
    at sun.awt.windows.WToolkit.eventLoop(Native Method)
    at sun.awt.windows.WToolkit.run(WToolkit.java:291)
    at java.lang.Thread.run(Thread.java:619)
    "AWT-Shutdown" prio=6 tid=0x354a1800 nid=0x268c in Object.wait() [0x3585f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:259)
    - locked <0x062208d8> (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:619)
    "Java2D Disposer" daemon prio=10 tid=0x354b7400 nid=0x5f4 in Object.wait() [0x3575f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x06220968> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at sun.java2d.Disposer.run(Disposer.java:125)
    at java.lang.Thread.run(Thread.java:619)
    "Low Memory Detector" daemon prio=6 tid=0x00ee1000 nid=0x2664 runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "CompilerThread0" daemon prio=10 tid=0x00edc000 nid=0x1f6c waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "Attach Listener" daemon prio=10 tid=0x00eda800 nid=0x206c runnable [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "Signal Dispatcher" daemon prio=10 tid=0x00ed9400 nid=0xf64 waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "Finalizer" daemon prio=8 tid=0x00ec6c00 nid=0x2348 in Object.wait() [0x34e7f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x061b0298> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0x00ec5800 nid=0x2484 in Object.wait() [0x34d7f000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x061b0320> (a java.lang.ref.Reference$Lock)
    "main" prio=6 tid=0x009f8000 nid=0x19e0 waiting on condition [0x00000000]
    java.lang.Thread.State: RUNNABLE
    "VM Thread" prio=10 tid=0x00ec1800 nid=0x22e0 runnable
    "VM Periodic Task Thread" prio=10 tid=0x00eec000 nid=0x2604 waiting on condition
    JNI global references: 5312
    Heap
    def new generation total 45376K, used 28499K [0x03080000, 0x061b0000, 0x061b0000)
    eden space 40384K, 67% used [0x03080000, 0x04af44c0, 0x057f0000)
    from space 4992K, 28% used [0x05cd0000, 0x05e30850, 0x061b0000)
    to space 4992K, 0% used [0x057f0000, 0x057f0000, 0x05cd0000)
    tenured generation total 604992K, used 364537K [0x061b0000, 0x2b080000, 0x2b080000)
    the space 604992K, 60% used [0x061b0000, 0x1c5ae548, 0x1c5ae600, 0x2b080000)
    compacting perm gen total 57344K, used 57260K [0x2b080000, 0x2e880000, 0x33080000)
    the space 57344K, 99% used [0x2b080000, 0x2e86b348, 0x2e86b400, 0x2e880000)
    No shared spaces configured.
    Edited by: mdaly on May 21, 2010 11:42 AM

    The cause of hang could be figured out from the thread dump.
    I have fixed that in development code. It was happening from PL/SQL editor trying to create tooltip which locked up the connection.
    I am not sure what would lead to memory bloat even if the tool is not in use. There could be memory leak from some component. I am investigating that.
    Suggest you to keep ObjectViewers, reports & PL/SQL editor closed when not in use.
    Have Sql Array Fetch Size preference to 50 if it's higher.
    Do not have auto-refresh for Report editor.
    Keep Sql History, Find Db Object, Snippets collapsed.
    See if that helps.
    -Raghu

  • How to get a thread dump, in case of an IDE freeze

    When you're working with the IDE and if it either freezes or becomes sluggish for no apparent reason, it's recommended to generate a thread dump, and report it to Studio feedback alias ([email protected]) with a description of what happened and attach the dump to it.
    A thread dump is an invaluable source of information to IDE developers when investigating deadlocks and some performance issues. It is a textual dump of all active threads and monitors of Java apps running in a Virtual Machine.
    The ways to generate a thread dump differ depending on the platform:
    * Windows systems
    Press Ctrl-Break in the command console you used to start the IDE.
    You won't be able to see the console in case you started the IDE using the default desktop icon created by the IDE installer. To be able to generate the thread dump, you must launch the IDE using a console window (invoke Command Prompt, go to {studio-install-dir}/bin and type runide.exe), or configure the desktop shortcut properties to launch the runide.exe program instead of the default runidew.exe.
    Note: You should increase the screen buffer size of the command prompt, so that it could retian the entire thread dump output. To do that, on the command prompt where IDE is running, open the properties dialog box by right clicking on the titlebar and selecting 'properties'. Then select the 'layout tab' and increase the 'screen buffer size' parameters to - width - 200, Height - 4000.
    * Unix systems
    Press Ctrl-\ in the terminal console you used to start the JSE.
    Alternatively, you can also generate a thread dump by sending the QUIT signal to the Java VM running the JSE
    kill -QUIT process_id
    where process_id is the process number of the respective java process.
    -------Here is a sample thread dump -------
    Full thread dump Java HotSpot(TM) Client VM (1.4.2_01-b06 mixed mode):
    "Text-Layout" daemon prio=2 tid=0x0886ebd8 nid=0x4e4 in Object.wait() [9e8f000..9e8fd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x11730910> (a org.netbeans.editor.view.spi.ViewLayoutQueue)
    at java.lang.Object.wait(Object.java:429)
    at org.netbeans.editor.view.spi.ViewLayoutQueue.waitForTask(ViewLayoutQueue.java:128)
    - locked <0x11730910> (a org.netbeans.editor.view.spi.ViewLayoutQueue)
    at org.netbeans.editor.view.spi.ViewLayoutQueue$LayoutThread.run(ViewLayoutQueue.java:182)
    "Compilation" daemon prio=2 tid=0x0883d878 nid=0xb1c in Object.wait() [9c8f000..9c8fd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1143dde0> (a java.util.LinkedList)
    at java.lang.Object.wait(Object.java:429)
    at org.netbeans.core.compiler.CompilationEngineImpl$CompilerThread.nextJobAndTask(CompilationEngineImpl.java:162)
    - locked <0x1143dde0> (a java.util.LinkedList)
    at org.netbeans.core.compiler.CompilationEngineImpl$CompilerThread.run(CompilationEngineImpl.java:175)
    "TimerQueue" daemon prio=5 tid=0x03575478 nid=0xb18 in Object.wait() [988f000..988fd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x10d4af60> (a javax.swing.TimerQueue)
    at javax.swing.TimerQueue.run(TimerQueue.java:231)
    - locked <0x10d4af60> (a javax.swing.TimerQueue)
    at java.lang.Thread.run(Thread.java:534)
    "AWT-EventQueue-1" prio=7 tid=0x0344fae0 nid=0xb14 in Object.wait() [3caf000..3cafd94]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x10d4afe8> (a java.awt.EventQueue)
    at java.lang.Object.wait(Object.java:429)
    at java.awt.EventQueue.getNextEvent(EventQueue.java:339)
    - locked <0x10d4afe8> (a java.awt.EventQueue)
    at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchThread.java:162)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:145)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:137)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:100)

    You can get a thread dump (Windows only) whitout any preparation. Simply use the Stack Trace tool from this web site:
    http://tmitevski.users.mcs2.netarray.com
    It works on Windows services too.

  • JNI - getting a thread dump

    Hi !
    I'm using JNI to create a JVM inside of my C daemon, and it's working OK except that the JVM is hanging sometimes (rarely), and I would like to get the thread dump, like when you send SIGQUIT to the java process. But it doesn't work with my program. Why?
    Thanks
    - Rustam -

    Do you have any signal handlers installed in your native code? If so, you may have to remove them - they can interfere with the JVM's use of signals, prior to Java 1.4.
    Java 1.4 introduces a "signal chaining" facility whereby you may use your own signal handlers in native code that will be chained after the JVM's own signal handlers are called. This version of JVM includes a library for determining whether a signal is intended for the JVM or for native code in the same process.

  • What is the problem in my Thread Dump's output?

    Dear all,
    Below is my Thread Dump output. I can not figure out what and where is the problem of this Thread Dump's output. Could some one please give me some hints?
    Thanks alot
    Tu
    ----------------Thread Dump's Output-----------------
    Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode, sharing):
    "TP-Monitor" daemon prio=1 tid=0xb0b14490 nid=0xfaa in Object.wait()
    [0xb08fe000..0xb08fee40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89433f70> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
    - locked <0x89433f70> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor4" daemon prio=1 tid=0xb0b13730 nid=0xfa9 runnable
    [0xb097e000..0xb097efc0]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x89438110> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)
    at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:647)
    at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:857)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor3" daemon prio=1 tid=0xb0b0f6b8 nid=0xfa8 in
    Object.wait() [0xb09fe000..0xb09fef40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89434250> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89434250> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor2" daemon prio=1 tid=0xb0b0fec8 nid=0xfa7 in
    Object.wait() [0xb0a7e000..0xb0a7f0c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x894342e8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x894342e8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "TP-Processor1" daemon prio=1 tid=0xb0b14978 nid=0xfa6 in
    Object.wait() [0xb0afe000..0xb0aff040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89434380> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89434380> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Monitor" prio=1 tid=0x08609b18 nid=0xfa5 in Object.wait()
    [0xb0cc9000..0xb0cc91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378950> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
    - locked <0x89378950> (a
    org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor25" daemon prio=1 tid=0x08608ce0 nid=0xfa4
    runnable [0xb0d48000..0xb0d49140]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x89010c58> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
    at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
    at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor24" daemon prio=1 tid=0x086085f8 nid=0xfa3 in
    Object.wait() [0xb0dc8000..0xb0dc8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378ab8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378ab8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor23" daemon prio=1 tid=0x08606dd0 nid=0xfa2 in
    Object.wait() [0xb0e48000..0xb0e48e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378b50> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378b50> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor22" daemon prio=1 tid=0x08605ed0 nid=0xfa1 in
    Object.wait() [0xb0ec8000..0xb0ec8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378be8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378be8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor21" daemon prio=1 tid=0x08604fd0 nid=0xfa0 in
    Object.wait() [0xb0f48000..0xb0f48f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378c80> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378c80> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor20" daemon prio=1 tid=0x086040f0 nid=0xf9f in
    Object.wait() [0xb0fc8000..0xb0fc90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378d18> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378d18> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor19" daemon prio=1 tid=0x08432538 nid=0xf9e in
    Object.wait() [0xb1048000..0xb1049040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378db0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378db0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor18" daemon prio=1 tid=0x08431638 nid=0xf9d in
    Object.wait() [0xb10c9000..0xb10c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378e48> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378e48> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor17" daemon prio=1 tid=0x08430738 nid=0xf9c in
    Object.wait() [0xb1148000..0xb1149140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378ee0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378ee0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor16" daemon prio=1 tid=0x0842f838 nid=0xf9b in
    Object.wait() [0xb11c8000..0xb11c8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89378f78> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89378f78> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor15" daemon prio=1 tid=0x0842e938 nid=0xf9a in
    Object.wait() [0xb1248000..0xb1248e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379010> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379010> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor14" daemon prio=1 tid=0x0842da38 nid=0xf99 in
    Object.wait() [0xb12c8000..0xb12c8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893790a8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893790a8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor13" daemon prio=1 tid=0x0842cb38 nid=0xf98 in
    Object.wait() [0xb1348000..0xb1348f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379140> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379140> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor12" daemon prio=1 tid=0x0842bc50 nid=0xf97 in
    Object.wait() [0xb13c8000..0xb13c90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893791d8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893791d8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor11" daemon prio=1 tid=0x084f75d0 nid=0xf96 in
    Object.wait() [0xb1448000..0xb1449040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379270> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379270> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor10" daemon prio=1 tid=0x084f66d0 nid=0xf95 in
    Object.wait() [0xb14c9000..0xb14c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379308> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379308> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor9" daemon prio=1 tid=0x084f57d0 nid=0xf94 in
    Object.wait() [0xb1548000..0xb1549140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893793a0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893793a0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor8" daemon prio=1 tid=0x08488970 nid=0xf93 in
    Object.wait() [0xb15c8000..0xb15c8ec0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379438> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379438> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor7" daemon prio=1 tid=0x08487a70 nid=0xf92 in
    Object.wait() [0xb1648000..0xb1648e40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893794d0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893794d0> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor6" daemon prio=1 tid=0x08486b90 nid=0xf91 in
    Object.wait() [0xb16c8000..0xb16c8fc0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379568> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379568> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor5" daemon prio=1 tid=0x084e3e30 nid=0xf90 in
    Object.wait() [0xb1748000..0xb1748f40]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379600> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379600> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor4" daemon prio=1 tid=0x084e2f90 nid=0xf8f in
    Object.wait() [0xb17c8000..0xb17c90c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379698> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379698> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor3" daemon prio=1 tid=0x084e2158 nid=0xf8e in
    Object.wait() [0xb1848000..0xb1849040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379730> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379730> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor2" daemon prio=1 tid=0x085e74d0 nid=0xf8d in
    Object.wait() [0xb18c9000..0xb18c91c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x893797c8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x893797c8> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "http-8080-Processor1" daemon prio=1 tid=0x085e6638 nid=0xf8c in
    Object.wait() [0xb1948000..0xb1949140]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x89379860> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Object.wait(Object.java:474)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
    - locked <0x89379860> (a
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
    at java.lang.Thread.run(Thread.java:595)
    "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=1
    tid=0x083719f8 nid=0xf8b waiting on condition [0xb1a85000..0xb1a85ec0]
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1547)
    at java.lang.Thread.run(Thread.java:595)
    "Low Memory Detector" daemon prio=1 tid=0x080a44f8 nid=0xf7f runnable
    [0x00000000..0x00000000]
    "CompilerThread0" daemon prio=1 tid=0x080a2f98 nid=0xf7e waiting on
    condition [0x00000000..0xb21b2928]
    "Signal Dispatcher" daemon prio=1 tid=0x080a2068 nid=0xf7d waiting on
    condition [0x00000000..0x00000000]
    "Finalizer" daemon prio=1 tid=0x0809c540 nid=0xf7c in Object.wait()
    [0xb24b2000..0xb24b3040]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=1 tid=0x0809a6b0 nid=0xf7b in
    Object.wait() [0xb2533000..0xb25331c0]
    at java.lang.Object.wait(Native Method)
    - waiting on <0x88f01e60> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:474)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x88f01e60> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x0805d1c8 nid=0xf79 runnable [0xbfbd9000..0xbfbda4f8]
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x8946f690> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:450)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:388)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:615)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:575)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)
    "VM Thread" prio=1 tid=0x08097b58 nid=0xf7a runnable
    "VM Periodic Task Thread" prio=1 tid=0x080a5998 nid=0xf80 waiting on condition

    Thanks alot for your reply. It is a web application
    using java and tomcat server. And the problem is that
    sometime when I click on a button or a link than the
    CPU goes to 100% and it hangs but normally it works
    smoothly without any problem.This often indicates a busy retry loop somewhere. You need to acquire the thread dump when the problem occurs, but be warned that depending on the problem it may not be possible to obtain a Java-level thread dump. In that case you need to try and take an OS level thread dump - eg using pstack on solaris/linux (and there's some win32 tool as well if I recall correctly).

  • 100% CPU load but no clue in thread dump, EP. Win a bottle of champagne...

    Hi,
    We are suddenly facing 100% CPU load in our EP cluster (5 x 8-way Xeon multiprocessor machines). We have a serious performance problem that is burning a lot of our time and causes a lot of stress for over 3 weeks now.
    We have taken tens of thread dumps from Application Nodes.
    - In none of the thread dumps we see the Finalizer thread
    running.
    - From the garbage collector log we see that full garbage
    collection runs occur very rarily (once an hour).
    - We use the compacting garbage collector.
    - In a lot of thread dumps we do not even see our own portal code in the stack traces of the threads!
    - In Windows Task Manager we see the jlaunch.exe processes
    consume all available CPU time.
    - Portal users see a blank portal page when the CPU load hits 100%. When the load goes down again, things return to normal.
    - There are no errors logged in any of the log files of the portal. We checked all of them. We expected a log full of errors somewhere but nothing even remotely interesting was found. Windows Event Viewer shows nothing either.
    - The amount of sockets in CLOSE_WAIT status is < 10 on every machine in the cluster.
    This forces me to conclude that something in the jlaunch.exe executable consumes the CPU time. This raises the following 3 questions:
    - What does the mystifying jlaunch.exe do besides executing java.exe ?
    - Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ? Can it be GZip compression ?
    - If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
    Our development- and support teams are desperate. All suggestions are welcome. The person that comes up with
    the solution to our problem gets a nice bottle of champagne.
    Regards,
    Chris Twigt

    Hi,
    - What does the mystifying jlaunch.exe do besides executing java.exe ?
    This is so that the startupframework can connect more easily and take control of the JVM in some situations
    - Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ?
    I assume when SAP release their JVM (in the next major release), that will be called directly.
    - If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
    Loads of thing, but there should be clues in the thread dumps.  (the reason why your code is not in those thread dumps is that your code is only active during the processing of a particular request, unless you have a service, afterwards there are no trace of it as the thread which does the processing goes back to sleep)
    I've experienced a similar situation with a an 6.40 portal, and it was then caused by the following:
    1. User A comes logs in and sees that some cache timeout has occured , therefore it issues a SQL query which does a full table scan on a table of approx. 1 GB (on of the UME tables)
    2. User B comes in just afterwards and also sees that the cache timeout has occured, and issues the same SQL query as User A
    3. User C .... and so on untill the query from User A eventually finishes
    So
    1. Check database. Any big queries running ?
    2. Check file activity, are there a lot of writing ?
    3. Check network activity, especially to the state controller which is the weakest link
    4. What about portal logs ? Any activity during the hang ?
    Also, please provide one of the thread dumps for further analysis..
    cheers
    Dagfinn

  • Weblogic 8.1 sp4 on Slowwaris 2.7 - Thread dump included

    Dear experts,
    We have a time response problem within our production environment, following is our configuration :
    Oracle 8.1.7.4
    Weblogic 8.1 sp4
    Solaris Sparc 2.7
    JSP is pre-compiled, though the time response behaviour is rather weird, sometime quick and most of the time very slow, that we can not stand with ie. 5 minutes for displaying the mainpage, worse is client IE browser should be shut and restart. Below is the thread dump of our server, we will appreciate if someone can interpret it and possibly lead us to pointers how to solve our problem.
    Full thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e1848 nid=0x34 in Object.wait() [ddf81000..ddf819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0e170> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0e170> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e0b38 nid=0x33 in Object.wait() [de081000..de0819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0d9d0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0d9d0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x005e08f0 nid=0x32 in Object.wait() [de181000..de1819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0d230> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0d230> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=5 tid=0x00359820 nid=0x31 in Object.wait() [de281000..de2819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4d0ca70> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe4d0ca70> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060d850 nid=0x30 waiting for monitor entry [de381000..de3819f0]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
         - waiting to lock <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060d0c8 nid=0x2f waiting for monitor entry [dea01000..dea019f0]
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)
         - waiting to lock <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0060cb28 nid=0x2e runnable [deb01000..deb019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe4cac810> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:95)
         - locked <0xe4cac810> (a [Lweblogic.socket.PosixSocketInfo$FdStruct;)
         - locked <0xe4c822d0> (a java.lang.String)
         at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ListenThread.Default" prio=5 tid=0x00609978 nid=0x2d runnable [dec01000..dec019f0]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <0xe4cb5d80> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:448)
         at java.net.ServerSocket.accept(ServerSocket.java:419)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Thread-6" daemon prio=5 tid=0x001d2ab8 nid=0x2c in Object.wait() [df701000..df7019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6a78a80> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xe6a78a80> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=5 tid=0x001d3b08 nid=0x2b in Object.wait() [dfe01000..dfe019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6a32e48> (a weblogic.jms.backend.BETimerTree)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <0xe6a32e48> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "weblogic.health.CoreHealthMonitor" daemon prio=5 tid=0x004ccf08 nid=0x2a waiting on condition [e0101000..e01019f0]
         at java.lang.Thread.sleep(Native Method)
         at weblogic.t3.srvr.CoreHealthMonitorThread.run(CoreHealthMonitorThread.java:128)
    "Thread-5" prio=5 tid=0x00387728 nid=0x29 in Object.wait() [e0201000..e02019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe66a3f78> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <0xe66a3f78> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "VDE Transaction Processor Thread" prio=2 tid=0x004a1138 nid=0x27 in Object.wait() [e0301000..e03019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b97d8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at java.lang.Object.wait(Object.java:429)
         at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:365)
         - locked <0xe65b97d8> (a com.octetstring.vde.backend.standard.TransactionProcessor)
         at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)
    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003cb6b0 nid=0x26 in Object.wait() [e0401000..e04019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9858> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9858> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003caac0 nid=0x25 in Object.wait() [e0501000..e05019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b98d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b98d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x003ca010 nid=0x24 in Object.wait() [e0601000..e06019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9958> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9958> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.admin.HTTP'" daemon prio=5 tid=0x003c9560 nid=0x23 in Object.wait() [e0701000..e07019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b99d8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b99d8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.admin.HTTP'" daemon prio=5 tid=0x003c9318 nid=0x22 in Object.wait() [e0801000..e08019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe65b9a58> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe65b9a58> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "weblogic.security.SpinnerRandomSource" daemon prio=5 tid=0x003c6900 nid=0x21 in Object.wait() [e0901000..e09019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651be08> (a java.lang.Object)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.security.SpinnerRandomBitsSource.run(SpinnerRandomBitsSource.java:60)
         - locked <0xe651be08> (a java.lang.Object)
         at java.lang.Thread.run(Thread.java:534)
    "weblogic.time.TimeEventGenerator" daemon prio=9 tid=0x003c5958 nid=0x20 runnable [e0a01000..e0a019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651be78> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <0xe651be78> (a weblogic.time.common.internal.TimeTable)
         at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:118)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x003c3988 nid=0x1f in Object.wait() [e0b01000..e0b019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651bef0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651bef0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fb810 nid=0x1e in Object.wait() [e0c01000..e0c019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651bf70> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651bf70> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fac20 nid=0x1d waiting on condition [e0d01000..e0d019f0]
         at weblogic.platform.SunVM.threadDump0(Native Method)
         - waiting to lock <0xe4c7cdd0> (a weblogic.platform.SunVM)
         at weblogic.platform.SunVM.threadDump(SunVM.java:106)
         at weblogic.t3.srvr.T3Srvr.getThreadDump(T3Srvr.java:1184)
         at weblogic.common.internal.AdminProxy.execute(AdminProxy.java:220)
         at weblogic.t3.srvr.ClientRequest$1.run(ClientContext.java:700)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
         at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
         at weblogic.t3.srvr.ClientRequest.execute(ClientContext.java:697)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001fa030 nid=0x1c in Object.wait() [e0e01000..e0e019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c070> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c070> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon prio=5 tid=0x001f9440 nid=0x1b in Object.wait() [e0f01000..e0f019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c0f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c0f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f8850 nid=0x1a in Object.wait() [e1001000..e10019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c170> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c170> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f7c60 nid=0x19 in Object.wait() [e1101000..e11019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c1f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c1f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f7070 nid=0x18 in Object.wait() [e1201000..e12019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c270> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c270> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f6480 nid=0x17 in Object.wait() [e1301000..e13019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c2f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c2f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x001f5890 nid=0x16 in Object.wait() [e1401000..e14019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c370> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c370> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d67b8 nid=0x15 in Object.wait() [e1501000..e15019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c3f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c3f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d5bc8 nid=0x14 in Object.wait() [e1601000..e16019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c470> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c470> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d4fd8 nid=0x13 in Object.wait() [e1701000..e17019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c4f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c4f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x004d43e8 nid=0x12 in Object.wait() [e1801000..e18019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c570> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c570> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b6d70 nid=0x11 in Object.wait() [e1901000..e19019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c5f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c5f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b61e8 nid=0x10 in Object.wait() [e1a01000..e1a019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c670> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c670> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b57a8 nid=0xf in Object.wait() [e1b01000..e1b019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c6f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c6f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x003b4e08 nid=0xe in Object.wait() [e1c01000..e1c019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c770> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c770> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x005e9030 nid=0xd in Object.wait() [e1d01000..e1d019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c7f0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c7f0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x005e8e90 nid=0xc in Object.wait() [e1e01000..e1e019f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe651c870> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <0xe651c870> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-1" daemon prio=5 tid=0x00201230 nid=0xb in Object.wait() [e4981000..e49819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe637e1d0> (a java.util.TaskQueue)
         at java.lang.Object.wait(Object.java:429)
         at java.util.TimerThread.mainLoop(Timer.java:403)
         - locked <0xe637e1d0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Signal Dispatcher" daemon prio=10 tid=0x000cdbc8 nid=0x8 runnable [0..0]
    "Finalizer" daemon prio=8 tid=0x000ca7b0 nid=0x6 in Object.wait() [fc381000..fc3819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6278d78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <0xe6278d78> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0x000c8e50 nid=0x5 in Object.wait() [fde81000..fde819f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe6278de0> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:429)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
         - locked <0xe6278de0> (a java.lang.ref.Reference$Lock)
    "main" prio=5 tid=0x00038510 nid=0x1 in Object.wait() [ffbed000..ffbed7dc]
         at java.lang.Object.wait(Native Method)
         - waiting on <0xe62eeff0> (a weblogic.t3.srvr.T3Srvr)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.t3.srvr.T3Srvr.waitForDeath(T3Srvr.java:1208)
         - locked <0xe62eeff0> (a weblogic.t3.srvr.T3Srvr)
         at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:390)
         at weblogic.Server.main(Server.java:32)
    "VM Thread" prio=5 tid=0x000c8008 nid=0x4 runnable
    "VM Periodic Task Thread" prio=10 tid=0x000d0a28 nid=0xa waiting on condition
    "Suspend Checker Thread" prio=10 tid=0x000cd260 nid=0x7 runnable
    Thanks in advance,
    saiful-france

    Hi All,
    It looks like I have the similar problem: I am using Weblogic 8.1 Sp3 platform. please help.
    Full thread dump Java HotSpot(TM) Server VM (1.4.2 1.4.2.03-040401-18:59-PA_RISC2.0 PA2.0 (aCC_AP) mixed mode):
    "Login" daemon prio=10 tid=01a0a0b0 nid=17817 lwp_id=1133989 in Object.wait() [0x2da54000..0x2da544f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92588248> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=01a09f48 nid=17816 lwp_id=1133988 in Object.wait() [0x2d90f000..0x2d90f4f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92580248> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=00c40450 nid=17815 lwp_id=1133969 in Object.wait() [0x2ed85000..0x2ed854f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <92446368> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "Login" daemon prio=10 tid=00c402e8 nid=17814 lwp_id=1133967 in Object.wait() [0x2db17000..0x2db174f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.doInvoke(DSUser.java:455)
         - locked <9242e368> (a com.inprise.vbroker.ds.DSUser$1$DSAMessageHandler)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:506)
         at com.inprise.vbroker.ds.DSUser$2.run(DSUser.java:712)
         at java.lang.Thread.run(Thread.java:534)
    "AreYouAlive" daemon prio=10 tid=077ca9b0 nid=11502 lwp_id=1080790 in Object.wait() [0x2d950000..0x2d9504f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fcb3390> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:115)
         - locked <3fcb3390> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
    "DatagramThread" daemon prio=10 tid=077ca848 nid=11501 lwp_id=1080789 runnable [0x2d991000..0x2d9914f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3fcb5458> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3fcb5488> (a java.net.DatagramPacket)
         - locked <3fcb54b0> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=077ca6e0 nid=11500 lwp_id=1080788 in Object.wait() [0x2d9d2000..0x2d9d24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fcb54d0> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3fcb54d0> (a com.inprise.vbroker.orb.GarbageCollector)
    "AreYouAlive" daemon prio=10 tid=030802e8 nid=11340 lwp_id=1078923 in Object.wait() [0x2da95000..0x2da954f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fc90508> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:115)
         - locked <3fc90508> (a com.inprise.vbroker.ds.DSUser$AreYouAlive)
    "DatagramThread" daemon prio=10 tid=03080180 nid=11339 lwp_id=1078922 runnable [0x2dad6000..0x2dad64f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3fc925d0> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3fc92600> (a java.net.DatagramPacket)
         - locked <3fc92628> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=03080018 nid=11338 lwp_id=1078921 in Object.wait() [0x2db58000..0x2db584f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3fc92648> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3fc92648> (a com.inprise.vbroker.orb.GarbageCollector)
    "AreYouAlive" daemon prio=10 tid=08005a10 nid=3830 lwp_id=947497 in Object.wait() [0x2db99000..0x2db994f0]
         at java.lang.Object.wait(Native Method)
         at com.inprise.vbroker.ds.DSUser.login(DSUser.java:736)
         - locked <3d4b1aa0> (a java.lang.Object)
         at com.inprise.vbroker.ds.DSUser.reconnect(DSUser.java:544)
         - locked <3d4b19b8> (a com.inprise.vbroker.ds.DSUser)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:513)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.fire(DSUser.java:95)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:117)
    "DatagramThread" daemon prio=10 tid=080058a8 nid=3829 lwp_id=947496 runnable [0x2dc1b000..0x2dc1b4f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3d4b3b20> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3d4b3b50> (a java.net.DatagramPacket)
         - locked <3d4b3b78> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=08005740 nid=3828 lwp_id=947495 in Object.wait() [0x2dbda000..0x2dbda4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3d4b3b98> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3d4b3b98> (a com.inprise.vbroker.orb.GarbageCollector)
    "ExecuteThread: '14' for queue: 'JmsDispatcher'" daemon prio=10 tid=030424a8 nid=142 lwp_id=679935 in Object.wait() [0x2dc5c000..0x2dc5c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b088> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b088> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '13' for queue: 'JmsDispatcher'" daemon prio=10 tid=03042340 nid=141 lwp_id=679934 in Object.wait() [0x2dc9d000..0x2dc9d4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b108> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b108> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '12' for queue: 'JmsDispatcher'" daemon prio=10 tid=030421d8 nid=140 lwp_id=679933 in Object.wait() [0x2dcde000..0x2dcde4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b188> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b188> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '11' for queue: 'JmsDispatcher'" daemon prio=10 tid=03042070 nid=139 lwp_id=679932 in Object.wait() [0x2dd1f000..0x2dd1f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b208> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b208> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041f08 nid=138 lwp_id=679931 in Object.wait() [0x2dd60000..0x2dd604f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b288> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b288> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041da0 nid=137 lwp_id=679930 in Object.wait() [0x2dda1000..0x2dda14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b308> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b308> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041c38 nid=136 lwp_id=679929 in Object.wait() [0x2dde2000..0x2dde24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b388> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b388> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041ad0 nid=135 lwp_id=679928 in Object.wait() [0x2de23000..0x2de234f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b408> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b408> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041968 nid=134 lwp_id=679927 in Object.wait() [0x2de64000..0x2de644f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b488> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b488> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041800 nid=133 lwp_id=679926 in Object.wait() [0x2dea5000..0x2dea54f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b508> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b508> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041698 nid=132 lwp_id=679925 in Object.wait() [0x2dee6000..0x2dee64f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b588> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b588> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041530 nid=131 lwp_id=679924 in Object.wait() [0x2df27000..0x2df274f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b608> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b608> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JmsDispatcher'" daemon prio=10 tid=030413c8 nid=130 lwp_id=679923 in Object.wait() [0x2df68000..0x2df684f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b688> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b688> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JmsDispatcher'" daemon prio=10 tid=03041260 nid=129 lwp_id=679922 in Object.wait() [0x2dfa9000..0x2dfa94f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b708> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b708> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JmsDispatcher'" daemon prio=10 tid=030410f8 nid=128 lwp_id=679921 in Object.wait() [0x2dfea000..0x2dfea4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b7a8> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3b81b7a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ListenThread.Default" prio=10 tid=02953558 nid=127 lwp_id=679919 runnable [0x2e02b000..0x2e02b4f0]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
         - locked <3b81b8c0> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:454)
         at java.net.ServerSocket.accept(ServerSocket.java:425)
         at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:26)
         at weblogic.t3.srvr.ListenThread.accept(ListenThread.java:735)
         at weblogic.t3.srvr.ListenThread.run(ListenThread.java:301)
    "Adapter Suspend/Resume Task Handler" daemon prio=10 tid=029533f0 nid=126 lwp_id=679918 in Object.wait() [0x2e06c000..0x2e06c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81b9a0> (a java.lang.String)
         at java.lang.Object.wait(Object.java:429)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:60)
         - locked <3b81b9a0> (a java.lang.String)
    "Adapter instance Auto-resume TaskHandler" daemon prio=10 tid=02953288 nid=125 lwp_id=679917 in Object.wait() [0x2e0ad000..0x2e0ad4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b81ba18> (a java.lang.String)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:55)
         - locked <3b81ba18> (a java.lang.String)
    "ApplicationView Auto-resume TaskHandler" daemon prio=10 tid=000b6360 nid=124 lwp_id=679915 in Object.wait() [0x2e0ee000..0x2e0ee4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b74b988> (a java.lang.String)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:55)
         - locked <3b74b988> (a java.lang.String)
    "ApplicationView Deployment TaskHandler" daemon prio=10 tid=000b61f8 nid=123 lwp_id=679914 in Object.wait() [0x2e12f000..0x2e12f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b74b648> (a java.lang.String)
         at java.lang.Object.wait(Object.java:429)
         at com.bea.wlai.TaskHandler.run(TaskHandler.java:60)
         - locked <3b74b648> (a java.lang.String)
    "AreYouAlive" daemon prio=10 tid=000b6090 nid=122 lwp_id=679910 in Object.wait() [0x2e170000..0x2e1704f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Thread.join(Thread.java:1001)
         - locked <8e395cc8> (a java.lang.Thread)
         at java.lang.Thread.join(Thread.java:1054)
         at com.inprise.vbroker.ds.DSUser.login(DSUser.java:752)
         at com.inprise.vbroker.ds.DSUser.reconnect(DSUser.java:544)
         - locked <3b6e4558> (a com.inprise.vbroker.ds.DSUser)
         at com.inprise.vbroker.ds.DSUser.invoke(DSUser.java:513)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.fire(DSUser.java:95)
         at com.inprise.vbroker.ds.DSUser$AreYouAlive.run(DSUser.java:117)
    "DatagramThread" daemon prio=10 tid=000b5f28 nid=121 lwp_id=679909 runnable [0x2e1b1000..0x2e1b14f0]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <3b6e6a18> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:728)
         - locked <3b6e9020> (a java.net.DatagramPacket)
         - locked <3b6e69f0> (a java.net.DatagramSocket)
         at com.inprise.vbroker.ds.DSUser$DatagramThread.run(DSUser.java:169)
    "GarbageCollector" daemon prio=10 tid=000b5dc0 nid=120 lwp_id=679908 in Object.wait() [0x2e1f2000..0x2e1f24f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3b6e3338> (a com.inprise.vbroker.orb.GarbageCollector)
         at com.inprise.vbroker.orb.GarbageCollector.run(GarbageCollector.java:62)
         - locked <3b6e3338> (a com.inprise.vbroker.orb.GarbageCollector)
    "Timer" daemon prio=10 tid=000b5af0 nid=119 lwp_id=679906 in Object.wait() [0x2e233000..0x2e2334f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b65ee40> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "Timer" daemon prio=10 tid=000b5c58 nid=118 lwp_id=679905 in Object.wait() [0x2e274000..0x2e2744f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b642528> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "Timer" daemon prio=10 tid=000b5820 nid=117 lwp_id=679904 in Object.wait() [0x2e2b5000..0x2e2b54f0]
         at java.lang.Object.wait(Native Method)
         at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:272)
         - locked <3b639540> (a weblogic.time.common.internal.TimeTable)
         at weblogic.management.timer.Timer.run(Timer.java:495)
         at java.lang.Thread.run(Thread.java:534)
    "ExecuteThread: '2' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca578 nid=116 lwp_id=679896 in Object.wait() [0x2e2f6000..0x2e2f64f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad75800> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca410 nid=115 lwp_id=679895 in Object.wait() [0x2e337000..0x2e3374f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3ad75328> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad75328> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'weblogic.kernel.Non-Blocking'" daemon prio=10 tid=077ca2a8 nid=114 lwp_id=679894 in Object.wait() [0x2e378000..0x2e3784f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3ad74e30> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3ad74e30> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_1>.ioThreadPool'" daemon prio=10 tid=000b5988 nid=113 lwp_id=679892 in Object.wait() [0x2e3b9000..0x2e3b94f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <3af7c258> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "Thread-20" prio=10 tid=000b56b8 nid=112 lwp_id=679851 in Object.wait() [0x2e3fa000..0x2e3fa4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3a9b0250> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3a9b0250> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-19" prio=10 tid=000b5550 nid=111 lwp_id=679850 in Object.wait() [0x2e43b000..0x2e43b4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3a9b02f0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3a9b02f0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-18" daemon prio=10 tid=01a09c78 nid=110 lwp_id=679822 in Object.wait() [0x2e47c000..0x2e47c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39f1b930> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <39f1b930> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "ExecuteThread: '11' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040f90 nid=109 lwp_id=679809 in Object.wait() [0x2e4bd000..0x2e4bd4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e53ac0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '10' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040e28 nid=108 lwp_id=679808 in Object.wait() [0x2e4fe000..0x2e4fe4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e535a8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '9' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040cc0 nid=107 lwp_id=679807 in Object.wait() [0x2e53f000..0x2e53f4f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e53090> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '8' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040b58 nid=106 lwp_id=679806 in Object.wait() [0x2e580000..0x2e5804f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e52b28> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '7' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030409f0 nid=105 lwp_id=679805 in Object.wait() [0x2e5c1000..0x2e5c14f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e52610> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '6' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040888 nid=104 lwp_id=679804 in Object.wait() [0x2e602000..0x2e6024f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e520f8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '5' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040720 nid=103 lwp_id=679803 in Object.wait() [0x2e643000..0x2e6434f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e51be0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '4' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030405b8 nid=102 lwp_id=679802 in Object.wait() [0x2e684000..0x2e6844f0]
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e516c8> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '3' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040450 nid=101 lwp_id=679801 in Object.wait() [0x2e6c5000..0x2e6c54f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e511b0> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e511b0> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '2' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=030402e8 nid=100 lwp_id=679800 in Object.wait() [0x2e706000..0x2e7064f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50c98> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50c98> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '1' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040180 nid=99 lwp_id=679799 in Object.wait() [0x2e747000..0x2e7474f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50780> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50780> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "ExecuteThread: '0' for queue: 'JMS.TimerClientPool'" daemon prio=10 tid=03040018 nid=98 lwp_id=679798 in Object.wait() [0x2e788000..0x2e7884f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39e50268> (a weblogic.kernel.ExecuteThread)
         at java.lang.Object.wait(Object.java:429)
         at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
         - locked <39e50268> (a weblogic.kernel.ExecuteThread)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)
    "SeedGenerator Thread" daemon prio=10 tid=000b53e8 nid=97 lwp_id=679782 in Object.wait() [0x2e7c9000..0x2e7c94f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39db8938> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)
         at java.lang.Object.wait(Object.java:429)
         at sun.security.provider.SeedGenerator$ThreadedSeedGenerator.run(SeedGenerator.java:334)
         - locked <39db8938> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator)
         at java.lang.Thread.run(Thread.java:534)
    "Noisy Thread" daemon prio=10 tid=000b5280 nid=96 lwp_id=679781 in Object.wait() [0x2e80a000..0x2e80a4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <39db89a0> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)
         at java.lang.Object.wait(Object.java:429)
         at sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread.run(SeedGenerator.java:452)
         - locked <39db89a0> (a sun.security.provider.SeedGenerator$ThreadedSeedGenerator$BogusThread)
    "ExecuteThread: '0' for queue: 'JMS.TimerTreePool'" daemon prio=10 tid=000b5118 nid=95 lwp_id=679741 in Object.wait() [0x2e84b000..0x2e84b4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <395c5970> (a weblogic.jms.backend.BETimerTree)
         at weblogic.jms.backend.BETimerTree.execute(BETimerTree.java:146)
         - locked <395c5970> (a weblogic.jms.backend.BETimerTree)
         at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
         at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
    "Thread-17" prio=10 tid=000b4fb0 nid=94 lwp_id=679731 in Object.wait() [0x2e88c000..0x2e88c4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e8a0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e8a0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-16" prio=10 tid=000b4e48 nid=93 lwp_id=679728 in Object.wait() [0x2e8cd000..0x2e8cd4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e940> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e940> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-15" prio=10 tid=000b4ce0 nid=92 lwp_id=679727 in Object.wait() [0x2e90e000..0x2e90e4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971e9e0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971e9e0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-14" prio=10 tid=000b4b78 nid=91 lwp_id=679726 in Object.wait() [0x2e94f000..0x2e94f4f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ea80> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ea80> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-13" prio=10 tid=000b4a10 nid=90 lwp_id=679724 in Object.wait() [0x2e990000..0x2e9904f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971eb20> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971eb20> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-12" prio=10 tid=000b48a8 nid=89 lwp_id=679720 in Object.wait() [0x2e9d1000..0x2e9d14f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ebc0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ebc0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-11" prio=10 tid=000b4740 nid=88 lwp_id=679719 in Object.wait() [0x2ea12000..0x2ea124f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ec60> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ec60> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-10" prio=10 tid=000b45d8 nid=87 lwp_id=679718 in Object.wait() [0x2ea53000..0x2ea534f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971ed00> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971ed00> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-9" prio=10 tid=000b4470 nid=86 lwp_id=679717 in Object.wait() [0x2ea94000..0x2ea944f0]
         at java.lang.Object.wait(Native Method)
         - waiting on <3971eda0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <3971eda0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-8" prio=10 tid=000b4308 nid=85 lwp_id=679715 i

  • WebLogic 8.1 SP4 on Linux - thread dump does not work

    I have setup WebLogic 8.1 SP4 on RedHat Linux AS 3.3 with Sun JVM (142_05). When I try to perform a thread dump using weblogic.Admin, I receive the following error in the stderr file. I have also tried the kill -3 but still no thread dump. Is there some trick to this ?
    cannot load libary 'stackdump': java.lang.UnsatisfiedLinkError: no stackdump in java.library.path

    I am getting the same error as well
    cannot load libary 'stackdump': java.lang.UnsatisfiedLinkError: no stackdump in java.library.path
    I am creating a simple domain using the domain config wizard and I hope all the PATH and LD_LIBRARY_PATH are set by the default scripts. But I am still getting this error.
    But Kill -3 <PID> works though!
    Is is a know problem OR am I missing something?!
    Bala

Maybe you are looking for

  • Running Independen​t VI's

    I've tired to have one vi open a second vi and then close the first vi leaving the second vi running. I can't seem to get this to work using the second vi opening a vi reference too itself as I was led to believe. Can anyone provide help or an exampl

  • How to send HTML email to End User using OOTB email processs?

    Hi, We are using OOTB Send email process to send email to end user. Templates has been created inside /etc/workflow/ProjectName/email folder. Its working properly for plain text email.but for html template ,It send the email with html tags. Any point

  • Runtime workbeanch RWB - Alert Configuration link not working

    Hi, We are having issue opening one of the link in our SAP PI system. The link is http://Hotname:51400/rwp -> AlertConfiguration and Alert inbox. All other links [Component Monitoring,Message Monitoring, Performance Monitoring .......] on this runtim

  • Amending order qty in prod order

    Hi, We need to amend the prod order qty incase of any rejections or scrap during manufacturing. How toimplement the same.

  • Run feedback Work flow with no edit permission

    Can I run a feedback collection workflow ( with task) to people who have no edit permission? If not how can I create a workflow to accomplish that?