Redirecting JVM thread dump to a file

The full thread dump of the JVM can be obtained by pressing ctrl-break in windows on the prompt from which the JVM is run and on solaris by executing the command "kill -QUIT pid" where pid is the process id of the JVM.
The thread dump will be printed on the console by default. But the requirement is to redirect this to a file. The following code redirects the dump to a file "Dump.txt".
System.out.close();
System.setOut(new PrintStream(new FileOutputStream("Dump.txt")));
Here we need to close the standard output stream "System.out" prior to redirecting it to a file. This is because if System.setOut is done without closing it the JVM still gives the thread dump on the initial standard output i.e., console. But if console is closed once we cannot get it back for the rest of the running time of the JVM.
IS THERE ANY SOLUTION TO THIS PROBLEM OF REDIRECTING DUMP TO A FILE, WITHOUT CLOSING THE STANDARD OUTPUT? The application should be able to get the standard output(console) back whenever it's required.

Hi,
I think that you can use a API for logs, as Log4J from Apache or Logger in java.util.logging... both are very simple...
good luck.
Mad.

Similar Messages

  • JVM Thread Dump

    Hello, I am using JES 2003Q4. I would like to be able to obtain a jvm thread dump on the portal server. However when I issue a kill -3 to any of the webservd processes, the web server simply shuts down. No thread dump is observed in the error log. I am not using the -Xrs option in my server.xml file. Is there any way to force a thread dump? Doing so without stopping the web server would be preferable but not absolutely necessary.
    Thanks!

    Well you could try debug and verbose. I've got a feeling that on windows you only see the parent process in the thread dump. Is it possible for you to try this on a 'nix platform? You might get more information.

  • Can't Redirect -classic thread dump/stack trace

    Hi,
    But how can you redirect the output so you can view it easily, instead of the info just scrolling by into oblivion? I'm using Win98 and JDK1.3.1_02, and w/ the default hotspot jvm, from the console, I could redirect the stack trace/thread dump from standard out (w/c it went into) into a file.
    But w/ the -classic option (from the SDK java version, not the JRE), the stack trace/thread dump does NOT seem to sent to standard out, because I can redirect standard out, but the trace/dump is STILL going to the screen. I have redirected standard err (from w/in my java program) as well, but still no dice!
    Help! TIA,
    Reggie

    Thanks for your reply guys! I also tried redirecting std err from inside my java app. My System.err outputs WERE redirected, but NOT the thread dump - that STILL went to the screen.
    Then I tried the "java 2>error.txt myclass" but win98 thought 2 was the name of my class and threw a NoClassDefFoundError ;-).
    Any more ideas? regards,
    Reggie

  • Need help interpreting jvm thread dump (linux)

    hi,
    i'm using jre1.4.2 and running AS3 Linux kernel version 2.4.
    i grep'ed for my java process id, and did a kill -3 on it to get the thread dump:
    ps -ef |grep Eatroot 3936 3845 68 16:12 pts/2 00:00:05 java EatCpu
    root 3948 30293 0 16:12 pts/3 00:00:00 grep Eat
    kill -3 3936
    thread dump:
    java EatCpuFull thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):
    "Thread-0" prio=1 tid=0x081126b8 nid=0xf60 runnable [aa7e0000..aa7e087c]
    at EatCpu.run(EatCpu.java:31)
    "Signal Dispatcher" daemon prio=1 tid=0x080a6a58 nid=0xf60 waiting on condition [0..0]
    "Finalizer" daemon prio=1 tid=0x08092ee8 nid=0xf60 in Object.wait() [aad4d000..aad4d87c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xaaed0490> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
    - locked <0xaaed0490> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=1 tid=0x08091498 nid=0xf60 in Object.wait() [aadce000..aadce87c]
    at java.lang.Object.wait(Native Method)
    - waiting on <0xaaed0380> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:429)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
    - locked <0xaaed0380> (a java.lang.ref.Reference$Lock)
    "main" prio=1 tid=0x0805bae0 nid=0xf60 runnable [bfffc000..bfffcc98]
    at EatCpu.main(EatCpu.java:22)
    "VM Thread" prio=1 tid=0x08090238 nid=0xf60 runnable
    "VM Periodic Task Thread" prio=1 tid=0x080a9248 nid=0xf60 waiting on condition
    "Suspend Checker Thread" prio=1 tid=0x080a6020 nid=0xf60 runnable
    i read that "nid" in the thread dump is suppose to correspond to PID. when i give the -m option for the "ps" command (for all threads), i see:
    ps -efm |grep Eatroot 3936 3845 43 16:12 pts/2 00:00:04 java EatCpu
    root 3937 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3938 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3939 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3940 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3941 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3942 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3943 3936 0 16:12 pts/2 00:00:00 java EatCpu
    root 3944 3936 30 16:12 pts/2 00:00:03 java EatCpu
    root 3950 30293 0 16:12 pts/3 00:00:00 grep Eat
    >
    but nid for all threads in the thead dump is the PID of my main process, 3936 (0xf60). is there a way to correlate the PIDs produced by "ps -efm" to the threads in the thread dump?
    thanks!
    -annie

    i upgraded to 1.5 version of java, and could see distinct "nids" in the thread dump after that..

  • Getting thread dump of JWS application

    Hi,
    I know this was discussed a few times before, but I still have no solution for my problem.
    I have a JWS GUI application that deadlocks sometimes (Windows 98). So I want to get the thread dump to analyze the deadlock. In Java Web Start I changed the java command from javaw.exe to java.exe, so I get a dos box. When I press CTRL-BREAK inside the dos box, I get the thread dump but because the dos box just shows a few lines I only get the last lines of the dump. Under Windows 98 I cannot resize the dos box.
    Is there a way to redirect the output of the dos box to a file? Or is there another way to get the thread dump of a JWS application?
    Thanks in advance
    Markus

    Here is 50% of the solution. To generate a thread dump programatically, you need to call JVM_DumpAllStacks through JNI. You need to link in "jvm.lib".
    JNIEXPORT void JNICALL JVM_DumpAllStacks(JNIEnv *env, jclass unused);
    JNIEXPORT void JNICALL Java_Services_generateThreadDump(JNIEnv *env, jobject thisObject) {
         // Produce a thread dump to VM standard out
         JVM_DumpAllStacks( env, NULL );
    }Here is the java class that calls the JVM_DumpAllStacks method through JNI.
    public class Services {
          * Send a message to the VM, forcing it to create a thread dump. The resulting
          * thread dump will be sent to stdout. Note that this is the VM's stdout, not
          * necessarily System.out.
         public static native void generateThreadDump();
    }When you reach the point in your code that you want to generate a thread dump, close stdout and reopen it with the thread dump destination stream. Next, call the JNI method. This will send the thread dump to stdout.
         // Replace stdout with a file output stream
         System.out.close();
         System.setOut( new PrintStream( new FileOutputStream( "stdout.log" )));
         // Generate thread dump
         Services.generateThreadDump();Why is this only 50% of the solution ? Because this works beautifully for regular applications but it does NOT work for WebStart applications. Not sure what is happening here, but so far I have not managed to get any thread dump in a file from an application that has been fired up through WebStart. Note that I CAN get a thread dump in a file if I fire up the same code from the command line.

  • Strange thread dump on deadlock

    We've been experiencing a strange deadlock during the startup of our java application. When I run jstack on the application to investigate, I see that the AWT-EventQueue is in Object.wait(), but the thread is still marked as RUNNABLE. I've included the relevent parts of the thread dump, and I'm hoping that someone can shed some light on this issue.
    "AWT-EventQueue-0" prio=6 tid=0x5f0a2400 nid=0x19e4 in Object.wait() [0x6007e000]
           java.lang.Thread.State: RUNNABLE
            at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)
              - locked <0x1fc87448> (a java.awt.Component$AWTTreeLock)
    "Thread-63-Pool-9" prio=6 tid=0x5f1a2800 nid=0x1f54 waiting for monitor entry [0x61a9f000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at java.awt.Component.setFont(Component.java:1777)
        - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
    "Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
       java.lang.Thread.State: BLOCKED (on object monitor)
        at java.awt.Component.setFont(Component.java:1777)
        - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
          ... In my search for the solution, I've noticed [this thread|http://stackoverflow.com/questions/64989/jvm-thread-dumps-containing-monitors-without-locking-threads], which mentions that accessing a static variable may be involved. This is also the case in our application. The line in getEAOTableLite in question references a static method.

    Thank you for the quick reply. I'm not sure how I missed it, but if I had read down the stack trace a little better I would have found that the issue was that the class loader for the EaoAlertManager never completed because it was blocked by the illegal call to setFont. This would cause the AWT-EventQueue thread to go into an infinite wait call on the ClassLoader.
    "Thread-289-Pool-3" prio=6 tid=0x60afe800 nid=0x12b8 waiting for monitor entry [0x623fe000]
       java.lang.Thread.State: BLOCKED (on object monitor)
         at java.awt.Component.setFont(Component.java:1777)
         - waiting to lock <0x1fc87448> (a java.awt.Component$AWTTreeLock)
         at java.awt.Container.setFont(Container.java:1554)
         at javax.swing.JComponent.setFont(JComponent.java:2723)
         at javax.swing.LookAndFeel.installColorsAndFont(LookAndFeel.java:191)
         at javax.swing.plaf.basic.BasicPanelUI.installDefaults(BasicPanelUI.java:49)
         at javax.swing.plaf.basic.BasicPanelUI.installUI(BasicPanelUI.java:39)
         at com.ge.med.ptk.laf.CuiPanelUI.installUI(CuiPanelUI.java:53)
         at javax.swing.JComponent.setUI(JComponent.java:662)
         at javax.swing.JPanel.setUI(JPanel.java:136)
         at javax.swing.JPanel.updateUI(JPanel.java:109)
         at javax.swing.JPanel.<init>(JPanel.java:69)
         at javax.swing.JPanel.<init>(JPanel.java:92)
         at javax.swing.JPanel.<init>(JPanel.java:100)
         at javax.swing.JRootPane.createGlassPane(JRootPane.java:528)
         at javax.swing.JRootPane.<init>(JRootPane.java:348)
         at javax.swing.JDialog.createRootPane(JDialog.java:611)
         at javax.swing.JDialog.dialogInit(JDialog.java:593)
         at com.ge.med.plaf.wrapper.WJDialog.dialogInit(WJDialog.java:42)
         at javax.swing.JDialog.<init>(JDialog.java:545)
         at javax.swing.JDialog.<init>(JDialog.java:515)
         at com.ge.med.plaf.wrapper.WJDialog.<init>(WJDialog.java:424)
         at com.ge.med.platinum.gui.util.PlatinumDialog.<init>(PlatinumDialog.java:138)
         at com.ge.med.platinum.gui.util.EAOAlertManager$EAOAlertDialog.<init>(EAOAlertManager.java:450)
         at com.ge.med.platinum.gui.util.EAOAlertManager.<clinit>(EAOAlertManager.java:77)
         at com.ge.med.platinum.work.isu.ExamTransaction.getEAOTableLite(ExamTransaction.java:1514)

  • Collect Thread Dump using Script and direct to separate log file

    Dear Legends,
    I request you to provide me some useful guide lines to collect Thread Dump using script in linux and redirect the output to a separate log file, not the STDOUT file. If it is redirecting to STDOUT then how to extract only the thread stack to a separate log file.
    i used
    ps -ef | grep java
    kill -3 <pid> >> ss1_td.log
    but it is not providing the thread stack to the log file.
    Anykind of help would be appreciated much.
    Regards,
    Karthik

    Hi Ram,
    Really a great and useful script for me at the right time. It solved my requirement and i made some changes to my requirement and the script it working fine to collect the thread dump and even i have checked the thread dump with SAMURAI works fine.
    Helpful site also refer sites like these which contains more valuable things for weblogic. We will meet in the forums again.
    Note: also please update several links available in the site like UNIX scripts, WLST, Python etc.
    Regards,
    Karthik

  • What's a good way to do a thread dump into a separate file

    What is a good way to do a thread dump automatically into a separate file.
    Example. I run a script to do the thread dump, but unfortunetly, it goes into my stdout log file with the rest of my weblogic errors.
    Any ideas? I want it in a separate file when I run my script?

    Do a Google search on "Drobo S" "benchmark."  I don't have a Drobo S, only the regular Drobo.  But here's a guy who tested one on Windows:
    http://mansurovs.com/drobo-s-review-usb-3-0-2nd-generation
    This one has it a bit faster:
    http://the-gadgeteer.com/2011/12/31/drobo-s-storage-array-review/
    Do read up on a few reviews of it, and be absolutely clear that interface speed (i.e. eSATA versus Firewire versus Thunderbolt) is NOT the same as the performance of the system.  The Drobo cannot keep up with any interfaces... at least the Drobo and the Drobo S cannot.
    I am not using the FS model which is a NAS.  I am using the plain old "Drobo" which is slower than the Drobo S, but that's not to say that the Drobo S is fast, because it is not.
    The Drobo in theory is really attractive: Dead simple to manage, can mix and match drive sizes, offers you some data protection, etc.  However do note that protected storage is not, in and of itself, a backup.  You need other backups besides just the data on the Drobo.  And, because it's so slow, it's really not a great fit for photo storage.  See this review from a guy who used to think the Drobo was great for that and then appended his review:
    http://www.stuckincustoms.com/drobo-review/
    To be as clear as possible, IMO the BEST backup strategy with something like Aperture (so long as your managed Aperture library is of a manageable size, like < 800 GB), is to get a few small portable Firewire 800 drives and keep vaults on each one.  They are great because they are easy to use, to have with you, are bus powered, and you WILL offsite them.

  • Thread dump running Classic JVM(jdk 1.2.2)

    Hi,
    We get a thread dump while running our application. The application is in java/jsp running on apache-tomcat. The tomcat version is 3.2.1. Java version being used is:
    java version "1.2.2"
    Classic VM (build JDK-1.2.2-007, native threads, symcjit)
    Does anyone have any idea how to identify the reason for the thread dump? The application eventually hangs after the thread dump. Thanks in advance. Below is the dump:
    <pre>
    Full thread dump Classic VM (JDK-1.2.2_007, native threads):
    "Thread-34" (TID:0x5faa550, sys_thread_t:0x2f4158, state:CW, native ID:0xb9c) prio=5
    "Thread-33" (TID:0x5faa1d0, sys_thread_t:0x22092f98, state:CW, native ID:0xa54) prio=5
         at java.lang.Object.wait(Native Method)
         at org.apache.tomcat.util.ThreadPool$MonitorRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-32" (TID:0x5faa220, sys_thread_t:0x2207df88, state:R, native ID:0xb18) prio=5
         at java.net.SocketInputStream.socketRead(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java, Compiled Code)
         at org.apache.tomcat.service.connector.TcpConnector.receiveFully(TcpConnector.java, Compiled Code)
         at org.apache.tomcat.service.connector.TcpConnector.receive(TcpConnector.java, Compiled Code)
         at org.apache.tomcat.service.connector.Ajp13ConnectionHandler.processConnection(Ajp13ConnectionHandler.java, Compiled Code)
         at org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java, Compiled Code)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-31" (TID:0x5fa9e48, sys_thread_t:0x22080228, state:CW, native ID:0xb2c) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java, Compiled Code)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-30" (TID:0x5fa9e98, sys_thread_t:0x22080140, state:CW, native ID:0x908) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java, Compiled Code)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-29" (TID:0x5fa9ee8, sys_thread_t:0x2207e758, state:CW, native ID:0x74c) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java, Compiled Code)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-28" (TID:0x5fa9f38, sys_thread_t:0x22080518, state:R, native ID:0x2fc) prio=5
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:414)
         at java.net.ServerSocket.implAccept(ServerSocket.java:242)
         at java.net.ServerSocket.accept(ServerSocket.java:224)
         at org.apache.tomcat.service.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:286)
         at org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java, Compiled Code)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-27" (TID:0x5fa9b58, sys_thread_t:0x2207ff00, state:CW, native ID:0xa7c) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-26" (TID:0x5fa9ba8, sys_thread_t:0x2207fd90, state:CW, native ID:0xa70) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-25" (TID:0x5fa9bf8, sys_thread_t:0x2207fb78, state:CW, native ID:0x4d4) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-24" (TID:0x5fa9ca0, sys_thread_t:0x2207f8f0, state:CW, native ID:0x7c8) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-23" (TID:0x5fa9c98, sys_thread_t:0x22055c18, state:CW, native ID:0xa60) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-22" (TID:0x5fa94a0, sys_thread_t:0x220556a0, state:CW, native ID:0x68c) prio=5
         at java.lang.Object.wait(Native Method)
         at org.apache.tomcat.util.ThreadPool$MonitorRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-21" (TID:0x5fa94f0, sys_thread_t:0x220555b8, state:R, native ID:0xb24) prio=5
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:414)
         at java.net.ServerSocket.implAccept(ServerSocket.java:242)
         at java.net.ServerSocket.accept(ServerSocket.java:224)
         at org.apache.tomcat.service.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:286)
         at org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java, Compiled Code)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-20" (TID:0x5fa9540, sys_thread_t:0x22055448, state:CW, native ID:0x664) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-19" (TID:0x5fa9590, sys_thread_t:0x22055360, state:CW, native ID:0x5c0) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-18" (TID:0x5fa9338, sys_thread_t:0x220551f0, state:CW, native ID:0x874) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-17" (TID:0x5fa9388, sys_thread_t:0x22055080, state:CW, native ID:0x4f4) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-16" (TID:0x5fa9480, sys_thread_t:0x22064cf8, state:CW, native ID:0x814) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-15" (TID:0x5fa9428, sys_thread_t:0x22064b88, state:CW, native ID:0x8ec) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-14" (TID:0x5fa9478, sys_thread_t:0x22064aa0, state:CW, native ID:0x24c) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-13" (TID:0x5fa8f68, sys_thread_t:0x22064930, state:CW, native ID:0x410) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-12" (TID:0x5fa8fb8, sys_thread_t:0x22064848, state:CW, native ID:0x3f0) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-11" (TID:0x5fa86b8, sys_thread_t:0x220607f0, state:CW, native ID:0xb28) prio=5
         at java.lang.Object.wait(Native Method)
         at org.apache.tomcat.util.ThreadPool$MonitorRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-10" (TID:0x5fa85f0, sys_thread_t:0x22061618, state:R, native ID:0x6fc) prio=5
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:414)
         at java.net.ServerSocket.implAccept(ServerSocket.java:242)
         at java.net.ServerSocket.accept(ServerSocket.java:224)
         at org.apache.tomcat.service.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:286)
         at org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java, Compiled Code)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-9" (TID:0x5fa8648, sys_thread_t:0x22061530, state:CW, native ID:0x484) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-8" (TID:0x5fa83f0, sys_thread_t:0x220613c0, state:CW, native ID:0x8f0) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-7" (TID:0x5fa8440, sys_thread_t:0x22061250, state:CW, native ID:0x94c) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-6" (TID:0x5fa8490, sys_thread_t:0x220610e0, state:CW, native ID:0x994) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-5" (TID:0x5fa84e0, sys_thread_t:0x2205a2a8, state:CW, native ID:0x530) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-4" (TID:0x5fa8120, sys_thread_t:0x2205a138, state:CW, native ID:0x730) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-3" (TID:0x5fa8170, sys_thread_t:0x2205a050, state:CW, native ID:0x688) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-2" (TID:0x5fa81c0, sys_thread_t:0x2205a560, state:CW, native ID:0x3c0) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-1" (TID:0x5fa8218, sys_thread_t:0x22053520, state:CW, native ID:0x8fc) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "StandardManager" (TID:0x5fa1158, sys_thread_t:0x220596a0, state:CW, native ID:0x724) prio=5
         at java.lang.Thread.sleep(Native Method)
         at org.apache.tomcat.session.StandardManager.threadSleep(StandardManager.java, Compiled Code)
         at org.apache.tomcat.session.StandardManager.run(StandardManager.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "StandardManager" (TID:0x5f9d178, sys_thread_t:0x22031d18, state:CW, native ID:0xaa0) prio=5
         at java.lang.Thread.sleep(Native Method)
         at org.apache.tomcat.session.StandardManager.threadSleep(StandardManager.java, Compiled Code)
         at org.apache.tomcat.session.StandardManager.run(StandardManager.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "StandardManager" (TID:0x5f872f8, sys_thread_t:0x21c8b718, state:CW, native ID:0x880) prio=5
         at java.lang.Thread.sleep(Native Method)
         at org.apache.tomcat.session.StandardManager.threadSleep(StandardManager.java, Compiled Code)
         at org.apache.tomcat.session.StandardManager.run(StandardManager.java, Compiled Code)
         at java.lang.Thread.run(Thread.java:479)
    "Thread-0" (TID:0x5f4e670, sys_thread_t:0x21b0eac0, state:CW, native ID:0x8bc) prio=5
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java, Compiled Code)
         at org.apache.tomcat.util.Queue.pull(Queue.java, Compiled Code)
         at org.apache.tomcat.logging.LogDaemon$1.run(TomcatLogger.java, Compiled Code)
         at org.apache.tomcat.logging.LogDaemon.run(TomcatLogger.java, Compiled Code)
    "SymcJIT-LazyCompilation-1" (TID:0x5f2e4f8, sys_thread_t:0x20e977d8, state:CW, native ID:0x458) prio=1
         at SymantecJITCompilationThread.DoCompileMethod(Native Method)
         at SymantecJITCompilationThread.run(JITcompilationthread.java, Compiled Code)
    "SymcJIT-LazyCompilation-0" (TID:0x5f2e540, sys_thread_t:0x20e9b6d0, state:CW, native ID:0x9bc) prio=1
         at SymantecJITCompilationThread.DoCompileMethod(Native Method)
         at SymantecJITCompilationThread.run(JITcompilationthread.java, Compiled Code)
    "SymcJIT-LazyCompilation-PA" (TID:0x5f2e508, sys_thread_t:0x20e9b560, state:CW, native ID:0x820) prio=10
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at SymantecJITCompilationThread.run(JITcompilationthread.java, Compiled Code)
    "Finalizer" (TID:0x5f29320, sys_thread_t:0x20dddec0, state:CW, native ID:0x4e8) prio=8
         at java.lang.Object.wait(Native Method)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java, Compiled Code)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java, Compiled Code)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:174)
    "Reference Handler" (TID:0x5f293b0, sys_thread_t:0x20ddc7d0, state:CW, native ID:0x55c) prio=10
         at java.lang.Object.wait(Native Method)
         at java.lang.Object.wait(Object.java:424)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:114)
    "Signal dispatcher" (TID:0x5f293e0, sys_thread_t:0x20dda218, state:R, native ID:0xa1c) prio=5
    Monitor Cache Dump:
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8178/6FF4140: <unowned>
         Waiting to be notified:
         "Thread-3" (0x2205a050)
    java.net.PlainSocketImpl@5FA7DC0/6FF18C8: owner "Thread-10" (0x22061618) 1 entry
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8128/6FF4248: <unowned>
         Waiting to be notified:
         "Thread-4" (0x2205a138)
    SymantecJITCompilationThread@5F2E540/6C1F6A0: <unowned>
         Waiting to be notified:
         "SymcJIT-LazyCompilation-PA" (0x20e9b560)
    org.apache.tomcat.util.ThreadPool$MonitorRunnable@5FAA1D8/6FFF340: <unowned>
         Waiting to be notified:
         "Thread-33" (0x22092f98)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA95D8/6FFADA8: <unowned>
         Waiting to be notified:
         "Thread-20" (0x22055448)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA81C8/6FF32F8: <unowned>
         Waiting to be notified:
         "Thread-2" (0x2205a560)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9598/6FFAC90: <unowned>
         Waiting to be notified:
         "Thread-19" (0x22055360)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8448/6FF4978: <unowned>
         Waiting to be notified:
         "Thread-7" (0x22061250)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9C50/6FFBAF8: <unowned>
         Waiting to be notified:
         "Thread-24" (0x2207f8f0)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9430/6FFA840: <unowned>
         Waiting to be notified:
         "Thread-15" (0x22064b88)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9C00/6FFC850: <unowned>
         Waiting to be notified:
         "Thread-25" (0x2207fb78)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA84E8/6FF4350: <unowned>
         Waiting to be notified:
         "Thread-5" (0x2205a2a8)
    org.apache.tomcat.util.ThreadPool$MonitorRunnable@5FA94A8/6FFAFB8: <unowned>
         Waiting to be notified:
         "Thread-22" (0x220556a0)
    org.apache.tomcat.util.Queue@5F4E6C0/6D4B798: <unowned>
         Waiting to be notified:
         "Thread-0" (0x21b0eac0)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8498/6FF4458: <unowned>
         Waiting to be notified:
         "Thread-6" (0x220610e0)
    java.net.PlainSocketImpl@5FA9768/6FFB970: owner "Thread-28" (0x22080518) 1 entry
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8F70/6FFA630: <unowned>
         Waiting to be notified:
         "Thread-13" (0x22064930)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9B60/6FFCA60: <unowned>
         Waiting to be notified:
         "Thread-27" (0x2207ff00)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9740/6FFB9F0: <unowned>
         Waiting to be notified:
         "Thread-23" (0x22055c18)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9340/6FFAB88: <unowned>
         Waiting to be notified:
         "Thread-18" (0x220551f0)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8F20/6FFA738: <unowned>
         Waiting to be notified:
         "Thread-14" (0x22064aa0)
    java.lang.ref.ReferenceQueue$Lock@5F29338/6BF89C0: <unowned>
         Waiting to be notified:
         "Finalizer" (0x20dddec0)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA83F8/6FF4A80: <unowned>
         Waiting to be notified:
         "Thread-8" (0x220613c0)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA93E0/6FFA948: <unowned>
         Waiting to be notified:
         "Thread-16" (0x22064cf8)
    java.net.PlainSocketImpl@5FA8FE8/6FF9860: owner "Thread-21" (0x220555b8) 1 entry
    java.lang.ref.Reference$Lock@5F293C0/6BF84B8: <unowned>
         Waiting to be notified:
         "Reference Handler" (0x20ddc7d0)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8FC0/6FF98E0: <unowned>
         Waiting to be notified:
         "Thread-12" (0x22064848)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9BB0/6FFC958: <unowned>
         Waiting to be notified:
         "Thread-26" (0x2207fd90)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9390/6FFAA80: <unowned>
         Waiting to be notified:
         "Thread-17" (0x22055080)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8650/6FF4B88: <unowned>
         Waiting to be notified:
         "Thread-9" (0x22061530)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9E50/6FFF128: <unowned>
         Waiting to be notified:
         "Thread-31" (0x22080228)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA8220/6FF31A8: <unowned>
         Waiting to be notified:
         "Thread-1" (0x22053520)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9EF0/6FFEF18: <unowned>
         Waiting to be notified:
         "Thread-29" (0x2207e758)
    org.apache.tomcat.util.ThreadPool$MonitorRunnable@5FA86C0/6FF5858: <unowned>
         Waiting to be notified:
         "Thread-11" (0x220607f0)
    org.apache.tomcat.util.ThreadPool$ControlRunnable@5FA9EA0/6FFF020: <unowned>
         Waiting to be notified:
         "Thread-30" (0x22080140)
    Registered Monitor Dump:
    SymcJIT Method Monitor: <unowned>
    SymcJIT Method Monitor: <unowned>
    SymcJIT Method Monitor: <unowned>
    SymcJIT Lazy Queue Lock: <unowned>
         Waiting to be notified:
         "SymcJIT-LazyCompilation-0" (0x20e9b6d0)
         "SymcJIT-LazyCompilation-1" (0x20e977d8)
    SymcJIT Method Monitor: <unowned>
    SymcJIT Method List Monitor: <unowned>
    SymcJIT Lock: <unowned>
    utf8 hash table: <unowned>
    JNI pinning lock: <unowned>
    JNI global reference lock: <unowned>
    BinClass lock: <unowned>
    Class linking lock: <unowned>
    System class loader lock: <unowned>
    Code rewrite lock: <unowned>
    Heap lock: <unowned>
    Monitor cache lock: owner "Signal dispatcher" (0x20dda218) 1 entry
    Thread queue lock: owner "Signal dispatcher" (0x20dda218) 1 entry
         Waiting to be notified:
         "Thread-34" (0x2f4158)
    Monitor registry: owner "Signal dispatcher" (0x20dda218) 1 entry
    </pre>
    Regards,
    Prashant

    No. we are getting the thread dumps only on the production server, and it being running on production, we cannot try it out with newer versions. However, we have so far been unable to simulate the thread dumps on our machines using the same version.

  • 100% CPU load but no clue in thread dump, EP. Win a bottle of champagne...

    Hi,
    We are suddenly facing 100% CPU load in our EP cluster (5 x 8-way Xeon multiprocessor machines). We have a serious performance problem that is burning a lot of our time and causes a lot of stress for over 3 weeks now.
    We have taken tens of thread dumps from Application Nodes.
    - In none of the thread dumps we see the Finalizer thread
    running.
    - From the garbage collector log we see that full garbage
    collection runs occur very rarily (once an hour).
    - We use the compacting garbage collector.
    - In a lot of thread dumps we do not even see our own portal code in the stack traces of the threads!
    - In Windows Task Manager we see the jlaunch.exe processes
    consume all available CPU time.
    - Portal users see a blank portal page when the CPU load hits 100%. When the load goes down again, things return to normal.
    - There are no errors logged in any of the log files of the portal. We checked all of them. We expected a log full of errors somewhere but nothing even remotely interesting was found. Windows Event Viewer shows nothing either.
    - The amount of sockets in CLOSE_WAIT status is < 10 on every machine in the cluster.
    This forces me to conclude that something in the jlaunch.exe executable consumes the CPU time. This raises the following 3 questions:
    - What does the mystifying jlaunch.exe do besides executing java.exe ?
    - Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ? Can it be GZip compression ?
    - If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
    Our development- and support teams are desperate. All suggestions are welcome. The person that comes up with
    the solution to our problem gets a nice bottle of champagne.
    Regards,
    Chris Twigt

    Hi,
    - What does the mystifying jlaunch.exe do besides executing java.exe ?
    This is so that the startupframework can connect more easily and take control of the JVM in some situations
    - Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ?
    I assume when SAP release their JVM (in the next major release), that will be called directly.
    - If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
    Loads of thing, but there should be clues in the thread dumps.  (the reason why your code is not in those thread dumps is that your code is only active during the processing of a particular request, unless you have a service, afterwards there are no trace of it as the thread which does the processing goes back to sleep)
    I've experienced a similar situation with a an 6.40 portal, and it was then caused by the following:
    1. User A comes logs in and sees that some cache timeout has occured , therefore it issues a SQL query which does a full table scan on a table of approx. 1 GB (on of the UME tables)
    2. User B comes in just afterwards and also sees that the cache timeout has occured, and issues the same SQL query as User A
    3. User C .... and so on untill the query from User A eventually finishes
    So
    1. Check database. Any big queries running ?
    2. Check file activity, are there a lot of writing ?
    3. Check network activity, especially to the state controller which is the weakest link
    4. What about portal logs ? Any activity during the hang ?
    Also, please provide one of the thread dumps for further analysis..
    cheers
    Dagfinn

  • Generate thread dump without sending ctrl-break on Win32

    I have a server application running as a service on Win2k (we use Service+ to do this). The output of the console is piped to a text file which we can monitor. But, I can't figure out how to generate a thread dump while running in this configuration. Does anyone know if Service+ can be configured to do this, or if it can be achieved programmatically (how does Sun spit out the thread info when it gets the request).
    Thanks,
    Larry

    If you are running a java application on a Windows server. You can press Ctrl-Break, and a thread dump will be displayed in the console. This thread dump is a list of all the threads running in the JVM, and the line of code they are each executing.
    We'd like to be able to generate this thread dump some other way than pressing Ctrl-Break, because we don't have a console, our application is running as a service using Service .
    Thanks for the reply,
    Larry0AThanks for the reply,
    Larry

  • WebLogic 8.1 SP4 on Linux - thread dump does not work

    I have setup WebLogic 8.1 SP4 on RedHat Linux AS 3.3 with Sun JVM (142_05). When I try to perform a thread dump using weblogic.Admin, I receive the following error in the stderr file. I have also tried the kill -3 but still no thread dump. Is there some trick to this ?
    cannot load libary 'stackdump': java.lang.UnsatisfiedLinkError: no stackdump in java.library.path

    I am getting the same error as well
    cannot load libary 'stackdump': java.lang.UnsatisfiedLinkError: no stackdump in java.library.path
    I am creating a simple domain using the domain config wizard and I hope all the PATH and LD_LIBRARY_PATH are set by the default scripts. But I am still getting this error.
    But Kill -3 <PID> works though!
    Is is a know problem OR am I missing something?!
    Bala

  • JVM using abt 100% CPU. JVM thrad dump

    I am having some problem in my Java Application. The idle CPU usage is always high to abt 91 % ..
    My java version is
    java version "1.4.2_04"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_04-b05)
    Java HotSpot(TM) Client VM (build 1.4.2_04-b05, mixed mode)
    Can someone help to analyze the thread dump taken from JVM.. given below...
    =================================================Thread dump ===========
    Full thread dump Java HotSpot(TM) Server VM (1.4.0_00-b05 mixed mode):
    "ORBacus:Client:ReceiverThread" prio=5 tid=0x673c660 nid=0x5627 runnable [df1ff000..df1ffc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.ooc.OCI.IIOP.Transport_impl.receive(Transport_impl.java:210)
         at com.ooc.OB.GIOPClientWorkerThreaded.receiverRun(GIOPClientWorkerThreaded.java:492)
         at com.ooc.OB.GIOPClientWorkerThreaded$ReceiverThread.run(GIOPClientWorkerThreaded.java:112)
    "ORBacus:Client:SenderThread" prio=5 tid=0x5d4ff80 nid=0x5626 waiting on monitor [e0bff000..e0bffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <e64107b0> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded.senderRun(GIOPClientWorkerThreaded.java:447)
         - locked <e64107b0> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded$SenderThread.run(GIOPClientWorkerThreaded.java:33)
    "ORBacus:Client:ReceiverThread" daemon prio=5 tid=0x674f6a0 nid=0x5624 runnable [df5ff000..df5ffc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.ooc.OCI.IIOP.Transport_impl.receive(Transport_impl.java:210)
         at com.ooc.OB.GIOPClientWorkerThreaded.receiverRun(GIOPClientWorkerThreaded.java:492)
         at com.ooc.OB.GIOPClientWorkerThreaded$ReceiverThread.run(GIOPClientWorkerThreaded.java:112)
    "ORBacus:Client:SenderThread" daemon prio=5 tid=0x63401d0 nid=0x5623 waiting on monitor [df2ff000..df2ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <e6418410> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded.senderRun(GIOPClientWorkerThreaded.java:447)
         - locked <e6418410> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded$SenderThread.run(GIOPClientWorkerThreaded.java:33)
    "ORBacus:Server:ReceiverThread" prio=5 tid=0x670fa28 nid=0x5621 runnable [dfdff000..dfdffc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.ooc.OCI.IIOP.Transport_impl.receive_detect(Transport_impl.java:259)
         at com.ooc.OB.GIOPServerWorkerThreaded.receiverRun(GIOPServerWorkerThreaded.java:338)
         at com.ooc.OB.GIOPServerWorkerThreaded$ReceiverThread.run(GIOPServerWorkerThreaded.java:106)
    "ORBacus:Server:SenderThread" prio=5 tid=0x6643eb0 nid=0x5620 waiting on monitor [e1d7f000..e1d7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ed721158> (a com.ooc.OB.GIOPServerWorkerThreaded)
         at com.ooc.OB.GIOPServerWorkerThreaded.senderRun(GIOPServerWorkerThreaded.java:286)
         - locked <ed721158> (a com.ooc.OB.GIOPServerWorkerThreaded)
         at com.ooc.OB.GIOPServerWorkerThreaded$SenderThread.run(GIOPServerWorkerThreaded.java:33)
    "RMI ConnectionExpiration-[10.0.0.23:65435]" daemon prio=5 tid=0x5f112c0 nid=0x5619 waiting on monitor [e06ff000..e06ffc50]
         at java.lang.Thread.sleep(Native Method)
         at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:447)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-5955" daemon prio=6 tid=0x5f388d8 nid=0x560d runnable [df3ff000..df3ffc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <ed700d18> (a java.io.BufferedInputStream)
         at java.io.FilterInputStream.read(FilterInputStream.java:66)
         at java.io.PushbackInputStream.read(PushbackInputStream.java:120)
         at org.apache.commons.net.io.FromNetASCIIInputStream.__read(FromNetASCIIInputStream.java:75)
         at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:170)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <ed701560> (a org.apache.commons.net.telnet.TelnetInputStream)
         at org.apache.commons.net.telnet.TelnetInputStream.__read(TelnetInputStream.java:114)
         at org.apache.commons.net.telnet.TelnetInputStream.run(TelnetInputStream.java:535)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-5954" daemon prio=6 tid=0x367b510 nid=0x560c runnable [e187f000..e187fc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <ed701e38> (a java.io.BufferedInputStream)
         at java.io.FilterInputStream.read(FilterInputStream.java:66)
         at java.io.PushbackInputStream.read(PushbackInputStream.java:120)
         at org.apache.commons.net.io.FromNetASCIIInputStream.__read(FromNetASCIIInputStream.java:75)
         at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:170)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <ed702680> (a org.apache.commons.net.telnet.TelnetInputStream)
         at org.apache.commons.net.telnet.TelnetInputStream.__read(TelnetInputStream.java:114)
         at org.apache.commons.net.telnet.TelnetInputStream.run(TelnetInputStream.java:535)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-5952" daemon prio=6 tid=0x633ad20 nid=0x5606 runnable [e05ff000..e05ffc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <ed703220> (a java.io.BufferedInputStream)
         at java.io.FilterInputStream.read(FilterInputStream.java:66)
         at java.io.PushbackInputStream.read(PushbackInputStream.java:120)
         at org.apache.commons.net.io.FromNetASCIIInputStream.__read(FromNetASCIIInputStream.java:75)
         at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:170)
         at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
         at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
         - locked <ed703a68> (a org.apache.commons.net.telnet.TelnetInputStream)
         at org.apache.commons.net.telnet.TelnetInputStream.__read(TelnetInputStream.java:114)
         at org.apache.commons.net.telnet.TelnetInputStream.run(TelnetInputStream.java:535)
         at java.lang.Thread.run(Thread.java:536)
    "ORBacus:Client:ReceiverThread" prio=5 tid=0x5b0c2c8 nid=0x5602 runnable [e267f000..e267fc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.ooc.OCI.IIOP.Transport_impl.receive(Transport_impl.java:210)
         at com.ooc.OB.GIOPClientWorkerThreaded.receiverRun(GIOPClientWorkerThreaded.java:492)
         at com.ooc.OB.GIOPClientWorkerThreaded$ReceiverThread.run(GIOPClientWorkerThreaded.java:112)
    "ORBacus:Client:SenderThread" prio=5 tid=0x63ba5a8 nid=0x5601 waiting on monitor [e03ff000..e03ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ed703cc0> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded.senderRun(GIOPClientWorkerThreaded.java:447)
         - locked <ed703cc0> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded$SenderThread.run(GIOPClientWorkerThreaded.java:33)
    "ORBacus:Server:ReceiverThread" prio=5 tid=0x5454f88 nid=0x55f3 runnable [e167f000..e167fc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.ooc.OCI.IIOP.Transport_impl.receive_detect(Transport_impl.java:259)
         at com.ooc.OB.GIOPServerWorkerThreaded.receiverRun(GIOPServerWorkerThreaded.java:338)
         at com.ooc.OB.GIOPServerWorkerThreaded$ReceiverThread.run(GIOPServerWorkerThreaded.java:106)
    "ORBacus:Server:SenderThread" prio=5 tid=0xefefd0 nid=0x55f2 waiting on monitor [df6ff000..df6ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ed6bfd50> (a com.ooc.OB.GIOPServerWorkerThreaded)
         at com.ooc.OB.GIOPServerWorkerThreaded.senderRun(GIOPServerWorkerThreaded.java:286)
         - locked <ed6bfd50> (a com.ooc.OB.GIOPServerWorkerThreaded)
         at com.ooc.OB.GIOPServerWorkerThreaded$SenderThread.run(GIOPServerWorkerThreaded.java:33)
    "ORBacus:Server:ReceiverThread" prio=5 tid=0x67ef270 nid=0x55ae runnable [e157f000..e157fc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.ooc.OCI.IIOP.Transport_impl.receive_detect(Transport_impl.java:259)
         at com.ooc.OB.GIOPServerWorkerThreaded.receiverRun(GIOPServerWorkerThreaded.java:338)
         at com.ooc.OB.GIOPServerWorkerThreaded$ReceiverThread.run(GIOPServerWorkerThreaded.java:106)
    "ORBacus:Server:SenderThread" prio=5 tid=0x5b93508 nid=0x55ad waiting on monitor [e09ff000..e09ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ed6b0350> (a com.ooc.OB.GIOPServerWorkerThreaded)
         at com.ooc.OB.GIOPServerWorkerThreaded.senderRun(GIOPServerWorkerThreaded.java:286)
         - locked <ed6b0350> (a com.ooc.OB.GIOPServerWorkerThreaded)
         at com.ooc.OB.GIOPServerWorkerThreaded$SenderThread.run(GIOPServerWorkerThreaded.java:33)
    "ORBacus:Client:ReceiverThread" prio=5 tid=0x5a16028 nid=0x558d runnable [dfaff000..dfaffc50]
         at java.net.SocketInputStream.socketRead0(Native Method)
         at java.net.SocketInputStream.read(SocketInputStream.java:116)
         at com.ooc.OCI.IIOP.Transport_impl.receive(Transport_impl.java:210)
         at com.ooc.OB.GIOPClientWorkerThreaded.receiverRun(GIOPClientWorkerThreaded.java:492)
         at com.ooc.OB.GIOPClientWorkerThreaded$ReceiverThread.run(GIOPClientWorkerThreaded.java:112)
    "ORBacus:Client:SenderThread" prio=5 tid=0x280f9b0 nid=0x558c waiting on monitor [df8ff000..df8ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ed6ad270> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded.senderRun(GIOPClientWorkerThreaded.java:447)
         - locked <ed6ad270> (a com.ooc.OB.GIOPClientWorkerThreaded)
         at com.ooc.OB.GIOPClientWorkerThreaded$SenderThread.run(GIOPClientWorkerThreaded.java:33)
    "Thread-3590" daemon prio=5 tid=0x248e790 nid=0x2ce5 runnable [e137f000..e137fc50]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
         - locked <ed55d900> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:438)
         at java.net.ServerSocket.accept(ServerSocket.java:409)
         at org.apache.log4j.net.SocketHubAppender$ServerMonitor.run(SocketHubAppender.java:336)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-2722" prio=5 tid=0x142fd30 nid=0x24dc waiting on monitor [e1c7f000..e1c7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ed517660> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <ed517660> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-2721" prio=5 tid=0x1838ae8 nid=0x24db waiting on monitor [dfbff000..dfbffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ed5176c0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <ed5176c0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-1869" daemon prio=5 tid=0x11d9f88 nid=0x1971 waiting on monitor [e177f000..e177fc50]
         at java.lang.Thread.sleep(Native Method)
         at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-806" daemon prio=5 tid=0x756930 nid=0xb3e waiting on monitor [e02ff000..e02ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebf302e0> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <ebf302e0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-193" daemon prio=5 tid=0x6b4858 nid=0x288 waiting on monitor [e0aff000..e0affc50]
         at java.lang.Thread.sleep(Native Method)
         at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-189" daemon prio=5 tid=0xc36fe8 nid=0x27e waiting on monitor [e1a7f000..e1a7fc50]
         at java.lang.Thread.sleep(Native Method)
         at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
         at java.lang.Thread.run(Thread.java:536)
    "GC Daemon" daemon prio=2 tid=0x783d10 nid=0x60 waiting on monitor [e10ff000..e10ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebe764c8> (a sun.misc.GC$LatencyLock)
         at sun.misc.GC$Daemon.run(GC.java:100)
         - locked <ebe764c8> (a sun.misc.GC$LatencyLock)
    "RMI Reaper" prio=5 tid=0x339e98 nid=0x5f waiting on monitor [e0fff000..e0fffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebe75b40> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <ebe75b40> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:330)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-30" daemon prio=5 tid=0xcdb3f8 nid=0x5e waiting on monitor [e237f000..e237fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebe75de0> (a java.util.TaskQueue)
         at java.lang.Object.wait(Object.java:426)
         at java.util.TimerThread.mainLoop(Timer.java:403)
         - locked <ebe75de0> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "RMI TCP Accept-0" daemon prio=5 tid=0x88d0f8 nid=0x5d runnable [e227f000..e227fc50]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
         - locked <ebe75b98> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:438)
         at java.net.ServerSocket.accept(ServerSocket.java:409)
         at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
         at java.lang.Thread.run(Thread.java:536)
    "AllocationEventDispatcher" prio=5 tid=0xbbd388 nid=0x5c waiting on monitor [e0eff000..e0effc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebcdd8e0> (a java.util.LinkedList)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.prov.client.EventUpdate_impl.run(EventUpdate_impl.java:61)
         - locked <ebcdd8e0> (a java.util.LinkedList)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-28" prio=5 tid=0x6e14c8 nid=0x58 waiting on monitor [e11ff000..e11ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebcdd958> (a java.util.LinkedList)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.prov.client.AlopaProvisioningClientManager.run(AlopaProvisioningClientManager.java:114)
         - locked <ebcdd958> (a java.util.LinkedList)
         at java.lang.Thread.run(Thread.java:536)
    "ProvClientRegistry-sweeper" daemon prio=5 tid=0x6fb338 nid=0x3e waiting on monitor [e197f000..e197fc50]
         at java.lang.Thread.sleep(Native Method)
         at com.alopa.prov.client.ProvisioningClientRegistry.run(ProvisioningClientRegistry.java:199)
    "Thread-20" prio=5 tid=0x2c7b8 nid=0x1 waiting on monitor [0..ffbfe4e0]
    "GeneratorManager" prio=5 tid=0x12cb6e0 nid=0x36 waiting on monitor [e1e7f000..e1e7fc50]
         at java.lang.Thread.sleep(Native Method)
         at com.alopa.metaserv.health.generator.GeneratorManager.run(GeneratorManager.java:102)
         at java.lang.Thread.run(Thread.java:536)
    "PSThread: 1" prio=5 tid=0xe2df20 nid=0x35 waiting on monitor [e1f7f000..e1f7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fe10> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fe10> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "PSThread: 2" prio=5 tid=0xe2ddd8 nid=0x34 waiting on monitor [e207f000..e207fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fdf0> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fdf0> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "PSThread: 3" prio=5 tid=0x1696788 nid=0x33 waiting on monitor [e217f000..e217fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fdd0> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fdd0> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "PSThread: 4" prio=5 tid=0x1696640 nid=0x32 waiting on monitor [e2a7f000..e2a7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fdb0> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fdb0> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "PSThread: 5" prio=5 tid=0x16964f8 nid=0x31 waiting on monitor [e297f000..e297fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fd90> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fd90> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "PSThread: 6" prio=5 tid=0x9d7038 nid=0x30 waiting on monitor [e337f000..e337fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fd70> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fd70> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "PSThread: 7" prio=5 tid=0x9d6ef0 nid=0x2f waiting on monitor [e327f000..e327fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fd50> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fd50> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "PSThread: 8" prio=5 tid=0xe2dc90 nid=0x2e waiting on monitor [e367f000..e367fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc9fd30> (a com.alopa.util.thread.ThreadPoolEntry)
         at java.lang.Object.wait(Object.java:426)
         at com.alopa.util.thread.ThreadPoolEntry.waitForRelease(ThreadPoolEntry.java:61)
         - locked <ebc9fd30> (a com.alopa.util.thread.ThreadPoolEntry)
         at com.alopa.util.thread.PoolThread.run(PoolThread.java:74)
    "ConnectionListener" prio=5 tid=0x1ddcc8 nid=0x2d runnable [e357f000..e357fc50]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
         - locked <ebc374a8> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:438)
         at java.net.ServerSocket.accept(ServerSocket.java:409)
         at com.alopa.util.misc.JythonServerStart.run(JythonServerStart.java:64)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-16" daemon prio=5 tid=0x8ab9d0 nid=0x28 waiting on monitor [e277f000..e277fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4ba50> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <ebc4ba50> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-15" daemon prio=5 tid=0x697218 nid=0x27 waiting on monitor [e287f000..e287fc50]
         at java.lang.Thread.sleep(Native Method)
         at org.apache.torque.oid.IDBroker.run(IDBroker.java:503)
         at java.lang.Thread.run(Thread.java:536)
    "ORBacus:ThreadPool-0:Dispatcher-6" prio=5 tid=0x1a92e0 nid=0x23 waiting on monitor [e2b7f000..e2b7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ThreadPool.get(ThreadPool.java:143)
         - locked <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at com.ooc.OB.ThreadPool.access$0(ThreadPool.java:137)
         at com.ooc.OB.ThreadPool$Dispatcher.run(ThreadPool.java:39)
    "ORBacus:ThreadPool-0:Dispatcher-5" prio=5 tid=0x1a9160 nid=0x22 waiting on monitor [e2c7f000..e2c7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ThreadPool.get(ThreadPool.java:143)
         - locked <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at com.ooc.OB.ThreadPool.access$0(ThreadPool.java:137)
         at com.ooc.OB.ThreadPool$Dispatcher.run(ThreadPool.java:39)
    "ORBacus:ThreadPool-0:Dispatcher-4" prio=5 tid=0xbe63a0 nid=0x21 waiting on monitor [e2d7f000..e2d7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ThreadPool.get(ThreadPool.java:143)
         - locked <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at com.ooc.OB.ThreadPool.access$0(ThreadPool.java:137)
         at com.ooc.OB.ThreadPool$Dispatcher.run(ThreadPool.java:39)
    "ORBacus:ThreadPool-0:Dispatcher-3" prio=5 tid=0xbe6258 nid=0x20 waiting on monitor [e2e7f000..e2e7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ThreadPool.get(ThreadPool.java:143)
         - locked <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at com.ooc.OB.ThreadPool.access$0(ThreadPool.java:137)
         at com.ooc.OB.ThreadPool$Dispatcher.run(ThreadPool.java:39)
    "ORBacus:ThreadPool-0:Dispatcher-2" prio=5 tid=0xce31d0 nid=0x1f waiting on monitor [e2f7f000..e2f7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ThreadPool.get(ThreadPool.java:143)
         - locked <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at com.ooc.OB.ThreadPool.access$0(ThreadPool.java:137)
         at com.ooc.OB.ThreadPool$Dispatcher.run(ThreadPool.java:39)
    "ORBacus:ThreadPool-0:Dispatcher-1" prio=5 tid=0xce3088 nid=0x1e waiting on monitor [e307f000..e307fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ThreadPool.get(ThreadPool.java:143)
         - locked <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at com.ooc.OB.ThreadPool.access$0(ThreadPool.java:137)
         at com.ooc.OB.ThreadPool$Dispatcher.run(ThreadPool.java:39)
    "ORBacus:ThreadPool-0:Dispatcher-0" prio=5 tid=0x90e208 nid=0x1d waiting on monitor [e317f000..e317fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ThreadPool.get(ThreadPool.java:143)
         - locked <ebc4bb50> (a com.ooc.OB.ThreadPool)
         at com.ooc.OB.ThreadPool.access$0(ThreadPool.java:137)
         at com.ooc.OB.ThreadPool$Dispatcher.run(ThreadPool.java:39)
    "ORBacus:Server:StarterThread" prio=5 tid=0x2e7148 nid=0x19 runnable [e347f000..e347fc50]
         at java.net.PlainSocketImpl.socketAccept(Native Method)
         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:343)
         - locked <ebc4bda8> (a java.net.PlainSocketImpl)
         at java.net.ServerSocket.implAccept(ServerSocket.java:438)
         at java.net.ServerSocket.accept(ServerSocket.java:409)
         at com.ooc.OCI.IIOP.Acceptor_impl.accept(Acceptor_impl.java:128)
         at com.ooc.OB.GIOPServerStarterThreaded.starterRun(GIOPServerStarterThreaded.java:227)
         at com.ooc.OB.GIOPServerStarterThreaded$StarterThread.run(GIOPServerStarterThreaded.java:33)
    "ORB Thread" prio=5 tid=0xb761e8 nid=0x15 waiting on monitor [e377f000..e377fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc3cda0> (a java.lang.Object)
         at java.lang.Object.wait(Object.java:426)
         at com.ooc.OB.ORBControl.run(ORBControl.java:313)
         - locked <ebc3cda0> (a java.lang.Object)
         at com.ooc.OBCORBA.ORB_impl.run(ORB_impl.java:1067)
         at com.alopa.metaserv.shared.startup.CorbaOrb.run(Unknown Source)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-11" prio=5 tid=0xce1870 nid=0x14 waiting on monitor [e387f000..e387fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc3a920> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <ebc3a920> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "FileTracker: generators.xml" prio=5 tid=0xc7ac80 nid=0x13 waiting on monitor [e397f000..e397fc50]
         at java.lang.Thread.sleep(Native Method)
         at com.alopa.metaserv.health.util.FileTracker.run(FileTracker.java:53)
    "Thread-10" prio=5 tid=0xc99560 nid=0x12 waiting on monitor [e3a7f000..e3a7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc379f8> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <ebc379f8> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "Thread-9" daemon prio=5 tid=0xcddd78 nid=0x11 runnable [e3b7f000..e3b7fc50]
         at java.net.PlainDatagramSocketImpl.receive(Native Method)
         - waiting to lock <ebc34518> (a java.net.PlainDatagramSocketImpl)
         at java.net.DatagramSocket.receive(DatagramSocket.java:670)
         - locked <ed7041a0> (a java.net.DatagramPacket)
         - locked <ebc34458> (a java.net.DatagramSocket)
         at com.alopa.mgmt.core.snmp.SnmpMonitor.run(SnmpMonitor.java:81)
    "Thread-8" daemon prio=5 tid=0x636ee8 nid=0x10 waiting on monitor [e3c7f000..e3c7fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebc32fb8> (a java.util.TaskQueue)
         at java.util.TimerThread.mainLoop(Timer.java:429)
         - locked <ebc32fb8> (a java.util.TaskQueue)
         at java.util.TimerThread.run(Timer.java:382)
    "PrimaryDBConnectionPoller-1" daemon prio=5 tid=0xb75270 nid=0xf waiting on monitor [e3d7f000..e3d7fc50]
         at java.lang.Thread.sleep(Native Method)
         at com.alopa.util.dbutils.FailSafeDataSource.run(FailSafeDataSource.java:249)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-5" daemon prio=5 tid=0xb73c00 nid=0xe waiting on monitor [e3e7f000..e3e7fc50]
         at java.lang.Thread.sleep(Native Method)
         at com.codestudio.util.LifeGuardThread.run(LifeGuardThread.java)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-4" daemon prio=5 tid=0xb73ab8 nid=0xd waiting on monitor [e3f7f000..e3f7fc50]
         at java.lang.Thread.sleep(Native Method)
         at com.codestudio.util.PoolSkimmerThread.run(PoolSkimmerThread.java)
         at java.lang.Thread.run(Thread.java:536)
    "Thread-1" daemon prio=5 tid=0x57f938 nid=0xb waiting on monitor [e407f000..e407fc50]
         at java.lang.Thread.sleep(Native Method)
         at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)
    "Signal Dispatcher" daemon prio=10 tid=0x10ef20 nid=0x7 waiting on monitor [0..0]
    "Finalizer" daemon prio=8 tid=0xf1608 nid=0x4 waiting on monitor [e637f000..e637fc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebb8c840> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
         - locked <ebb8c840> (a java.lang.ref.ReferenceQueue$Lock)
         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
    "Reference Handler" daemon prio=10 tid=0xf0530 nid=0x3 waiting on monitor [fa4ff000..fa4ffc50]
         at java.lang.Object.wait(Native Method)
         - waiting on <ebb8c8a8> (a java.lang.ref.Reference$Lock)
         at java.lang.Object.wait(Object.java:426)
         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
         - locked <ebb8c8a8> (a java.lang.ref.Reference$Lock)
    "VM Thread" prio=5 tid=0xef9b0 nid=0x2 runnable
    "VM Periodic Task Thread" prio=10 tid=0x10dc60 nid=0x5 waiting on monitor
    "Suspend Checker Thread" prio=10 tid=0x10e5c0 nid=0x6 runnable

    Hi,
    - What does the mystifying jlaunch.exe do besides executing java.exe ?
    This is so that the startupframework can connect more easily and take control of the JVM in some situations
    - Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ?
    I assume when SAP release their JVM (in the next major release), that will be called directly.
    - If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?
    Loads of thing, but there should be clues in the thread dumps.  (the reason why your code is not in those thread dumps is that your code is only active during the processing of a particular request, unless you have a service, afterwards there are no trace of it as the thread which does the processing goes back to sleep)
    I've experienced a similar situation with a an 6.40 portal, and it was then caused by the following:
    1. User A comes logs in and sees that some cache timeout has occured , therefore it issues a SQL query which does a full table scan on a table of approx. 1 GB (on of the UME tables)
    2. User B comes in just afterwards and also sees that the cache timeout has occured, and issues the same SQL query as User A
    3. User C .... and so on untill the query from User A eventually finishes
    So
    1. Check database. Any big queries running ?
    2. Check file activity, are there a lot of writing ?
    3. Check network activity, especially to the state controller which is the weakest link
    4. What about portal logs ? Any activity during the hang ?
    Also, please provide one of the thread dumps for further analysis..
    cheers
    Dagfinn

  • How to find out the JVM thread that is consuming high CPU

    I am using WL 10.3, JRockit JVM on Linux. I have taken thread dumps. How to find out the JVM thread that is consuming high CPU?
    I know how to map it Sun JVM on Solaris. I haven't done in on JRockit so far. Please advise.

    Check out some of Marcus Hirt's blog entries for details about how to use JRockit Mission Control. It's a very nice tool that should provide a lot of insight:
    http://blogs.oracle.com/hirt/

  • Thread Dump issue with LD_ASSUME_KERNEL=2.4.1

    Hi ,
    When I take Thread Dump using 'jstack <PID>' in JDK 1.5 it givss me "sun.jvm.hotspot.debugger.DebuggerException" in the dump nothing more
    I set LD_ASSUME_KERNEL=2.4.1 in my server to avaoid some other issue ( JVM crash some times)
    Surprisingly, I can not stop my server ( my java process ) after that using our Shutdow scripts , Ctrl C or even "kill -9 <PID>",
    I have to restart the machine or manually release uncleaned resources that my server occupied and restart the server
    This happens Redhar 9 as well as in Linux ES.
    Anybody faced similar problem?
    Any help or information regarding this is highly apprecialted
    Vasu
    Thread Dump Output:
    Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Meth
    od)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:34)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(Li
    nuxDebuggerLocal.java:431)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(Linux
    DebuggerLocal.java:109)
    sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_re
    gs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(L
    inuxDebuggerLocal.java:134)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebugge
    rLocal.java:437)
    at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:48)
    at sun.jvm.hotspot.runtime.linux_x86.LinuxX86JavaThreadPDAccess.getCurrentFrameGuess(LinuxX86
    JavaThreadPDAccess.java:75)
    at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:252)
    at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:211)
    at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:42)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:58)
    Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Meth
    od)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$800(LinuxDebuggerLocal.java:34)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(Li
    nuxDebuggerLocal.java:431)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(Linux
    DebuggerLocal.java:109)
    sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_re
    gs failed for a lwp
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(L
    inuxDebuggerLocal.java:134)
    at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebugge
    rLocal.java:437)

    @brain0
    I've downloaded the glic-2.3.6 sources from gnu, so I could build it from those. I'm however reluctant to do this because I really don't want to break my install.
    I do agree with you on the NPTL statement, but pvs relies on allegro, which relies on LinuxThreads. Allegro is not being ported to new versions of glibc, so that approach is unfortunately not viable.
    @iphitus
    I wasn't very specific - it's allegro as in a lisp environment.
    I think I'll try and install an old version of arch on wmware instead. Is there anywhere you can check out glibc version numbering on old arch install isos (ie. do I need arch-0.[1-9].iso)? And anywhere you can download the old isos (tried filewatcher, but a lot of the older sites seem broken)?
    Thanks for the replies,
    Mads
    PS. I noticed that you recommended slackware for old kernels in another thread. I'm however in a bit different situation as I need old versions of glibc. Furthermore I would prefer sticking to arch, but was wondering whether there were any specific reasons for not doing that.

Maybe you are looking for

  • How do I create an FLV Preloader?

    Hello everyone, I am an Action Script illiterate or... let's put it this way, I am currently learning it. I have created a video in After Effects that I exported to an FLV movie. I would like to use it as an intro for my website but I would like to h

  • I want to create a bean wich on a keystroke add a button to a JtextPane

    The bean have two JComponents : a JTextPane and a JButton. Innitialy the bean must show just the JTextPane. When I press key F1 I want that the JButton to be added on the right part of the JTextPane, and be painted inside JTextPane. I've tried someth

  • Photoshop CS6 Photomerge for astrophotography

    I have been have been unsuccessful using photomerge to stitch 3 shot panoramas of the night sky (milky way and stars only) with manual settings, using a tilt shift lens with plenty of overlap.  Perhaps the part of the problem is that the photos have

  • Annoying little O2 Icon

    So two days into my S6 experience and this is annoying me!   If you click on it you get this  These are various numbers - just out of interest the Customer Srv number is for PAYG - I am on refresh. Despite my best efforts I can't delete this O2 icon,

  • Buil-in microphone frequency response graph

    I'm curious as to where I can find a frequency response graph for the built-in mic in the 15" MBP 2.8. I know it's a silly thing to want, but... Sometimes I don't have the time or resources to set-up an external mic with a flat frequency response, an