Server Socket Leaks on stopping Java Processes

I have a strange problem on Windows 2000 Advanced Server with JDK1.5.10. Our application runs on a Tomcat 5.5.12 and is opening 2 server sockets along with Tomcat's own 8080 port. Also we deployed a stand alone application that also opens a server socket. Whenever we stop Tomcat service or our Application service (Windows services) these server sockets are not getting closed. If we use "netstat" to check if any sockets are still open it is NOT showing any sockets.
But when we use "netstat -a" command it is showing entries like
TCP    10.40.1.162:8080    10.40.1.162:0  LISTENING
TCP    10.40.1.162:22222    10.40.1.162:0  LISTENING
TCP    10.40.1.162:33334    10.40.1.162:0  LISTENINGHere 8080 (tomcat), 22222, 33334 are the ports on which the server sockets are opened earlier. We check in Task Manager but there is no java process running. If we try to restart the same java process it is throwing JVM_Bind exception. If we reboot the box the we are able to restart the services. Similar behaviour we observed sometimes on Linux also. I guess some settings in the OS we need to tweak to fix this issue. Can somebody please help me what are those settings for Windows and Linux platforms which reclaims unused server sockets faster??

The state is LISTENING not TIME_WAIT. There is a cleanly closure of sockets provided in the code. It happens to Tomcat's own ports also. If you telnet to these sockets from another box, it is connecting and staying instead of "Connection Failed" error. But in Task Manager or "Services.msc" Console no service or process running! One more observation is that there is heavy load of threads and a huge number of clients socktets were interacting with these sockets just before shutdown.

Similar Messages

  • How do I stop Java processes from appearing on the dock?

    I have a Java application that has a fairly extensive build process using ant. It builds fine on OSX, but constantly annoys me. There are a large number of unit tests that run using JUnit, and each test starts in a new Java process. The problem is that when each process starts (and there are about 100 during the course of the build), a new Java icon appears on the dock and stays until the process finishes. First of all, that slows down the build (since it has to do all that graphics work to keep changing the appearance of the dock). Secondly, every time a new process starts, it takes away the keyboard focus from whatever I'm currently working on. I either have to click in the window I want again, or else wait for the process to finish.
    On Windows, this is handled with an alternative version of the JVM called "javaw.exe" that does not get a foreground window. However, I see no "javaw" in the Java directory on OSX. Is there some option to start Java with that will stop this dock updating? Note that there are times when I want the dock to be updated (when I'm running a "normal" Java app).

    Even when the Guest Account / Access is disabled, the login option / and lock screen, still have a guest login.
    This is because in Preferences/ Security & Privacy/General the bottom item is turned on.
    Turn on: Disable restarting to Safari when screen is locked

  • Socket stays open after java process exits, Runtime.exec()

    I have a program that does the following:
    opens a socket
    Does a runtime.exec() of another program
    then the main program exits.
    what i am seeing is, as long as the exec'd program is running, the socket remains open.
    What can i do to get the socket to close?
    I even tried to explicity call close() on it, and that didn't work. Any ideas would be great.
    I am running this on WindowsXP using netstat to monitor the port utilization.
    here is some sample code
    import java.io.*;
    import java.net.*;
    public class ForkTest
        public static void main(String[] args)
            try
                DatagramSocket s = new DatagramSocket(2006);
                Process p = Runtime.getRuntime().exec("notepad.exe");
                System.out.println("Press any key to exit");
                System.in.read();
            catch (IOException ex)
                ex.printStackTrace();
    }

    java.net.BindException: Address already in use: Cannot bind
            at java.net.PlainDatagramSocketImpl.bind(Native Method)
            at java.net.DatagramSocket.bind(DatagramSocket.java:368)
            at java.net.DatagramSocket.<init>(DatagramSocket.java:210)
            at java.net.DatagramSocket.<init>(DatagramSocket.java:261)
            at java.net.DatagramSocket.<init>(DatagramSocket.java:234)
            at ForkTest.main(ForkTest.java:11)

  • OBIEE Server Hangs how to stop the process?

    Hi,
    I schedule one report in BI publisher and in history its showing status as ‘Running’ from a long time, I want to terminate that process and want to stop the running report.
    But as I don’t no much about OBIEE I am unable to do that. Could you please help me with this.
    Thank you.

    Hi,
      At times even I have faced this situation.  During those times, running the Function module RSPC_PROCESS_FINISH by passing the right parameters came to my rescue.
    Regards,
    Raj

  • How to create a server socket

    I want to make a server socket.
    anybody provide me java store procedure code for socket, so that i can send data from my plsql procedure to my c# client also receive data which is send my client.

    The database listener is providing you perfect server socket.
    If you insist in opening the socket in java, I would suggest using a java app server, instead of the in-database jvm.

  • Could not stop SDM server. Cannot execute Java process for stopSDM with parameters 50018

    dear SAP
    i have a system Solman 7.1
    i'm upgrading Solman from SP04 to SP13 using SUM (SP12 patch 10) on windows 2012
    and i faced  error  during  Excution phase 5.6  in JAVA Stack.
    phase : Excution - Actual Update
    Error : An error has occurred during the execution of the Deploy software components step.
    Could not stop SDM server. Cannot execute Java process for stopSDM with parameters 50018. Return code condition success evaluated to false for process java.exe for action ACTION_STOP
    You can find more information in the files D:\usr\sap\SOL\SUM\sdt\log\SUM\DEPLOY-COMPONENTS_22.LOG and ProcessOverview.html.
    To troubleshoot the problem use the provided information. An SAP Note may provide a solution to this problem. Search for SAP Notes with the following keywords:
    SUM??execute??component-deploy-700??deploy-components??com.sap.sdt.j2ee.services.servicesimpl.SDMService??class com.sap.sdt.j2ee.tools.sdmmanagement.OperateSdmException
    i try to restart server and killed sapstartsrv..  but the problem is not solved. 
    please let me know what i check more?
    D:\usr\sap\SOL\SUM\sdt\log\SUM\DEPLOY-COMPONENTS_22.LOG
    <!--LOGHEADER[START]/-->
    <!--HELP[Manual modification of the header may cause parsing problem!]/-->
    <!--LOGGINGVERSION[2.0.7.1006]/-->
    <!--NAME[D:\usr\sap\SOL\SUM\sdt\log\SUM\DEPLOY-COMPONENTS_22.LOG]/-->
    <!--PATTERN[DEPLOY-COMPONENTS_22.LOG]/-->
    <!--FORMATTER[com.sap.tc.logging.TraceFormatter(%d [%6s]: %m)]/-->
    <!--ENCODING[UTF8]/-->
    <!--LOGHEADER[END]/-->
    Apr 19, 2015 3:59:00 AM [Info  ]: Stopping SDM server...
    Apr 19, 2015 3:59:00 AM [Info  ]: AS Java process ID 11 has been started.
    Apr 19, 2015 3:59:00 AM [Info  ]:   Command line: D:\usr\sap\SOL\DVEBMGS00\exe\sapjvm_4\bin\java.exe -Xmx900M -Djava.ext.dir=d:/usr/sap/SOL/DVEBMGS00/SDM/program/lib;D:/usr/sap/SOL/DVEBMGS00/exe/sapjvm_4/jre/lib/ext -cp . -jar d:\usr\sap\SOL\DVEBMGS00\SDM\program\bin\SDM.jar shutdown Sdmguiport=50018 sdmHome=d:/usr/sap/SOL/DVEBMGS00/SDM/program logfile=D:\usr\sap\SOL\SUM\sdt\log\SUM\OperateSDM_60.LOG
    Apr 19, 2015 3:59:00 AM [Info  ]:   Standard out: D:\usr\sap\SOL\SUM\sdt\log\SUM\OPERATESDM_44.OUT
    Apr 19, 2015 3:59:00 AM [Info  ]: Process ID 11 has been started.
    Apr 19, 2015 3:59:00 AM [Info  ]: Waiting for process ID 11, name java.exe to finish.
    Apr 19, 2015 3:59:02 AM [Info  ]: Process ID 11, name java.exe has been finished, exit code 16.
    Apr 19, 2015 3:59:02 AM [Info  ]: Process ID 11, name java.exe has been terminated.
    Apr 19, 2015 3:59:02 AM [Error ]: The following problem has occurred during step execution: com.sap.sdt.j2ee.tools.sdmmanagement.OperateSdmException: Could not stop SDM server.
    Cannot execute Java process for stopSDM with parameters 50018.
    Return code condition success evaluated to false for process java.exe for action ACTION_STOP.
    Best regards..

    Could not stop SDM server. Cannot execute Java process for stopSDM with parameters 50018. Return code condition success evaluated to false for process java.exe for action ACTION_STOP
    Please follow the solution mentioned in the below link.
    Error During Deployment - Java Support Package Manager (OBSOLETE) - SAP Library
    regards,
    pavan

  • FU Ant task failure: java.util.concurrent.ExecutionException: could not close client/server socket

    We sometimes see this failure intermitently when using the FlexUnit Ant task to run tests in a CI environment. The Ant task throws this exception:
    java.util.concurrent.ExecutionException: could not close client/server socket
    I have seen this for a while now, and still see it with the latest 4.1 RC versions.
    Here is the console output seen along with the above exception:
    FlexUnit player target: flash
    Validating task attributes ...
    Generating default values ...
    Using default working dir [C:\DJTE\commons.formatter_swc\d3flxcmn32\extracted\Source\Flex]
    Using the following settings for the test run:
    FLEX_HOME: [C:\dev\vert-d3flxcmn32\302100.41.0.20110323122739_d3flxcmn32]
    haltonfailure: [false]
    headless: [false]
    display: [99]
    localTrusted: [true]
    player: [flash]
    port: [1024]
    swf: [C:\DJTE\commons.formatter_swc\d3flxcmn32\extracted\build\commons.formatter.tests.unit.sw f]
    timeout: [1800000ms]
    toDir: [C:\DJTE\commons.formatter_swc\d3flxcmn32\reports\xml]
    Setting up server process ...
    Entry  [C:\DJTE\commons.formatter_swc\d3flxcmn32\extracted\build] already  available in local trust file at  [C:\Users\user\AppData\Roaming\Macromedia\Flash  Player\#Security\FlashPlayerTrust\flexUnit.cfg].
    Executing 'rundll32' with arguments:
    'url.dll,FileProtocolHandler'
    'C:\DJTE\commons.formatter_swc\d3flxcmn32\extracted\build\commons.formatter.tests.unit.swf '
    The ' characters around the executable and arguments are
    not part of the command.
    Starting server ...
    Opening server socket on port [1024].
    Waiting for client connection ...
    Client connected.
    Setting inbound buffer size to [262144] bytes.
    Receiving data ...
    Sending acknowledgement to player to start sending test data ...
    Stopping server ...
    End of test data reached, sending acknowledgement to player ...
    When the problem occurs, it is not always during the running of any particular test (that I am aware of). Recent runs where this failure was seen had the following number of tests executed (note: the total number that should be run is 45677): 18021, 18, 229.
    Here is a "good" run when the problem does not occur:
    Setting inbound buffer size to [262144] bytes.
    Receiving data ...
    Sending acknowledgement to player to start sending test data ...
    Stopping server ...
    End of test data reached, sending acknowledgement to player ...
    Closing client connection ...
    Closing server on port [1024] ...
    Analyzing reports ...
    Suite: com.formatters.help.TestGeographicSiteUrls
    Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec
    Suite: com.formatters.functionalUnitTest.testCases.TestNumericUDF
    Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.071 sec
    Results :
    Tests run: 45,677, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 201.186 sec
    Has anyone else ran across this problem?
    Thanks,
    Trevor

    I am not sure if this information will help everyone, but here goes...
    For us, these problems with FlexUnit tests crashing the Flash Player appear to be related to couple of factors. Recently, we moved up from Flex 3.2 to Flex 4.1 as our development baseline.  Many people complained that their development environment (Flash Builder, etc.) was much more unstable.  Apparently, 4.1 produces SWFs that require more memory to run than 3.2 does?  Anyway, we still had Flash Player 10.1 as our runtime baseline.  Apparently, that version of the player was not as capable of running larger FlexUnit test SWFs, and would crash (as I posted months earlier).  I upgraded to the latest 10.3 standalone player versions, and the crashes have now ceased.  It would be nice to know exactly what was causing the crashes, but memory management (or lack of) is my best guess.
    So, if you are seeing these issues, try upgrading to the latest Flash Player version.
    Regards,
    Trevor

  • Interesting! C client socket communicating with java server sockets after..

    Hi,
    I have a client(in C) open a socket and server(in java; jvm 1.2) listening on the same port(e.g. 4444) accepts the request and establishes a socket conn. Now the sockets are kept open indefinitely. If I kill the server process and bring it backup, the client(in C) cannot communicate with the server(in java).
    Why would this be happening? Aren't the server socket garbage collected?
    How do I make sure that the next time I bring up the server again, the client communicates with it.
    Any inputs is really appreciated.
    thanks
    atul.

    Check the client is closing the connection before it stops.
    I've had similar problems with JDBC before where if you don't explicitly close the connection it remains bound to the port, hence if your client only ever tries to connect to that port it will fail.
    I think.
    MrChris

  • Java process didn't stop.

    hello,
    i'm runing oas10gR with Forms services. i have 2 java processes consuming 100 % of the cpu time ( nice time ).
    when a stop the oas with "opmnctl stopall" the java processes hagings didn't end !
    i'm asking why these java processes are spending all the time in "nice time" and where they didn't end ?

    Sounds like the processes are waiting for something (check out the logfiles in $ORACLE_HOME/opmn/logs). If you want to shutdown them forcefully you can issue opmnctl shutdown instead of opmnctl stopall.

  • Does different Java processes spawn different JVMs running on a server ?

    Hi,
    I have 2 applications running on AIX server, one is weblogic server and the other one is inhouse developed Java application. I have set the heap sizes for both the applications and when I do
    ps ax | grep JavaI get
    /usr/java14/bin/java -Xms2048m -Xmx2048m (weblogic)
    /usr/java14/jre/bin/java -Xmx320m -Dvert.log.prefix (inhouse Java)I was wondering if these two processes have their own JVM or as they are running on the same server they share the same JVM. I read in some Java related article that even though there are multiple Java processes, they share the same JVM when run on a single server.
    So from the above Java processes is it true that the total Java heap size is 2048+320=2368M and is used by only one JVM?
    I appreciate if anyone who can answer or direct me to some resources which might give me some answers.
    Thanks,
    kraziabtu

    No, you have two totally independent JVM's. As with many O/S's AIX (and I'm so sorry for you on that :) will only load a single copy of things like read only data segments, shared libraries and some other things but the processes are separate.
    Your first process is allowed to grow to a maximum of 2GB and your second one is set at 320 MB. Again, as there are two of them they are not related.
    Need Java help? Want to help people who do? Sit down with a cup of Java at the hotjoe forums.
    Sure they're new - come get them started!

  • Java Process Consuming 100% of Server CPU

    Hi,
    We have a new instance of CQ5 (version 5.4) running on a Windows 2003 SP2 Server. For some reason, which we can't determine, after a period of time a Java process on the system begins to consume 100% of the CPU and the system becomes totally unresponsive. Has anyone seen this before? Is there a fix for this?
    RK         

    Does the it eventually recover on it's own? It's possible that the JVM is doing full garbage collection.  How big is your max heap size?  I suggest to add some garbage collection debugging in the JVM_OPTS to see if it is Java's garbage collection or not.  Might as well check the heap too, to see if it's running our of memory.
    Example;
    -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -Xloggc:/tmp/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp
    Ron

  • JDK 1.6 on Solaris. Multiple java processes and thread freezes

    Hi, we've come across a really weird behavior on the Solaris JVM, reported by a customer of ours.
    Our server application consists of multiple threads. Normally we see them all running within a single Java process, and all is fine.
    At some point in time, and only on Solaris 10, it seems that the main Java process starts a second Java process. This is not our code trying to execute some other application/command. It's the JVM itself forking a new copy of itself. I assumed this was because of some JVM behaviour on Solaris that uses multiple processes if the number of threads is > 128. However at the time of spawn there are less than 90 threads running.
    In any case, once this second process starts, some of the threads of the application (incidentally, they're the first threads created by the application at startup, in the first threadgroup) stop working. Our application dumps a list of all threads in the system every ten minutes, and even when they're not working, the threads are still there. Our logs also show that when the second process starts, these threads were not in the running state. They had just completed their operations and were sleeping in their thread pool, in a wait() call. Once the second process starts, jobs for these threads just queue up, and the wait() does not return, even after another thread has done a notify() to inform them of the new jobs.
    Even more interesting, when the customer manually kills -9 the second process, without doing anything in our application, all threads that were 'frozen' start working again, immediately. This (and the fact that this never happens on other OSes) makes us think that this is some sort of problem (or misconfiguration) specific to the Solaris JVM, and not our application.
    The customer initially reported this with JDK 1.5.0_12 , we told them to upgrade to the latest JDK 1.6 update 6, but the problem remains. There are no special JVM switches (apart from -Xms32m -Xmx256m) used. We're really at a dead end here in diagnosing this problem, as it clearly seems to be outside our app. Any suggestion?

    Actually, we've discovered that that's not really what was going on. I still believe there's a bug in the JVM, but the fork was happening because our Java code tries to exec a command line tool once a minute. After hours of this, we get a rogue child process with this stack (which is where we are forking this command line tool once a minute):
    JVM version is 1.5.0_08-b03
    Thread t@38: (state = IN_NATIVE)
    - java.lang.UNIXProcess.forkAndExec(byte[], byte[], int, byte[], int, byte[], boolean, java.io.FileDescriptor, java.io.FileDescriptor, java.io.FileDescriptor) @bci=168980456 (Interpreted frame)
    - java.lang.UNIXProcess.forkAndExec(byte[], byte[], int, byte[], int, byte[], boolean, java.io.FileDescriptor, java.io.FileDescriptor, java.io.FileDescriptor) @bci=0 (Interpreted frame)
    - java.lang.UNIXProcess.<init>(byte[], byte[], int, byte[], int, byte[], boolean) @bci=62, line=53 (Interpreted frame)
    - java.lang.ProcessImpl.start(java.lang.String[], java.util.Map, java.lang.String, boolean) @bci=182, line=65 (Interpreted frame)
    - java.lang.ProcessBuilder.start() @bci=112, line=451 (Interpreted frame)
    - java.lang.Runtime.exec(java.lang.String[], java.lang.String[], java.io.File) @bci=16, line=591 (Interpreted frame)
    - java.lang.Runtime.exec(java.lang.String, java.lang.String[], java.io.File) @bci=69, line=429 (Interpreted frame)
    - java.lang.Runtime.exec(java.lang.String) @bci=4, line=326 (Interpreted frame)
    - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)There are also several dozen other threads all with the same stack:
    Thread t@32: (state = BLOCKED)
    Error occurred during stack walking:
    sun.jvm.hotspot.debugger.DebuggerException: can't map thread id to thread handle!
         at sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
         at sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.getThreadIntegerRegisterSet(ProcDebuggerLocal.java:364)
         at sun.jvm.hotspot.debugger.proc.sparc.ProcSPARCThread.getContext(ProcSPARCThread.java:35)
         at sun.jvm.hotspot.runtime.solaris_sparc.SolarisSPARCJavaThreadPDAccess.getCurrentFrameGuess(SolarisSPARCJavaThreadPDAccess.java:108)
         at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:252)
         at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:211)
         at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
         at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
         at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
         at sun.jvm.hotspot.tools.JStack.main(JStack.java:58)I'm pretty sure this is because the fork part of the UnixProcess.forkAndExec is using the Solaris fork1 system call, and thus all the Java context thinks all those threads exist, whereas the actual threads don't exist in that process.
    It seems to me that something is broken in UnixProcess.forkAndExec in native code; it did the fork, but not the exec, and this exec thread just sits there forever. And of course, it's still holding all the file descriptors of the original process, which means that if we decide to restart our process, we can't reopen our sockets for listening or whatever else we want to do.
    There is another possibility, which I can't completely rule out: this child process just happened to be the one that was fork'd when the parent process called Runtime.halt(), which is how the Java process exits. We decided to exit halfway through a Runtime.exec(), and got this child process stuck. But I don't think that's what happens... from what I understand that we collected, we see this same child process created at some point in time, and it doesn't go away.
    Yes, I realize that my JVM is very old, but I cannot find any bug fixes in the release notes that claim to fix something like this. And since this only happens once every day or two, I'm reluctant to just throw a new JVM at this--although I'm sure I will shortly.
    Has anyone else seen anything like this?

  • Weblogic portal server getting problem it stops responding again again

    i am using weblogic portal 10.1 from last few weeks i am getting continuous problem as the server stops responding. in the admin server logs i have found errors mentioned below
    ####<Aug 31, 2009 11:59:48 AM FJT> <Error> <ContentManagement> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-74D57518D9A8B9659F4E> <> <1251676788241> <BEA-000000> <
    com.bea.content.RepositoryException: java.sql.SQLException: The transaction is no longer active - status: 'Marked rollback. [Reason=weblogic.transaction.internal.TimedOutException: Transaction timed out after 66 seconds
    BEA1-74D57518D9A8B9659F4E]'. No further JDBC access is allowed within this transaction.
         at com.bea.content.internal.server.dao.DaoBase.doDatabaseAction(DaoBase.java:287)
         at com.bea.content.internal.server.dao.DaoBase.doQueryActionAndCloseConnection(DaoBase.java:195)
         at com.bea.content.repo.internal.server.dao.NodeDao.hasChildren(NodeDao.java:595)
         at com.bea.content.repo.internal.server.persister.JDBCNodePersister.hasChildren(JDBCNodePersister.java:71)
         at com.bea.content.repo.internal.server.logic.NodeOpsLogic.hasChildren(NodeOpsLogic.java:1031)
         at com.bea.content.repo.internal.server.bean.NodeOpsBean.hasChildren(NodeOpsBean.java:321)
         at com.bea.content.repo.internal.server.bean.RepoNodeOps_ehgg7y_ELOImpl.hasChildren(RepoNodeOps_ehgg7y_ELOImpl.java:1229)
         at com.bea.content.repo.internal.client.common.Node.hasChildren(Node.java:233)
         at com.bea.content.spi.internal.NodeOpsBase.constructNode(NodeOpsBase.java:132)
         at com.bea.content.spi.internal.NodeOpsImpl.getNodesFromRepo(NodeOpsImpl.java:306)
         at com.bea.content.spi.internal.ExtendedNodeOpsImpl.getNodes(ExtendedNodeOpsImpl.java:55)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodesImpl(InternalNodeOpsBean.java:952)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodes(InternalNodeOpsBean.java:1950)
         at com.bea.content.manager.internal.NodeOpsEJB_ihurl6_ELOImpl.getNodes(NodeOpsEJB_ihurl6_ELOImpl.java:2776)
         at com.bea.content.manager.internal.NodeOpsImpl.getNodes(NodeOpsImpl.java:240)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.retrieveNodes(SearchTag.java:598)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.doStartTag(SearchTag.java:246)
         at jsp_servlet.__searchwap._jspService(__searchwap.java:351)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:60)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.continueChainNoWrapper(PageFlowPageFilter.java:455)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.runPage(PageFlowPageFilter.java:432)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:284)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(Unknown Source)
         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
    java.sql.SQLException: The transaction is no longer active - status: 'Marked rollback. [Reason=weblogic.transaction.internal.TimedOutException: Transaction timed out after 66 seconds
    BEA1-74D57518D9A8B9659F4E]'. No further JDBC access is allowed within this transaction.
         at weblogic.jdbc.wrapper.JTSConnection.checkIfRolledBack(JTSConnection.java:178)
         at weblogic.jdbc.wrapper.JTSConnection.checkConnection(JTSConnection.java:188)
         at weblogic.jdbc.wrapper.ResultSet.preInvocationHandler(ResultSet.java:57)
         at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
         at com.bea.content.internal.server.common.dbaction.QueryAction.runExecuteStep(QueryAction.java:25)
         at com.bea.content.internal.server.common.dbaction.SqlStatementAction.execute(SqlStatementAction.java:115)
         at com.bea.content.internal.server.dao.DaoBase.doDatabaseAction(DaoBase.java:280)
         at com.bea.content.internal.server.dao.DaoBase.doQueryActionAndCloseConnection(DaoBase.java:195)
         at com.bea.content.repo.internal.server.dao.NodeDao.hasChildren(NodeDao.java:595)
         at com.bea.content.repo.internal.server.persister.JDBCNodePersister.hasChildren(JDBCNodePersister.java:71)
         at com.bea.content.repo.internal.server.logic.NodeOpsLogic.hasChildren(NodeOpsLogic.java:1031)
         at com.bea.content.repo.internal.server.bean.NodeOpsBean.hasChildren(NodeOpsBean.java:321)
         at com.bea.content.repo.internal.server.bean.RepoNodeOps_ehgg7y_ELOImpl.hasChildren(RepoNodeOps_ehgg7y_ELOImpl.java:1229)
         at com.bea.content.repo.internal.client.common.Node.hasChildren(Node.java:233)
         at com.bea.content.spi.internal.NodeOpsBase.constructNode(NodeOpsBase.java:132)
         at com.bea.content.spi.internal.NodeOpsImpl.getNodesFromRepo(NodeOpsImpl.java:306)
         at com.bea.content.spi.internal.ExtendedNodeOpsImpl.getNodes(ExtendedNodeOpsImpl.java:55)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodesImpl(InternalNodeOpsBean.java:952)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodes(InternalNodeOpsBean.java:1950)
         at com.bea.content.manager.internal.NodeOpsEJB_ihurl6_ELOImpl.getNodes(NodeOpsEJB_ihurl6_ELOImpl.java:2776)
         at com.bea.content.manager.internal.NodeOpsImpl.getNodes(NodeOpsImpl.java:240)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.retrieveNodes(SearchTag.java:598)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.doStartTag(SearchTag.java:246)
         at jsp_servlet.__searchwap._jspService(__searchwap.java:351)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:60)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.continueChainNoWrapper(PageFlowPageFilter.java:455)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.runPage(PageFlowPageFilter.java:432)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:284)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(Unknown Source)
         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
    >
    ####<Aug 31, 2009 11:59:51 AM FJT> <Error> <ContentManagement> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-74D57518D9A8B9659F4E> <> <1251676791539> <BEA-000000> <
    com.bea.content.RepositoryException: An error occurred while trying to find children for node id 162,041.
         at com.bea.content.repo.internal.server.dao.NodeDao.hasChildren(NodeDao.java:606)
         at com.bea.content.repo.internal.server.persister.JDBCNodePersister.hasChildren(JDBCNodePersister.java:71)
         at com.bea.content.repo.internal.server.logic.NodeOpsLogic.hasChildren(NodeOpsLogic.java:1031)
         at com.bea.content.repo.internal.server.bean.NodeOpsBean.hasChildren(NodeOpsBean.java:321)
         at com.bea.content.repo.internal.server.bean.RepoNodeOps_ehgg7y_ELOImpl.hasChildren(RepoNodeOps_ehgg7y_ELOImpl.java:1229)
         at com.bea.content.repo.internal.client.common.Node.hasChildren(Node.java:233)
         at com.bea.content.spi.internal.NodeOpsBase.constructNode(NodeOpsBase.java:132)
         at com.bea.content.spi.internal.NodeOpsImpl.getNodesFromRepo(NodeOpsImpl.java:306)
         at com.bea.content.spi.internal.ExtendedNodeOpsImpl.getNodes(ExtendedNodeOpsImpl.java:55)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodesImpl(InternalNodeOpsBean.java:952)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodes(InternalNodeOpsBean.java:1950)
         at com.bea.content.manager.internal.NodeOpsEJB_ihurl6_ELOImpl.getNodes(NodeOpsEJB_ihurl6_ELOImpl.java:2776)
         at com.bea.content.manager.internal.NodeOpsImpl.getNodes(NodeOpsImpl.java:240)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.retrieveNodes(SearchTag.java:598)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.doStartTag(SearchTag.java:246)
         at jsp_servlet.__searchwap._jspService(__searchwap.java:351)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:60)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.continueChainNoWrapper(PageFlowPageFilter.java:455)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.runPage(PageFlowPageFilter.java:432)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:284)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(Unknown Source)
         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
    com.bea.content.RepositoryException: java.sql.SQLException: The transaction is no longer active - status: 'Marked rollback. [Reason=weblogic.transaction.internal.TimedOutException: Transaction timed out after 66 seconds
    BEA1-74D57518D9A8B9659F4E]'. No further JDBC access is allowed within this transaction.
         at com.bea.content.internal.server.dao.DaoBase.doDatabaseAction(DaoBase.java:287)
         at com.bea.content.internal.server.dao.DaoBase.doQueryActionAndCloseConnection(DaoBase.java:195)
         at com.bea.content.repo.internal.server.dao.NodeDao.hasChildren(NodeDao.java:595)
         at com.bea.content.repo.internal.server.persister.JDBCNodePersister.hasChildren(JDBCNodePersister.java:71)
         at com.bea.content.repo.internal.server.logic.NodeOpsLogic.hasChildren(NodeOpsLogic.java:1031)
         at com.bea.content.repo.internal.server.bean.NodeOpsBean.hasChildren(NodeOpsBean.java:321)
         at com.bea.content.repo.internal.server.bean.RepoNodeOps_ehgg7y_ELOImpl.hasChildren(RepoNodeOps_ehgg7y_ELOImpl.java:1229)
         at com.bea.content.repo.internal.client.common.Node.hasChildren(Node.java:233)
         at com.bea.content.spi.internal.NodeOpsBase.constructNode(NodeOpsBase.java:132)
         at com.bea.content.spi.internal.NodeOpsImpl.getNodesFromRepo(NodeOpsImpl.java:306)
         at com.bea.content.spi.internal.ExtendedNodeOpsImpl.getNodes(ExtendedNodeOpsImpl.java:55)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodesImpl(InternalNodeOpsBean.java:952)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodes(InternalNodeOpsBean.java:1950)
         at com.bea.content.manager.internal.NodeOpsEJB_ihurl6_ELOImpl.getNodes(NodeOpsEJB_ihurl6_ELOImpl.java:2776)
         at com.bea.content.manager.internal.NodeOpsImpl.getNodes(NodeOpsImpl.java:240)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.retrieveNodes(SearchTag.java:598)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.doStartTag(SearchTag.java:246)
         at jsp_servlet.__searchwap._jspService(__searchwap.java:351)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:60)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.continueChainNoWrapper(PageFlowPageFilter.java:455)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.runPage(PageFlowPageFilter.java:432)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:284)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(Unknown Source)
         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
    java.sql.SQLException: The transaction is no longer active - status: 'Marked rollback. [Reason=weblogic.transaction.internal.TimedOutException: Transaction timed out after 66 seconds
    BEA1-74D57518D9A8B9659F4E]'. No further JDBC access is allowed within this transaction.
         at weblogic.jdbc.wrapper.JTSConnection.checkIfRolledBack(JTSConnection.java:178)
         at weblogic.jdbc.wrapper.JTSConnection.checkConnection(JTSConnection.java:188)
         at weblogic.jdbc.wrapper.ResultSet.preInvocationHandler(ResultSet.java:57)
         at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
         at com.bea.content.internal.server.common.dbaction.QueryAction.runExecuteStep(QueryAction.java:25)
         at com.bea.content.internal.server.common.dbaction.SqlStatementAction.execute(SqlStatementAction.java:115)
         at com.bea.content.internal.server.dao.DaoBase.doDatabaseAction(DaoBase.java:280)
         at com.bea.content.internal.server.dao.DaoBase.doQueryActionAndCloseConnection(DaoBase.java:195)
         at com.bea.content.repo.internal.server.dao.NodeDao.hasChildren(NodeDao.java:595)
         at com.bea.content.repo.internal.server.persister.JDBCNodePersister.hasChildren(JDBCNodePersister.java:71)
         at com.bea.content.repo.internal.server.logic.NodeOpsLogic.hasChildren(NodeOpsLogic.java:1031)
         at com.bea.content.repo.internal.server.bean.NodeOpsBean.hasChildren(NodeOpsBean.java:321)
         at com.bea.content.repo.internal.server.bean.RepoNodeOps_ehgg7y_ELOImpl.hasChildren(RepoNodeOps_ehgg7y_ELOImpl.java:1229)
         at com.bea.content.repo.internal.client.common.Node.hasChildren(Node.java:233)
         at com.bea.content.spi.internal.NodeOpsBase.constructNode(NodeOpsBase.java:132)
         at com.bea.content.spi.internal.NodeOpsImpl.getNodesFromRepo(NodeOpsImpl.java:306)
         at com.bea.content.spi.internal.ExtendedNodeOpsImpl.getNodes(ExtendedNodeOpsImpl.java:55)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodesImpl(InternalNodeOpsBean.java:952)
         at com.bea.content.manager.internal.InternalNodeOpsBean.getNodes(InternalNodeOpsBean.java:1950)
         at com.bea.content.manager.internal.NodeOpsEJB_ihurl6_ELOImpl.getNodes(NodeOpsEJB_ihurl6_ELOImpl.java:2776)
         at com.bea.content.manager.internal.NodeOpsImpl.getNodes(NodeOpsImpl.java:240)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.retrieveNodes(SearchTag.java:598)
         at com.bea.content.manager.servlets.jsp.taglib.SearchTag.doStartTag(SearchTag.java:246)
         at jsp_servlet.__searchwap._jspService(__searchwap.java:351)
         at weblogic.servlet.jsp.JspBase.service(JspBase.java:34)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:60)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.continueChainNoWrapper(PageFlowPageFilter.java:455)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.runPage(PageFlowPageFilter.java:432)
         at org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:284)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(Unknown Source)
         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)

    and a lot of errors like these and finally in logs at the end we find
    ####<Aug 31, 2009 3:27:02 PM FJT> <Error> <WebLogicServer> <content-oam> <AdminServer> <[STANDBY] ExecuteThread: '268' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1251689222980> <BEA-000337> <[STUCK] ExecuteThread: '207' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "2,040" seconds working on the request "Http Request: /WapWeb/siteController.jsp", which is more than the configured time (StuckThreadMaxTime) of "2,000" seconds. Stack trace:
         weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:35)
         org.apache.beehive.netui.pageflow.internal.DefaultReloadableClassHandler.loadClass(DefaultReloadableClassHandler.java:188)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.getFlowControllerClass(FlowControllerFactory.java:533)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.createPageFlow(FlowControllerFactory.java:189)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.getPageFlowForPath(FlowControllerFactory.java:163)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.getPageFlowForRequest(FlowControllerFactory.java:127)
         org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:244)
         weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         weblogic.security.service.SecurityManager.runAs(Unknown Source)
         weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
    >
    ####<Aug 31, 2009 3:27:03 PM FJT> <Error> <WebLogicServer> <content-oam> <AdminServer> <[STANDBY] ExecuteThread: '268' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1251689223109> <BEA-000337> <[STUCK] ExecuteThread: '206' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "2,040" seconds working on the request "Http Request: /WapWeb/siteController.jsp", which is more than the configured time (StuckThreadMaxTime) of "2,000" seconds. Stack trace:
         weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:35)
         org.apache.beehive.netui.pageflow.internal.DefaultReloadableClassHandler.loadClass(DefaultReloadableClassHandler.java:188)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.getFlowControllerClass(FlowControllerFactory.java:533)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.createPageFlow(FlowControllerFactory.java:189)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.getPageFlowForPath(FlowControllerFactory.java:163)
         org.apache.beehive.netui.pageflow.FlowControllerFactory.getPageFlowForRequest(FlowControllerFactory.java:127)
         org.apache.beehive.netui.pageflow.PageFlowPageFilter.doFilter(PageFlowPageFilter.java:244)
         weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         weblogic.security.service.SecurityManager.runAs(Unknown Source)
         weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
    >
    ####<Aug 31, 2009 3:27:55 PM FJT> <Warning> <Socket> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '212' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1251689275482> <BEA-000449> <Closing socket as no data read from it during the configured idle timeout of 5 secs>
    ####<Aug 31, 2009 3:29:59 PM FJT> <Error> <PortalAdminTools> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '83' for queue: 'weblogic.kernel.Default (self-tuning)'> <Webmasters> <> <> <1251689399276> <BEA-415121> <the treeMode cannot be found>
    ####<Aug 31, 2009 3:29:59 PM FJT> <Error> <org.apache.beehive.netui.pageflow.internal.DefaultExceptionsHandler> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '83' for queue: 'weblogic.kernel.Default (self-tuning)'> <Webmasters> <> <> <1251689399282> <BEA-000000> <Could not find exception handler method handleException for java.lang.NullPointerException.>
    ####<Aug 31, 2009 3:29:59 PM FJT> <Error> <org.apache.beehive.netui.pageflow.internal.DefaultExceptionsHandler> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '83' for queue: 'weblogic.kernel.Default (self-tuning)'> <Webmasters> <> <> <1251689399283> <BEA-000000> <Could not find exception handler method handleException for java.lang.RuntimeException.>
    ####<Aug 31, 2009 3:29:59 PM FJT> <Error> <org.apache.beehive.netui.pageflow.internal.DefaultExceptionsHandler> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '83' for queue: 'weblogic.kernel.Default (self-tuning)'> <Webmasters> <> <> <1251689399283> <BEA-000000> <Could not find message-resources for bundle org.apache.struts.action.MESSAGE>
    ####<Aug 31, 2009 3:29:59 PM FJT> <Warning> <global.internal.AbstractBaseController$LoggerCatalog> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '83' for queue: 'weblogic.kernel.Default (self-tuning)'> <Webmasters> <> <> <1251689399283> <BEA-000000> <Exception occured trying to instantiate the null bean.
    java.lang.NullPointerException
         at util.tree.TreeController.constructRootWithoutChildren(TreeController.java:483)
         at util.tree.TreeController.view(TreeController.java:172)
         at sun.reflect.GeneratedMethodAccessor637.invoke(Unknown Source)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:585)
         at org.apache.beehive.netui.pageflow.FlowController.invokeActionMethod(FlowController.java:879)
         at org.apache.beehive.netui.pageflow.FlowController.getActionMethodForward(FlowController.java:809)
         at org.apache.beehive.netui.pageflow.FlowController.internalExecute(FlowController.java:478)
         at org.apache.beehive.netui.pageflow.PageFlowController.internalExecute(PageFlowController.java:306)
         at global.internal.AbstractBaseController.internalExecute(AbstractBaseController.java:348)
         at org.apache.beehive.netui.pageflow.FlowController.execute(FlowController.java:336)
         at org.apache.beehive.netui.pageflow.internal.FlowControllerAction.execute(FlowControllerAction.java:52)
         at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
         at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.access$201(PageFlowRequestProcessor.java:97)
         at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor$ActionRunner.execute(PageFlowRequestProcessor.java:2044)
         at org.apache.beehive.netui.pageflow.interceptor.action.internal.ActionInterceptors.wrapAction(ActionInterceptors.java:91)
         at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.processActionPerform(PageFlowRequestProcessor.java:2116)
         at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
         at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.processInternal(PageFlowRequestProcessor.java:556)
         at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:853)
         at org.apache.beehive.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:631)
         at org.apache.beehive.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:158)
         at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
         at org.apache.beehive.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1170)
         at org.apache.beehive.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1201)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:686)
         at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.renderInternal(ScopedContentCommonSupport.java:266)
         at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.render(PageFlowStubImpl.java:135)
         at com.bea.netuix.servlets.controls.content.NetuiContent.preRender(NetuiContent.java:290)
         at com.bea.netuix.nf.ControlLifecycle$6.visit(ControlLifecycle.java:426)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:712)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:724)
         at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:146)
         at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:375)
         at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:341)
         at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:188)
         at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:142)
         at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:377)
         at com.bea.netuix.servlets.manager.UIServlet.doPost(UIServlet.java:253)
         at com.bea.netuix.servlets.manager.UIServlet.doGet(UIServlet.java:206)
         at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:191)
         at com.bea.netuix.servlets.manager.SingleFileServlet.service(SingleFileServlet.java:266)
         at com.bea.netuix.servlets.manager.PortalServlet.service(PortalServlet.java:656)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
         at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
         at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
         at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
         at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.jsptools.servlet.PagedResultServiceFilter.doFilter(PagedResultServiceFilter.java:82)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.portal.tools.servlet.http.HttpContextFilter.doFilter(HttpContextFilter.java:60)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:315)
         at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
         at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3393)
         at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
         at weblogic.security.service.SecurityManager.runAs(Unknown Source)
         at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2140)
         at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
         at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
         at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
         at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)
    >
    ####<Aug 31, 2009 3:31:31 PM FJT> <Critical> <Health> <content-oam> <AdminServer> <weblogic.GCMonitor> <<anonymous>> <> <> <1251689491464> <BEA-310003> <Free memory in the server is 4,431,384 bytes. There is danger of OutOfMemoryError>
    ####<Aug 31, 2009 3:33:25 PM FJT> <Critical> <Health> <content-oam> <AdminServer> <weblogic.GCMonitor> <<anonymous>> <> <> <1251689605339> <BEA-310003> <Free memory in the server is 2,960,928 bytes. There is danger of OutOfMemoryError>
    ####<Aug 31, 2009 3:34:07 PM FJT> <Warning> <Socket> <content-oam> <AdminServer> <[ACTIVE] ExecuteThread: '133' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1251689647279> <BEA-000449> <Closing socket as no data read from it during the configured idle timeout of 5 secs>

  • BPEL server is unable to run any process - message ORABPEL-02152 in logs

    Hello,
    We have a problem with Oracle BPEL Process Manager on production environment. 2 days ago processes using table pooling stopped working correctly. Status for new record in pooled table was changed (logical
    delete - table update) but process didn't execute next steps (no log messages).
    Restarting OC4J_BPEL resolved problem.
    But now message ORABPEL-02152 constantly appears in domain log and all processes stop working. Deployed
    simple HelloWorld from examples - it does not work too.
    Meantime some bug fixes mainly for connection to RAC database were applied. Patch list
    1. patch 4869010
    2. patch 4915565
    3. patch 2617419
    4. patch 5030051
    5. patch 5638122
    6. patch 5925254
    7. patch 5903451
    Steps taken for now.
    1. Check connection to BPEL store - ok free space in database exists
    2. This is not an issue with BPEL console - process instance does not show in BPEL console and does not work at all
    3. Search forum and metalink for similar issues ;-)
    I think this a problem with store but how can I diagnose this. I read Metalink document Note:372456.1 "How to configure BPEL 10.1.2.0.2 to operate with an Oracle 10g RAC dehydration store" but I don't understand if changing data-source class to "com.evermind.sql.ICCXADataSource" is a must or is is optional task.
    So, any suggestions how to start resolving this issue. What should i check first ?
    Thanks for help...
    Regards
    Tom
    Configuration details:
    OS version: Linux oradb01 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 i686 i686 i386
    GNU/Linux
    DB version: Oracle Database 10g Release 10.2.0.2.0 - Production (RAC) - cluster enviroment
    BPEL: BPEL_10.1.2.0.2_GENERIC_060110.1200
    Domain.log fragment for simple HelloWorld process below.
    2007-05-07 15:58:31,405> <DEBUG> <default.collaxa.cube.activation> <Database Adapter::Inbound> <oracle.tip.adapter.db.TopLinkLogger log> client released
    <2007-05-07 15:58:31,979> <DEBUG> <default.collaxa.cube.engine.data> <ConnectionFactory::getConnection> GOT CONNECTION 1 Autocommit = true
    <2007-05-07 15:58:32,034> <ERROR> <default.collaxa.cube> <BaseCubeSessionBean::logError> Error while invoking bean "finder": Instance not found in datasource.
    The process domain was unable to fetch the instance with key "b7fd898ac6095465:15aed57:11266ad2a65:-7feb" from the datasource.
    Please check that the instance key "b7fd898ac6095465:15aed57:11266ad2a65:-7feb" refers to a valid instance that has been started and not removed from the process domain.
    ORABPEL-02152
    Instance not found in datasource.
    The process domain was unable to fetch the instance with key "b7fd898ac6095465:15aed57:11266ad2a65:-7feb" from the datasource.
    Please check that the instance key "b7fd898ac6095465:15aed57:11266ad2a65:-7feb" refers to a valid instance that has been started and not removed from the process domain.
         at com.collaxa.cube.engine.data.AdminFinder.__lookupInstanceMetaData(AdminFinder.java:314)
         at com.collaxa.cube.engine.data.AdminFinder.__lookupInstanceMetaData(AdminFinder.java:262)
         at com.collaxa.cube.engine.data.AdminFinder.lookupInstanceByConversationId(AdminFinder.java:235)
         at com.collaxa.cube.ejb.impl.FinderBean.lookupInstanceByConversationId(FinderBean.java:177)
         at IFinderBean_StatelessSessionBeanWrapper42.lookupInstanceByConversationId(IFinderBean_StatelessSessionBeanWrapper42.java:669)
         at com.oracle.bpel.client.Locator.lookupInstance(Locator.java:420)
         at displayInstance.jspService(_displayInstance.java:111)
         at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56)
         at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:350)
         at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:509)
         at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:413)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:824)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:330)
         at com.evermind.server.http.ServletRequestDispatcher.forward(ServletRequestDispatcher.java:222)
         at com.collaxa.cube.fe.DomainFilter.doFilter(DomainFilter.java:152)
         at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:663)
         at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:330)
         at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:830)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:224)
         at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:133)
         at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
         at java.lang.Thread.run(Thread.java:534)
    <2007-05-07 15:58:32,037> <DEBUG> <default.collaxa.cube.engine.data> <ConnectionFactory::closeConnection> CLOSE CONNECTION 0
    <2007-05-07 15:58:32,270> <DEBUG> <default.collaxa.cube.activation> <Database Adapter::Inbound> <oracle.tip.adapter.db.TopLinkLogger log> client acquired

    Hello,
    I try verify if BPEL store is working. In error log I can see
    The process domain was unable to fetch the instance with key "b7fd898ac6095465:15aed57:11266ad2a65:-7feb" from the datasource.
    Is this select should return any rows if the process instance is correctly stored in database ?
    select * from cube_instance where
    conversation_id like '%b7fd898ac6095465:15aed57:11266ad2a65:-7feb%'
    Regards
    Tom

  • Signal ILL trying to debug java process with dbx

    Hello,
    I'm trying to debug a java process to find an error in JNI code. I get this message when dbx stops in a C breakpoint:
    t@18 (l@19) signal ILL (illegal trap) in (unknown) t@18 (l@19) signal ILL (illegal trap) in (unknown) at 0xf848ed28
    0xf848ed28:     tne     0x12This signal is not launched when running application without debugger, and after launching this exception and exiting dbx, the java process generates a core.
    I've tried to find out unsuccessfully whether my dbx version is compatible with my Java version, but it seems this could be the problem.
    This is my current configuration:
    java �version:
    java version "1.5.0_06"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
    Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode)
    opt/SUNWspro/bin/version:
    Machine hardware:   sun4u
    OS version:         5.8
    Processor type:     sparc
    Hardware:           SUNW,Sun-Blade-2500
    The following components are installed on your system:
    Forte C++ 6 update 2
            Sun WorkShop 6 update 2 Compilers C    
            Sun WorkShop 6 update 2 Compilers C++
            Sun WorkShop 6 update 2 Tools.h++ 7.1
            Sun WorkShop 6 update 2 Standard 64-bit Class Library for C++
            Sun WorkShop 6 update 2 Memory Monitor for C++
            Sun WorkShop 6 update 2 Visual
            Sun WorkShop 6 update 2 Development Environment
            Sun WorkShop 6 update 2 TeamWare
            Sun WorkShop 6 update 2 FileMerge
            Sun WorkShop 6 update 2 DBX Debugging Tools
            Sun WorkShop 6 update 2 Performance Analyzer
            Sun WorkShop 6 update 2 LockLint
            XEmacs 20.4 Source Distribution
            Sun WorkShop 6 update 2 Documentation Set
    Sun WorkShop 6 update 2 Building SoftwareAnd this is how I launch java application:
    java -Djava.compiler=NONE -Xdebug -Xnoagent -Xrundbx_agent -jar jarname.jar -mainOptionsDid I miss something important? Does anyone know if it is possible debugging with this configuration?
    Best Regards,
    Alberto

    - Native code has been compiled with -g option.
    - It fails in my code, but only when debugging.
    - Stack trace (executing "where" in dbx):
    (dbx) where
    current thread: t@18
      [1] 0xf848ef68(0xf01537d8, 0xf443b7b0, 0x0, 0xf84181b0, 0x21e9, 0xef581050), at 0xf848ef68
      [2] 0xf8458354(0xf15faea8, 0xf01537d8, 0xf0151a88, 0xf8419368, 0x21e8, 0x0), at 0xf8458354
      [3] 0xf848c9a0(0xf15faea8, 0xf01537d8, 0xf0151a58, 0xf0151a38, 0xf0153830, 0x0), at 0xf848c9a0
      [4] 0xf84577c8(0xf15faea8, 0xb6, 0x0, 0xf84181b0, 0xf0158f80, 0xef581118), at 0xf84577c8
      [5] 0xf8405764(0xf15faea8, 0xb6, 0x0, 0xf8419320, 0xf0158f80, 0xef5811c8), at 0xf8405764
      [6] 0xf8405764(0xf15faea8, 0xb6, 0x0, 0xf84195a0, 0xf0158f80, 0xef581250), at 0xf8405764
      [7] 0xf8405764(0xf0151a38, 0xb8, 0x0, 0xf8419320, 0xf0125258, 0xef5812d8), at 0xf8405764
      [8] 0xf8405874(0xf15fb6c8, 0xb8, 0x0, 0xf8419720, 0xf0125258, 0xef581360), at 0xf8405874
      [9] 0xf8405764(0xf15fae98, 0xb6, 0x0, 0xf8419770, 0xf00f0088, 0xef5813f0), at 0xf8405764
      [10] 0xf8405874(0xf15fae98, 0xb6, 0x0, 0xf8419320, 0x1465, 0xef581478), at 0xf8405874
      [11] 0xf8405874(0xf16315b0, 0xb6, 0x0, 0xf8419320, 0xf00f0088, 0xef581500), at 0xf8405874
      [12] 0xf8405764(0xf16315b0, 0xf45744c0, 0x0, 0xf8419320, 0xf45744c0, 0xef581588), at 0xf8405764
      [13] 0xf8405c2c(0xf16315d0, 0xb7, 0x0, 0xf84198a0, 0xf00f0088, 0xef581618), at 0xf8405c2c
      [14] 0xf8405764(0xf009c018, 0xf402c8b8, 0x0, 0xf84195a0, 0x8800, 0xef5816a8), at 0xf8405764
      [15] 0xf8405c2c(0x2000, 0x2, 0x0, 0xf84198f0, 0xef50e0f8, 0xef581720), at 0xf8405c2c
      [16] 0xf8400218(0xef581808, 0xef581a08, 0xa, 0xf402dae8, 0xf840ba20, 0xef581920), at 0xf8400218
      [17] JavaCalls::call_helper(0x1, 0x65d188, 0xef581918, 0xef581818, 0x4, 0xef581a08), at 0xfdd9a808
      [18] JavaCalls::call_virtual(0x7a9c, 0x65d188, 0x3570f8, 0x357104, 0x357100, 0xf00bc770), at 0xfdec319c
      [19] thread_entry(0xf402fc58, 0x65d188, 0x3574e4, 0xfe424640, 0xfe424710, 0xfe424130), at 0xfdee2700
      [20] JavaThread::run(0x65d188, 0x18, 0xfe40c790, 0x6c00, 0x7dd4, 0x7c00), at 0xfdede29c
      [21] _start(0x65d188, 0x3b62, 0xfe3be000, 0x0, 0x4f24, 0x4c00), at 0xfe264240 - Stack trace (obtained from core generated after ILL signal):
    -----------------  lwp# 19 / thread# 18  --------------------
    ff3791f0 __sigprocmask (ff37b7f4, 0, 0, ef581d98, ff38c000, 0) + 8
    ff36dd0c _sigon   (ef581d98, ff3938a8, 6, ef58083c, ef581d98, ff39321c) + d0
    ff370d4c _thrp_kill (0, 12, 6, ff38c000, 12, ff2c0458) + f8
    ff24bce0 raise    (6, 0, 0, ffffffff, ff2c03c4, 8400) + 40
    ff235984 abort    (ff2bc008, fe3be000, fe26a8e0, fe4133c4, 0, 331c8) + 100
    fe264724 __1cCosFabort6Fi_v_ (1, 0, fe3f2614, fe3be000, 73b4, 7000) + 58
    fe2f08f0 __1cHVMErrorOreport_and_die6M_v_ (0, fe419ad8, fe41339c, 1, fe268a14, fe41339c) + c84
    fde76878 JVM_handle_solaris_signal (4, ef580fd8, ef580d20, 7800, 11, 65d188) + aac
    ff37b118 __sighndlr (4, ef580fd8, ef580d20, fde75d94, ef581e40, ef581e30) + c
    ff37811c sigacthandler (4, ef581d98, 0, 0, 0, ff38c000) + 708
    --- called from signal handler with signal 4 (SIGILL) ---
    f848ef68 ???????? (f15faea8, f01537d8, f0151a88, f8419368, 21e8, 0)
    f848c9a0 ???????? (f15faea8, f01537d8, f0151a58, f0151a38, f0153830, 0)
    f84577c8 ???????? (f15faea8, b6, 0, f84181b0, f0158f80, ef581118)
    f8405764 ???????? (f15faea8, b6, 0, f8419320, f0158f80, ef5811c8)
    f8405764 ???????? (f15faea8, b6, 0, f84195a0, f0158f80, ef581250)
    f8405764 ???????? (f0151a38, b8, 0, f8419320, f0125258, ef5812d8)
    f8405874 ???????? (f15fb6c8, b8, 0, f8419720, f0125258, ef581360)
    f8405764 ???????? (f15fae98, b6, 0, f8419770, f00f0088, ef5813f0)
    f8405874 ???????? (f15fae98, b6, 0, f8419320, 1465, ef581478)
    f8405874 ???????? (f16315b0, b6, 0, f8419320, f00f0088, ef581500)
    f8405764 ???????? (f16315b0, f45744c0, 0, f8419320, f45744c0, ef581588)
    f8405c2c ???????? (f16315d0, b7, 0, f84198a0, f00f0088, ef581618)
    f8405764 ???????? (f009c018, f402c8b8, 0, f84195a0, 8800, ef5816a8)
    f8405c2c ???????? (2000, 2, 0, f84198f0, ef50e0f8, ef581720)
    f8400218 ???????? (ef581808, ef581a08, a, f402dae8, f840ba20, ef581920)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 65d188, ef581918, ef581818, 4, ef581a08) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 65d188, 3570f8, 357104, 357100, f00bc770) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 65d188, 3574e4, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (65d188, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (65d188, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (65d188, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 20 / thread# 19  --------------------
    ff29f474 _lwp_sema_wait (ff3934c8, ff392e40, 0, 1, 0, 0) + c
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 21  --------------------------------
    ff379200 private___lwp_cond_wait (fa5a3d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa5a3d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 22  --------------------------------
    ff379200 private___lwp_cond_wait (fa4d3d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa4d3d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 23  --------------------------------
    ff379200 private___lwp_cond_wait (fa4c3d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa4c3d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 24  --------------------------------
    ff379200 private___lwp_cond_wait (fa493d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa493d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 25  --------------------------------
    ff379200 private___lwp_cond_wait (fa483d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa483d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 26  --------------------------------
    ff379200 private___lwp_cond_wait (fa473d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa473d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 1 / thread# 1  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, 18e, 37020, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (1c08, fe406e44, 37020, fe412d84, 38680, fe3be000) + 260
    f840e22c ???????? (b9, ffbee73c, ffbee6b8, ffffff80, 0, 0)
    f840e1d0 ???????? (0, b8, 0, 8, f402fc58, ffbee6d0)
    f8405764 ???????? (f15996c8, fe4163d4, 0, f8419770, 8800, ffbee768)
    f8405764 ???????? (f01778f8, fe4163d4, 0, f8419370, 8800, ffbee820)
    f8405764 ???????? (2000, 2, 0, f8419320, ffb7c110, ffbee8b8)
    f8400218 ???????? (ffbee9a0, ffbeeb00, a, f41c4038, f840ba20, ffbeea88)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 37020, ffbeea80, ffbee9b0, 4, ffbeeb00) + 5a0
    fdedea14 jni_CallStaticVoidMethod (fe412d84, 381ec, 36750, 37020, 36748, 37cb8) + 500
    000121fc main     (38648, 1e7fa2, fe40a630, 1d8, fded413c, 1) + 115c
    00011088 _start   (0, 0, 0, 0, 0, 0) + 108
    -----------------  lwp# 2 / thread# 2  --------------------
    ff29edc4 _signotifywait (ff38c000, 0, 0, 0, 0, 0) + 8
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 3  --------------------------------
    ff379200 private___lwp_cond_wait (ff325d98, ff38cd6c, ff38c000, 0, 0, 4) + 8
    ff29c9c0 _door_return (ff325cd8, ff36a358, 0, 0, 0, 0) + 68
    -----------------  lwp# 4 / thread# 4  --------------------
    ff29f428 ___lwp_cond_wait (35d68, 35d50, fdb81978, 0, 0, fdb81970) + 8
    ff296a7c _lwp_cond_timedwait (2958790, 44b36755, fdb81a50, 35d50, 35d68, a95a) + 98
    fddc3258 __1cHMonitorEwait6Mil_i_ (35cf8, 11dce0, fff0bc00, fe3be000, 6000, 35d28) + 328
    fdf6b4fc __1cIVMThreadDrun6M_v_ (fe4124d8, fe420c84, 0, fe420c70, fe420c80, 1) + 1b4
    fe264240 __1cG_start6Fpv_0_ (11dce0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (11dce0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 5  --------------------------------
    ff29c968 _door_return (4, ff38d658, ff38d670, 3, ff38c000, 1) + 10
    ff36a358 _lwp_start (fe785d98, 0, 6000, ff325b9c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 6 / thread# 5  --------------------
    ff29f428 ___lwp_cond_wait (120148, 120130, 0, ff2968f0, 0, 1) + 8
    fdeb4768 __1cNObjectMonitorEwait6MxipnGThread__v_ (120370, 9408, 8c00, fe3be000, 11fdc0, 120390) + 59c
    fdeb5978 __1cSObjectSynchronizerEwait6FnGHandle_xpnGThread__v_ (120372, 86d4, 0, 11fdc0, 8400, fe3be000) + e8
    fdeb5628 JVM_MonitorWait (0, e7a14, 11fdc0, fe412d84, 0, 1) + 31c
    f840e22c ???????? (51, fda816ec, fda81670, ffffff80, fdd2e808, 0)
    f840e1d0 ???????? (f15817e0, b6, 0, c, 0, fda81688)
    f8405764 ???????? (f15817e0, b6, 0, f8419354, 80000000, fda81708)
    f8405764 ???????? (0, fe411508, 0, f8419770, 7bac, fda817a0)
    f8400218 ???????? (fda81888, fda81a88, a, f40901d0, f840ba20, fda819a0)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 11fdc0, fda81998, fda81898, 4, fda81a88) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 11fdc0, e7a00, e7a0c, e7a08, f0000760) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 11fdc0, e7dec, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (11fdc0, 0, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (11fdc0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (11fdc0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 7 / thread# 6  --------------------
    ff29f428 ___lwp_cond_wait (121cc8, 121cb0, 0, ff2968f0, 0, 1) + 8
    fdeb4768 __1cNObjectMonitorEwait6MxipnGThread__v_ (1203b8, 9408, 8c00, fe3be000, 121530, 1203d8) + 59c
    fdeb5978 __1cSObjectSynchronizerEwait6FnGHandle_xpnGThread__v_ (1203ba, 86d4, 0, 121530, 8400, fe3be000) + e8
    fdeb5628 JVM_MonitorWait (0, 121714, 121530, fe412d84, 0, 1) + 31c
    f840e22c ???????? (59, fb3817ec, fb381768, ffffff80, 0, 0)
    f840e1d0 ???????? (f1581760, b6, 0, c, 80000000, fb381780)
    f8405764 ???????? (f1581768, b6, 0, f8419354, 8800, fb381820)
    f8405874 ???????? (f1581768, b6, 0, f8419354, f402af58, fb3818a0)
    f8405874 ???????? (0, fe411508, 0, f8419720, 7bac, fb381920)
    f8400218 ???????? (fb381a08, fb381c08, a, f4090ab8, f840ba20, fb381b20)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 121530, fb381b18, fb381a18, 4, fb381c08) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 121530, 121700, 12170c, 121708, f0000858) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 121530, 121aec, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (121530, 0, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (121530, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (121530, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 8 / thread# 7  --------------------
    ff29f428 ___lwp_cond_wait (1b65f0, 1b65d8, fb281890, ff296918, ff29691c, fb281888) + 8
    ff296a7c _lwp_cond_timedwait (2964ae0, 44b36756, fb281910, 1b65d8, 1b65f0, 153180) + 98
    fe2606e0 __1cCosHSolarisFEventEpark6Mx_i_ (1b65b0, 6a60, fe4238d8, 1b65f0, fe406ef0, fb281910) + 1bc
    fe25ff28 __1cNObjectMonitorIraw_wait6MxipnGThread__i_ (1b65b0, fe214b70, 7bac, 0, 1b59e8, 12a188) + 20c
    fe1f4b8c __1cIJvmtiEnvORawMonitorWait6MpnPJvmtiRawMonitor_x_nKjvmtiError__ (fb2, 12a188, 0, 7d0, 4, 1b59e8) + a8
    fe7d87ec process_next_event (fe40b864, 10000, fe7f24a0, fe7f4b90, fe7f25f0, fe7f25ec) + 108
    fe7c7630 agent_bar (1b5aa4, fe7f2ae8, ff38c000, fe7f24c8, 0, 0) + 1c
    fe7c7bc4 jvmti_debug_agent (fe7f2400, 1b5aa4, 0, 1678, 1, fe7f252c) + d4
    fe211c74 __1cQJvmtiAgentThreadTcall_start_function6M_v_ (1b59e8, 7800, 8c94, fe3be000, fe4123c4, fe4163d4) + 150
    fdede29c __1cKJavaThreadDrun6M_v_ (1b59e8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b59e8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b59e8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 9 / thread# 8  --------------------
    ff29f474 _lwp_sema_wait (fb181e60, ff38c000, 0, fb181d98, 0, 0) + c
    ff3690d8 _swtch   (fb181d98, ffffffff, ff38c000, 5, 1000, fe4170c4) + 158
    ff36c860 sema_wait (fe4217e8, ff39785c, 5000, 10000, 534d, 1) + bc
    fdf61dc0 __1cVcheck_pending_signals6Fi_i_ (1, fe4124d8, fe4123c0, 1b6ab8, fe3be000, a) + 254
    fdf58ec8 __1cTsignal_thread_entry6FpnKJavaThread_pnGThread__v_ (7428, 1b6ab8, 1b7088, fe3be000, 4ff61c, 1b7080) + 34
    fdede29c __1cKJavaThreadDrun6M_v_ (1b6ab8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b6ab8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b6ab8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 10 / thread# 9  --------------------
    ff29f428 ___lwp_cond_wait (36548, 36530, fe423d60, 0, 0, ff29695c) + 8
    fddc33ec __1cHMonitorEwait6Mil_i_ (364d8, 1b7998, 1b8510, fe3be000, 1b7998, 36508) + 4bc
    fdf35d00 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (fe4124d8, 1b54c0, 1b7998, fe407360, 8c00, 364d8) + 22c
    fdede29c __1cKJavaThreadDrun6M_v_ (1b7998, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b7998, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b7998, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 11 / thread# 10  --------------------
    fdd0c200 __1cENodeSremove_dead_region6MpnIPhaseGVN_i_i_ (9132d4, fa6809f4, 1, 9132d4, 9132f4, 68ad00) + 2c
    fdd3332c __1cHMemNodeMIdeal_common6MpnIPhaseGVN_i_pnENode__ (fa6809f4, fa6809f4, 1, 800, 9132d4, 9132f4) + 3c
    fdd68a78 __1cJStoreNodeFIdeal6MpnIPhaseGVN_i_pnENode__ (9132d4, fa6809f4, 1, 0, fdd4be74, fe3fbb44) + c
    fdcf887c __1cMPhaseIterGVNNtransform_old6MpnENode__2_ (fa6809f4, 9132d4, 89e938, fdd68a6c, 20000000, c7) + 28
    fdde3b98 __1cMPhaseIterGVNIoptimize6M_v_ (fa6809f4, 14, 9132d4, 0, fa680b44, ffffffff) + ac
    fde048d8 __1cOPhaseIdealLoop2t6MrnMPhaseIterGVN_pk0i_v_ (fa68087c, ffffffff, fa680fb0, 2, 0, fa680880) + c34
    fde7fb28 __1cHCompileIOptimize6M_v_ (fa680fb0, 1b55b0, 0, 0, fe3be000, 1000) + 200
    fde82740 __1cHCompile2t6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_ (fa680fb0, 0, 36ee0, 0, fe3384f8, fa680fb0) + be8
    fde786f8 __1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_ (fa6818ac, fe324608, 9269b8, ffffffff, 36ee0, 1000001) + b0
    fde793b4 __1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_ (597690, 1b8eb4, 1b55b0, 1b88d8, fe338ea7, fa6818ac) + 4cc
    fdf35f20 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (fe4124d8, 1b5550, 1b88d8, f4368380, 1b8eb0, 0) + 44c
    fdede29c __1cKJavaThreadDrun6M_v_ (1b88d8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b88d8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b88d8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 12 / thread# 11  --------------------
    ff29f428 ___lwp_cond_wait (365d8, 365c0, fe423d60, 0, 0, ff29695c) + 8
    fddc33ec __1cHMonitorEwait6Mil_i_ (36568, 1b9818, 1ba390, fe3be000, 1b9818, 36598) + 4bc
    fdf35d00 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (fe4124d8, 1b5550, 1b9818, fe407360, 1b9df0, 36568) + 22c
    fdede29c __1cKJavaThreadDrun6M_v_ (1b9818, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b9818, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b9818, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 13 / thread# 12  --------------------
    ff29f428 ___lwp_cond_wait (34858, 34840, 1, 0, 1, 1bb248) + 8
    fddc309c __1cHMonitorEwait6Mil_i_ (347e8, 1bb1a0, fff0bc00, fe3be000, f4000, 34818) + 16c
    fdf64b44 __1cRLowMemoryDetectorbGlow_memory_detector_thread_entry6FpnKJavaThread_pnGThread__v_ (1bb1a0, 347e8, 9000, fe411604, fe3be000, 1) + 2ec
    fdede29c __1cKJavaThreadDrun6M_v_ (1bb1a0, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1bb1a0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1bb1a0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 14 / thread# 13  --------------------
    ff29d600 _poll    (fe4124d8, 32, 0, 0, 8400, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, 1, 0, 2, 4) + 25c
    fdfa5194 __1cNWatcherThreadDrun6M_v_ (1bc4c8, 1b5460, 32, fe3be000, fe417db8, 0) + e8
    fe264240 __1cG_start6Fpv_0_ (1bc4c8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1bc4c8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 15 / thread# 14  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, f1, 3baea0, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (ba8, fe406e44, 3baea0, fe412d84, 3bb180, fe3be000) + 260
    f840e22c ???????? (c9, eff818f4, eff81870, ffffff80, f00e5250, 0)
    f840e1d0 ???????? (0, b8, 0, 8, f00e5250, eff81888)
    f8405764 ???????? (2000, 2, 0, f8419754, eff0e110, eff81920)
    f8400218 ???????? (eff81a08, eff81c08, a, f423b868, f840ba20, eff81b20)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 3baea0, eff81b18, eff81a18, 4, eff81c08) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 3baea0, 254948, 254954, 254950, f01aaaf8) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 3baea0, 254d34, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (3baea0, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (3baea0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (3baea0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 16 / thread# 15  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, f0, 358da0, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (368, fe406e44, 358da0, fe412d84, 358f98, fe3be000) + 260
    f840e22c ???????? (91, efe81874, efe817f0, ffffff80, f00dfeb0, 0)
    f840e1d0 ???????? (0, b8, 0, 8, f00dfeb0, efe81808)
    f8405764 ???????? (2000, 2, 0, f8419754, efe0e110, efe818a0)
    f8400218 ???????? (efe81988, efe81b88, a, f423b868, f840ba20, efe81aa0)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 358da0, efe81a98, efe81998, 4, efe81b88) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 358da0, 254d48, 254d54, 254d50, f01b4270) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 358da0, 255134, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (358da0, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (358da0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (358da0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 17 / thread# 16  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, f5, 3dd8a8, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (1628, fe406e44, 3dd8a8, fe412d84, 3c5f98, fe3be000) + 260
    f840e22c ???????? (c1, efb015fc, efb01570, ffffff80, efb, 0)
    f840e1d0 ???????? (0, b8, 0, 8, 8800, efb01588)
    f8405764 ???????? (2000, 2, 0, f8419770, efa8e110, efb01620)
    f8400218 ???????? (efb01708, efb01908, a, f422e650, f840ba20, efb01820)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 3dd8a8, efb01818, efb01718, 4, efb01908) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 3dd8a8, 3568f8, 356904, 356900, f007a910) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 3dd8a8, 356ce4, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (3dd8a8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (3dd8a8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (3dd8a8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 18 / thread# 17  --------------------
    ff29f428 ___lwp_cond_wait (65b7a0, 65b788, ef681128, fe4123c4, ff2968f4, ef681120) + 8
    ff296a7c _lwp_cond_timedwait (2ea2b88, 44b36759, ef681248, 65b788, 65b7a0, ff2969e4) + 98
    fdeb4974 __1cNObjectMonitorEwait6MxipnGThread__v_ (120400, fe406ef0, fe4238d8, fe3be000, 65b550, 120420) + 7a8
    fdeb5978 __1cSObjectSynchronizerEwait6FnGHandle_xpnGThread__v_ (120402, 86d4, 0, 65b550, 8400, fe3be000) + e8
    fdeb5628 JVM_MonitorWait (0, 3dcdb4, 65b550, fe412d84, 0, 1) + 31c
    f840e22c ???????? (91, ef6814b4, ef681430, ffffff80, ef6, 0)
    f840e1d0 ???????? (f1631648, b6, 0, c, f00d7f28, ef681448)
    f8405764 ???????? (f007ea58, fe4163d4, 0, f8419354, 8800, ef681508)
    f8405764 ???????? (2000, 2, 0, f84195a0, ef60e110, ef6815a0)
    f8400218 ???????? (ef681688, ef681888, a, f456d6b8, f840ba20, ef6817a0)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 65b550, ef681798, ef681698, 4, ef681888) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 65b550, 3dcda0, 3dcdac, 3dcda8, f007ea58) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 65b550, 3dd18c, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (65b550, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (65b550, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (65b550, 0, 0, 0, 0, 0) + 40
    --------------------------  thread# 3  --------------------
    ff36d9b8 _reap_wait (ff390980, 1e924, 0, ff38c000, 0, 0) + 38
    ff36d710 _reaper  (ff38ce00, ff392708, ff390980, ff38cdd8, 1, fe400000) + 38
    ff37b01c _thread_start (0, 0, 0, 0, 0, 0) + 40

Maybe you are looking for