JDK 1.6 on Solaris. Multiple java processes and thread freezes

Hi, we've come across a really weird behavior on the Solaris JVM, reported by a customer of ours.
Our server application consists of multiple threads. Normally we see them all running within a single Java process, and all is fine.
At some point in time, and only on Solaris 10, it seems that the main Java process starts a second Java process. This is not our code trying to execute some other application/command. It's the JVM itself forking a new copy of itself. I assumed this was because of some JVM behaviour on Solaris that uses multiple processes if the number of threads is > 128. However at the time of spawn there are less than 90 threads running.
In any case, once this second process starts, some of the threads of the application (incidentally, they're the first threads created by the application at startup, in the first threadgroup) stop working. Our application dumps a list of all threads in the system every ten minutes, and even when they're not working, the threads are still there. Our logs also show that when the second process starts, these threads were not in the running state. They had just completed their operations and were sleeping in their thread pool, in a wait() call. Once the second process starts, jobs for these threads just queue up, and the wait() does not return, even after another thread has done a notify() to inform them of the new jobs.
Even more interesting, when the customer manually kills -9 the second process, without doing anything in our application, all threads that were 'frozen' start working again, immediately. This (and the fact that this never happens on other OSes) makes us think that this is some sort of problem (or misconfiguration) specific to the Solaris JVM, and not our application.
The customer initially reported this with JDK 1.5.0_12 , we told them to upgrade to the latest JDK 1.6 update 6, but the problem remains. There are no special JVM switches (apart from -Xms32m -Xmx256m) used. We're really at a dead end here in diagnosing this problem, as it clearly seems to be outside our app. Any suggestion?

Actually, we've discovered that that's not really what was going on. I still believe there's a bug in the JVM, but the fork was happening because our Java code tries to exec a command line tool once a minute. After hours of this, we get a rogue child process with this stack (which is where we are forking this command line tool once a minute):
JVM version is 1.5.0_08-b03
Thread t@38: (state = IN_NATIVE)
- java.lang.UNIXProcess.forkAndExec(byte[], byte[], int, byte[], int, byte[], boolean, java.io.FileDescriptor, java.io.FileDescriptor, java.io.FileDescriptor) @bci=168980456 (Interpreted frame)
- java.lang.UNIXProcess.forkAndExec(byte[], byte[], int, byte[], int, byte[], boolean, java.io.FileDescriptor, java.io.FileDescriptor, java.io.FileDescriptor) @bci=0 (Interpreted frame)
- java.lang.UNIXProcess.<init>(byte[], byte[], int, byte[], int, byte[], boolean) @bci=62, line=53 (Interpreted frame)
- java.lang.ProcessImpl.start(java.lang.String[], java.util.Map, java.lang.String, boolean) @bci=182, line=65 (Interpreted frame)
- java.lang.ProcessBuilder.start() @bci=112, line=451 (Interpreted frame)
- java.lang.Runtime.exec(java.lang.String[], java.lang.String[], java.io.File) @bci=16, line=591 (Interpreted frame)
- java.lang.Runtime.exec(java.lang.String, java.lang.String[], java.io.File) @bci=69, line=429 (Interpreted frame)
- java.lang.Runtime.exec(java.lang.String) @bci=4, line=326 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)There are also several dozen other threads all with the same stack:
Thread t@32: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: can't map thread id to thread handle!
     at sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
     at sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.getThreadIntegerRegisterSet(ProcDebuggerLocal.java:364)
     at sun.jvm.hotspot.debugger.proc.sparc.ProcSPARCThread.getContext(ProcSPARCThread.java:35)
     at sun.jvm.hotspot.runtime.solaris_sparc.SolarisSPARCJavaThreadPDAccess.getCurrentFrameGuess(SolarisSPARCJavaThreadPDAccess.java:108)
     at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:252)
     at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:211)
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
     at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
     at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
     at sun.jvm.hotspot.tools.JStack.main(JStack.java:58)I'm pretty sure this is because the fork part of the UnixProcess.forkAndExec is using the Solaris fork1 system call, and thus all the Java context thinks all those threads exist, whereas the actual threads don't exist in that process.
It seems to me that something is broken in UnixProcess.forkAndExec in native code; it did the fork, but not the exec, and this exec thread just sits there forever. And of course, it's still holding all the file descriptors of the original process, which means that if we decide to restart our process, we can't reopen our sockets for listening or whatever else we want to do.
There is another possibility, which I can't completely rule out: this child process just happened to be the one that was fork'd when the parent process called Runtime.halt(), which is how the Java process exits. We decided to exit halfway through a Runtime.exec(), and got this child process stuck. But I don't think that's what happens... from what I understand that we collected, we see this same child process created at some point in time, and it doesn't go away.
Yes, I realize that my JVM is very old, but I cannot find any bug fixes in the release notes that claim to fix something like this. And since this only happens once every day or two, I'm reluctant to just throw a new JVM at this--although I'm sure I will shortly.
Has anyone else seen anything like this?

Similar Messages

  • Understanding asynchronous BPEL processes and threading

    Hi everybody,
    I have a question regarding asynchronous BPEL processes and threading.
    I have an asynchronous BPEL process which delivers a message (picked up from a JMS topic) to a repository (via for instance an ICAN server). This transaction must be time-sensitive, i.e. first-in is first-out.
    If I have several threads running concurrently I have no guarantee that the messages are delivered in the same order that they where generated. It seems that the suggested way around this problem is to use single-threading.
    So here's the part that I do not understand:
    An asynchronous request does not wait for a response. If I use single-threading then what happens to that single-thread if for some reason (for instance the ICAN server is not responding) the BPEL instance is dehydrated? Is the thread still allocated for waiting for the first response or does it go back and pick up the next thing on the topic? If the next thing on the topic is picked up then how can I be sure that the messages are delivered in the correct order? If the thread is still allocated to the single instance then does that not kindda go against the whole concept of an asynchronous process?
    I hope some one can explain this to me.
    Thank you in advance.
    Kind regards,
    Mathias
    PS. This question is an attempt of a different angle on another forum thread: Asynchronuos BPEL - how to get the order of messages right?

    Hi again,
    A quick follow up question:
    Is there any way to have several BPEL processes running on the same server with different settings for how many threads to use?
    It seems that the only way to set number of threads used by BPEL is to use the configuration in the BPEL console? This is an domain setting - which means that all processes deployed to the server is effected the same way?
    I am asking because I am facing a scenario where I need one BPEL process to read from a AQ queue (well actually a topic) with a single thread and another BPEL process - deployed to the same BPEL server - to read from the same AQ queue with multiple threads.
    Does anyone have any idea of how to do this? Can it be done?
    Any suggestions, hints or reference to any documentation would be very much appreciated.
    Thank you in advance,
    Aagaard

  • Process and Thread

    I am new to the servlet and JSP technology.It is said that "Servlets and CGI differ mainly in Process and Thread.CGI used processes and Servlet technology uses Threads".I would like to know what exacly is the meaning of a process and a thread.How do we say that a process uses higher resources and hence a higher resource and memory overhead.
    What do we exactly mean by thread and how r they using less resources?When we say Thread t=new Thread(); what exactly is happening inside?

    A thread runs with in the context of the JVM process.
    http://java.sun.com/docs/books/tutorial/essential/threads/definition.html
    A process is the running of an executable. A cgi can be a C .exe file or a perl script run under perl.exe.
    The thread is faster because the jvm process has been loaded into memory and initialized. For each cgi call the operating system has to load the code into memeory and intialize all the variables and resources.
    http://c2.com/cgi/wiki?CgiVsServlet
    http://www.disc.com/jdbcserv.html

  • Process and threads

    Dear community
    Very kindly pleasant about information that is fiber in thread process and threads it interface between user and application ?
    is this answer on user ask in application run ?
    About any advises who concerns this matter , please .

    Hello RB_1.
    You can find more informations about fibers
    here and
    here on the MSDN Library.
    Bye.
    Luigi Bruno
    MCP, MCTS, MOS, MTA

  • OracleIAS and Information about Process and Threads

    Hi all,
    I need to know information about the running process and threads inside a Conteiner or an Enterprise Application, is it possible?
    Thanks in advance.

    Hi,
    Probably all of mentioned in some extent - depends, which data system needs.
    regards,
    wojciech

  • I'm running Lion on a Late 2008 MacBook Air.  With no other applications running, I can no longer Check for Software Updates.  It gets abut half way through the checking process and then freezes.  The MBA passes Disk Utility. Thoughts?

    I'm running Lion on a Late 2008 MacBook Air. 
    With no other applications running, I can no longer Check for Software Updates.  It gets about half way through the checking process and then freezes. 
    The MBA passes Disk Utility. Thoughts?
    Thanks
    Jim Taylor

    How long do you leave it?
    Mine can sometimes sit for quite a while (maybe 10 minutes??) before it finally finishes.
    You could look at the relevant logs via Console (Utilities folder) to see if Software Update is complaining about anything.
    charlie

  • GUI ..Process and threads

    Hi..Folks
    Is it possible to read the output of a class by starting it as a separate thread in a GUI application?
    Currently we follow this approach. We start the class as a separate process (passing in what ever parameters needed) in the GUI(using RunTime.exec()). Read the output.
    But we need a reference to the class that is being started.
    Is it possible to get a reference to the class?
    Thanks in advance.

    Thanks for your replies.
    Oki this time i will be more specific..I want to start and read the output of a thread..So while the thread is executing I can hook in to the output stream and read the output simultaneously...Actually I came across this link [Convert a Java OutputStream to an InputStream|http://ostermiller.org/convert_java_outputstream_inputstream.html] ... which suggests some idea to it...But the code gives compilation error (the bold line) ..."Cannot refer to a non-final variable in inside an inner class defined in a different method"..I have attached the code..Any ideas?
    Readin.java creates the thread and sends in the PipedInputStream to tt.java's putDataOnOutputStream() which writes to the stream back...
    Readin.java
    byte[] b = null;
               PipedInputStream in = new PipedInputStream();
               PipedOutputStream out ;
              try {
                   out = new PipedOutputStream(in);
              } catch (IOException e) {
                   e.printStackTrace();
                new Thread(
                  new Runnable(){
                    public void run(){
                      try {
                        *     new tt().putDataOnOutputStream(new PipedOutputStream(in));*
                        } catch (IOException e) {
                             // TODO Auto-generated catch block
                             e.printStackTrace();
                ).start();
               try {
                   in.read(b);
              } catch (IOException e) {
                   e.printStackTrace();
               String str = new String(b);
               System.out.println(str);
    ...          And the code for the other class is
    ... tt.java.....
    public  void putDataOnOutputStream(ByteArrayOutputStream out){
              String str="test";
              try {
                   out.write(str.getBytes());
              } catch (IOException e) {
                   // TODO Auto-generated catch block
                   e.printStackTrace();
         }

  • What is the difference between Process and Threads

    What is the difference between Process and Threads

    http://forum.java.sun.com/thread.jsp?forum=45&thread=525518&start=0&range=15#2519429

  • Java swing and threading issue

    i am trying to update swing components from some thread. at my first try, i noticed this was a disaster. upon further research, i found out about the single-threaded model of swing events (i.e. event dispatching thread).
    i did more research and could NOT find a good example on how to update swing components from other threads. many examples were showing too much.
    can someone post a simple example on here? i just want to see how to properly update a swing component from a non-swing class using threading.

    I think its a simple example:
    http://forum.java.sun.com/thread.jspa?forumID=57&threadID=621226
    Let us know what you think.

  • Process and thread CPU%

    Hello,
    Is there a utility to allow me to view CPU usage of my process and its inidividual threads (i.e. not "top").
    Thanks,
    Neil

    tray
    sar n
    where n is the time period
    Ahmed fathy

  • Logging Java objects and threads. Application sampling.

    I am using a laptop to run my application on. It uses windows XP and it is the only application running on that machine. I use TCP and serial port to communicate to other devices. My application is supposed to use up to 300M of memory but apparently it using much more than that. The laptop physical memory size is 512M. I am wondering if there is a way to sample my application periodically. What I mean by that is, after the applications runs I want to run at the same time kind of profiler that every, let us say, 10 minutes logs the number of objects created and what those objects are or what the current ones are. The number of threads running and what are their names. I want to know what is using all of this memory. The memory usage goes up to about 420M. As a result, the application throws out of memory exception. I believe I can get the number of threads running but do not know how to get their names. I need to know all of that so I would go into my code and see where it is happening and try to solve the problem. One of the ways I have in my mind now is to put a print out everywhere I create an object or thread, but that�s very expensive. Any hints or tips could be useful.
    Thank you in advance.
    P.S.:
    I do not want to run a profiler such as JProfiler because the application is running live and I do not have access to it. I want to embed these logs into the application itself and rerun it on a live system.

    As a first thought, I'm guessing that you're merely thinking that your program will not exceed 300 megabytes just based on the -Xmx300M command you may be passing your program. This is a max heap size, not the max footprint your program will take. Be aware that this can go beyond the scope of 300 MB....
    Now, if you want very detailed information about processes, memory usage, etc, you might be able to keep track of it yourself, but if you want this information from a system-level sort of thing you will not get it, as one of the major points of Java is that it does not care about a lot of system specifics.
    Now, if you're really interested in keeping track of threads, I don't see why you could not just write to a log file every time you spawn a new thread of a particular kind.

  • Process and thread execution time

    HI all
    I am doing a project to develop a high level simulation framework. In the project I need to calculate the number of cycles of a snippet of code in a thread. Like
    void func()
    event1():
    int x;
    x = x * x;
    for();
    while();
    exec(numberofcyles)
    event2();
    Here I want to calculate the number of cycles between event1 and event2 and pass these number of cycles to the exec(numberofcycles) which will later on be simulated. I investigated a number of tools like gprof, Dtrace, linux process statistics, rdstc, getrusage(). None of these seems to be very relevent.
    I tried linux process statistics i.e. /proc/<pid>/task/<tid>/stat. I can access the execution time of threads, but all the time I get 0 execution time. What I think that it reads the execution time of threads when it was started. Is there any way to get the updated execution time of thread?
    Any help will be highly appreciated.
    Irfan

    I suggest reposting in the Unix forum here:
    http://discussions.apple.com/forum.jspa?forumID=735

  • Re: Java Images and Threads prob Help!!

    Without going through it in detail the problem may be that you're doing the repaint which you hope will draw the image before the image has completed loading.
    Create your ImageIcon (loading the "xxxx.jpeg" files) during program initialisation. And check the load status of the icon, at least in a test version.

    I still cant get the reqd output. Only the 1st image is dispayed and when i click the second code, the 2nd image doesnt come(the first one does not disappear, and even when i click the code 1 i have to minimize/ maximize the screen, to get the image and after that it does not go for the s2nd image 2 come!! )
    Create your ImageIcon (loading the "xxxx.jpeg" files) during program >>initialisation. And check the load status of the icon, at least in a test version. How do u load the image during initialization??? and can ny1 plz tell me how to get the flashing threads with msgs????

  • How to check actual allocated and used memory for java process in solaris?

    Hi,
    I'm testing performance for java application on solaris 10. And I would like to know how to measure actual memory allocated and used for java process.
    I'm setting -Xms512m -Xmx512m for my java process and I use prstat command to monitor it's memory. But I found that when I run prstat to check, in SIZE column is more than I setting (found 644).
    So I'm don't know the actual memory that java process used. (In this case,is it mean the process use memory (644) more than setting (512)?)
    Thank you.

    With Xms/Xmx you specify the Java heap size. On top of that comes the permanent generation (default max size 64m) and the C part of the process (the JVM itself with all its libraries and data).
    With "ps -e -o pid,vsz,rss,args" you get the virtual and set resident size of your processes.
    Nick.

  • Java Processes for Planning Applications

    Hi There,
    We've experienced performance issues where we have reached the limit of the java process for Hyperion Planning - WAS element. My question! Is it possible to create multiple java processes for Hyperion Planning, for example: If we have 7 Planning applications could we have 7 Java processes running - one for each application?
    Thanks
    Mark

    Couple of thoughts on scaling:
    1. Change the mx setting to 1200m -- this is the most you can scale a single instance on 32-bit
    2. Use a 64-bit platform you can set memory settings really high (AIX/Solaris)
    3. Multiple servers with planning on it using a load balancer
    Regards,
    John A. Booth
    http://www.metavero.com

Maybe you are looking for