JVM hand on heavy load

Question

Hi,
We encounter a recurent problem with JVM hanging on production platform (RH9, JVM 1.4.1) and locked thread in a "curious" state, here is (one of) our stack trace :
"reaper:user" daemon prio=1 tid=0x0x698267f0 nid=0x6e3d in Object.wait() [69afe000..69aff010]
at java.lang.Object.wait(Native Method)
- waiting on <0x47cf4e38> (a java.util.Hashtable)
at org.webmacro.broker.ResourceManager$1.run(ResourceManager.java:164)
- locked <0x47cf4e38> (a java.util.Hashtable)
"reaper:template" daemon prio=1 tid=0x0x69826620 nid=0x6e3d in Object.wait() [69a7e000..69a7f090]
at java.lang.Object.wait(Native Method)
- waiting on <0x47cf4eb0> (a java.util.Hashtable)
at org.webmacro.broker.ResourceManager$1.run(ResourceManager.java:179)
- locked <0x47cf4eb0> (a java.util.Hashtable)
"reaper:config" daemon prio=1 tid=0x0x69825ae8 nid=0x6e3d in Object.wait() [699ff000..699ff110]
at java.lang.Object.wait(Native Method)
- waiting on <0x47cf4f28> (a java.util.Hashtable)
at org.webmacro.broker.ResourceManager$1.run(ResourceManager.java:179)
- locked <0x47cf4f28> (a java.util.Hashtable)
Our product is a mobile "web" (in fact wap) application and use an old webmacro version, but we encounter the same problem in other code section: a thread which own a lock wait later in the program for this lock ?
Have anybody encounter this problem ? Is this related to an internal JVM process which will own the lock (GC ?) ...
We have planned to update the JVM version to a more recent 1.4.2 and I've also read RH9/NTP may have some pthreads problem but I'm not sure this will solve the problem.

Answer 1

A code defection is always possible, but our platform handle 200 to 1000 req/s perfectly for hours, load balanced with more than ten java server where some of them don't restart for several days.
For a long time we note java server hanging and we have in our production environment external process examining server activity (logs) and restart them if required, sending a "kill -3" to output a stack trace we can examine later (very useful to find and workaround application bottleneck).
Reading carefully the stack trace show that all waiting threads are "waiting on condition" or "- waiting on <...>", all other threads are runnable. Reading http://java.sun.com/developer/technicalArticles/Programming/Stacktrace, the threads are in CW or MW states which are "normal states" but I don't know for our problem where is the monitor at this time !
We encounter this condition each time a code is using a "synch. object": Vector, Hashtable ... with additionnal synchronized blocks, for example :
Hashtable xyz;
synchronized(xyz) {
xyz.put(...);
Replacing the Hashtable with a simple HashMap and managing the lock ourself solve the problem (we don't encounter other hang in this code later). Sometimes the problem occur in code we don't own : JVM internal: Class loading, String.intern(), Date format "static synched" ... or external components we use.
I admit the problem may be elsewhere and the stack trace we examine are perhaps "normal JVM states", showing only slower because "critical" code section. But it may be also a JVM monitor problem which show in stack trace where our application need synchronized objects !
We will try upgrading the JVM version.