JDK1.3: 100% CPU and total hanging up

Question

Our environemt is HP-UX with 4 CPU, 2.5Gb of memory. Java is 1.3.13 (latest release). All necessary Unix patches installed.
We perform a load test over our application server with 40 users only. The test is pretty simple and doesn't deserve to be explained here (it uses no special system resources). In an unpredictable moment the java process utilizes 100% CPU and doesn't react to 'kill -3' but to 'kill -9' only. The problem is it happens whenever it wishes. No logic behind it. Every test loop takes about 1 minute to complete and the server gets stucked sometimes after 10 mins and sometimes after 3 houres of work. Changing java heap parameters and playing with new/old generation ratio also doesn't lead to any conclusion. Sometimes it happens before full gc but other settings allow full gc to happen every minute and the server works for hours.
What can it be? Any ideas? Any directions of investingation the problem?

Answer 1

What can it be? Any ideas? Any directions of investingation the problem?An endless do-nothing loop. I've seen HotSpot optimize such a loop to a single "jump-to-self" CPU instruction. HotSpot doesn't know how to insert a suspension safepoint into a loop that consists of a single instruction. When GC next kicks in it suspends every thread (including the signal dispatcher, which is responsible for a thread dump) but the nop-loop thread continues with free reign of the CPU.
In my case the loop looked like the following. A thread is started running this class, later another thread passes it some data by calling put(). There is no synchronization so the former thread is not obliged to see the effects of any other thread. The optimizer legally considered this loop to be do-nothing.
class ... implements Runnable {
    private Object data;
    public void put(Object data) {
        this.data = data;
    public void run() {
        while (true) {
            if (data == null) continue;
            // process data
}Since you can't get a thread dump it's very hard to pinpoint the problematic piece of code (unless you have Sun's poprietary tools that restore Java state from a core file).
If your code base is reasonably small, review it thoroughly for what can be an endless do-nothing loop. Another approach is to incrementally insert System.err.println("") into suspicious loops. This will add a call to native code and thus a suspension safepoint. Once you no longer can reproduce the problem, you must have hit the problemmatic loop. Go back and review it carefully.