Java process - high CPU usage

Question

Hi,
I'm describing a high CPU scenario which gets triggered randomly ( I'm not able to replicate it on my lab setup).
There are around 120 threads which are running in my java process. The jvm is running on a high traffic (through put) site, where there are a lot of async events coming to the java process.( around 220 events per 60 seconds ). The java process works fine in this scenario, the normal CPU consumption hovers around 1.5 % to 2.0 %.
But, at times, I've seen CPU to be as high as 43 %, and it stays at that value for hours altogther. In those situations, I usually do a failover to standby java process. I tried debugging the issue to see which java thread could be causing the issue, but, I could not come to a conclusion or replicate the situation in lab environment.
Here are the details of the execution environment
java -version
java version "1.4.2_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_11-b06)
Java HotSpot(TM) Client VM (build 1.4.2_11-b06, mixed mode)
prstat during high CPU
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
10485 root 120 10 0 570M 381M cpu1 268:10 43.64% java
prstat -Lm -p output
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
10485 root 53 0.5 0.2 0.0 0.0 30 0.2 16 69 1K 118 0 java/2
10485 root 31 0.0 0.1 0.0 0.0 53 0.2 16 23 778 93 0 java/26
10485 root 0.4 0.0 0.0 0.0 0.0 99 0.0 0.1 10 16 106 0 java/12
10485 root 0.1 0.0 0.0 0.0 0.0 100 0.0 0.0 3 2 7 0 java/15
10485 root 0.1 0.0 0.0 0.0 0.0 97 0.0 2.4 120 3 128 0 java/41
10485 root 0.1 0.0 0.0 0.0 0.0 97 0.0 2.5 120 4 131 0 java/410
Some more points about the last prstat -Lm output.
java/2 is "VM Thread" ( responsible for GC). "VM Thread" is having a NORMAL priority ( 5 )
java/26 is a "Worker" thread, with priority MINIMUM ( 1 ).
Could you suggest what could be issue, and what other information I could collect to find out the issue. Its difficult to profile the process because the problem scenario is difficult to ascertain and the process is running on a production setup.
Any help is appreciated.
Thanks
Sanjay

Answer 1

Hi,
Thanks for your response. Both, the production setup and lab setup have have 2 physical CPUs.
Actually, there are two java threads ( machine is solaris 10) one is "VM Thread" and other is my applications worker thread. (there are 10 of them with priority 1). If you look at the top two lwps in the prstat -Lm , both are showing high value of ICX.
I'm still not able to drill down to my code level. (Worker thread is waiting on a queue to de-queue server request). Could you give some hint to move foward?
rgds
Sanjay