CPU high utilization on OAS Server, how to find the cause of problem ?

Question

Hi all,
We are running ADFBC application on OAS 10.1.3.3 on RH Linux ES 4. The database is Oracle 10g on RH Linux also.
here is the setting in opmn.xml :
<data id="java-bin" value="/opt/oracle/OraHome_oc4j/jdk64/jdk1.5.0_17/bin/java"/>
<data id="java-options" value="-server -XX:CompileCommand=exclude,oracle/sql/NUMBER,toBytes
-ms3072M -mx3072M -XX:+AggressiveHeap -XX:MaxPermSize=1024M -XX:AppendRatio=3 -Djava.security.policy=$ORACLE_HOME/j2ee/ittas/config/java2.policy
-Djava.awt.headless=true -Dhttp.webdir.enable=false"/>
Most of time I see (with linux "top" command) that CPU utilizationis 99.9 %
And sometimes at that time user gets very slow that we have to restart the OC4J to the it normal again.
How do I know what is the cause of that problem ?
Thank you very much,
xtanto

Answer 1

Maybe this could help.
Regars
From note 467624.1
Applies to:
Oracle Containers for J2EE - Version: 10.1.2.0.0 to 11.0.0.0
Linux x86
Purpose
To find out the java thread that is taking high CPU usage in OC4J
This article is specific to Linux due to its threading model and makes specific use of parameters for the ps command that only exist in Linux but not in other operating systems. The general idea from this note might, however, be used on other platforms also, but then requires deeper knowledge of that operating system.
Scope and Application
If you are using the JDK 1.4 then you need to set the LD_ASSUME_KERNEL value to 2.4.1 to get the unique nid value.
In JDK 1.5 setting the LD_ASSUME_KERNEL to 2.4.1 is not required.
Add this variable to ORACLE_HOME\opmn\conf\opmn.xml file as follows:
<ias-instance id="j2ee3.host.domain">
<environment>
<variable id="TMP" value="/tmp"/>
<variable id="LD_ASSUME_KERNEL" value="2.4.1"/>
</environment>
Find Out The Thread That Is Taking High CPU Usage In OC4J
Step 1: Find out the Process ID of the OC4J container
You can find out the pid from the output of the following command:
ORACLE_HOME\opmn\bin> opmnctl status
For example
O_H\opmn\bin> opmnctl status
Processes in Instance: j2ee3.host.domain
------------------------------------------------+---------
ias-component | process-type | pid | status
------------------------------------------------+---------
DSA | DSA | N/A | Down
LogLoader | logloaderd | N/A | Down
dcm-daemon | dcm-daemon | N/A | Down
OC4J | home | 28370 | Alive
WebCache | WebCache | 28380 | Alive
WebCache | WebCacheAdmin | 28371 | Alive
HTTP_Server | HTTP_Server | 28373 | Alive
In this note, we will try to trace the java thread that is using high CPU in OC4J 'home'. The pid of 'home' container is 28370 according to the example.
Step 2: Find the actual process ID
Use the following command to find out the actual process ID:
ps -ef | grep 28370
rac 28370 28354 0 11:23 ? 00:00:01 /home/rac/j2ee2/jdk/bin/java
rac 28377 28370 0 11:23 ? 00:00:00 /home/rac/j2ee2/jdk/bin/java
In this case the actual java process is 28377.
Step 3: Find out the thread which takes lot of CPU time
Run the command
watch "ps -eLo pid,ppid,tid,pcpu,comm | grep 28377"
The thread ID of the thread that consumes high CPU time will be shown in the first column of the output
Step 4: Take thread dump to trace the java thread with the thread ID.
Take the thread dump using the following command:
kill -3 28370 (process-id-from-opmn-status-output)
The thread dump will be written to the log file at ORACLE_HOME\opmn\logs\OC4J~<OC4J_Container_Name>~default_island~1.
Step 5: Trace the java thread.
Convert the Thread ID you got from the command,
ps -eLo pid,ppid,tid,pcpu,comm | grep 28377
to hexadecimal and then search for this value in the thread dump.
For example, if you get the thread id '5828' in the first column of this command, convert '5828' to hexadecimal:
5828 (dec) corresponds to 0x16c4 (hex)
Now search for nid=0x16c4 in the thread dump, and you may find something like:
"RMIServer [0.0.0.0:12401] count:1" prio=1 tid=0xa7c1ee80 nid=0x16c4 runnable [a777e000..a777e22c]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0xac0d5b50> (a java.net.PlainSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at com.evermind.server.rmi.RMIServer.run(RMIServer.java:464)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:186)
at java.lang.Thread.run(Thread.java:534)