How to kill Forms Runaway Process using 95% CPU and running for 2 hours.

Question

We had a situation at E-Business Suite customer (using Oracle VM server) where some of Form processes were not being cleared by form timeout settings automatically.
Also when user exits the form session from front end, the linux form process (PID) and DB session did not exit properly, so they got hung.
They were spiking CPU and memory usage and causing e-business suite to perform slowely and ultimately causing VM host to reboot the production VM guest (running on Linux).
We could see the form processes (PIDs) using almost 100% cpu with "top" command and running for a long time.
Also we verified those Form Sessions did not exist in the application itself.
ie. Using from Grid Control -> OAM-> Site Map -> Monitoring (tab) -> "Form Sessions".
It means that we could safely kill that form process from Linux using "kill -9 <PID>" command.
But that required a continuous monitoring and manual DBA intervention as customer is 24x7 customer.
So, I wrote a shell script to do the following;
•     Cron job runs every half an hour 7 days a week which calls this shell script.
•     Shell script runs and tries to find "top two" f60webmx processes (form sessions) using over 95% cpu with 2 minutes interval.
•     If no process is found or CPU% is less than 95%, it exits and does nothing.
•     If top process is found, it searches for its DB session using apps login (with hidden apps password file - /home/applmgr/.pwd).
a.     If DB session is NOT found (which means form process is hung), it kills the process from unix and emails results to <[email protected]>
b.     If DB session is found, it waits for 2 hours so that form process times automatically via form session timeout setting.
It also emails the SQL to check the DB session for that form process.
c.     If DB session is found and it does not timeout after 2 hours,
it kills the process from unix (which in turn kills the DB session). Output is emailed.
This are the files required for this;
1. Cron job which calls the shell script looks like this;
# Kill form runaway process, using over 95% cpu having no DB session or DB session for > 2hrs
00,30 * * * * /home/applmgr/forms_runaway.sh 2>&1
2. SQL that this script calls is /home/applmgr/frm_runaway.sql and looks like;
set head off
set verify off
set feedback off
set pagesize 0
define form_client_PID = &1
select count(*) from v$session s , v$process p, FND_FORM_SESSIONS_V f where S.AUDSID=f.audsid and p.addr=s.paddr and s.process='&form_client_PID';
3. Actual shell script is /home/applmgr/forms_runaway.sh and looks like;
# Author : Amandeep Singh
# Description : Kills runaway form processes using more than 95% cpu
# and Form Session with no DB session or DB session > 2hrs
# Dated : 11-April-2012
#!/bin/bash
. /home/applmgr/.bash_profile
PWD=`cat ~/.pwd`
export PWD
echo "`date`">/tmp/runaway_forms.log
echo "----------------------------------">>/tmp/runaway_forms.log
VAR1=`top -b -u applmgr -n 1|grep f60webmx|grep -v sh|grep -v awk|grep -v top|sort -nrk9|head -2|sed 's/^[ \t]*//;s/[ \t]*$//'| awk '{ if ($9 > 95 && $12 = "f60webmx") print $1 " "$9 " "$11 " "$12; }'`
PID1=`echo $VAR1|awk '{print $1}'`
CPU1=`echo $VAR1|awk '{print $2}'`
TIME1=`echo $VAR1|awk '{print $3}'`
PROG1=`echo $VAR1|awk '{print $4}'`
PID_1=`echo $VAR1|awk '{print $5}'`
CPU_1=`echo $VAR1|awk '{print $6}'`
TIME_1=`echo $VAR1|awk '{print $7}'`
PROG_1=`echo $VAR1|awk '{print $8}'`
echo "PID1="$PID1", CPU%="$CPU1", Running Time="$TIME1", Program="$PROG1>>/tmp/runaway_forms.log
echo "PID_1="$PID_1", CPU%="$CPU_1", Running Time="$TIME_1", Program="$PROG_1>>/tmp/runaway_forms.log
echo " ">>/tmp/runaway_forms.log
sleep 120
echo "`date`">>/tmp/runaway_forms.log
echo "----------------------------------">>/tmp/runaway_forms.log
VAR2=`top -b -u applmgr -n 1|grep f60webmx|grep -v sh|grep -v awk|grep -v top|sort -nrk9|head -2|sed 's/^[ \t]*//;s/[ \t]*$//'| awk '{ if ($9 > 95 && $12 = "f60webmx") print $1 " "$9 " "$11 " "$12; }'`
PID2=`echo $VAR2|awk '{print $1}'`
CPU2=`echo $VAR2|awk '{print $2}'`
TIME2=`echo $VAR2|awk '{print $3}'`
PROG2=`echo $VAR2|awk '{print $4}'`
PID_2=`echo $VAR2|awk '{print $5}'`
CPU_2=`echo $VAR2|awk '{print $6}'`
TIME_2=`echo $VAR2|awk '{print $7}'`
PROG_2=`echo $VAR2|awk '{print $8}'`
HRS=`echo $TIME1|cut -d: -f1`
exprHRS=`expr "$HRS"`
echo "PID2="$PID2", CPU%="$CPU2", Running Time="$TIME2", Program="$PROG2>>/tmp/runaway_forms.log
echo "PID_2="$PID_2", CPU%="$CPU_2", Running Time="$TIME_2", Program="$PROG_2>>/tmp/runaway_forms.log
echo " ">>/tmp/runaway_forms.log
# If PID1 or PID2 is NULL
if [ -z ${PID1} ] || [ -z ${PID2} ]
then
echo "no top processes found. Either PID is NULL OR CPU% is less than 95%. Exiting...">>/tmp/runaway_forms.log
elif
# If PID1 is equal to PID2 or PID1=PID_2 or PID_1=PID2 or PID_1=PID_2
[ ${PID1} -eq ${PID2} ] || [ ${PID1} -eq ${PID_2} ] || [ ${PID_1} -eq ${PID2} ] || [ ${PID_1} -eq ${PID_2} ];
then
DB_SESSION=`$ORACLE_HOME/bin/sqlplus -S apps/$PWD @/home/applmgr/frm_runaway.sql $PID1 << EOF
EOF`
echo " ">>/tmp/runaway_forms.log
echo "DB_SESSION ="$DB_SESSION >>/tmp/runaway_forms.log
# if no DB session found for PID
if [ $DB_SESSION -eq 0 ] then
echo " ">>/tmp/runaway_forms.log
echo "Killed Following Runaway Forms Process:">>/tmp/runaway_forms.log
echo "-------------------------------------------------------------------">>/tmp/runaway_forms.log
echo "PID="$PID1", CPU%="$CPU1", Running Time="$TIME1", Program="$PROG1>>/tmp/runaway_forms.log
kill -9 $PID1
#Email the output
mailx -s "Killed: `hostname -a` Runaway Form Processes" [email protected] </tmp/runaway_forms.log
cat /tmp/runaway_forms.log
else
# If DB session exists for PID
if [ ${exprHRS} -gt 120 ]; then
echo $DB_SESSION "of Database sessions exist for this forms process-PID="$PID1". But its running for more than 2 hours. ">>/tmp/runaway_forms.log
echo "Process running time is "$exprHRS" minutes.">>/tmp/runaway_forms.log
echo "Killed Following Runaway Forms Process:">>/tmp/runaway_forms.log
echo "-------------------------------------------------------------------">>/tmp/runaway_forms.log
echo "PID="$PID1", CPU%="$CPU1", Running Time="$TIME1", Program="$PROG1>>/tmp/runaway_forms.log
kill -9 $PID1
#Email the output
mailx -s "`hostname -a`: Runaway Form Processes" [email protected] </tmp/runaway_forms.log
cat /tmp/runaway_forms.log
else
echo "Process running time is "$exprHRS" minutes.">>/tmp/runaway_forms.log
echo $DB_SESSION "of Database sessions exist for PID="$PID1" and is less than 2 hours old. Not killing...">>/tmp/runaway_forms.log
echo "For more details on this PID, run following SQL query;">>/tmp/runaway_forms.log
echo "-----------------------------------------------------------------------">>/tmp/runaway_forms.log
echo "set pages 9999 lines 150">>/tmp/runaway_forms.log
echo "select f.user_form_name, f.user_name, p.spid DB_OS_ID , s.process client_os_id,, s.audsid, f.PROCESS_SPID Forms_SPID,">>/tmp/runaway_forms.log
echo "to_char(s.logon_time,'DD-Mon-YY hh:mi:ss'), s.seconds_in_wait">>/tmp/runaway_forms.log
echo "from v\$session s , v\$process p, FND_FORM_SESSIONS_V f">>/tmp/runaway_forms.log
echo "where S.AUDSID=f.audsid and p.addr=s.paddr and s.process='"$PID1"' order by p.spid;">>/tmp/runaway_forms.log
mailx -s "`hostname -a`: Runaway Form Processes" [email protected] </tmp/runaway_forms.log
cat /tmp/runaway_forms.log
fi
fi
else
#if PID1 and PID2 are not equal or CPU% is less than 95%.
echo "No unique CPU hogging form processes found. Exiting...">>/tmp/runaway_forms.log
cat /tmp/runaway_forms.log
fi
If you have the same problem with some other unix and DB processes, the script can be easily modified and used.
But use this with thorough testing first (by commenting out <kill -9 $PID1> lines.
Good luck.
Edited by: R12_AppsDBA on 19/04/2012 13:10

Answer 1

Thanks for sharing the script!
Hussein