Terminated by signal 11 error for reports

Question

Hi,
I observed in internal concurrent manager log is sh: line 71: 3501 Segmentation fault
and like showing the below
========================================================================
Starting PROD_0826@PROD Internal Concurrent Manager -- shell process ID 1081
logfile=/oracle/apps/PROD/prodcomn/admin/log/PROD_oraapps01/PROD_0826.mgr
PRINTER=noprint
mailto=applprod
restart=N
diag=N
sleep=60 (default)
pmon=20 (default)
quesiz=1 (default)
sh: line 71: 1086 Segmentation fault FNDLIBR FND CPMGR "FNDCPMBR sysmgr=\"\" $maxreq $sleep $pmon $quesiz $diag logfile=$logfile $target" <<STOP
$sysmanager
STOP
The PROD_0826@PROD internal concurrent manager has terminated with status 139 - giving up.
========================================================================
Starting PROD_0826@PROD Internal Concurrent Manager -- shell process ID 5471
logfile=/oracle/apps/PROD/prodcomn/admin/log/PROD_oraapps01/PROD_0826.mgr
PRINTER=noprint
mailto=applprod
restart=N
diag=N
sleep=60 (default)
pmon=20 (default)
quesiz=1 (default)
Application Object Library: Concurrent Processing version 11.5
Process monitor session ended : 26-AUG-2011 10:14:41
Process monitor session started : 26-AUG-2011 10:16:41
Found running request 1620562 attached to dead manager process.
Setting request status to completed.
Found dead process: spid=(6657), cpid=(121680), ORA pid=(59), manager=(0/0)
Starting STANDARD Concurrent Manager : 26-AUG-2011 10:16:45
Process monitor session ended : 26-AUG-2011 10:16:45
Process monitor session started : 26-AUG-2011 10:18:45
Process monitor session ended : 26-AUG-2011 10:18:47
than manager are bring down like CM status shows
than we checking the adcmctl.sh status apps/apps
CM status shows ICM status could not be determined.
try to open request log than we are getting error like
'Error connecting to database PROD: ORA-3113' and also all requests completed with errors.
than stop all services here CM failed to stop
/oracle/apps/PROD/prodcomn/admin/scripts/PROD_oraapps01/adcmctl.sh: line 244: 9681 Segmentation fault $FND_TOP/bin/CONCSUB "$unpw" SYSADMIN $WAIT_PARAM 'System Administrator' SYSADMIN CONCURRENT FND SHUTDOWN
[Service Control Execution Report]
The report format is:
<Service> <Script> <Status>
Oracle Apache Server PROD_oraapps01 adapcctl.sh Stopped
OracleTNSListener80APPS_PROD_oraapps01 adalnctl.sh Stopped
OracleConcMgrPROD_oraapps01 adcmctl.sh Failed
OracleFormsServer-Forms60PROD_oraapps01 adfrmctl.sh Stopped
OracleReportServer-Rep60_PROD adrepctl.sh Stopped
Oracle Metrics Client PROD_oraapps01 adfmcctl.sh Stopped
Oracle Metrics Server PROD_oraapps01 adfmsctl.sh Stopped
Oracle Fulfillment Server PROD_oraapps01 jtffmctl.sh Stopped
Oracle Discoverer services PROD_oraapps01 addisctl.sh Stopped
after this again stop the CM by adcmctl stop apps/apps
than it stopped after that start the all services than works fine,after starting services all requests completed with terminated by signal 11 while CM down ,it happens everyday.
everyday midnight we take cold backup by scheduling cronjobs
and also after cloning the ICM did not show the sigmentation fault and in TEST some requests completed with terminated by signal 11
please give me solution for this issue.
Thanks,
mahi

Answer 1

Mahi,
This appears to be a serious database issue. Unfortunately, there is no simple magical fix I guess. Going through Oracle Support, instead of applying some patch, is the better route, i believe.
To me, it appears more of bug# 3804515. Please check the following:
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx?id=3804515.8
You may want to review the following patches too:
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 6945111]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 6661430]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 6666626]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 6682463]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 6729613]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 6931682]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 6661435]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 7303352]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 7303650]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 7350779]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 7466966]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 7555753]
"KWQICACLCUR" SHOULD NOT WRITE TRACE FILES FOR ORA-1013 NOR ORA-3113          [Patch 7611844]
Thanks,
Raj Mareddi