CMS crash with core files and multiple report output generation

Question

Happy new year to everyone,
Our BOXIR3.1SP6FP2 env has recently started behaving weirdly by triggering multiple output to users inbox and email notification out of scheduled reports. Also we have noticed the CMS crash with core file (almost 4GB) generation at the time of multiple report output.
Most of the times, CMC crashes and recycles itself. At few times, CMS services alone went shut down.
OS details: RHEL 5.5, 32 GB RAM, 8 core processor on each of the clustered node, Oracle 10GR2.4 CMS DB server, 11GR2.4 oracle reporting DB server and oracle 11.1.0.6 client.
2015/01/21 23:54:37.946|>=| | |28123|1534131088|{|||||||||||||||DBQueue::Read
2015/01/21 23:54:37.946|==| | |28123|1496185744|
|||||||||||||||(OracleStatement.cpp:156) Prepare: SQL: SELECT ObjectID,
Version, LastModifyTime, CRC, Properties FROM CMS_InfoObjects6 WHERE ObjectID
IN (1004050) ORDER BY ObjectID
2015/01/21 23:54:37.946|==| | |28123|1496185744| ||||||||||||||(OracleStatement.cpp:183) Prepared statement Execute
2015/01/21 23:54:37.965|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString 50293
2015/01/21 23:54:37.966|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString Unknown exception in database thread
2015/01/21 23:54:37.967|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString 33007
2015/01/21 23:54:37.967|==| | |28123|1496451984| |||||||||||||||SResourceSource::LoadString CMS is unstable and will shut down immediately. Reason: %1...
2015/01/21 23:54:38.506|==| | |28123|1496185744| |||||||||||||||(OracleStatement.cpp:156) Prepare: SQL: SELECT ObjectID,
Version, LastModifyTime, CRC, Properties FROM CMS_InfoObjects6 WHERE ObjectID IN (1009213) ORDER BY ObjectID
2015/01/21 23:54:38.506|==| | |28123|1496185744| |||||||||||||||(OracleStatement.cpp:183) Prepared statement Execute
2015/01/21 23:54:38.512|==| | |28123|1455592672| |||||||||||||||(sidaemon.cpp:549) SUNIXDaemon::run: server restart flag is 1..
2015/01/21 23:54:38.513|==| | |28123|1455592672| |||||||||||||||(sidaemon.cpp:552) SUNIXDaemon::run: in abort ...
2015/01/21 23:54:38.513|==| | |28123|1455592672| |||||||||||||||(sidaemon.cpp:555) SUNIXDaemon::run: doing the WithAbort case ...
2015/01/21 23:54:38.520|==| | |28123|1496185744| |||||||||||||||(dbq.cpp:1357) DBQ: Time required to read 1 objects: 20.000000 ms
Thank you,
Karthik

Answer 1

Hi Denis,
I'm trying my best for the last few weeks to understand the core issue along with SAP however it is still a mystery.
>Ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 270335
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 270335
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Below is the observation as part of troubleshooting:
1. CMS breaks at threshold of 3.9 G.
2. CMS DB sits in a different Linux server than BOE server.
3. All core files were generated by boe_cmsd process and are almost 4GB in size (same as max threshold which it breaks).
4. Shell script which I've added in the BOE servers shows that the CMS DB is available/connecting at the time of CMS crash.
5. SAP analysed the Core files and skeptical about the below lines.
     #3  0x58687b80 in skgesigCrash ()
      from /opt/oracle/product/11.1.0/client_1/lib32/libclntsh.so
     #4  0x58687e0d in skgesig_sigactionHandler ()
I'll continue troubleshooting with a hope to fix it at the earliest.
Thanks,
Karthik