Intermittent JVM SIGSEV crashes in symbols of various GC strategies

Question

h2. Hello!
Our JVM crashes on a very regular base with a SIGSEV located in symbols located in the GC implementations.
We tried to split up our deployment into multiple, smaller JVMs as well as switching over to CMS based GC.
Both did not help.
Here 3 typical crash frames:
# V [libjvm.so+0x69c800] MarkSweep::IsAliveClosure::do_object(oopDesc*)+0x20
# V [libjvm.so+0x756687] PSMarkSweepDecorator::precompact()+0x307
# V [libjvm.so+0x4b0f1a] instanceKlass::oop_follow_contents(oopDesc*)+0x13a
The deployed applications produce a complex and wide web of object graphs, but has only a moderate CPU load.
Any hints what might be the problem and on how to tackle them?
Thank your very much for your support
- Benjamin
h2. JVM Arguments
*"Old" Parallel Version*
jvm_args: -XX:NewRatio=2 -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:MaxPermSize=512m -Xms1350M -Xmx4096M -Xss1024k -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/srv/tomcat/logs/gc.log
CMS GC
jvm_args: -XX:+UseConcMarkSweepGC -XX:+CMSScavengeBeforeRemark -XX:+UseParNewGC -XX:+CMSConcurrentMTEnabled -XX:NewRatio=2 -XX:MaxPermSize=512m -Xms1350M -Xmx3072M -Xss1024k -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/srv/tomcat/logs/gc.log
h2. Full Crash Stacks
Stack: [0x00007fc7bb6e0000,0x00007fc7bb7e1000], sp=0x00007fc7bb7df5a8, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x69c800] MarkSweep::IsAliveClosure::do_object(oopDesc*)+0x20
V [libjvm.so+0x46fb0f] GenMarkSweep::invoke_at_safepoint(int, ReferenceProcessor*, bool)+0x28f
V [libjvm.so+0x3c01b9] CMSCollector::do_compaction_work(bool)+0x1f9
V [libjvm.so+0x3bfbd0] CMSCollector::acquire_control_and_collect(bool, bool)+0x160
V [libjvm.so+0x3bf8d8] ConcurrentMarkSweepGeneration::collect(bool, bool, unsigned long, bool)+0xf8
V [libjvm.so+0x46ccaa] GenCollectedHeap::do_collection(bool, bool, unsigned long, bool, int)+0x51a
V [libjvm.so+0x46e1b3] GenCollectedHeap::do_full_collection(bool, int)+0x53
V [libjvm.so+0x85fd36] VM_GenCollectFull::doit()+0x66
V [libjvm.so+0x86d7fa] VM_Operation::evaluate()+0x4a
V [libjvm.so+0x86cde2] VMThread::evaluate_operation(VM_Operation*)+0x82
V [libjvm.so+0x86d053] VMThread::loop()+0x193
V [libjvm.so+0x86cb5e] VMThread::run()+0x6e
V [libjvm.so+0x710adf] java_start(Thread*)+0x13f
VM_Operation (0x00007fc7ba70c520): GenCollectFull, mode: safepoint, requested by thread 0x000000004167f800
Stack: [0x00007f330cca1000,0x00007f330cda2000], sp=0x00007f330cda04f0, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x4b0f1a] instanceKlass::oop_follow_contents(oopDesc*)+0x13a
V [libjvm.so+0x69c33b] MarkSweep::follow_stack()+0x7b
V [libjvm.so+0x755e14] PSMarkSweep::mark_sweep_phase1(bool)+0xe4
V [libjvm.so+0x754cc1] PSMarkSweep::invoke_no_policy(bool)+0x481
V [libjvm.so+0x754803] PSMarkSweep::invoke(bool)+0x83
V [libjvm.so+0x8608f7] VM_ParallelGCSystemGC::doit()+0xf7
V [libjvm.so+0x86d7fa] VM_Operation::evaluate()+0x4a
V [libjvm.so+0x86cde2] VMThread::evaluate_operation(VM_Operation*)+0x82
V [libjvm.so+0x86d053] VMThread::loop()+0x193
V [libjvm.so+0x86cb5e] VMThread::run()+0x6e
V [libjvm.so+0x710adf] java_start(Thread*)+0x13f
VM_Operation (0x00007f3307dfc710): ParallelGCSystemGC, mode: safepoint, requested by thread 0x00000000409cd000
Stack: [0x00007f3d8893f000,0x00007f3d88a40000], sp=0x00007f3d88a3e6f0, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x756687] PSMarkSweepDecorator::precompact()+0x307
V [libjvm.so+0x757cd4] PSOldGen::precompact()+0x24
V [libjvm.so+0x754d24] PSMarkSweep::invoke_no_policy(bool)+0x4e4
V [libjvm.so+0x754803] PSMarkSweep::invoke(bool)+0x83
V [libjvm.so+0x8608f7] VM_ParallelGCSystemGC::doit()+0xf7
V [libjvm.so+0x86d7fa] VM_Operation::evaluate()+0x4a
V [libjvm.so+0x86cde2] VMThread::evaluate_operation(VM_Operation*)+0x82
V [libjvm.so+0x86d053] VMThread::loop()+0x193
V [libjvm.so+0x86cb5e] VMThread::run()+0x6e
V [libjvm.so+0x710adf] java_start(Thread*)+0x13f
VM_Operation (0x00007f3d83996890): ParallelGCSystemGC, mode: safepoint, requested by thread 0x00007f3d844a7800
h2. Heap distributions
ParallelGC
PSYoungGen total 432768K, used 9064K [0x00000007aaab0000, 0x00000007c6cb0000, 0x0000000800000000)
eden space 405632K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007c36d0000)
from space 27136K, 33% used [0x00000007c36d0000,0x00000007c3faa130,0x00000007c5150000)
to space 27136K, 0% used [0x00000007c5230000,0x00000007c5230000,0x00000007c6cb0000)
PSOldGen total 921600K, used 765369K [0x0000000700000000, 0x0000000738400000, 0x00000007aaab0000)
object space 921600K, 83% used [0x0000000700000000,0x000000072eb6e7c8,0x0000000738400000)
PSPermGen total 199552K, used 158747K [0x00000006e0000000, 0x00000006ec2e0000, 0x0000000700000000)
object space 199552K, 79% used [0x00000006e0000000,0x00000006e9b06db0,0x00000006ec2e0000)
CMS
par new generation total 414784K, used 69179K [0x0000000720000000, 0x000000073c210000, 0x0000000760000000)
eden space 368704K, 6% used [0x0000000720000000, 0x000000072168ef80, 0x0000000736810000)
from space 46080K, 100% used [0x0000000739510000, 0x000000073c210000, 0x000000073c210000)
to space 46080K, 0% used [0x0000000736810000, 0x0000000736810000, 0x0000000739510000)
concurrent mark-sweep generation total 1046400K, used 625509K [0x0000000760000000, 0x000000079fde0000, 0x00000007e0000000)
concurrent-mark-sweep perm gen total 85788K, used 51439K [0x00000007e0000000, 0x00000007e53c7000, 0x0000000800000000)
h2. Machine spec
JRE version: 6.0_26-b03
Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)
uname:Linux 2.6.32.29-0.3-default #1 SMP 2011-02-25 13:36:59 +0100 x86_64
vm_info: Java HotSpot(TM) 64-Bit Server VM (20.1-b02) for linux-amd64 JRE (1.6.0_26-b03), built on May 4 2011 01:13:47 by "java_re" with gcc 3.2.2 (SuSE Linux)
libc:glibc 2.11.1 NPTL 2.11.1
rlimit: STACK 8192k, CORE 1k, NPROC infinity, NOFILE 8192, AS 59803600k
load average:0.04 0.02 0.01
MemTotal: 37009784 kB
Memory: 4k page, physical 37009784k(2027440k free), swap 37744708k(37744496k free)
CPU:total 16 (4 cores per cpu, 2 threads per core) family 6 model 26 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht

Answer 1

Hi jschell! Thanks for your pointers!
We do not use JNI. We DO use the latest JVM. Furthermore all code frames point to "libjvm.so" and "VMThread" with "VM_Operation".
I it really looks more to some 64-bit JVM problem.
I did a forum and bug search in advance to bothering, but was not sucessfull.
A repeated search for a helping topics gave me following topics which might be related:
<li> {thread:id=1540042}
<li> {thread:id=1557245}
<li> {message:id=6438249}
Is the thread you were thinking of in the list? As a result of this search I will try:
<li> disabling -XX:+HeapDumpOnOutOfMemoryError
<li> try -XX:-ReduceInitialCardMarks
<li> ensure memory overcomittment is not the problem
Futhermore I found the resource "Troubleshooting Guide for Java SE 6 with HotSpot VM" very helpful more of the background details,
though I was not able to distill a solution out of it: http://www.oracle.com/technetwork/java/javase/crashes-137240.html