Solaris 10, amd64, JVM crash at short time after startup

Question

Hi All,
I have an application which allocate 10G of memory for his Heap, run on 4 dual core amd64 CPUs, 16G RAM machine and crash in 98% of the cases at short time after startup. This is a customer machine.
I have a similar machine for tests with only 2 dual core amd64 CPU and 12G RAM and I don't experience any crashes running application with similar configuration, similar version of JDK and similar input data.
I have tried with different GC algorithms, but we obtain crashes sonner or later. I rare cases when application manage to start seems that CMS offer the greatest performance.
There is a bug in JVM? Knows anybody an workaround for this?
# An unexpected error has been detected by Java Runtime Environment:
# SIGSEGV (0xb) at pc=0xfffffd7ffebbdb33, pid=21678, tid=5
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05 mixed mode)
# Problematic frame:
# V [libjvm.so+0x5bdb33]
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
--------------- T H R E A D ---------------
Current thread (0x000000000042e000): GCTaskThread [id=5]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000000000000a8
Registers:
RAX=0x0000000000000000, RBX=0x0000000000000002, RCX=0x0000000000000000, RDX=0x0000000000000000
RSP=0xfffffd7ffad37800, RBP=0xfffffd7ffad37830, RSI=0x0000000000000000, RDI=0x00000000004e6790
, R8=0x0000000000000000, R9=0x0000000000000002, R10=0x0000000000000000, R11=0xfffffd7d7bddd058, R12=0x00000000004e6790, R13=0x0000000000000000, R14=0x0000000000000000, R15=0x0000000000429650
RIP=0xfffffd7ffebbdb33, RFLAGS=0x0000000000010246
Top of Stack: (sp=0xfffffd7ffad37800)
0xfffffd7ffad37800: fffffd7fff1051c8 000000000042a1f0
0xfffffd7ffad37810: fffffd7d72bd05a0 0000000000000000
0xfffffd7ffad37820: fffffd7d72bd05a0 0000000000000001
0xfffffd7ffad37830: fffffd7ffad37840 fffffd7ffebcfdd1
0xfffffd7ffad37840: fffffd7ffad37cc0 fffffd7ffee8953b
0xfffffd7ffad37850: fffffd7d72bd05a0 0000000000000001
0xfffffd7ffad37860: 000000000042a370 000000000042e000
0xfffffd7ffad37870: 0000000000000002 0000000000000000
0xfffffd7ffad37880: 0000000000000000 0000000000422e80
0xfffffd7ffad37890: 000000000042c120 000000000042e320
0xfffffd7ffad378a0: 000000000042e330 000000000042e708
0xfffffd7ffad378b0: 000000000042e000 0000000000424180
0xfffffd7ffad378c0: 000000000042e720 000000000042e730
0xfffffd7ffad378d0: 000000000042eb08 00000000004241c0
0xfffffd7ffad378e0: 00000000004224a0 0000000000000000
0xfffffd7ffad378f0: 0000000000000000 0000000000000000
0xfffffd7ffad37900: 0000000000000000 0000000000000400
0xfffffd7ffad37910: fffffd7d7a92c5f8 fffffd7d7a92c5f8
0xfffffd7ffad37920: fffffd7d7a92c5f8 fffffd7d7a92c5f8
0xfffffd7ffad37930: 0000000000000000 0000000000000000
0xfffffd7ffad37940: 0000000000000000 0000000000038800
0xfffffd7ffad37950: 0000000000000479 0000000000000000
0xfffffd7ffad37960: 0000000000000000 0000000000000000
0xfffffd7ffad37970: 0000000000000000 fffffd7ffad37b10
0xfffffd7ffad37980: fffffd7ffad37b58 fffffd7ffad37ba0
0xfffffd7ffad37990: fffffd7ffad37c30 0000000000430240
0xfffffd7ffad379a0: 0000000000429650 fffffd7d7a930000
0xfffffd7ffad379b0: 000000026ed7667f 0000000000000000
0xfffffd7ffad379c0: 000000000003a890 0000000000000000
0xfffffd7ffad379d0: 0000000000000000 0000000000000000
0xfffffd7ffad379e0: 0000000000000000 0000000000000000
0xfffffd7ffad379f0: 0000000000000000 0000000000000000
Instructions: (pc=0xfffffd7ffebbdb33)
0xfffffd7ffebbdb23: 24 20 4c 8b 44 24 20 4d 89 03 eb 18 4d 8b 43 08
0xfffffd7ffebbdb33: 4d 8b 80 a8 00 00 00 4c 89 44 24 28 4c 8b 44 24
Stack: [0x0000000000000000,0x0000000000000000), sp=0xfffffd7ffad37800, free space=1610527966k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x5bdb33]
V [libjvm.so+0x5cfdd1]
V [libjvm.so+0x88953b]
V [libjvm.so+0x98e8a6]
V [libjvm.so+0x98e799]
V [libjvm.so+0x87b18c]
C [libc.so.1+0xcb34b] thrslot_offset+0x31b
C [libc.so.1+0xcb580] thrslot_offset+0x550
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000000000a6b400 JavaThread "Timer-0" [_thread_blocked, id=19]
0x0000000000743400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=17]
0x0000000000742000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=16]
0x0000000000740c00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=15]
0x000000000072c400 JavaThread "JDWP Event Helper Thread" daemon [_thread_blocked, id=14]
0x0000000000729800 JavaThread "JDWP Transport Listener: dt_socket" daemon [_thread_in_native, id=13]
0x000000000071c400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=12]
0x000000000071b800 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=11]
0x00000000006f3000 JavaThread "Finalizer" daemon [_thread_blocked, id=10]
0x00000000006f2800 JavaThread "Reference Handler" daemon [_thread_blocked, id=9]
0x0000000000428400 JavaThread "main" [_thread_blocked, id=2]
Other Threads:
0x00000000006ee400 VMThread [id=8]
0x0000000000744c00 WatcherThread [id=18]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00000000004273a0/0x0000000000427400] Threads_lock - owner thread: 0x00000000006ee400
[0x0000000000423500/0x0000000000427b10] Heap_lock - owner thread: 0x0000000000428400
Heap
par new generation total 74560K, used 63936K [0xfffffd7d75600000, 0xfffffd7d7a930000, 0xfffffd7d7a930000)
eden space 63936K, 100% used [0xfffffd7d75600000, 0xfffffd7d79470000, 0xfffffd7d79470000)
from space 10624K, 0% used [0xfffffd7d79470000, 0xfffffd7d79470000, 0xfffffd7d79ed0000)
to space 10624K, 100% used [0xfffffd7d79ed0000, 0xfffffd7d7a930000, 0xfffffd7d7a930000)
concurrent mark-sweep generation total 10400576K, used 29146K [0xfffffd7d7a930000, 0xfffffd7ff5600000, 0xfffffd7ff5600000)
concurrent-mark-sweep perm gen total 21248K, used 6589K [0xfffffd7ff5600000, 0xfffffd7ff6ac0000, 0xfffffd7ffaa00000)
Dynamic libraries:
0x0000000000400000      /export/home/tg/jre1.6.0_03/bin/amd64/java
0xfffffd7fff380000      /lib/64/libthread.so.1
0xfffffd7fff360000      /export/home/tg/jre1.6.0_03/bin/amd64/../../lib/amd64/jli/libjli.so
0xfffffd7fff390000      /lib/64/libdl.so.1
0xfffffd7fff240000      /lib/64/libc.so.1
0xfffffd7ffe600000      /export/home/tg/jre1.6.0_03/lib/amd64/server/libjvm.so
0xfffffd7fff1e0000      /lib/64/libsocket.so.1
0xfffffd7fff340000      /usr/lib/64/libsched.so.1
0xfffffd7fff1a0000      /lib/64/libm.so.1
0xfffffd7fff170000      /usr/lib/64/libCrun.so.1
0xfffffd7fff140000      /lib/64/libdoor.so.1
0xfffffd7ffe530000      /lib/64/libnsl.so.1
0xfffffd7ffe4b0000      /lib/64/libm.so.2
0xfffffd7ffe470000      /lib/64/libscf.so.1
0xfffffd7ffe440000      /lib/64/libuutil.so.1
0xfffffd7ffe420000      /lib/64/libmd5.so.1
0xfffffd7ffe3f0000      /lib/64/libmp.so.2
0xfffffd7ffe270000      /export/home/tg/jre1.6.0_03/lib/amd64/native_threads/libhpi.so
0xfffffd7ffe210000      /export/home/tg/jre1.6.0_03/lib/amd64/libjdwp.so
0xfffffd7ffe1e0000      /export/home/tg/jre1.6.0_03/lib/amd64/libnpt.so
0xfffffd7ffe190000      /usr/lib/iconv/amd64/UTF-8%646.so
0xfffffd7ffe160000      /usr/lib/iconv/amd64/646%UTF-8.so
0xfffffd7ffe110000      /export/home/tg/jre1.6.0_03/lib/amd64/libverify.so
0xfffffd7ffe0c0000      /export/home/tg/jre1.6.0_03/lib/amd64/libjava.so
0xfffffd7ffe080000      /export/home/tg/jre1.6.0_03/lib/amd64/libzip.so
0xfffffd7ffe010000      /export/home/tg/jre1.6.0_03/lib/amd64/libdt_socket.so
0xfffffd7d75460000      /lib/64/nss_files.so.1
VM Arguments:
jvm_args: -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -Dapp.dir=/opt/tg/cos-deploy-0.35/bin/.. -Dlog.dir=log -Dapp.name=process -Detc.dir=/opt/tg/cos-deploy-0.35/bin/../etc -Dconfig.file=/opt/tg/cos-deploy-0.35/bin/../etc/process.properties -Dlog4j.configuration=etc/log4j.xml -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -Xms10G -Xmx10G
java_command: tg.cos.server.Main
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/export/home/tg/jre1.6.0_03
PATH=/opt/tg/fcs-foundation/perl/bin:/opt/tg/fcs-foundation/perl/bin:/sbin:/usr/sbin:/usr/sfw/bin:/opt/sfw/bin:/usr/X/bin:/usr/ccs/bin:/usr/local/bin:/usr/bin:/usr/sbin:/export/home/oracle_install/product/client/10.2/bin:/opt/tg/POOL/bin:/opt/tg/MailSender/bin:/export/home/tg/bin:/usr/sbin:/export/home/oracle_install/product/client/10.2/bin:/opt/tg/POOL/bin:/opt/tg/MailSender/bin
LD_LIBRARY_PATH=/export/home/tg/jre1.6.0_03/lib/amd64/server:/export/home/tg/jre1.6.0_03/lib/amd64:/export/home/tg/jre1.6.0_03/../lib/amd64
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x9883b0], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0x9883b0], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x315b10], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x315b10], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x315b10], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGHUP: [libjvm.so+0x87ca80], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0x87ca80], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x87ca80], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x87ca80], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x87ca80], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x87ca80], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0x87f4d0], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x315b10], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
--------------- S Y S T E M ---------------
OS: Solaris 10 11/06 s10x_u3wos_10 X86
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 14 November 2006
uname:SunOS 5.10 Generic_118855-33 i86pc (T2 libthread)
rlimit: STACK 10240k, CORE infinity, NOFILE 65536, AS infinity
load average:3.72 3.68 3.23
CPU:total 8 (2 cores per cpu, 1 threads per core) family 15 model 65 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext
Memory: 4k page, physical 16775096k(1672236k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05) for solaris-amd64, built on Sep 24 2007 22:29:24 by "" with unknown Workshop:0x580
I created a small test which somehow emulate the real application and I obtain the same results: on test machine run without any issues on live machine it crashes all the time.
What can be the cause?
import java.util.*;
public class MemoryTest {
     private List<Page> pages = new LinkedList<Page>();
     public void initTest() {
          int counter = 1;
          while( canAddOneMorePage() ) {
               Page page = new Page(counter, (byte)counter++);
               pages.add(page);
     private boolean canAddOneMorePage() {
          boolean result = true;
          long freeMem = Runtime.getRuntime().freeMemory() / (1024*1024) ;
          if (freeMem < 512) {
               result = false;
          return result;
     private boolean validatePages() {
          boolean arePagesValid = true;
          for(Page p : pages) {
               boolean isPageValid = p.isValid();
               if ( !isPageValid ) {
                    System.out.println("Page "+p.id+ " contains corrupted data");
                    arePagesValid = false;
          return arePagesValid;
     private void updatePages(byte value) {
          for(Page p : pages) {
               p.updateValues(value);
     public void runTest() {
          int TESTS_NUM = 100;
          System.out.println("There are "+pages.size()+" pages.");
          System.out.println("Currently the pages "+ (validatePages() ? "contain no errors" : "contain errors"));
          for(int i=1; i<TESTS_NUM; i++) {
               System.out.println("Test "+i+" of "+TESTS_NUM);
               updatePages((byte)i);
               validatePages();
     public static void main(String[] args) {
          MemoryTest main = new MemoryTest();
          main.initTest();
          main.runTest();
class Page {
     public static final int LENGTH = 1024*1024;
     List <byte[]> list;
     public byte value = 0;
     public int id;
     public Page(int id, byte value) {
          this.list = new LinkedList<byte[]>();
          this.value = value;
          this.id = id;
          for(int i=0; i<LENGTH; i++) {
               byte[] data = new byte [512];
               for( int j=0; j<512; j++) {
                    data[j] = value;
               list.add(data);
     public void updateValues(byte value) {
          this.value = value;
          this.list = new LinkedList<byte[]>();
          System.out.println("Updating page "+id);
          for(int i=0; i<LENGTH; i++) {
               byte[] data = new byte [512];
               for( int j=0; j<512; j++) {
                    data[j] = value;
               list.add(data);
     public boolean isValid() {
          boolean result = true;
          for( byte[] data: list) {
               for( byte b : data) {
                    if (value != b) {
                         result = false;
                         break;
               if ( !result ) {
                    break;
          return result;
}Edited by: blue67 on Dec 10, 2007 12:57 AM

Answer 1

I've experienced the same crash problem for my application in a Sun Fire x4600 M2..
My application which allocates 50G of memory for his Heap, runs on 8 quad core amd64 CPUs, 64G RAM machine and crash in 98% of the cases at short time after startup.
If I run my application on a different machine, for example a Sun Fire 280R with 2 CPUs UltraSPARC-III+ 1.2Ghz 8MB, there are no crashes.
I think the problem is related to the AMD 64 bits CPU architecture.
I filed a bug to the Sun Bug DB with ID 1415527, but it hasn't been published yet.
I cannot understand why, since it seems to be a problem common also to other users.
Here's part of my last hs_err_pidXXX.log file:
# An unexpected error has been detected by Java Runtime Environment:
# SIGSEGV (0xb) at pc=0xfffffd7ffebbd0bd, pid=25799, tid=50
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode solaris-amd64)
# Problematic frame:
# V [libjvm.so+0x5bd0bd]
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
T H R E A D
Current thread (0x0000000000b21c00): GCTaskThread [stack: 0x0000000000000000,0x0000000000000000] [id=50]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xfffffd7ffebbd0bd
Registers:
RAX=0x0000000000000000, RBX=0x0000000000000008, RCX=0x0000000000000002, RDX=0xfffffd73b56f1ce8
RSP=0xfffffd73b56f1cb0, RBP=0xfffffd73b56f1e00, RSI=0xfffffd7c55dd2f08, RDI=0xfffffd7c55dd2bb9
, R8=0xb000000000000000, R9=0x0000000000c38ff0, R10=0x00000000042d0e97, R11=0xfffffd7c55dd2d88, R12=0x0000000000b38f00, R13=0x000000000045d070, R14=0x000000000045d070, R15=0x0000000000003c4f
RIP=0xfffffd7ffebbd0bd, RFLAGS=0x0000000000010246
Top of Stack: (sp=0xfffffd73b56f1cb0)
0xfffffd73b56f1cb0: fffffd7fff0f15a8 0000000000000000
0xfffffd73b56f1cc0: fffffd73b56f1e48 fffffd73b45e0790
0xfffffd73b56f1cd0: 0000000000000008 fffffd7ffebbc239
0xfffffd73b56f1ce0: fffffd7fff0f15a8 fffffd7fff0ec260
0xfffffd73b56f1cf0: 0000000000000000 0000000000b0c5b0
0xfffffd73b56f1d00: fffffd73fbc00000 000000017fa80000
0xfffffd73b56f1d10: 0000000000b0c710 0000000000b0c850
0xfffffd73b56f1d20: 0000000000000000 000000000045d070
0xfffffd73b56f1d30: 0000000000000000 0000000000b0c850
0xfffffd73b56f1d40: fffffd73b45e07f8 0000000000b0c8d8
0xfffffd73b56f1d50: 0000000900b0c5b0 fffffd792b971d60
0xfffffd73b56f1d60: fffffd7fff103d04 0000441ef1e00000
0xfffffd73b56f1d70: fffffd7c55dd2f08 fffffd7f00003c4d
0xfffffd73b56f1d80: fffffd73b45e0790 441f3c4d00000000
0xfffffd73b56f1d90: 441f3c4d441f3c4d 441f3c4d441f3c4d
0xfffffd73b56f1da0: 441e3c4d441f3c4d 441f3c4d441e3c4d
0xfffffd73b56f1db0: 43c03c1e441e3c4d 441e3c4d441e3c4d
0xfffffd73b56f1dc0: 5e3634bf5e3634bf 10830ae010830ae0
0xfffffd73b56f1dd0: 036b3673036b3674 036b3673036b3673
0xfffffd73b56f1de0: 5e3634c05e3634c1 5e3634c05e3634c0
0xfffffd73b56f1df0: fffffd73b45e0790 00000054b45e0790
0xfffffd73b56f1e00: fffffd73b56f1eb0 fffffd7ffebbbd47
0xfffffd73b56f1e10: fffffd73b56f1ef8 0000000000b0ce20
0xfffffd73b56f1e20: 0000000000b21c00 fffffd7fff04a509
0xfffffd73b56f1e30: 0000000000000008 00000000ffffffff
0xfffffd73b56f1e40: fffffd73b56f1e70 0000000000000000
0xfffffd73b56f1e50: 000002185a539ab2 fffffd7fff103d01
0xfffffd73b56f1e60: 000000000042c560 0000000000b21e90
0xfffffd73b56f1e70: 0000000000b21ea0 0000000000b22278
0xfffffd73b56f1e80: 0000000000b21c00 000000000045a780
0xfffffd73b56f1e90: 0000000000b22290 0000000000b222a0
0xfffffd73b56f1ea0: 0000000000b22678 000000000045a7c0
Thanks