JDK1.5.0_12 crash when I try to startup a tomcat app server with it.

Question

Hi,
I am support a Blackboard application running on a PrimePower box.
Recently I installed JDK1.5.0_12 on the box. When I re-configured Blackboard to use the new JDK and tried to restart the server, the tomcat portion will keep getting the JVM Hotspot crash error as illustrated below.
The startup is invoked from a shell script that starts the web server as well as the tomcat (app) server. As long as the root partition has enough diskspace, the script will try to "start" the tomcat server.
The JVM will keep throwing core dumps and issuing messages like the following, with each core dump taking about 100 to 150MB in space from the root partition.
Only when there isn't enough space in the root partition for another core dump, did I notice that it stopped.
My sys admin has since cleaned up the dump files and I have reverted back to the earlier minor release of JDK1.5.0.
I have used ldd -r libjvm.so and the command could return me the list of libraries that the libjvm.so depends on (not a problem with the loading of the dynamic libraries and such).
I have submitted the JVM crash report. But no one has gotten back to me.
Has anyone encountered the same problems / have any suggestions on what could be cause and how to resolved this?
Thanks.
Regards,
Hon Peng.
=============================
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGSEGV (0xb) at pc=0xfea84b00, pid=13715, tid=1
# Java VM: Java HotSpot(TM) Server VM (1.5.0_12-b04 mixed mode)
# Problematic frame:
# V [libjvm.so+0x284b00]
--------------- T H R E A D ---------------
Current thread (0x00036e20): JavaThread "main" [_thread_in_vm, id=1]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x57726974
Registers:
O0=0x00000000 O1=0x00000000 O2=0x0000015c O3=0x00000000
O4=0x0000015c O5=0x0000015c O6=0xffbfe6f8 O7=0x0000015c
G1=0x00000000 G2=0x0000015c G3=0x0000015c G4=0x00000000
G5=0x00000000 G6=0x00000000 G7=0xff350000 Y=0x00000000
PC=0xfea84b00 nPC=0xfea84b04
Top of Stack: (sp=0xffbfe6f8)
0xffbfe6f8: 000373fc 57726974 000adfb0 ff028d48
0xffbfe708: 00032778 fefdc000 001b3990 b4805210
0xffbfe718: ffbfe84c 00000000 00000000 ff010c40
0xffbfe728: 00009c00 80000000 ffbfe7e8 fea84674
0xffbfe738: 00000000 e3908660 ffbfe7c0 f900587c
0xffbfe748: b4a64018 007f9a80 ffbfe7b4 b4a5e2c5
0xffbfe758: ffbfe7b8 ffbfe8e0 f9015b04 00000000
0xffbfe768: 00000000 f900576c b4a64018 007f9a80
Instructions: (pc=0xfea84b00)
0xfea84af0: 80 a5 a0 00 04 40 00 15 b4 10 20 00 e2 04 a0 08
0xfea84b00: d4 04 60 00 d2 04 60 04 8b 2a b0 20 88 12 40 05
Stack: [0xffb80000,0xffc00000), sp=0xffbfe6f8, free space=505k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x284b00]
V [libjvm.so+0x28467c]
j java.lang.Object.notify()V+0
j blackboard.util.Queue.put(Ljava/lang/Object;)V+19
j blackboard.platform.log.impl.QueuedLogger.doLogMessage(Ljava/lang/String;)V+77
j blackboard.platform.log.impl.BaseLogger.log(Ljava/lang/String;Lblackboard/platform/log/LogService$Verbosity;)V+10
j blackboard.platform.log.impl.BaseLogger.logInfo(Ljava/lang/String;)V+5
j blackboard.platform.log.impl.BbLogServiceImpl.logInfo(Ljava/lang/String;)V+5
j blackboard.platform.BbServiceManager.init(Ljava/io/File;Ljava/util/Properties;Z)V+249
j blackboard.platform.BbServiceManager.init(Ljava/lang/String;Ljava/util/Properties;Z)V+44
j blackboard.platform.ApplicationLauncher.main([Ljava/lang/String;)V+479
v ~StubRoutines::call_stub
V [libjvm.so+0x19b4dc]
V [libjvm.so+0x2dd2cc]
C [java+0x222c] main+0x1164
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.lang.Object.notify()V+0
j blackboard.util.Queue.put(Ljava/lang/Object;)V+19
j blackboard.platform.log.impl.QueuedLogger.doLogMessage(Ljava/lang/String;)V+77
j blackboard.platform.log.impl.BaseLogger.log(Ljava/lang/String;Lblackboard/platform/log/LogService$Verbosity;)V+10
j blackboard.platform.log.impl.BaseLogger.logInfo(Ljava/lang/String;)V+5
j blackboard.platform.log.impl.BbLogServiceImpl.logInfo(Ljava/lang/String;)V+5
j blackboard.platform.BbServiceManager.init(Ljava/io/File;Ljava/util/Properties;Z)V+249
j blackboard.platform.BbServiceManager.init(Ljava/lang/String;Ljava/util/Properties;Z)V+44
j blackboard.platform.ApplicationLauncher.main([Ljava/lang/String;)V+479
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0037fbe8 JavaThread "FileLogSinkImpl Daemon" [_thread_blocked, id=15]
0x0015ddc0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=13]
0x0015c4e8 JavaThread "CompilerThread1" daemon [_thread_blocked, id=12]
0x0015b670 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11]
0x0015a7f8 JavaThread "AdapterThread" daemon [_thread_blocked, id=10]
0x001599e8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=9]
0x0014d768 JavaThread "Finalizer" daemon [_thread_blocked, id=8]
0x0014d228 JavaThread "Reference Handler" daemon [_thread_blocked, id=7]
=>0x00036e20 JavaThread "main" [_thread_in_vm, id=1]
Other Threads:
0x0014ab40 VMThread [id=6]
0x000aff90 WatcherThread [id=14]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00032778/0x00035dd0] SignatureHandlerLibrary_lock - owner thread: 0x00036e20
Heap
PSYoungGen total 39424K, used 5458K [0xe3400000, 0xe6000000, 0xf8c00000)
eden space 33792K, 16% used [0xe3400000,0xe3954b40,0xe5500000)
from space 5632K, 0% used [0xe5a80000,0xe5a80000,0xe6000000)
to space 5632K, 0% used [0xe5500000,0xe5500000,0xe5a80000)
PSOldGen total 90112K, used 0K [0xb8800000, 0xbe000000, 0xe3400000)
object space 90112K, 0% used [0xb8800000,0xb8800000,0xbe000000)
PSPermGen total 16384K, used 4896K [0xb4800000, 0xb5800000, 0xb8800000)
object space 16384K, 29% used [0xb4800000,0xb4cc83e8,0xb5800000)
Dynamic libraries:
0x00010000      /usr/local/jdk1.5.0_12/bin/java
0xff370000      /usr/lib/libthread.so.1
0xff3fa000      /usr/lib/libdl.so.1
0xff280000      /usr/lib/libc.so.1
0xff3a0000      /usr/platform/FJSV,GPUS/lib/libc_psr.so.1
0xfe800000      /usr/local/jdk1.5.0_12/jre/lib/sparc/server/libjvm.so
0xff240000      /usr/lib/libsocket.so.1
0xff220000      /usr/lib/libsched.so.1
0xff1f0000      /usr/lib/libCrun.so.1
0xff1a0000      /usr/lib/libm.so.1
0xff080000      /usr/lib/libnsl.so.1
0xff170000      /usr/lib/libmp.so.2
0xfe7e0000      /usr/local/jdk1.5.0_12/jre/lib/sparc/native_threads/libhpi.so
0xfe7b0000      /usr/local/jdk1.5.0_12/jre/lib/sparc/libverify.so
0xfe770000      /usr/local/jdk1.5.0_12/jre/lib/sparc/libjava.so
0xfe740000      /usr/local/jdk1.5.0_12/jre/lib/sparc/libzip.so
0xf8cd0000      /usr/local/jdk1.5.0_12/jre/lib/sparc/libj2pkcs11.so
0xb3580000      /usr/lib/libpkcs11.so
0xf8c50000      /usr/lib/libasn1.so
0xb3400000      /usr/lib/libcmicmn.so
0xb47a0000      /opt/FJSVsmee/lib/libinfoca.so
0xf8f90000      /opt/FJSVsmee/lib/libinfocacom.so
0xf8c30000      /opt/FJSVsmee/lib/libsctl15d.so
VM Arguments:
java_command: blackboard.platform.ApplicationLauncher -S ../config/update-tool-vm/install-tool-vm-servicedef.properties -A blackboard.apis.updatetools.CommandLineToolInvokerApp blackboard.apis.updatetools.tooldefs.install.ServiceControllerTool services.stop
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/local/jdk1.5.0_12
CLASSPATH=
PATH=/usr/local/blackboard/apps/openssl/bin:/usr/local/blackboard/apps/bbcms/bin:/usr/local/jdk1.5.0_12/bin:/usr/bin:/usr/ucb:/etc:/usr/local/bin:.
LD_LIBRARY_PATH=/usr/local/jdk1.5.0_12/jre/lib/sparc/server:/usr/local/jdk1.5.0_12/jre/lib/sparc:/usr/local/jdk1.5.0_12/jre/../lib/sparc
SHELL=/bin/ksh
HOSTTYPE=sparc
OSTYPE=solaris2.9
MACHTYPE=sparc-sun-solaris2.9
Signal Handlers:
SIGSEGV: [libjvm.so+0x70cb50], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGBUS: [libjvm.so+0x70cb50], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGFPE: [libjvm.so+0x2743ac], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x2743ac], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x2743ac], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGUSR1: [libjvm.so+0x67f5f0], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIGUSR2: [libjvm.so+0x2743ac], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
SIGHUP: [libjvm.so+0x67e22c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGINT: [libjvm.so+0x67e22c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGQUIT: [libjvm.so+0x67e22c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
SIGTERM: [libjvm.so+0x67e22c], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
--------------- S Y S T E M ---------------
OS: Solaris 9 4/04 s9s_u6wos_08a SPARC
Copyright 2004 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 22 March 2004
uname:SunOS 5.9 Generic_118558-33 sun4us (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 5120, AS infinity
load average:0.05 0.12 0.16
CPU:total 4 has_v8, has_v9, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 8388608k(6679936k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_12-b04) for solaris-sparc, built on May 2 2007 02:33:21 by unknown with unknown Workshop:0x550

Answer 1

Hi,
Thanks for the reply. I have tried this on another machine running the same version of Blackboard. It works fine.
Only difference is that the machine that has the JVM crash is running on Fujitsu PrimePower and the OS is Solaris 9 (but twigged for PrimePower).
The one where it runs and starts up successfully is running on SunFire (OS is Solaris 10).
I am wondering if there is a problem with the JVM loading required libraries because of the OS being different versions.
Any leads on this is welcomed. Thank you.
Regards,
Hon Peng
===================
On Fujitsu Primepower
SunOS 5.9 Generic_118558-33 sun4us sparc FJSV,GPUS
the Dependencies for libjvm.so are as follows:
===================
bash-2.05$ ldd -r libjvm.so
libsocket.so.1 => /usr/lib/libsocket.so.1
libsched.so.1 => /usr/lib/libsched.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libCrun.so.1 => /usr/lib/libCrun.so.1
libm.so.1 => /usr/lib/libm.so.1
libthread.so.1 => /usr/lib/libthread.so.1
libc.so.1 => /usr/lib/libc.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/FJSV,GPUS/lib/libc_psr.so.1
=====================================
On SunFire, Sun machine
SunOS 5.10 Generic_118833-36 sun4u sparc SUNW,Sun-Fire-280R
the dependencies are different for the libjvm.so:
=====================================
bash-2.05$ ldd -r libjvm.so
libsocket.so.1 => /lib/libsocket.so.1
libsched.so.1 => /usr/lib/libsched.so.1
libdl.so.1 => /lib/libdl.so.1
libCrun.so.1 => /usr/lib/libCrun.so.1
libm.so.1 => /lib/libm.so.1
libthread.so.1 => /lib/libthread.so.1
libc.so.1 => /lib/libc.so.1
libnsl.so.1 => /lib/libnsl.so.1
libmp.so.2 => /lib/libmp.so.2
libmd5.so.1 => /lib/libmd5.so.1
libscf.so.1 => /lib/libscf.so.1
libdoor.so.1 => /lib/libdoor.so.1
libuutil.so.1 => /lib/libuutil.so.1
/platform/SUNW,Sun-Fire-280R/lib/libc_psr.so.1
libm.so.2 => /lib/libm.so.2
/platform/SUNW,Sun-Fire-280R/lib/libmd5_psr.so.1