Java 1.5.0_17 core dumps
Hi All,
We have a java application which works in protocol level. Our java application uses JNI to interact with others modules which are written in C++.
We are doing stress testing on the application. after that, we are stopping the java application and starting it. when this action happens java core dumps.
when i see the core dump using gdb it show the following message.
Core was generated by `java -server -Xmn64M -Xms170M -Xmx170M -cp :../lib/servic
es.jar:../lib/commons-'.
Program terminated with signal 3, Quit.
#0 0x400ea621 in nanosleep () from /lib/libc.so.6
(gdb) where
#0 0x400ea621 in nanosleep () from /lib/libc.so.6
#1 0x400208ee in __pthread_timedsuspend_new (self=0xbf1ffc00,
abstime=0xbf1ff914) at pthread.c:1125
#2 0x4001d183 in pthread_cond_timedwait_relative (cond=0x805ce38,
mutex=0x805ce20, abstime=0xbf1ff914) at restart.h:45
#3 0x405a666c in os::Linux::safe_cond_timedwait ()
from /sw/linuxapps/usr/java/jre1.5.0_17/lib/i386/server/libjvm.so
#4 0x4058db51 in Monitor::wait ()
from /sw/linuxapps/usr/java/jre1.5.0_17/lib/i386/server/libjvm.so
#5 0x40689c6b in VMThread::loop ()
from /sw/linuxapps/usr/java/jre1.5.0_17/lib/i386/server/libjvm.so
#6 0x406898d0 in VMThread::run ()
from /sw/linuxapps/usr/java/jre1.5.0_17/lib/i386/server/libjvm.so
#7 0x405a7408 in _start ()
from /sw/linuxapps/usr/java/jre1.5.0_17/lib/i386/server/libjvm.so
#8 0x4001dfb7 in pthread_start_thread (arg=0xbf1ffc00) at manager.c:284
#9 0x4011d84a in clone () from /lib/libc.so.6
Please provide your thoughts or suggestions. is this a known issue in JVM? or this is not related to JVM issue?
Thanks
iramar.
"stopping" and "starting" are two actions here. Which action triggered core dumps? How did you stop the application?
From your gdb message, the core dump was generated when your java application received a "kill -QUIT" signal.
In my Solaris java 1.5.0_10 environment, if you had a "-Xs" in your java option, the "kill -QUIT" signal will generate a core dump
Similar Messages
-
Dump complete java VM state as core dump (not via OS) possible?
Hi everyone,
I've a problem debugging an application.
Background: Sometimes my application comes to a very unlikely state, which at the moment results in an error message. The stack trace alone has no great value, since this state is cause by the interaction of more than one thread. The state is resolved throwing an exception, the program continues normally.
Goal: If I reach this state, I want to suspend the application, dump the complete state of all java threads, objects, ... (complete java memory core dump) to analyse it later.
Question: Is there a possibility to generate such dumps?
Just thinking:
One posibility would be, to send a signal from the vm to some external program (e.g. udp packet), this program attaches a standard OS-level debugger (e.g. gdb under linux), dumps the core, resumes app, detach. But I guess reconstruction of internal VM state from complete dump is rather hard, is it?
I've looked at jdb, to see if it has some java-core-dump functions, but it seems not to be so. Are there alternative implementations, or helper scripts that make jdb loop over all java object addresses and dump them?
Does someone known about the jdb remote debugging interface? Is the protocol public, can I implement such a feature on myself?Hi everyone,
I was looking for a way to dump the complete state of a JVM (all runing threads, objects, ...) and afterwards to analize the dumped data. I have found a thread(http://forum.java.sun.com/thread.jspa?threadID=735052) here in the forums with some explainations how this could be done using the Java 2 sdk 1.5.0 - troubleshooting tools (http://java.sun.com/j2se/1.5.0/docs/tooldocs/experimentaltools.html).
I've made a small application, which starts a thread and just prints out continously some messages to the console.
After that I attached with gdb to the running process as described, and generated a core file.
Unfortunatelly on calling jdsadebugd with the core dump it throws the following exception:
jsadebugd /..../jdk1.5.0_07/bin/java core.18294 DebServ
Attaching to core core.18294 from executable /.../jdk1.5.0_07/bin/java and starting RMI services, please wait...
Error attaching to core file or starting server: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the core file
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:258)
at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:624)
at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:610)
at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:323)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:298)
at sun.jvm.hotspot.HotSpotAgent.startServer(HotSpotAgent.java:234)
at sun.jvm.hotspot.DebugServer.run(DebugServer.java:94)
at sun.jvm.hotspot.DebugServer.main(DebugServer.java:29)
at sun.jvm.hotspot.jdi.SADebugServer.main(SADebugServer.java:46)
Even if I try to print the shared objects mapping with jmap it fails with the following message:
jmap /.../jdk1.5.0_07/bin/java core.18294
Attaching to core core.18294 from executable /.../jdk1.5.0_07/bin/java, please wait...
Error attaching to core file: Can't attach to the core file
Does anyone have an idea what goes wrong here? -
Hi,
One of my Java Processes on Solaris Core Dumped due to SIGNAL 11 (SIGENV) and SIGNAL 10 (SIGBUS). SIgnal 11 is caused due to a Segmentation fault and Signal 10 is caused due to a 'Bus error' (hardware error).
I would like to know, which is the best debugger to debug this core dump (java) and know the root cause. I tried 'gdb' but I am not sure if 'gdb' works for java. I would like to get a neat stack trace of the error.
Any help is appreciated.
Thanks.Check out this page: http://www-128.ibm.com/developerworks/java/library/j-jnidebug/index.html
-
Application cause java core dump, how can i do?
Hi
In the case, we're using JDK 1.4 and Hibernate on Aix 64 bit, but seems we've got lots of java core even occupied full hard space and cause application crash
system info :
JavaVersion is J2RE 1.4.2 IBM AIX 5L for PowerPC (64 bit JVM)
build caix64142ifx-20051115 (SR3 + 94164 + 97403 + 97482)
(note the 64 bit JVM).
-Xmx=1024 and -xms=512, free heap is 330,415,568 bytes, and allocated
is 1073,740,288 bytes.
some core dump :
WebContainer : 4
at java.io.ObjectInputStream.readString()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0() at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readObject()
at java.util.HashMap.readObject()
at sun.reflect.GeneratedMethodAccessor399.invoke()
at sun.reflect.DelegatingMethodAccessorImpl.invoke()
at java.lang.reflect.Method.invoke()
at java.io.ObjectStreamClass.invokeReadObject()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readObject()
at com.ibm.ws.naming.util.Serialization$1.run()
at com.ibm.ws.security.util.AccessController.doPrivileged()
at com.ibm.ws.naming.util.Serialization.deserializeObject()
at com.ibm.ws.naming.util.Helpers
.processSerializedObjectForLookupExt()
at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookup()
at com.ibm.ws.naming.jndicos.CNContextImpl.processResolveResults()
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup()
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup()
at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt()
at com.ibm.ws.naming.jndicos.CNContextImpl.lookup()
at com.ibm.ws.naming.util.WsnInitCtx.lookup()
at javax.naming.InitialContext.lookup()
at com.ibm.wsspi.RegistryLoader.getPluginRegistry()
at com.ibm.wsspi.IPluginRegistryFactory.getPluginRegistry()
at ibmjsp.com2E_ibm_2E_ws_2E_console_2E_appmanagement
._collectionTableLayout._jspService()
at com.ibm.ws.jsp.runtime.HttpJspBase.service()
at javax.servlet.http.HttpServlet.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest()
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper
.handleRequest()
at com.ibm.ws.jsp.webcontainerext.JSPExtensionServletWrapper
.handleRequest()
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.include()
at org.apache.jasper.runtime.JspRuntimeLibrary.include()
at org.apache.jasper.runtime.PageContextImpl.include()
at org.apache.struts.tiles.TilesUtilImpl.doInclude()
at org.apache.struts.tiles.TilesUtil.doInclude()
at org.apache.struts.taglib.tiles.InsertTag.doInclude()
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag()
at org.apache.struts.taglib.tiles.InsertTag.doEndTag()
at ibmjsp.secure.layouts.vboxLayout._jspService()
at com.ibm.ws.jsp.runtime.HttpJspBase.service()
at javax.servlet.http.HttpServlet.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter()
... more stack lines not shown ===================================
"WebContainer: 2" shows much
the same stack trace, but extends to include
@com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState()
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter()
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest()
Through IBM, we got those answer :
Looking at the javacore files, we're seeing OOM errors. However,
looking at a heapdump, we're not seeing total heap space exhaustion -
around half of the heap is occupied, and occupancy by the vartious
objects seems, to me at least, to be normal.
The threads
seem to be using the compound class loader to load the customer's
"Hibernate" code - and in the largest of the threads in the heapdump -
(it consumes 258,720,168 bytes), the class org/hibernate/impl/Session
Impl and its children - consume 129,540,864 bytes.
The other 128,785,680 is consumed by org/hibernate/proxy/CGLIBLazyInitia
lizer. There are some other classes that make up the small difference
if you added these numbers.
The class com/softleader/hibernate/util/HibernateUtil and its children
consume 74,932,848 bytes.
I could go on through the heapdump, but there are very large classes
in here, and it may be more advantageous to get a verbosegc turned on
to show the patterns of JVM Heap requirements.
I suppose that the customer is developing this app for a 32bit
architecture market, since they are keeping their JVM Heap within the
32 bit limits. If this is the case, then Xmx=1280 and Xms=384 seems a
more reasonable JVM setting. If they are not, and they want to try
Xmx above 1280, then we have not yet had much experience with 64 bit
but I'd expect that the larger heapsize would benefit the application.
Either way, we need verbosegc turned on to monitor the heap requests.
That mean, i have to turn up JVM Heap size? or what can i do for it?
Thanks a lot.I'd hook up a profiler to your app and monitor it to see what's happening and what objects aren't playing nice.
-
JNI - core dump - internal error on linux after calling Java method
I'm getting a core dump after calling athe main statric method using JNNI.
On linux.
I can get the class id correcttly but when I attempt to call the method it craches with an internal error , anyone know why it would crash instead of just not work.
if(cls)
main_methodID = env->GetStaticMethodID(cls, "main", "([Ljava/lang/String;)V");
printf("Class Found Successfully\n");
else
printf ( "cls not found\n");
return 0;
if(main_methodID)
jstring first_str = env->NewStringUTF("The First String");//create string
jobjectArray args = (jobjectArray)env->NewObjectArray(1,env->FindClass("java/lang/String"), first_str);//new array with 2 elements
env->SetObjectArrayElement(args, 2, first_str);//insert the second string into index 1 of the array
jstring second_str = env->NewStringUTF("The Second String");//create string
env->SetObjectArrayElement(args, 1, second_str);//insert the second string into index 1 of the array
env->CallStaticVoidMethod(cls, main_methodID, args);//pass the array to the Java main method
The JAVA method is
public static void main(String[] args) {
System.out.println("Main method in Framework");
Framework framework = new Framework();
I see yere points but what I see on linux is it makes it to the constructor of the Java object and somewhere afterwards it bails out, the Java app does work alone. It appears to have problems initializing the JFrame, could there be a problem with the Java inheritance when a JVM is invoked throurgh native invocation??
Here's some of the Java stuff
public class Framework extends WindowAdapter {
public int numWindows = 0;
private Point lastLocation = null;
private int maxX = 500;
private int maxY = 500;
public Framework() {
System.out.println("JAVA Framework cnst");//GETTING HERE
makeNewWindow();
public void makeNewWindow() {
System.out.println("JAVA makeWindow"); //GETTING HERE
JFrame frame = new MyFrame(this); //NOT GETTING HERE!!!!!!!!!
numWindows++;
System.out.println("Number of windows: " + numWindows);
System.out.println("Frame location: " + lastLocation);
frame.setVisible(true);
System.out.println("Post Java set frame visible");
public static void main(String[] args) {
System.out.println("Main method in Framework");
Framework framework = new Framework();//GETTING HERE
class MyFrame extends JFrame {
protected Dimension defaultSize = new Dimension(200, 200);
protected Framework framework = null;
public MyFrame(Framework controller) {
super("New Frame");
System.out.println("MyFrame cnst ");//NOT GETTING HERE!!!!!!
framework = controller;
setDefaultCloseOperation(DISPOSE_ON_CLOSE);
addWindowListener(framework);
JMenu menu = new JMenu("Window");
menu.setMnemonic(KeyEvent.VK_W);
setSize(defaultSize);
} -
Java crashes on a solaris 9 randomly with a core dump
Hello
Currently we are using JBoss Application server on solaris 9. Over a period of time. Java crashes suddenly. The condition under which this can be reproduced is currently not reproduceable. But over a period of time, this crash just happens.
The following is the stack trace of the core file. Can somebody see this and let me know what could be the possible problem.
bash-2.05# adb java.3558.1133780573
core file = java.3558.1133780573 -- program `` /global/netmfs/netmbase/NetM62/licensing/jre/bin/java'' on platform SUNW,Sun-Fire-V440
SIGABRT: Abort
$c
libc.so.1`_lwp_kill+8(6, 0, acf7b6e8, 2, 0, af2d482c) libc.so.1`abort+0x100(2d80f8, b, b3ef0040, 0, 48, 0) libkernel32.so`__1cUSehScanInvokeTryList6FpnQSEH_THREAD_BLOCK__i_+0x334(2d9790,
af1cd7f8, af2d7324, 0, 2, 0)
libkernel32.so`__1cOSignal_HandlerFraise6FIpviipL_i_+0xfc(c0000005, acf7ba20, 0 , 2, acf7b858, 1800) libkernel32.so`__1cPRaise_Exception2f6MipnHsiginfo_pv_v_+0xa4(516f28, b, acf7bcd8, acf7ba20, af2ea1e4, 2c) libthread.so.1`__sighndlr+0xc(b, acf7bcd8, acf7ba20, af1cd854, 0, 0) libthread.so.1`call_user_handler+0x234(b, acf7bcd8, acf7ba20, 0, 0, 0) libthread.so.1`sigacthandler+0x64(b, acf7bcd8, acf7ba20, 1, f6e89af0, 3ba) 0xfa0f4520(b5fec400, f6e89af0, b5fec550, 3ba, fa015624, b5fed448) 0xfa43586c(f5c770a8, b5fec530, bcdddf90, b5fec530, bcdddf90, b5fec400) 0xfa005b10(f6e8a270, b6, 8, fa0155e0, f6e847a0, acf7bf70) 0xfa005b10(2d6b00, b8, acf7c0e4, fa0152c0, fa015624, acf7c000) 0xfa005b10(2d6b00, b8, 8, fa015590, feca1858, acf7c078) 0xfa005b10(2d6b00, 0, 0, fa0155e0, 3349f0, acf7c108) 0xfa000118(acf7c1f0, acf7c388, a, f6e89818, fa00aae0, acf7c328) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(acf7c380, acf7c318, acf7c320, 2d6b00, 2d6b00, 34e9cc
libjvm.so`JVM_DoPrivileged+0x4c0(632adc, ff0104d0, acf7c65c, 0, 1, 1) libjava.so`Java_java_security_AccessController_doPrivileged__Ljava_security_Priv
ilegedExceptionAction_2+0x14(2d6b94, acf7c5e0, acf7c65c, ffffffff, fa015624, 0) 0xfa00be48(2d6b00, b8, 15c, 4, f6e847a0, acf7c5f8) 0xfa005b10(2d6b00, 0, 0, fa015480, 3349f0, acf7c690) 0xfa000118(acf7c778, acf7c838, a, f6e87d18, fa00aae0, acf7c844) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(acf7c830, acf7c82c, acf7c840, 2d6b00, 2d6b00,
acf7d170)
libjvm.so`__1cNinstanceKlassbBcall_class_initializer_impl6FnTinstanceKlassHandle
pnGThread_v_+0xfc(acf7c950, 2d6b00, 10b, acf7c9e8, 1, 0) libjvm.so`__1cNinstanceKlassPinitialize_impl6FnTinstanceKlassHandle_pnGThread__v
+0x4f4(acf7ca98, 2d6b00, acf7d490, 2d741c, 2d6b00, 0) libjvm.so`_1cNinstanceKlassKinitialize6MpnGThread__v_+0x84(f6e87dd0, 2d6b00, 1b8, f5c01e00, acf7cb5c, 1) libjvm.so`__1cMLinkResolverTresolve_static_call6FrnICallInfo_rnLKlassHandle_nMsy
mbolHandle_53iipnGThread__v_+0x190(0, 2d6b00, ff0104d0, acf7cc54, acf7cc50, 1) libjvm.so`__1cMLinkResolverOresolve_invoke6FrnICallInfo_nGHandle_nSconstantPoolH
andle_inJBytecodesECode_pnGThread__v_+0xd8(acf7cf44, acf7cf0c, acf7cf08, 12, b8 , 2d6b00) libjvm.so`__1cSInterpreterRuntimeOresolve_invoke6FpnKJavaThread_nJBytecodesECode
__v_+0x6ac(2d6b00, b8, 8, b5feaea8, f6e80c78, 0) 0xfa01561c(2d6b00, b8, b5feaf20, fa015590, fa015624, acf7d0d0) 0xfa005b10(2d6b00, b8, 1b9, fa015590, f6dc6a20, acf7d170) 0xfa005b10(2d6b00, b8, 8, fa0155e0, f6dc6a20, acf7d208) 0xfa005b10(2d6b00, b8, acf7d3ac, fa015590, f6dc9888, acf7d2b0) 0xfa005b10(2d6b00, b8, 8, fa015590, f6dc6a20, acf7d340) 0xfa005b10(2d6b00, 0, 0, fa0155e0, 3349f0, acf7d3c8) 0xfa000118(acf7d4b0, acf7d570, a, f6dc87d0, fa00aae0, acf7d57c) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(acf7d568, acf7d564, acf7d578, 2d6b00, 2d6b00, 0) libjvm.so`__1cNinstanceKlassbBcall_class_initializer_impl6FnTinstanceKlassHandle
pnGThread_v_+0xfc(acf7d688, 2d6b00, 10b, acf7d720, 1, 0) libjvm.so`__1cNinstanceKlassPinitialize_impl6FnTinstanceKlassHandle_pnGThread__v
+0x4f4(acf7d7d0, 2d6b00, acf7e118, 2d72c4, 2d6b00, b5c49570) libjvm.so`_1cNinstanceKlassKinitialize6MpnGThread__v_+0x84(f6dc9700, 2d6b00, 1b8, f5c01e00, acf7d894, 1) libjvm.so`__1cMLinkResolverTresolve_static_call6FrnICallInfo_rnLKlassHandle_nMsy
mbolHandle_53iipnGThread__v_+0x190(0, 2d6b00, ff0104d0, acf7d98c, acf7d988, 1) libjvm.so`__1cMLinkResolverOresolve_invoke6FrnICallInfo_nGHandle_nSconstantPoolH
andle_inJBytecodesECode_pnGThread__v_+0xd8(acf7dc7c, acf7dc44, acf7dc40, 6d, b8 , 2d6b00) libjvm.so`__1cSInterpreterRuntimeOresolve_invoke6FpnKJavaThread_nJBytecodesECode
__v_+0x6ac(2d6b00, b8, acf7de5c, fa0155d8, fa015624, acf7dd50) 0xfa01561c(2d6b00, b8, b5c47f38, fa015590, f67cf4d0, acf7dde8) 0xfa005a44(2d6b00, b8, 3, fa015590, fa015304, acf7deb8) 0xfa005b54(be833b70, b6, acf7e024, fa015590, fa015304, acf7df48) 0xfa005b54(be833b78, b6, acf7e0a4, fa015270, b6080000, acf7dfc8) 0xfa005b54(2d6b00, 0, 0, fa015270, 3349f0, acf7e048) 0xfa000118(acf7e138, acf7e270, a, f67d9ba0, fa00aae0, acf7e2ac) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(acf7e268, acf7e264, acf7e29c, 2d6b00, 2d6b00,
fece7f70)
libjvm.so`__1cKReflectionGinvoke6FnTinstanceKlassHandle_nMmethodHandle_nGHandle_
inOobjArrayHandle_nJBasicType_4ipnGThread__pnHoopDesc__+0xeec(0, acf7e3c0, ff0104d0, 1, f67cf810, a) libjvm.so`__1cKReflectionNinvoke_method6FpnHoopDesc_nGHandle_nOobjArrayHandle_pn
GThread__2_+0x230(be3e4900, acf7e448, acf7e444, 2d6b00, feda0bd0, acf7e810) libjvm.so`JVM_InvokeMethod+0x26c(acf7e5b0, acf7e5ac, ff005ddc, ff0073e8, c6, 0) libjava.so`Java_sun_reflect_NativeMethodAccessorImpl_invoke0+0x10(2d6b94,
acf7e528, acf7e5b4, acf7e5b0, acf7e5ac, 0) 0xfa00be48(b5fed8d0, b8, acf7e634, c, b6080000, acf7e540) 0xfa005b10(b5fefd98, be8339c0, b5fed8d0, fa015590, f6b5c960, acf7e5d8) 0xfa2267c4(b5fefdb0, be8339c0, b60a0028, fa015270, 1, acf7e648) 0xfa2264e0(be3f5ca8, be8339c0, b60a0028, be3f5c70, f5c2a550, f5c2a550) 0xfa005b10(be41d6d8, f65ee028, acf7e878, fa015270, f68e6d70, acf7e778) 0xfa005fd8(be40db50, f65ee028, acf7e914, fa0156f0, f6229940, acf7e810) 0xfa005fd8(be3f6f48, b6, 49, fa0156f0, c6, acf7e8b0) 0xfa005b10(be3f6f48, b7, acf7ea70, fa0152b8, 0, acf7e978) 0xfa005b10(be3f6f48, f65ee028, acf7eb00, fa015430, fa00aae0, acf7ea10) 0xfa005fd8(be3ebba8, f65ee028, acf7eb98, fa0156f0, 2d6b00, acf7eaa0) 0xfa005fd8(be481ee8, f65ee028, acf7ec30, fa0156f0, 42e6f04c, acf7eb38) 0xfa005fd8(be47c270, f65ee028, acf7ecc0, fa0156f0, 1, acf7ebd0) 0xfa005fd8(be448ef8, b6, acf7ed90, fa0156f0, f620a188, acf7ec50) 0xfa005b10(be448ef8, b6, 0, fa015270, b6080000, acf7ed18) 0xfa005b10(be449a38, b6, acf7eed0, fa015270, c6, acf7edc8) 0xfa005b10(be6e0488, f5ee2070, 4, fa015270, f5c045c0, acf7ee58) 0xfa005fd8(2d6b00, 0, 0, fa015740, 3349f0, acf7ef00) 0xfa000118(acf7eff0, acf7f128, a, f67f5f50, fa00aae0, acf7f164) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(acf7f120, acf7f11c, acf7f154, 2d6b00, 2d6b00,
fece7f70)
libjvm.so`__1cKReflectionGinvoke6FnTinstanceKlassHandle_nMmethodHandle_nGHandle_
inOobjArrayHandle_nJBasicType_4ipnGThread__pnHoopDesc__+0xeec(0, acf7f278, ff0104d0, 1, f67cf810, a) libjvm.so`__1cKReflectionNinvoke_method6FpnHoopDesc_nGHandle_nOobjArrayHandle_pn
GThread__2_+0x230(be849d88, acf7f300, acf7f2fc, 2d6b00, feda0bd0, acf7f6c8) libjvm.so`JVM_InvokeMethod+0x26c(acf7f468, acf7f464, ff005ddc, ff0073e8, 20, 0) libjava.so`Java_sun_reflect_NativeMethodAccessorImpl_invoke0+0x10(2d6b94,
acf7f3e0, acf7f46c, acf7f468, acf7f464, 0) 0xfa00be48(b5fed810, b8, acf7f4ec, c, b6080000, acf7f3f8) 0xfa005b10(b5fed8a8, be6d7ab8, b5fed810, fa015590, 3230bc, acf7f490) 0xfa2267c4(b5fed8c0, be6d7ab8, b60a0000, fa015270, be843a68, acf7f500) 0xfa2264e0(b5fed858, be6d7ab8, b60a0000, b5fed858, 1, 0) 0xfa005b10(be6de080, b6, b5fed858, fa015270, be6d4478, acf7f620) 0xfa005b10(be6de080, b6, acf7f7c8, fa015270, 96, acf7f6c8) 0xfa005b10(be6de080, f5ee2070, 4, fa015270, f5c045c0, acf7f750) 0xfa005fd8(be6d6a18, f67d91d0, 8, fa015740, fa015304, acf7f7f8) 0xfa00601c(be6a3ac0, b6, 8, fa015590, 8, acf7f8a0) 0xfa005b54(be6a3ac0, f640af90, acf7fa7c, fa015270, f640af90, acf7f978) 0xfa005fd8(be8338f8, b7, 0, fa0156f0, f640af90, acf7fa08) 0xfa005c64(be8338f8, b7, acf7fb18, fa015430, be8338f8, acf7fab0) 0xfa005c64(acf7fba8, 0, 0, fa015430, 3349f0, acf7fb48) 0xfa000118(acf7fc30, acf7fe98, a, f6496780, fa00aae0, acf7fdb8) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(acf7fe90, acf7fcf8, acf7fdb0, 2d6b00, 2d6b00,
acf7fd08)
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle
_4pnRJavaCallArguments_pnGThread__v_+0x164(feff0000, 2d71e8, acf7fda4, acf7fda0 , acf7fdb0, 2d6b00) libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsym
bolHandle_5pnGThread__v_+0x60(acf7fe90, acf7fe8c, acf7fe84, acf7fe7c, acf7fe74,
2d6b00)
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x128(2d6b00, 2d6b00, 2d6190, 2d71e8, 319508, fecd6a5c) libjvm.so`__1cKJavaThreadDrun6M_v_+0x288(2d6b00, 0, ff00f808, ffff8000, 0, 0) libjvm.so`_start+0x134(2d6b00, 0, 0, 0, 0, 0) libthread.so.1`_lwp_start(0, 0, 0, 0, 0, 0)
Second Core dump with adb
bash-2.05# adb java.9337.1133744404
.core file = java.9337.1133744404 -- program `` /global/netmfs/netmbase/NetM62/licensing/jre/bin/java'' on platform SUNW,Sun-Fire-V440
SIGABRT: Abort
.$c
$c
libc.so.1`_lwp_kill+8(6, 0, ad0fc950, 2, 0, aeed482c) libc.so.1`abort+0x100(bd90f8, b, b3670200, 0, 48, 0) libkernel32.so`__1cUSehScanInvokeTryList6FpnQSEH_THREAD_BLOCK__i_+0x334(6197b0,
aedcd7f8, aeed7324, 0, 2, 0)
libkernel32.so`__1cOSignal_HandlerFraise6FIpviipL_i_+0xfc(c0000005, ad0fcc88, 0 , 2, ad0fcac0, 1800) libkernel32.so`__1cPRaise_Exception2f6MipnHsiginfo_pv_v_+0xa4(50c028, b, ad0fcf40, ad0fcc88, aeeea1e4, 2c) libthread.so.1`__sighndlr+0xc(b, ad0fcf40, ad0fcc88, aedcd854, 0, 0) libthread.so.1`call_user_handler+0x234(b, ad0fcf40, ad0fcc88, 0, 0, 0) libthread.so.1`sigacthandler+0x64(b, ad0fcf40, ad0fcc88, fa015590, b6130000,
ad0fcfc0)
0xfa0157f0(b5cb5908, b6, 8, fa015740, 0, ad0fd070) 0xfa005c64(b5cb5908, b7, ad0fd208, fa015270, b5cb1c90, ad0fd118) 0xfa005c64(b5cb5908, b7, 0, fa015430, b6130000, ad0fd1a8) 0xfa005c64(be56d460, b6, 8, fa015430, f691f2b0, ad0fd240) 0xfa005b10(be56d498, b6, be574c70, fa015270, 4889ba9b, ad0fd330) 0xfa005b10(be56d4a8, b7, ad0fd4d4, fa015270, 9, ad0fd3c8) 0xfa005b10(be56d4a8, b7, 0, fa015430, b6130000, ad0fd448) 0xfa005b10(be56d4a8, f6915008, 8, fa015430, 42a1665e, ad0fd540) 0xfa005fd8(bd44ecd0, f63b2338, ad0fd6c4, fa0156f0, b5cb4b38, ad0fd5d8) 0xfa005fd8(b5cb4a60, b7, ad0fd6d0, fa0156f0, b5cb4b48, ad0fd660) 0xfa005b10(b5cb4a60, b6, ad0fd818, fa015430, f65b7528, ad0fd718) 0xfa005b10(bd41bcf0, f63b1548, ad0fd8a0, fa015270, b6130000, ad0fd7b8) 0xfa005fd8(bd410620, b7, ad0fd934, fa0156f0, b6130000, ad0fd840) 0xfa005b10(bd410620, b6, ad0fd9c4, fa015430, b6130000, ad0fd8d0) 0xfa005b10(bd410620, b6, ad0fda44, fa015270, b6130000, ad0fd968) 0xfa005b10(bd8d1420, f63aa0c8, ad0fdad4, fa015270, 4400, ad0fd9e8) 0xfa005fd8(bd8c7f60, f65c2490, ad0fdb5c, fa0156f0, f6906720, ad0fda70) 0xfa005fd8(be548400, f68f50b0, ad0fdbe8, fa0156f0, f65b4158, ad0fdaf0) 0xfa005fd8(b5caf718, f6d147c0, 0, fa0156f0, b6130000, ad0fdb80) 0xfa005fd8(b5caeb10, b7, ad0fdcf8, fa0156f0, b6130000, ad0fdc10) 0xfa005c64(b5caeb10, f6a62080, ad0fdd8c, fa015430, f6ad6088, ad0fdc90) 0xfa005fd8(b5caf718, f6d147c0, ad0fde3c, fa0156f0, b6130000, ad0fdd20) 0xfa005fd8(be6abc20, b6, 0, fa015738, b6130000, ad0fddd0) 0xfa005b10(be6abc20, b7, ad0fdf94, fa015270, 4400, ad0fdea0) 0xfa005b10(be6abc20, b7, 0, fa015478, b6130000, ad0fdf38) 0xfa005b10(be6abc20, b7, 0, fa015478, f6864318, ad0fdfc8) 0xfa005b10(be6abc20, b6, ad0fe14c, fa015430, 3230bc, ad0fe058) 0xfa005b10(be6abc20, b7, ad0fe1d8, fa015270, b6130000, ad0fe0f0) 0xfa005b10(be6abc20, f6ad6088, ad0fe264, fa015430, f6ad6088, ad0fe178) 0xfa005fd8(be695da8, f6ab5750, 0, fa0156f0, b6130000, ad0fe208) 0xfa005fd8(b5caeb10, b7, ad0fe3ac, fa0156f0, f65b4158, ad0fe2a8) 0xfa005b10(b5caeb10, b7, ad0fe43c, fa015478, b6130000, ad0fe350) 0xfa005b10(b5caeb10, f6a62080, ad0fe4c8, fa015478, f5c355f8, ad0fe3e0) 0xfa005fd8(be7db4e8, b6, ad0fe55c, fa0156f0, aeed28, ad0fe468) 0xfa005b10(be7db4e8, b7, 0, fa015590, b6130000, ad0fe4f0) 0xfa005b10(be7db4e8, f6da8e80, ad0fe664, fa0156f0, fa0154c4, ad0fe580) 0xfa005fd8(be7d8e20, b7, ad0fe700, fa015270, f68e2490, ad0fe608) 0xfa005c64(be7d8e20, b7, ad0fe788, fa015430, fa0154c4, ad0fe6a0) 0xfa005c64(be7d8e20, b7, ad0fe808, fa015270, fa015624, ad0fe728) 0xfa005c64(aeed28, b8, ad0fe888, fa015430, f5c05cb0, ad0fe7a8) 0xfa005c64(b5c556a8, f66f5b10, ad0fe910, fa015270, f66f5b10, ad0fe828) 0xfa00612c(aeed28, 0, 0, fa0156f0, 3349f0, ad0fe8b0) 0xfa000118(ad0fe9a0, ad0fead8, a, f66f5668, fa00aae0, ad0feb10) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(ad0fead0, ad0feacc, ad0feb04, aeed28, aeed28,
fece7f70)
libjvm.so`__1cKReflectionGinvoke6FnTinstanceKlassHandle_nMmethodHandle_nGHandle_
inOobjArrayHandle_nJBasicType_4ipnGThread__pnHoopDesc__+0xeec(0, ad0fec28, ff0104d0, 1, f655dbd0, e) libjvm.so`__1cKReflectionNinvoke_method6FpnHoopDesc_nGHandle_nOobjArrayHandle_pn
GThread__2_+0x230(be023bb0, ad0fecb0, ad0fecac, aeed28, feda0bd0, ad0ff080) libjvm.so`JVM_InvokeMethod+0x26c(ad0fede8, ad0fede4, ff005ddc, ff0073e8, f6159f00, ad0ff120) libjava.so`Java_sun_reflect_NativeMethodAccessorImpl_invoke0+0x10(aeedbc,
ad0fed84, ad0fedec, ad0fede8, ad0fede4, fece6088) 0xfa4067a8(be07f5a8, be7d8938, b5c53fc0, fa015430, 20, 0) 0xfa4062a8(be7d8e28, be7d8938, b5c53fc0, fa013bc8, bd0ac0f0, ad0fee58) 0xfa226284(be7d8e40, be7d8938, b5c53fc0, bd0ac040, f6159f00, 1) 0xfa225fa0(be07f5a8, be7d8938, b5c53fc0, 1, f5c2a550, f5c2a550) 0xfa005b10(be035120, f660cb18, ad0ff0e8, fa015270, f661f5d0, ad0fefe8) 0xfa005fd8(be0281a0, f660cb18, ad0ff184, fa0156f0, f6226670, ad0ff080) 0xfa005fd8(be023b70, b6, 49, fa0156f0, f6159f00, ad0ff120) 0xfa005b10(be023b70, b7, ad0ff2e0, fa0137d8, 0, ad0ff1e8) 0xfa005b10(be023b70, f660cb18, ad0ff370, fa015430, 20, ad0ff280) 0xfa005fd8(be0741e0, f660cb18, ad0ff408, fa0156f0, bd0ac0f0, ad0ff310) 0xfa005fd8(be066d20, f660cb18, ad0ff4a0, fa0156f0, 42a165e0, ad0ff3a8) 0xfa005fd8(be05f6d0, f660cb18, ad0ff530, fa0156f0, 1, ad0ff440) 0xfa005fd8(be05a1b8, b6, ad0ff600, fa0156f0, f620a1f8, ad0ff4c0) 0xfa005b10(be05a1b8, b6, 0, fa015270, b6130000, ad0ff588) 0xfa005b10(be023af0, b6, 0, fa015270, f5c045c0, ad0ff630) 0xfa005b10(be0d4ed0, f65f5b80, ad0ff7d8, fa015270, f6d80ae8, ad0ff6c8) 0xfa00612c(be0ab2f0, f65f5b80, ad0ff7e0, fa0156f0, f6560da8, ad0ff778) 0xfa00612c(be0b84d0, b6, ad0ff8f8, fa0156f0, 0, ad0ff810) 0xfa005c64(be0b84d0, b6, ad0ff900, fa015270, f6560da8, ad0ff898) 0xfa005c64(be0af510, f5c1e740, ad0ffa24, fa015270, fa013848, ad0ff930) 0xfa00612c(be0ab2f0, b7, ad0ffaac, fa0156f0, 0, ad0ff9c0) 0xfa005c64(be0ab2f0, f5c1e740, ad0ffb2c, fa015430, 0, ad0ffa48) 0xfa00612c(b5d4ea98, f5c1e740, ad0ffba4, fa0156f0, 0, ad0ffad0) 0xfa00612c(ad0ffba8, 0, 0, fa0156f0, 3349f0, ad0ffb48) 0xfa000118(ad0ffc30, ad0ffe98, a, f5c1efd8, fa00aae0, ad0ffdb8) libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallAr
guments_pnGThread__v_+0x274(ad0ffe90, ad0ffcf8, ad0ffdb0, aeed28, aeed28,
ad0ffd08)
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle
_4pnRJavaCallArguments_pnGThread__v_+0x164(feff0000, ae82b8, ad0ffda4, ad0ffda0 , ad0ffdb0, aeed28) libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsym
bolHandle_5pnGThread__v_+0x60(ad0ffe90, ad0ffe8c, ad0ffe84, ad0ffe7c, ad0ffe74,
aeed28)
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x128(aeed28, aeed28, af6758, ae82b8, 319508, fecd6a5c) libjvm.so`__1cKJavaThreadDrun6M_v_+0x288(aeed28, ffffffe2, ff00f808, ffff8000, 0 , 0) libjvm.so`_start+0x134(aeed28, 0, 0, 0, 0, 0) libthread.so.1`_lwp_start(0, 0, 0, 0, 0, 0)Hi there
Not sure which jdk you are using but I am guessing you are using native code?
libkernel32.so
See http://java.sun.com/j2se/1.4.2/docs/guide/vm/signal-chaining.html
Start your application this way, this should solve your problem.
export LD_PRELOAD=<libjvm.so dir>/libjsig.so; java_application (ksh)
setenv LD_PRELOAD <libjvm.so dir>/libjsig.so; java_application (csh)
Also, even if u are using 1.3.1, u can get the libjsig.so from 1.4.2 and do the same.
Hope this helps. -
Hi all
We have some problems on our production system that it keeps on dumping core dumps of about 3gigs each night and just kills the system. Looks like something wrong with java?
Directory: /usr/sap/<SID>/DVEBMGS00/j2ee/cluster/server0
Files:
core.20090817.174103.15073438.0009.dmp
Snap.20090817.174103.15073438.0010.trc
heapdump.20090817.174103.15073438.0011.phd
javacore.20090817.174103.15073438.0012.txt
2.49 core.20090817.161324.15073438.0001.dmp
2.53 core.20090817.163951.15073438.0005.dmp
2.58 core.20090817.174103.15073438.0009.dmp
Any help will be much appreciated.Hello Stephan
Most likely these dumps are being caused by an OutOfMemoryError in the java stack. Check note #1289240 How to deal with out of memory errors, for some hints on what to do.
Regards
Mark -
Java Core Dump Issue with jdk 1.4.1
I have an application which is run as a batch process. it handles data ranging from 100 thousand to 3 million. This process was running without any problem with jdk1.3 , but with 1.4.1 i am not able to finish this process , it generates core dump every time and the process stops. The process is asyncronus , it spons two threads to do the processing. Is there any way to handle this ?
JDK 1.4.1 is not supported anymore by Sun. See also http://java.sun.com/products/archive/
"... These products are down-revision products that may have various bugs, Y2000, and possibly security issues associated with them. Sun in no way recommends these products be used in a live, production environment ..."
Try the latest 1.4.2 release. Go to http://java.sun.com/j2se/1.4.2/download.html or even better, try 5.0 at http://java.sun.com/j2se/1.5.0/download.jsp
-jonelo -
JAVA CORE DUMP IN AIX MACHINES
Hi,
we got a java core dump in a AIX machine .
The excerpts of the core dump file is as below. Please can anyone suggest me what the core dump is all about and what i need to do to fix the problem :
NULL ------------------------------------------------------------------------
0SECTION XHPI subcomponent dump routine
NULL ==============================
1XHTIME Wed Sep 14 09:48:39 2005
1XHSIGRECV SIGILL received at 0xd2a36cfc in /iscp/node/lib/libmocj.a. Processing terminated.
1XHFULLVERSION J2RE 1.4.1 IBM AIX build ca1411ifx-20040810 (141SR3)
NULL
1XHCURRENTTHD Current Thread Details
NULL ----------------------
2XHCURRSYSTHD "Thread-16" sys_thread_t:0x36866028
3XHNATIVESTACK Native Stack
NULL ------------
3XHSTACKLINE at 0xD2A2FC40 in changeTimeOut__FP8__clienti
3XHSTACKLINE at 0xD2A36CFC in ismc_getUsersOnsite__FRCQ2_3std4listXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ2_3std9allocatorXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc____RQ2_3std4listXTQ2_3std12basic_stringXTcTQ2_3std11c
3XHSTACKLINE at 0xD2A4D8D0 in Java_com_telcordia_iscp_moc_libj_cgp_RPC_getUsersOnsite
3XHSTACKLINE at 0xD2821CE8 in mmipInvoke_O_O
3XHSTACKLINE at 0xD27E9370 in mmipExecuteJava
3XHSTACKLINE at 0xD27E1FB4 in xeRunJavaVarArgMethod
3XHSTACKLINE at 0xD27E2260 in xeRunDynamicMethod
3XHSTACKLINE at 0xD27CFFFC in threadRT0
3XHSTACKLINE at 0xD293CD10 in xmExecuteThread
3XHSTACKLINE at 0xD2940BF4 in threadStart
3XHSTACKLINE at 0xD2A75408 in _start
3XHSTACKLINE at 0xD005277C in pthreadbody
NULL
1XHOPENV Operating Environment
NULL ---------------------
2XHHOSTNAME Host : cu11sv01_if0:10.104.148.79
2XHOSLEVEL OS Level : AIX 5.2.0.0
2XHCPUS Processors -
3XHCPUARCH Architecture : POWER_PC (impl: unknown, ver: unknown)
3XHNUMCPUS How Many : 2
/jni
4XESTACKTRACE at java.lang.Object.wait(Native Method)
4XESTACKTRACE at java.lang.Object.wait(Object.java:438)
4XESTACKTRACE at com.inprise.vbroker.orb.ORB.run(ORB.java:261)
4XESTACKTRACE at com.telcordia.iscp.moc.server.config.ismc_config_mgr.main(ismc_config_mgr.java:125)
3XHNATIVESTACK Native Stack
NULL ------------
3XHSTACKLINE at 0xD005D814 in eventwait
3XHSTACKLINE at 0xD0069740 in condwait_local
3XHSTACKLINE at 0xD0069B98 in condwait
3XHSTACKLINE at 0xD006A6F8 in pthread_cond_wait
3XHSTACKLINE at 0xD2A7C8AC in condvarWait
3XHSTACKLINE at 0xD2A7B718 in sysMonitorWait
3XHSTACKLINE at 0xD286B1F8 in lkMonitorWait
3XHSTACKLINE at 0xD27E0234 in JVM_MonitorWait
3XHSTACKLINE at 0xD2832168 in sysInvokeNative
3XHSTACKLINE at 0xD281CE58 in mmipInvokeJniMethod
3XHSTACKLINE at 0xD27E9370 in mmipExecuteJava
3XHSTACKLINE at 0xD27E192C in xeRunJniMethod
3XHSTACKLINE at 0xD2819F58 in invokeJniMethod
3XHSTACKLINE at 0xD2813D4C in jni_CallStaticVoidMethod
NULL
NULL ----------------------------------------------------
2XMFULLTHDDUMP Full thread dump Classic VM (J2RE 1.4.1 IBM AIX build ca1411ifx-20040810 (141SR3), native threads):
3XMTHREADINFO "Thread-16" (TID:0x300E9E88, sys_thread_t:0x36866028, state:R, native ID:0xF37) prio=5
4XESTACKTRACE at com.telcordia.iscp.moc.libj.cgp.RPC.getUsersOnsite(Native Method)
4XESTACKTRACE at com.telcordia.iscp.moc.libj.cgp.RPC.getUsersFromSite(RPC.java:27)
4XESTACKTRACE at com.telcordia.iscp.moc.server.config.RPCThread.rpcGetUsers(RPCThread.java:111)
4XESTACKTRACE at com.telcordia.iscp.moc.server.config.RPCThread.run(RPCThread.java:60)
4XESTACKTRACE at java.lang.Thread.run(Thread.java:568)
3XHNATIVESTACK Native Stack
NULL ------------
3XHSTACKLINE at 0xD2A2FC40 in changeTimeOut__FP8__clienti
3XHSTACKLINE at 0xD2A36CFC in ismc_getUsersOnsite__FRCQ2_3std4listXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ2_3std9allocatorXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc____RQ2_3std4listXTQ2_3std12basic_stringXTcTQ2_3std11c
3XHSTACKLINE at 0xD2A4D8D0 in Java_com_telcordia_iscp_moc_libj_cgp_RPC_getUsersOnsite
3XHSTACKLINE at 0xD2821CE8 in mmipInvoke_O_O
3XHSTACKLINE at 0xD27E9370 in mmipExecuteJava
3XHSTACKLINE at 0xD27E1FB4 in xeRunJavaVarArgMethod
3XHSTACKLINE at 0xD27E2260 in xeRunDynamicMethod
3XHSTACKLINE at 0xD27CFFFC in threadRT0
3XHSTACKLINE at 0xD293CD10 in xmExecuteThread
3XHSTACKLINE at 0xD2940BF4 in threadStart
3XHSTACKLINE at 0xD2A75408 in _start
3XHSTACKLINE at 0xD005277C in pthreadbody
NULL
ThanksTake a look at this IBM article to see if it helps:
http://www-128.ibm.com/developerworks/eserver/library/es-javaonaix_core.html -
Help on JVM Crash with core dump on solaris - 1.5_17
Some times in my load test scenarios on sun os boxes JVM crashing with core dump. Here is some dump from the file
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGSEGV (0xb) at pc=0xfea07f40, pid=1564, tid=10
# Java VM: Java HotSpot(TM) Server VM (1.5.0_17-b04 mixed mode)
# Problematic frame:
# V [libjvm.so+0x207f40]
--------------- T H R E A D ---------------
Current thread (0x0014e220): JavaThread "CompilerThread1" daemon [_thread_in_native, id=10]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000000
Registers:
O0=0x00000010 O1=0x019c1960 O2=0x01e00ec0 O3=0x002bdc48
O4=0x01042c68 O5=0xc467eb4c O6=0xc467e330 O7=0x01042c68
G1=0x01e00ea0 G2=0xff014c94 G3=0x000000e6 G4=0x01c5a4e4
G5=0x01736e20 G6=0x00000000 G7=0xfb9e4200 Y=0x00000000
PC=0xfea07f40 nPC=0xfea07f44
--------------- S Y S T E M ---------------
OS: Solaris 10 5/08 s10s_u5wos_10 SPARC
Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 24 March 2008
uname:SunOS 5.10 Generic_127127-11 sun4v (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:2.73 2.67 2.21
CPU:total 32 has_v8, has_v9, has_vis1, has_vis2, is_ultra3, is_sun4v, is_niagara1
Memory: 8k page, physical 8257536k(366576k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_17-b04) for solaris-sparc, built on Nov 10 2008 01:58:40 by unknown with unknown Workshop:0x550
Here is the stack dump of the kill quit thread
----------------- lwp# 10 / thread# 10 --------------------
ff2c5bf0 lwpkill (6, 0, ff2f2e10, ff2a8bd0, ffffffff, 6) + 8
ff2410f8 abort (7400, 1, 7c00, ad314, ff2f12d8, 0) + 110
fee7e58c __1cCosFabort6Fi_v_ (1, 0, ff013084, fefde000, 7d94, 7c00) + 58
fef0de48 __1cHVMErrorOreport_and_die6M_v_ (0, ff03a640, ff033ff4, 1, fee82c88, ff033ff4) + c84
fea74138 JVM_handle_solaris_signal (b, c467e2b0, c467dff8, 8000, ff032fa0, 14e220) + ab4
ff2c4b28 __sighndlr (b, c467e2b0, c467dff8, fea7364c, 0, 1) + c
ff2b9b00 call_user_handler (b, ffbffeff, c, 0, fb9e4200, c467dff8) + 3b8
fea07f40 __1cMPhaseChaitinFSplit6MI_I_ (c467ec2c, 0, 0, 3677ac, 398, c) + 3410
fea13c68 __1cMPhaseChaitinRRegister_Allocate6M_v_ (c467eb4c, e88, dc0, ff0137d8, c467fb14, 48d) + 720
fea17c64 __1cHCompileICode_Gen6M_v_ (c467f218, 9e0c, 9c00, fef56b15, 0, c467ec2c) + 2b0
fea7ff14 __1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_ (c467f218, 0, 346c8, 0, fef569b8, 0) + c08
fea75fb8 __1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_ (c467fb14, fef42a90, 1e40f58, 244, 346c8, d1800000) + b0
fea76b68 __1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_ (908928, 14e7fc, 13c900, 14e220, fef57367, c467fb14) + 4cc
feb3357c __1cNCompileBrokerUcompiler_thread_loop6F_v_ (ff0330b8, 13c8a0, 14e220, c5e67700, 14e7f8, 0) + 44c
feadbd20 __1cKJavaThreadDrun6M_v_ (14e220, ff037040, 7820, 0, 7800, 9400) + 2b0
fee7e0a8 __1cG_start6Fpv_0_ (14e220, 61c, fefde000, 0, 5874, 5800) + 208
ff2c49fc lwpstart (0, 0, 0, 0, 0, 0)
Any idea on this dump, helps me a lot.
Thanks.[http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/crashes.html]
-
Hotspot core dumping during JVM garbage collection ?
We have an application which calls a 3rd party supplied server API which has recently been upgraded to use Java 1.5
We are getting the following error reported by our client application. The application is also now running Java 1.5 but references many classes in jar files which would have quite old code in.
The supplier of the API has stated that the problem requires us to recompile all our jar files using v 1.5 ( including things like jconnect and jms ?!?!? ). This sounds like a bit of a cop-out to me, not to mention being impossible since we don't have the source for things like jconnect.
I suspect that there is a garbage collection problem at the bottom of all this, but I'm not sure how I can "prove" this, nor do I currently have any real clue as to how to fix any GC problem that may exist.
The application is supposed to wait for a message on a MQSeries queue and then transforms it via Xalan XSLT and sends it to a Server application. I've tried playing around with heap sizes etc but that just seems to make it worse. If I leave it at the settings that the previous version used then the client at least manages to process a couple of messages before core dumping. There doesn't seem to be a consistent trigger event to cause the core dump ( it's not a message arriving on a queue for example ) but it does seem to be fairly consistent timewise, i.e. after
Any ideas gratefully accepted.
Here's a logfile excerpt from my applications showing the Hotspot error message :
=====================================================================================
08-Jul-2008 10:01:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:02:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:03:05 Waiting for messages from COLT.BBFS
405.815: [GC [PSYoungGen: 17331K->9244K(37632K)] 111702K->103615K(192128K), 0.1615910 secs]
405.977: [Full GC#
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
# An error report file with more information is saved as hs_err_pid2600.log
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
=====================================================================================
The logfile referred to in the error message contains the following.
=====================================================================================
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
--------------- T H R E A D ---------------
Current thread (0x001484d8): VMThread [id=8]
siginfo:si_signo=10, si_errno=0, si_code=1, si_addr=0x0000080f
Registers:
O0=0x00487588 O1=0xfe7d6454 O2=0x000079b0 O3=0x00007800
O4=0x00008868 O5=0x00147d48 O6=0xf8781460 O7=0xfe0f7938
G1=0xe52aaae8 G2=0x00000003 G3=0x00000003 G4=0x001484d8
G5=0xf8781d98 G6=0x00000002 G7=0xf8781d98 Y=0x805683e2
PC=0xfe141348 nPC=0xfe14134c
Top of Stack: (sp=0xf8781460)
0xf8781460: fe786000 00c6ba20 fe10013c e54bab68
0xf8781470: 0000080f e54bab6c 0000080f 00000004
0xf8781480: 00487588 00000134 e54bab6c 00000004
0xf8781490: 00000001 00000000 f87814c0 fe17aef4
0xf87814a0: fe6485b4 fe7d899c 001484d8 0011da40
0xf87814b0: 00148988 00148c10 00148d7c f8781880
0xf87814c0: 007c3389 007c3c0e 00000f87 00008868
0xf87814d0: 00008800 00487588 fe141310 fe7d6454
Instructions: (pc=0xfe141348)
0xfe141338: ec 06 c0 1a 80 a5 a0 00 22 40 00 0a ba 07 60 01
0xfe141348: f2 05 a0 00 ae 0e 60 03 80 a5 e0 03 22 40 00 05
Stack: [0xf8702000,0xf8781d98), sp=0xf8781460, free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x141348]
V [libjvm.so+0x17aefc]
V [libjvm.so+0x2d557c]
V [libjvm.so+0x300ef8]
V [libjvm.so+0x301e84]
V [libjvm.so+0x2ff950]
V [libjvm.so+0x29df30]
V [libjvm.so+0x362b44]
V [libjvm.so+0x6436f0]
VM_Operation (0xe03012b0): parallel gc system gc, mode: safepoint, requested by thread 0x0031bca0
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00b2c028 JavaThread "Thread-4" [_thread_in_native, id=85]
0x007f5048 JavaThread "Thread-0" [_thread_blocked, id=84]
0x00c27cf0 JavaThread "Notification Delivery" [_thread_blocked, id=81]
0x0026fa08 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=73]
0x00821048 JavaThread "RMI RenewClean-[162.11.2.32:44425]" daemon [_thread_blocked, id=70]
0x0031bca0 JavaThread "GC Daemon" daemon [_thread_blocked, id=67]
0x00cd5d28 JavaThread "RMI Reaper" [_thread_blocked, id=66]
0x003c9300 JavaThread "Timer-0" daemon [_thread_blocked, id=65]
0x00929fe0 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=64]
0x0089bf18 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=42]
0x00c47248 JavaThread "Pool thread #7" daemon [_thread_blocked, id=38]
0x00c466a0 JavaThread "Pool thread #6" daemon [_thread_blocked, id=37]
0x00311850 JavaThread "Pool thread #5" daemon [_thread_blocked, id=36]
0x00287a40 JavaThread "Pool thread #4" daemon [_thread_blocked, id=35]
0x00286e98 JavaThread "Pool thread #3" daemon [_thread_blocked, id=34]
0x00c134b0 JavaThread "Pool thread #2" daemon [_thread_blocked, id=33]
0x00ad09e0 JavaThread "Pool thread #1" daemon [_thread_blocked, id=32]
0x00286cd8 JavaThread "PoolThreadManager" daemon [_thread_blocked, id=31]
0x00c129e0 JavaThread "Channel Reaper" daemon [_thread_blocked, id=30]
0x00c669e8 JavaThread "ORB Daemon Thread" daemon [_thread_blocked, id=29]
0x00b10170 JavaThread "Worker for ServerProtocol: (iiop) /0.0.0.0:20168" daemon [_thread_blocked, id=22]
0x008a17e0 JavaThread "Syn~ Client" daemon [_thread_blocked, id=21]
0x003dc378 JavaThread "PoolScavenger0" daemon [_thread_blocked, id=20]
0x0015a928 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=15]
0x00159880 JavaThread "CompilerThread1" daemon [_thread_blocked, id=14]
0x00158a18 JavaThread "CompilerThread0" daemon [_thread_blocked, id=13]
0x00157b98 JavaThread "AdapterThread" daemon [_thread_blocked, id=12]
0x00156dc8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11]
0x0014ccd8 JavaThread "Finalizer" daemon [_thread_blocked, id=10]
0x0014ad90 JavaThread "Reference Handler" daemon [_thread_blocked, id=9]
0x00038238 JavaThread "main" [_thread_in_native, id=1]
Other Threads:
=>0x001484d8 VMThread [id=8]
0x0015c3b0 WatcherThread [id=16]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00037728/0x00037758] Threads_lock - owner thread: 0x001484d8
[0x00033650/0x00037ba8] Heap_lock - owner thread: 0x0031bca0
Heap
PSYoungGen total 37632K, used 9244K [0xf2eb0000, 0xf7ba0000, 0xf8400000)
eden space 28352K, 0% used [0xf2eb0000,0xf2eb0000,0xf4a60000)
from space 9280K, 99% used [0xf4a60000,0xf5367080,0xf5370000)
to space 25216K, 0% used [0xf6300000,0xf6300000,0xf7ba0000)
PSOldGen total 154496K, used 94371K [0xe8400000, 0xf1ae0000, 0xf2eb0000)
object space 154496K, 61% used [0xe8400000,0xee028e78,0xf1ae0000)
PSPermGen total 35584K, used 18260K [0xe4400000, 0xe66c0000, 0xe8400000)
object space 35584K, 51% used [0xe4400000,0xe55d5158,0xe66c0000)
Dynamic libraries:
0x00010000 /dsdvlp/java/jvm/jdk1.5.0_03/bin/java
0xff350000 /usr/lib/libthread.so.1
0xff340000 /usr/lib/libdl.so.1
0xff200000 /usr/lib/libc.so.1
0xff390000 /usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1
0xfe000000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server/libjvm.so
0xff1e0000 /usr/lib/libsocket.so.1
0xff2d0000 /usr/lib/libsched.so.1
0xff1b0000 /usr/lib/libCrun.so.1
0xff160000 /usr/lib/libm.so.1
0xff080000 /usr/lib/libnsl.so.1
0xff060000 /usr/lib/libmp.so.2
0xff030000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/native_threads/libhpi.so
0xfdfc0000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libverify.so
0xfdf80000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libjava.so
0xfdf50000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libzip.so
0xfb7e0000 /usr/lib/locale/en_GB.ISO8859-1/en_GB.ISO8859-1.so.2
0xe4190000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libnet.so
0xe3bd0000 /dsdvlp/lib/5/libSolarisNatives.so
0xe3e90000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/librmi.so
VM Arguments:
jvm_args: -Djava.ext.dirs=/dsdvlp/java/tmijar/firs7/lib/cli:/dsdvlp/java/tmijar/firs7/lib/cli/ext:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB/ext:/dsdvlp/java/tmijar/firs7/lib/cmn:/dsdvlp/java/tmijar/firs7/lib/cmn/ext:/dsdvlp/java/tmijar/firs7/daemonlib -Duser.dir=/dsdvlp/java/tmijar/firs7 -Dopenorb.config=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB/config/SynOpenORB.xml -Dopenorb.home=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB -Dcom.coexis.syn.general.orbbinding=com.coexis.syn.general.orbbinding.openorb.OpenORBBinding_1_4 -Dsun.rmi.dgc.client.gcInterval=360000 -Dsun.rmi.dgc.server.gcInterval=360000000 -Xms32m -Xmx256m -Dcom.coexis.syn.clientcommandsconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/clientcommands.xml -Dcom.coexis.syn.clientconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/fsbbtd_client.xml -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
java_command: com.coexis.syn.mqmessaging.daemon.RunDaemon -p /dsdvlp/bin/5/lndsfsd_fsbbtd.properties start
Environment Variables:
JAVA_HOME=/dsdvlp/java/jvm/jdk150
CLASSPATH=.:/dsdvlp/java/jar/jconnect520.jar:/dsdvlp/java/jar/vbjapp340.jar:/dsdvlp/java/jar/vbjorb340.jar:/dsdvlp/java/jar/javax_jndi120.jar
PATH=/usr/local/etc:/usr/lang:/usr/openwin/bin:/usr/ucb:/bin:/usr/etc:/usr/local/5/bin:/dsdvlp/bin/5:/dsdvlp/bin/4:/home/app/sybase/5/bin:/home/app/sybase/5/localscripts:/home/app/sybase/5/sqr:/home/app/lang:/home/app/lang/SC2.0.1:/usr/ccs/bin:/usr/local/opt/Acrobat3/bin:/dsdvlp/bin:.
LD_LIBRARY_PATH=/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server:/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc:/dsdvlp/java/jvm/jdk1.5.0_03/jre/../lib/sparc:/usr/lib:/usr/openwin/lib:/usr/local/5/lib:/dsdvlp/lib/5:/dstest/lib/5:/home/app/sybase/5/lib:/dstest/cats/sun4/lib:/tmitest/Opus/opus/lib
SHELL=/bin/csh
DISPLAY=CLI00184.mfil.local:1.0
OS=5
--------------- S Y S T E M ---------------
OS: Solaris 8 2/02 s28s_u7wos_08a SPARC
Copyright 2002 Sun Microsystems, Inc. All Rights Reserved.
Assembled 18 December 2001
uname:SunOS 5.8 Generic_117350-20 sun4u (T1 libthread)
rlimit: STACK 8192k, CORE 9216k, NOFILE 4096, AS infinity
load average:2.24 2.67 2.68
CPU:total 4 has_v8, has_v9, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 8388608k(166384k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_03-b07) for solaris-sparc, built on Apr 13 2005 03:31:26 by unknown with unknown Workshop:0x550The very first suggestion I have is to move your VM to a more recent update of 1.5.0.
It looks like you are crashing with 5.0u3, and I'm pretty sure 5.0u16 is available. You
don't want to waste your time chasing a bug that's already been fixed. -
Webcenter crashing with core dump
Hi All,
Once in a while Webcenter(11.1.1.5) is crashing and generating core dump and the error message is same evertime. We tried to research on addPartialTriggerListeners and found this kind of behaviour is a bug in apache trinidad version 1.2.5-core and is fixed in 1.2.7-core. But the webcenter we are using as version 1.2.12. Also the issue seems to be comming from webcenter/adf framework itself as none of our code uses addPartialTriggerListeners.
Please help us resolve this issue as we are getting this issue in our production environment and we need to fix this asap.
# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb) at pc=0xffffffff7318caf0, pid=19921, tid=149
# JRE version: 6.0_29-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.4-b02 mixed mode solaris-sparc compressed oops)
*# Problematic frame:*
+*# J org.apache.myfaces.trinidadinternal.context.RequestContextImpl.addPartialTriggerListeners(Ljavax/faces/component/UIComponent;[Ljava/lang/String;)V*#+
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
--------------- T H R E A D ---------------
Current thread (0x00000001145ba000): JavaThread "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_Java, id=149, stack(0xfffffffea6800000,0xfffffffea6900000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000000
Registers:
G1=0xffffffff503d5098 G2=0x00000001145ba000 G3=0xffffffff73516254 G4=0x00000000000000d1
ThanksOracle Support has a thread and work around for this.
DocID 1514563.1
Applies to:
Oracle WebCenter Portal - Version 11.1.1.5.0 and later
Information in this document applies to any platform.
Purpose
Java HotSpot Virtual Machine, a core component of Oracle's Java SE platform, known to core dump (resulting in a server crash) while optimizing certain methods in Oracle WebCenter.
If you can switch to JRockit you can get around it also. Otherwise follow the note and setup the appropriate pre-compile excludes in your WC startup script. -
Investigation on JVM crash without core dump (by JIT compiler)
Hi, All
I posted "JVM crash without core dump due to CompilerThread1" couple months ago.
http://forum.java.sun.com/thread.jspa?threadID=5253434
I would like show what we found, and ask couple questions
1) The reason of JVM crash is "CompilerThread0/1 wanted to allocate more native memory and eventually exceeded the limit of 32-bit Linux (on Redhat, the limit of VIRT is 3G)
2) After we lower the heap size (to reduce process size) and added process size monitor (track process size every minute) and JIT compilation log (-XX:+PrintCompilation),
We found sometimes JVM process jumped more than 800M when compiling one method. with the following log
Total time for which application threads were stopped: 0.1997400 seconds
5828 xxx.xxx.SomeClass::someMethod (1507 bytes)
5828 COMPILE SKIPPED: out of nodes during split (not retryable)
549002.449: [GC [PSYoungGen: 517897K->28390K(551296K)] 942855K->453348K(1229952K), 0.0726350 secs]
if we saw "COMPILE SKIPPED: out of nodes during split (not retryable)", the process size of JVM (VIRT/RES) always jumped 800M~1000M, and sometimes the memory get reverted in 30mins to couple hours, and sometimes it lasted forever. (so before we have larger footprint, this jump will kill JVM, right now, if the jump lasts forever, any more allocation on top of it also can kill jvm (much rare))
3) This is definitely a bug of JVM, because
it only happened on server mode, not client mode
it is random, that method can be compiled success on other JVM or next restart (we have 20 JVMs)
the method is not that complicate, like 100 lines, (bigger method gets compiled success)
this only happen on PROD environment, we can't reproduce it locally or QA (the method is always compiled success)
It mostly like under some condition (maybe node space is not enough), to compile that method will trigger JVM trying to allocate much more native memory.
4) We are going to disable this method by ".hotspot_compiler" to fix it, right now we are using lower footprint, JVM dies rarely.
+To understand more about this, I have some questions about JIT compiling+
I saw same method are compiled more than once from JIT log, like the method caused our problem, it only happens on 2nd time compilation.
Is it that JIT compiler will recompile the method with deeper optimization level some time, and more optimization it uses, the more memory it requires?
(like gcc has -O2 -O3)
Thanks!
NeoToday, one JVM crashed again, with
{Heap before gc invocations=4188:
PSYoungGen total 540864K, used 487958K [0x8ba50000, 0xb1250000, 0xb1250000)
eden space 467328K, 100% used [0x8ba50000,0xa82b0000,0xa82b0000)
from space 73536K, 28% used [0xa82b0000,0xa96d5850,0xaca80000)
to space 71680K, 0% used [0xacc50000,0xacc50000,0xb1250000)
ParOldGen total 1024000K, used 493477K [0x4d250000, 0x8ba50000, 0x8ba50000)
object space 1024000K, 48% used [0x4d250000,0x6b439610,0x8ba50000)
PSPermGen total 101760K, used 101240K [0x2d250000, 0x335b0000, 0x4d250000)
object space 101760K, 99% used [0x2d250000,0x3352e1f8,0x335b0000)
405290.711: [GC [PSYoungGen: 487958K->17397K(544192K)] 981435K->518248K(1568192K), 0.0769350 secs]
Heap after gc invocations=4188:
PSYoungGen total 544192K, used 17397K [0x8ba50000, 0xb1250000, 0xb1250000)
eden space 472512K, 0% used [0x8ba50000,0x8ba50000,0xa87c0000)
from space 71680K, 24% used [0xacc50000,0xadd4d480,0xb1250000)
to space 70208K, 0% used [0xa87c0000,0xa87c0000,0xacc50000)
ParOldGen total 1024000K, used 500851K [0x4d250000, 0x8ba50000, 0x8ba50000)
object space 1024000K, 48% used [0x4d250000,0x6bb6cf70,0x8ba50000)
PSPermGen total 101760K, used 101240K [0x2d250000, 0x335b0000, 0x4d250000)
object space 101760K, 99% used [0x2d250000,0x3352e1f8,0x335b0000)
Total time for which application threads were stopped: 0.0779620 seconds
Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 4522768 bytes for Chunk::new. Out of swap space? -
JDB crashes while lodading core dump or throws errors
I'm having here RHEL 5.1 system and some core dumps generated by our 32 bit java application. This application uses JNI and for some reason (that I'm still trying to figure out) is crashing while executing JNI calls.
I can load the core dumps inside gdb and even backtrace the native stack of the program and see the variables inside the shared library where it's crashing. But I need to take a closer look on the java stack and some of the variable values on the java side also.
I did try to use JDB to do this like it's described in the Troubleshooting guide for Java 6 but for some unknown reason when I do try to attach JDB to the core dumps I either get some error thrown on the console or JDB crashes and produces a hs_err_pit*.log file. I have used this technique already few times and didn't had any problems with JDB. I'm wondering if it could be that the core dump files are too big - some of them are around 3 Gb. But one of them is ~600Mb and also is crashing. So is ti possible bug in JDB or I'm missing something. I'm posting the error and the one of the hs_err_pid files.
Appreciate any help and ideas
$ jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=/usr/java/jdk/bin/java,core=core.31919
java.io.IOException
at sun.jvm.hotspot.jdi.SACoreAttachingConnector.attach(SACoreAttachingConnector.java:123)
at com.sun.tools.example.debug.tty.VMConnection.attachTarget(VMConnection.java:358)
at com.sun.tools.example.debug.tty.VMConnection.open(VMConnection.java:168)
at com.sun.tools.example.debug.tty.Env.init(Env.java:64)
at com.sun.tools.example.debug.tty.TTY.main(TTY.java:1010)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.jvm.hotspot.jdi.SACoreAttachingConnector.createVirtualMachine(SACoreAttachingConnector.java:80)
at sun.jvm.hotspot.jdi.SACoreAttachingConnector.attach(SACoreAttachingConnector.java:108)
... 4 more
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the core file
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:258)
at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:625)
at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:322)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:297)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:157)
at sun.jvm.hotspot.jdi.VirtualMachineImpl.createVirtualMachineForCorefile(VirtualMachineImpl.java:190)
... 10 more
Fatal error:
Unable to attach to target VM.
# An unexpected error has been detected by Java Runtime Environment:
# SIGSEGV (0xb) at pc=0xafcfa6bd, pid=3653, tid=4160613264
# Java VM: Java HotSpot(TM) Server VM (1.6.0_03-b05 mixed mode)
# Problematic frame:
# C [libsaproc.so+0x36bd]
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
--------------- T H R E A D ---------------
Current thread (0x08058000): JavaThread "main" [_thread_in_native, id=3654]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000151
Registers:
EAX=0xf7ffdabc, EBX=0xafcff25c, ECX=0xafd72d08, EDX=0x00000149
ESP=0xf7fdc520, EBP=0xf7fdc538, ESI=0xafd72d08, EDI=0x08058000
EIP=0xafcfa6bd, CR2=0x00000151, EFLAGS=0x00010206
Top of Stack: (sp=0xf7fdc520)
0xf7fdc520: 464c457f 00010101 000000a2 0000009c
0xf7fdc530: 0000009f afcff25c f7fdc588 afcfb01a
0xf7fdc540: afd67780 f7ffdabc 00200034 afcfaff4
0xf7fdc550: 004a004b 00000000 00200034 0028000a
0xf7fdc560: 004a004b afba13e0 00001110 afcff25c
0xf7fdc570: afb8e2e8 08058000 f7fdc5a8 afcf93b3
0xf7fdc580: 00000004 afcff25c f7fdc5a8 afcf9964
0xf7fdc590: afd67780 f7ffdabc f7fdd72c 00000004
Instructions: (pc=0xafcfa6bd)
0xafcfa6ad: 8d 14 85 00 00 00 00 8b 41 24 8b 14 10 8b 45 0c
0xafcfa6bd: 3b 42 08 72 08 8b 45 f8 89 45 f4 eb b1 8b 45 f8
Stack: [0xf7f8e000,0xf7fdf000), sp=0xf7fdc520, free space=313k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libsaproc.so+0x36bd]
C [libsaproc.so+0x401a]
C [libsaproc.so+0x2964] ps_pdread+0x1c
C [libsaproc.so+0x4d3e]
C [libsaproc.so+0x4ffd] Pgrab_core+0x281
C [libsaproc.so+0x56c7] Java_sun_jvm_hotspot_debugger_linux_LinuxDebuggerLocal_attach0__Ljava_lang_String_2Ljava_lang_String_2+0x99
j sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Ljava/lang/String;Ljava/lang/String;)V+0
j sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(Ljava/lang/String;Ljava/lang/String;)V+29
j sun.jvm.hotspot.HotSpotAgent.attachDebugger()V+43
j sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux()V+122
j sun.jvm.hotspot.HotSpotAgent.setupDebugger()V+95
j sun.jvm.hotspot.HotSpotAgent.go()V+1
j sun.jvm.hotspot.HotSpotAgent.attach(Ljava/lang/String;Ljava/lang/String;)V+56
j sun.jvm.hotspot.jdi.VirtualMachineImpl.createVirtualMachineForCorefile(Lcom/sun/jdi/VirtualMachineManager;Ljava/lang/String;Ljava/lang/String;I)Lsun/jvm/hotspot/jdi/VirtualMachineImpl;+58
v ~StubRoutines::call_stub
V [libjvm.so+0x2c5ecd]
V [libjvm.so+0x4523b8]
V [libjvm.so+0x2c5d60]
V [libjvm.so+0x49a08f]
V [libjvm.so+0x49ca8c]
V [libjvm.so+0x331e88]
C [libjava.so+0x15224] Java_sun_reflect_NativeMethodAccessorImpl_invoke0+0x34
j sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0
j sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+87
j sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+6
j java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+161
j sun.jvm.hotspot.jdi.SACoreAttachingConnector.createVirtualMachine(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;)Lcom/sun/jdi/VirtualMachine;+122
j sun.jvm.hotspot.jdi.SACoreAttachingConnector.attach(Ljava/util/Map;)Lcom/sun/jdi/VirtualMachine;+90
j com.sun.tools.example.debug.tty.VMConnection.attachTarget()Lcom/sun/jdi/VirtualMachine;+13
j com.sun.tools.example.debug.tty.VMConnection.open()Lcom/sun/jdi/VirtualMachine;+33
j com.sun.tools.example.debug.tty.Env.init(Ljava/lang/String;ZI)V+28
j com.sun.tools.example.debug.tty.TTY.main([Ljava/lang/String;)V+1207
v ~StubRoutines::call_stub
V [libjvm.so+0x2c5ecd]
V [libjvm.so+0x4523b8]
V [libjvm.so+0x2c5d60]
V [libjvm.so+0x2ef186]
V [libjvm.so+0x2e082b]
C [jdb+0x1af8] JavaMain+0x2c8
C [libpthread.so.0+0x543b]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Ljava/lang/String;Ljava/lang/String;)V+0
j sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(Ljava/lang/String;Ljava/lang/String;)V+29
j sun.jvm.hotspot.HotSpotAgent.attachDebugger()V+43
j sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux()V+122
j sun.jvm.hotspot.HotSpotAgent.setupDebugger()V+95
j sun.jvm.hotspot.HotSpotAgent.go()V+1
j sun.jvm.hotspot.HotSpotAgent.attach(Ljava/lang/String;Ljava/lang/String;)V+56
j sun.jvm.hotspot.jdi.VirtualMachineImpl.createVirtualMachineForCorefile(Lcom/sun/jdi/VirtualMachineManager;Ljava/lang/String;Ljava/lang/String;I)Lsun/jvm/hotspot/jdi/VirtualMachineImpl;+58
v ~StubRoutines::call_stub
j sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+0
j sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+87
j sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+6
j java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+161
j sun.jvm.hotspot.jdi.SACoreAttachingConnector.createVirtualMachine(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;)Lcom/sun/jdi/VirtualMachine;+122
j sun.jvm.hotspot.jdi.SACoreAttachingConnector.attach(Ljava/util/Map;)Lcom/sun/jdi/VirtualMachine;+90
j com.sun.tools.example.debug.tty.VMConnection.attachTarget()Lcom/sun/jdi/VirtualMachine;+13
j com.sun.tools.example.debug.tty.VMConnection.open()Lcom/sun/jdi/VirtualMachine;+33
j com.sun.tools.example.debug.tty.Env.init(Ljava/lang/String;ZI)V+28
j com.sun.tools.example.debug.tty.TTY.main([Ljava/lang/String;)V+1207
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0xafd08c00 JavaThread "Thread-1" daemon [_thread_blocked, id=3682]
0x0819b000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3680]
0x08199400 JavaThread "CompilerThread1" daemon [_thread_blocked, id=3679]
0x08198000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=3678]
0x08196c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3677]
0x08183c00 JavaThread "Finalizer" daemon [_thread_blocked, id=3676]
0x08183400 JavaThread "Reference Handler" daemon [_thread_blocked, id=3675]
=>0x08058000 JavaThread "main" [_thread_in_native, id=3654]
Other Threads:
0x08180800 VMThread [id=3671]
0x0819c800 WatcherThread [id=3681]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGen total 9280K, used 2718K [0xedca0000, 0xee6f0000, 0xf4e60000)
eden space 8000K, 33% used [0xedca0000,0xedf47b40,0xee470000)
from space 1280K, 0% used [0xee5b0000,0xee5b0000,0xee6f0000)
to space 1280K, 0% used [0xee470000,0xee470000,0xee5b0000)
PSOldGen total 84992K, used 0K [0xb4e60000, 0xba160000, 0xedca0000)
object space 84992K, 0% used [0xb4e60000,0xb4e60000,0xba160000)
PSPermGen total 16384K, used 3088K [0xb0e60000, 0xb1e60000, 0xb4e60000)
object space 16384K, 18% used [0xb0e60000,0xb1164398,0xb1e60000)
Dynamic libraries:
00b31000-00b4a000 r-xp 00000000 fd:00 5932045 /lib/ld-2.5.so
00b4a000-00b4b000 r-xp 00019000 fd:00 5932045 /lib/ld-2.5.so
00b4b000-00b4c000 rwxp 0001a000 fd:00 5932045 /lib/ld-2.5.so
00b4e000-00c88000 r-xp 00000000 fd:00 5932057 /lib/libc-2.5.so
00c88000-00c8a000 r-xp 00139000 fd:00 5932057 /lib/libc-2.5.so
00c8a000-00c8b000 rwxp 0013b000 fd:00 5932057 /lib/libc-2.5.so
00c8b000-00c8e000 rwxp 00c8b000 00:00 0
00c90000-00ca3000 r-xp 00000000 fd:00 5932059 /lib/libpthread-2.5.so
00ca3000-00ca4000 r-xp 00012000 fd:00 5932059 /lib/libpthread-2.5.so
00ca4000-00ca5000 rwxp 00013000 fd:00 5932059 /lib/libpthread-2.5.so
00ca5000-00ca7000 rwxp 00ca5000 00:00 0
00ca9000-00cab000 r-xp 00000000 fd:00 5932058 /lib/libdl-2.5.so
00cab000-00cac000 r-xp 00001000 fd:00 5932058 /lib/libdl-2.5.so
00cac000-00cad000 rwxp 00002000 fd:00 5932058 /lib/libdl-2.5.so
00caf000-00cb6000 r-xp 00000000 fd:00 5932062 /lib/librt-2.5.so
00cb6000-00cb7000 r-xp 00006000 fd:00 5932062 /lib/librt-2.5.so
00cb7000-00cb8000 rwxp 00007000 fd:00 5932062 /lib/librt-2.5.so
00d9b000-00dc0000 r-xp 00000000 fd:00 5932060 /lib/libm-2.5.so
00dc0000-00dc1000 r-xp 00024000 fd:00 5932060 /lib/libm-2.5.so
00dc1000-00dc2000 rwxp 00025000 fd:00 5932060 /lib/libm-2.5.so
06000000-065a0000 r-xp 00000000 fd:00 16777229 /usr/java/jdk1.6.0_03/jre/lib/i386/server/libjvm.so
065a0000-065db000 rwxp 005a0000 fd:00 16777229 /usr/java/jdk1.6.0_03/jre/lib/i386/server/libjvm.so
065db000-069fc000 rwxp 065db000 00:00 0
08048000-08052000 r-xp 00000000 fd:00 16810695 /usr/java/jdk1.6.0_03/bin/jdb
08052000-08053000 rwxp 00009000 fd:00 16810695 /usr/java/jdk1.6.0_03/bin/jdb
08053000-08289000 rwxp 08053000 00:00 0 [heap]
afa00000-afa39000 rwxp afa00000 00:00 0
afa39000-afb00000 ---p afa39000 00:00 0
afb00000-afbf2000 rwxp afb00000 00:00 0
afbf2000-afc00000 ---p afbf2000 00:00 0
afcf7000-afcff000 r-xp 00000000 fd:00 16777232 /usr/java/jdk1.6.0_03/jre/lib/i386/libsaproc.so
afcff000-afd00000 rwxp 00007000 fd:00 16777232 /usr/java/jdk1.6.0_03/jre/lib/i386/libsaproc.so
afd00000-afe00000 rwxp afd00000 00:00 0
afe60000-afe63000 ---p afe60000 00:00 0
afe63000-afeb1000 rwxp afe63000 00:00 0
afeb1000-afeb6000 r-xp 00000000 fd:00 5931062 /lib/libthread_db-1.0.so
afeb6000-afeb7000 r-xp 00004000 fd:00 5931062 /lib/libthread_db-1.0.so
afeb7000-afeb8000 rwxp 00005000 fd:00 5931062 /lib/libthread_db-1.0.so
afecb000-afede000 r-xp 00000000 fd:00 16777249 /usr/java/jdk1.6.0_03/jre/lib/i386/libnet.so
afede000-afedf000 rwxp 00013000 fd:00 16777249 /usr/java/jdk1.6.0_03/jre/lib/i386/libnet.so
afedf000-afefe000 r-xs 00152000 fd:00 16810648 /usr/java/jdk1.6.0_03/lib/sa-jdi.jar
afefe000-aff5b000 r-xs 00b3c000 fd:00 16811549 /usr/java/jdk1.6.0_03/lib/tools.jar
aff5b000-aff5c000 ---p aff5b000 00:00 0
aff5c000-affdc000 rwxp aff5c000 00:00 0
affdc000-affdf000 ---p affdc000 00:00 0
affdf000-b002d000 rwxp affdf000 00:00 0
b002d000-b0030000 ---p b002d000 00:00 0
b0030000-b00ae000 rwxp b0030000 00:00 0
b00ae000-b00b1000 ---p b00ae000 00:00 0
b00b1000-b012f000 rwxp b00b1000 00:00 0
b012f000-b0132000 ---p b012f000 00:00 0
b0132000-b0180000 rwxp b0132000 00:00 0
b0180000-b0380000 r-xp 00000000 fd:00 15535906 /usr/lib/locale/locale-archive
b0380000-b0383000 ---p b0380000 00:00 0
b0383000-b03d1000 rwxp b0383000 00:00 0
b03d1000-b03d4000 ---p b03d1000 00:00 0
b03d4000-b0422000 rwxp b03d4000 00:00 0
b0422000-b0423000 ---p b0422000 00:00 0
b0423000-b04d3000 rwxp b0423000 00:00 0
b04d3000-b064f000 r-xs 02c8f000 fd:00 16777319 /usr/java/jdk1.6.0_03/jre/lib/rt.jar
b064f000-b0650000 ---p b064f000 00:00 0
b0650000-b06d0000 rwxp b0650000 00:00 0
b06d0000-b06d1000 ---p b06d0000 00:00 0
b06d1000-b0751000 rwxp b06d1000 00:00 0
b0751000-b0752000 ---p b0751000 00:00 0
b0752000-b07d2000 rwxp b0752000 00:00 0
b07d2000-b07d3000 ---p b07d2000 00:00 0
b07d3000-b0853000 rwxp b07d3000 00:00 0
b0853000-b0854000 ---p b0853000 00:00 0
b0854000-b08d4000 rwxp b0854000 00:00 0
b08d4000-b08d5000 ---p b08d4000 00:00 0
b08d5000-b0955000 rwxp b08d5000 00:00 0
b0955000-b0956000 ---p b0955000 00:00 0
b0956000-b09d6000 rwxp b0956000 00:00 0
b09d6000-b09d7000 ---p b09d6000 00:00 0
b09d7000-b0a5f000 rwxp b09d7000 00:00 0
b0a5f000-b0a77000 rwxp b0a5f000 00:00 0
b0a77000-b0aa1000 rwxp b0a77000 00:00 0
b0aa1000-b0c3f000 rwxp b0aa1000 00:00 0
b0c3f000-b0c47000 rwxp b0c3f000 00:00 0
b0c47000-b0c5f000 rwxp b0c47000 00:00 0
b0c5f000-b0c89000 rwxp b0c5f000 00:00 0
b0c89000-b0e26000 rwxp b0c89000 00:00 0
b0e26000-b0e2c000 rwxp b0e26000 00:00 0
b0e2c000-b0e5f000 rwxp b0e2c000 00:00 0
b0e5f000-b1e60000 rwxp b0e5f000 00:00 0
b1e60000-b4e60000 rwxp b1e60000 00:00 0
b4e60000-ba160000 rwxp b4e60000 00:00 0
ba160000-edca0000 rwxp ba160000 00:00 0
edca0000-ee6f0000 rwxp edca0000 00:00 0
ee6f0000-f4e60000 rwxp ee6f0000 00:00 0
f4e63000-f4e6c000 rwxp f4e63000 00:00 0
f4e6c000-f4f23000 rwxp f4e6c000 00:00 0
f4f23000-f5163000 rwxp f4f23000 00:00 0
f5163000-f7f23000 rwxp f5163000 00:00 0
f7f23000-f7f32000 r-xp 00000000 fd:00 16777245 /usr/java/jdk1.6.0_03/jre/lib/i386/libzip.so
f7f32000-f7f34000 rwxp 0000e000 fd:00 16777245 /usr/java/jdk1.6.0_03/jre/lib/i386/libzip.so
f7f34000-f7f57000 r-xp 00000000 fd:00 16777241 /usr/java/jdk1.6.0_03/jre/lib/i386/libjava.so
f7f57000-f7f59000 rwxp 00023000 fd:00 16777241 /usr/java/jdk1.6.0_03/jre/lib/i386/libjava.so
f7f59000-f7f62000 r-xp 00000000 fd:00 5931048 /lib/libnss_files-2.5.so
f7f62000-f7f63000 r-xp 00008000 fd:00 5931048 /lib/libnss_files-2.5.so
f7f63000-f7f64000 rwxp 00009000 fd:00 5931048 /lib/libnss_files-2.5.so
f7f64000-f7f77000 r-xp 00000000 fd:00 5931145 /lib/libnsl-2.5.so
f7f77000-f7f78000 r-xp 00012000 fd:00 5931145 /lib/libnsl-2.5.so
f7f78000-f7f79000 rwxp 00013000 fd:00 5931145 /lib/libnsl-2.5.so
f7f79000-f7f7b000 rwxp f7f79000 00:00 0
f7f82000-f7f8d000 r-xp 00000000 fd:00 16777240 /usr/java/jdk1.6.0_03/jre/lib/i386/libverify.so
f7f8d000-f7f8e000 rwxp 0000b000 fd:00 16777240 /usr/java/jdk1.6.0_03/jre/lib/i386/libverify.so
f7f8e000-f7f91000 ---p f7f8e000 00:00 0
f7f91000-f7fe1000 rwxp f7f91000 00:00 0
f7fe1000-f7fe8000 r-xp 00000000 fd:00 16777243 /usr/java/jdk1.6.0_03/jre/lib/i386/jli/libjli.so
f7fe8000-f7fea000 rwxp 00006000 fd:00 16777243 /usr/java/jdk1.6.0_03/jre/lib/i386/jli/libjli.so
f7fec000-f7ff4000 rwxs 00000000 fd:00 19562502 /tmp/hsperfdata_omnitrak/3653
f7ff4000-f7ffa000 r-xp 00000000 fd:00 16777227 /usr/java/jdk1.6.0_03/jre/lib/i386/native_threads/libhpi.so
f7ffa000-f7ffb000 rwxp 00006000 fd:00 16777227 /usr/java/jdk1.6.0_03/jre/lib/i386/native_threads/libhpi.so
f7ffb000-f7ffc000 rwxp f7ffb000 00:00 0
f7ffc000-f7ffd000 r-xp f7ffc000 00:00 0
f7ffd000-f7ffe000 rwxp f7ffd000 00:00 0
ff9fb000-ffa03000 rwxp ff9fb000 00:00 0 [stack]
ffffe000-fffff000 r-xp ffffe000 00:00 0
VM Arguments:
jvm_args: -Dapplication.home=/usr/java/jdk1.6.0_03
java_command: com.sun.tools.example.debug.tty.TTY -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=/usr/java/jdk/bin/java,core=core.25602
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/java/jdk/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin
LD_LIBRARY_PATH=/usr/java/jdk1.6.0_03/jre/lib/i386/server:/usr/java/jdk1.6.0_03/jre/lib/i386:/usr/java/jdk1.6.0_03/jre/../lib/i386
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x53c560], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x53c560], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x451a50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x451a50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x451a50], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x453a80], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x4534a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x4534a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x4534a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x4534a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR2: [libjvm.so+0x453a80], sa_mask[0]=0x00000000, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Client release 5.1 (Tikanga)
uname:Linux 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 53248, NOFILE 1024, AS infinity
load average:0.07 0.22 0.42
CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 7 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 6113940k(245496k free), swap 2097144k(2097144k free)
vm_info: Java HotSpot(TM) Server VM (1.6.0_03-b05) for linux-x86, built on Sep 24 2007 22:32:39 by "java_re" with gcc 3.2.1-7a (J2SE release)Thank you for your responds!
I'm not trying at all to transfer the core to a different system. I'm debugging on the same linux red hat box where the cores where produced and I'm using the JDB version from the same JDK that was used to run the app.
I'm wondering what makes you think that I'm possibly mixing core files?
Here is the output that JDB generates when I define LIBSAPROC_DEBUG=1. This is the case when JDB crashes:
$ jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=/usr/java/jdk/bin/java,core=core.23850
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24310
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x0
libsaproc DEBUG: ebx = 0x5d2a
libsaproc DEBUG: ecx = 0x5ef6
libsaproc DEBUG: edx = 0x6
libsaproc DEBUG: esp = 0x324fd0e4
libsaproc DEBUG: ebp = 0x324fd0e4
libsaproc DEBUG: esi = 0x324fd19c
libsaproc DEBUG: edi = 0x78eff4
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 3 and n_descsz = 124
libsaproc DEBUG: Note header with n_type = 6 and n_descsz = 144
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 27070
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffe00
libsaproc DEBUG: ebx = 0xa
libsaproc DEBUG: ecx = 0x312edad8
libsaproc DEBUG: edx = 0x33f4d1a4
libsaproc DEBUG: esp = 0x312edac0
libsaproc DEBUG: ebp = 0x312edac0
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x2000
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 27063
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffe00
libsaproc DEBUG: ebx = 0xa
libsaproc DEBUG: ecx = 0x334fccd8
libsaproc DEBUG: edx = 0x33f4d1a4
libsaproc DEBUG: esp = 0x334fccc0
libsaproc DEBUG: ebp = 0x334fccc0
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x2000
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 1754
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x32baab38
libsaproc DEBUG: ecx = 0x1
libsaproc DEBUG: edx = 0x6ddd00
libsaproc DEBUG: esp = 0x32baaaf0
libsaproc DEBUG: ebp = 0x32baaaf0
libsaproc DEBUG: esi = 0x1eed0
libsaproc DEBUG: edi = 0x78eff4
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24325
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x834486c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x306bf
libsaproc DEBUG: esp = 0x32bfdda8
libsaproc DEBUG: ebp = 0x32bfdda8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x306bf
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24319
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x831e8cc
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x2d49f7
libsaproc DEBUG: esp = 0x33259cb0
libsaproc DEBUG: ebp = 0x33259cb0
libsaproc DEBUG: esi = 0x33259cc4
libsaproc DEBUG: edi = 0x2d49f7
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24318
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x83783a4
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x38dd3
libsaproc DEBUG: esp = 0x3330ab30
libsaproc DEBUG: ebp = 0x3330ab30
libsaproc DEBUG: esi = 0x3330ab44
libsaproc DEBUG: edi = 0x38dd3
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24317
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x1
libsaproc DEBUG: ebx = 0x827bb60
libsaproc DEBUG: ecx = 0x1
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x3335bb0c
libsaproc DEBUG: ebp = 0x3335bb0c
libsaproc DEBUG: esi = 0x3335bbb8
libsaproc DEBUG: edi = 0x827bb60
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24316
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8082d14
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x33d5ca98
libsaproc DEBUG: ebp = 0x33d5ca98
libsaproc DEBUG: esi = 0x33d5caac
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24315
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8378204
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x334adac8
libsaproc DEBUG: ebp = 0x334adac8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24314
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x19
libsaproc DEBUG: ebx = 0x33f164cc
libsaproc DEBUG: ecx = 0x9b
libsaproc DEBUG: edx = 0x39f99940
libsaproc DEBUG: esp = 0x320f9a14
libsaproc DEBUG: ebp = 0x1e
libsaproc DEBUG: esi = 0x2
libsaproc DEBUG: edi = 0x39f975a0
libsaproc DEBUG: eip = 0x33f0e054
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24313
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x39c534e0
libsaproc DEBUG: ebx = 0x33f164cc
libsaproc DEBUG: ecx = 0x321fabba
libsaproc DEBUG: edx = 0x6
libsaproc DEBUG: esp = 0x321faa70
libsaproc DEBUG: ebp = 0x321faa7c
libsaproc DEBUG: esi = 0x2
libsaproc DEBUG: edi = 0x39c53478
libsaproc DEBUG: eip = 0x33f1028e
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24312
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x29
libsaproc DEBUG: ebx = 0x39bd4780
libsaproc DEBUG: ecx = 0x1
libsaproc DEBUG: edx = 0x2400d1a2
libsaproc DEBUG: esp = 0x322fbc50
libsaproc DEBUG: ebp = 0x322fbc7c
libsaproc DEBUG: esi = 0x33f167a8
libsaproc DEBUG: edi = 0xf9000607
libsaproc DEBUG: eip = 0x33f0ef54
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24311
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x827ac04
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x137ee7
libsaproc DEBUG: esp = 0x323fcd28
libsaproc DEBUG: ebp = 0x323fcd28
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x137ee7
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24309
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x6
libsaproc DEBUG: ebx = 0x3a5c0970
libsaproc DEBUG: ecx = 0x2
libsaproc DEBUG: edx = 0xb
libsaproc DEBUG: esp = 0x325febd0
libsaproc DEBUG: ebp = 0x3a5c1cdc
libsaproc DEBUG: esi = 0x6
libsaproc DEBUG: edi = 0x4
libsaproc DEBUG: eip = 0x33f0ef06
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24308
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x5
libsaproc DEBUG: ebx = 0x3a98cb48
libsaproc DEBUG: ecx = 0x2
libsaproc DEBUG: edx = 0x9
libsaproc DEBUG: esp = 0x32cfea50
libsaproc DEBUG: ebp = 0x3a98dde8
libsaproc DEBUG: esi = 0x5
libsaproc DEBUG: edi = 0x5
libsaproc DEBUG: eip = 0x33f0f084
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24307
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0x8
libsaproc DEBUG: ebx = 0x3a1fd104
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x6
libsaproc DEBUG: esp = 0x3345c984
libsaproc DEBUG: ebp = 0x3345c984
libsaproc DEBUG: esi = 0x29
libsaproc DEBUG: edi = 0x0
libsaproc DEBUG: eip = 0x33f0f210
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24304
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x82ee0ac
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x332aad90
libsaproc DEBUG: ebp = 0x332aad90
libsaproc DEBUG: esi = 0x332aada4
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24015
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x856427c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x33b5ca90
libsaproc DEBUG: ebp = 0x33b5ca90
libsaproc DEBUG: esi = 0x33b5caa4
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24014
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x818b8e4
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x3
libsaproc DEBUG: esp = 0x33bada98
libsaproc DEBUG: ebp = 0x33bada98
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x3
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24013
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffe00
libsaproc DEBUG: ebx = 0x5
libsaproc DEBUG: ecx = 0x33bfedd0
libsaproc DEBUG: edx = 0x33f4d1a4
libsaproc DEBUG: esp = 0x33bfedb8
libsaproc DEBUG: ebp = 0x33bfedb8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x827e400
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24011
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x828951c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x33dadcd8
libsaproc DEBUG: ebp = 0x33dadcd8
libsaproc DEBUG: esi = 0x33dadcec
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 24010
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x849caac
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x33dfec68
libsaproc DEBUG: ebp = 0x33dfec68
libsaproc DEBUG: esi = 0x33dfec7c
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23868
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x80c213c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x34090f78
libsaproc DEBUG: ebp = 0x34090f78
libsaproc DEBUG: esi = 0x34090f8c
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23867
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x819e294
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0x340e2018
libsaproc DEBUG: ebp = 0x340e2018
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23866
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8199a5c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x434
libsaproc DEBUG: esp = 0x34162e48
libsaproc DEBUG: ebp = 0x34162e48
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x434
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23865
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8199a5c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x432
libsaproc DEBUG: esp = 0x341e3ec8
libsaproc DEBUG: ebp = 0x341e3ec8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x432
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23864
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x65e80a0
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x0
libsaproc DEBUG: esp = 0x34234fb0
libsaproc DEBUG: ebp = 0x34234fb0
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0xffc
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23863
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x818ba8c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x289
libsaproc DEBUG: esp = 0x34485de0
libsaproc DEBUG: ebp = 0x34485de0
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x289
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23862
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x81887d4
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x2b3
libsaproc DEBUG: esp = 0x344d6c90
libsaproc DEBUG: ebp = 0x344d6c90
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x2b3
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23861
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x31e0548c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x3
libsaproc DEBUG: esp = 0x345580d8
libsaproc DEBUG: ebp = 0x345580d8
libsaproc DEBUG: esi = 0x345580ec
libsaproc DEBUG: edi = 0x3
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23860
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e9b
libsaproc DEBUG: esp = 0x34784f68
libsaproc DEBUG: ebp = 0x34784f68
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e9b
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23859
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e9a
libsaproc DEBUG: esp = 0x34805fe8
libsaproc DEBUG: ebp = 0x34805fe8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e9a
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23858
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e9c
libsaproc DEBUG: esp = 0x34886e68
libsaproc DEBUG: ebp = 0x34886e68
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e9c
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23857
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e99
libsaproc DEBUG: esp = 0x34907ee8
libsaproc DEBUG: ebp = 0x34907ee8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e99
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23856
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e98
libsaproc DEBUG: esp = 0x34989168
libsaproc DEBUG: ebp = 0x34989168
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e98
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23855
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e97
libsaproc DEBUG: esp = 0x34a0a1e8
libsaproc DEBUG: ebp = 0x34a0a1e8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e97
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23854
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e96
libsaproc DEBUG: esp = 0x34a8b068
libsaproc DEBUG: ebp = 0x34a8b068
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e96
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23853
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x8096674
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x161e95
libsaproc DEBUG: esp = 0x34b0c0e8
libsaproc DEBUG: ebp = 0x34b0c0e8
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x161e95
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23852
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0x33e5e19c
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x1
libsaproc DEBUG: esp = 0xf7fde228
libsaproc DEBUG: ebp = 0xf7fde228
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x1
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: Note header with n_type = 1 and n_descsz = 144
libsaproc DEBUG: got integer regset for lwp 23850
libsaproc DEBUG: integer regset
libsaproc DEBUG: eax = 0xfffffffc
libsaproc DEBUG: ebx = 0xf7fdebd8
libsaproc DEBUG: ecx = 0x0
libsaproc DEBUG: edx = 0x5d2c
libsaproc DEBUG: esp = 0xffb93630
libsaproc DEBUG: ebp = 0xffb93630
libsaproc DEBUG: esi = 0x0
libsaproc DEBUG: edi = 0x7a8ff4
libsaproc DEBUG: eip = 0xffffe410
libsaproc DEBUG: Note header with n_type = 2 and n_descsz = 108
libsaproc DEBUG: Note header with n_type = 1189489535 and n_descsz = 512
libsaproc DEBUG: ELF interpreter /lib/ld-linux.so.2
libsaproc DEBUG: address of _DYNAMIC is 0x8052548
libsaproc DEBUG: ---- sorted virtual address map ----
libsaproc DEBUG: base = 0x64f000 size = 4096
libsaproc DEBUG: base = 0x650000 size = 4096
libsaproc DEBUG: base = 0x78d000 size = 8192
libsaproc DEBUG: base = 0x78f000 size = 4096
libsaproc DEBUG: base = 0x790000 size = 12288
libsaproc DEBUG: base = 0x7a8000 size = 4096
libsaproc DEBUG: base = 0x7a9000 size = 4096
libsaproc DEBUG: base = 0x7aa000 size = 8192
libsaproc DEBUG: base = 0x7b0000 size = 4096
libsaproc DEBUG: base = 0x7b1000 size = 4096
libsaproc DEBUG: base = 0x879000 size = 4096
libsaproc DEBUG: base = 0x87a000 size = 4096
libsaproc DEBUG: base = 0x884000 size = 4096
libsaproc DEBUG: base = 0x885000 size = 4096
libsaproc DEBUG: base = 0xc37000 size = 16384
libsaproc DEBUG: base = 0xc3b000 size = 4096
libsaproc DEBUG: base = 0xc3c000 size = 24576
libsaproc DEBUG: base = 0xd38000 size = 4096
libsaproc DEBUG: base = 0x6000000 size = 5898240
libsaproc DEBUG: base = 0x65a0000 size = 241664
libsaproc DEBUG: base = 0x65db000 size = 4329472
libsaproc DEBUG: base = 0x8048000 size = 38100
libsaproc DEBUG: base = 0x8052000 size = 4096
libsaproc DEBUG: base = 0x8053000 size = 14032896
libsaproc DEBUG: base = 0x30f00000 size = 1007616
libsaproc DEBUG: base = 0x31100000 size = 1048576
libsaproc DEBUG: base = 0x312a0000 size = 12288
libsaproc DEBUG: base = 0x312a3000 size = 319488
libsaproc DEBUG: base = 0x3133a000 size = 16384
libsaproc DEBUG: base = 0x31344000 size = 753664
libsaproc DEBUG: base = 0x313fc000 size = 16384
libsaproc DEBUG: base = 0x31400000 size = 929792
libsaproc DEBUG: base = 0x31500000 size = 1028096
libsaproc DEBUG: base = 0x31600000 size = 966656
libsaproc DEBUG: base = 0x31700000 size = 1028096
libsaproc DEBUG: base = 0x31800000 size = 991232
libsaproc DEBUG: base = 0x31900000 size = 1044480
libsaproc DEBUG: base = 0x31a00000 size = 1032192
libsaproc DEBUG: base = 0x31b00000 size = 1032192
libsaproc DEBUG: base = 0x31c00000 size = 1003520
libsaproc DEBUG: base = 0x31d51000 size = 319488
libsaproc DEBUG: base = 0x31df9000 size = 28672
libsaproc DEBUG: base = 0x31e00000 size = 1048576
libsaproc DEBUG: base = 0x31f03000 size = 552960
libsaproc DEBUG: base = 0x31f8a000 size = 20480
libsaproc DEBUG: base = 0x31f90000 size = 331776
libsaproc DEBUG: base = 0x31fe1000 size = 8192
libsaproc DEBUG: base = 0x31ffa000 size = 12288
libsaproc DEBUG: base = 0x31ffd000 size = 1040384
libsaproc DEBUG: base = 0x320fb000 size = 12288
libsaproc DEBUG: base = 0x320fe000 size = 1040384
libsaproc DEBUG: base = 0x321fc000 size = 12288
libsaproc DEBUG: base = 0x321ff000 size = 1040384
libsaproc DEBUG: base = 0x322fd000 size = 12288
libsaproc DEBUG: base = 0x32300000 size = 1040384
libsaproc DEBUG: base = 0x323fe000 size = 12288
libsaproc DEBUG: base = 0x32401000 size = 1040384
libsaproc DEBUG: base = 0x324ff000 size = 12288
libsaproc DEBUG: base = 0x32502000 size = 1957888
libsaproc DEBUG: base = 0x32700000 size = 1024000
libsaproc DEBUG: base = 0x32800000 size = 2072576
libsaproc DEBUG: base = 0x32a00000 size = 1040384
libsaproc DEBUG: base -
We have a weblogic 9 cluster where a node crash/shutdown with no trace. Sometimes its node 2 and sometimes node 3, but not at the same time.
We can't find any core dump, hs_err_pid or any stracktrace in the logs or std out/err.
Specifying where hs_err files goes with -XX:ErrorFile should not be necessary as it should write in current directory or if not possible due to permissions etc, in the OS tmp dir.
Are there any Weblogic specific JVM or system property we can specify to force some kind of trace when the process just seem to crash?
We see this in the log from time to time, perhaps its related:
<2012-jun-08 kl 13:53 CEST> <Error> <Security> <BEA-090060> <The AccessDecision class "weblogic.security.providers.realmadapter.AuthorizationProviderImpl" returned an error: java.lang.SecurityException: Realm Adapter ACL Mapping Failed.>
Can the above hide a possible out of memory error trace to appear in the logs?
SERVER: Weblogic 9.2.3 with some additional patches (and compatibility mode)
OS: HP-UX B.11.23 U ia64
JAVA: Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0.20-_28_apr_2010_03_15)
Java HotSpot(TM) Server VM (build 1.5.0.20 jinteg:04.28.10-02:28 IA64, mixed mode)http://docs.oracle.com/cd/B28359_01/java.111/b31224/instclnt.htm gives some info about the used libraries (libocijdbc11.so)
Could you check the environment variable - http://docs.oracle.com/cd/B28359_01/java.111/b31224/getsta.htm#i1005378
"On Sun Solaris or Linux, set the LD_LIBRARY_PATH environment variable as follows:
ORACLE_HOME/lib
This directory contains the libocijdbc11.so shared object library.
Note:
If you are running a 32-bit Java Virtual Machine (JVM) against a 64-bit client or database, then you must also add ORACLE_HOME/lib32 to the LD_LIBRARY_PATH environment variable."
Maybe you are looking for
-
Error while loading create database charecter set
Hello, I have completed instalation Oracle 9iV2 on Redhat 7.3 without any problems. After I reboot the system, and try to start database, I have got error mesage: ORA-12709: error while loading create database character set Could anyone help? Thanks
-
Sender Communication Channel Error
Hi all, iam working on JDBC scenario.For this iam using party and business service. iam getting following error at jdbc sender communication channel. Error occurred while sending message: com.sap.aii.af.ra.ms.api.ConfigException: SLDAccess set to tru
-
Regarding Agent Rule GRAC_REQ_MITIG_MONITOR_AGENT
Can anyone help me in determining if i can make use of agent rule GRAC_REQ_MITIG_MONITOR_AGENT to look for the approver at mitigation monitor stage under process ID SAP_GRAC_ACCESS_REQUEST_HR ?? SAP has suggested that this agent rule is only specific
-
RB_SPLIT and RB_CLASSIC
Hi Experts, I am working on PI7.0. I was trying to learn more on RB_SPLIT and RB_CLASSIC. As per the blogs in SDN, it says , I can see the same in INTERFACE DETERMINATION. But honestly speaking, I cant get to see these 2 features in INTERFACE DET
-
Can I get a quick response please
When I try and open any website on my iPhone, while its loading if I hit the X or try to change the web address, SAFARI closes to Home Screen. Is this how it functions or is this related to my phone?