Application cause java core dump, how can i do?

Question

Hi
In the case, we're using JDK 1.4 and Hibernate on Aix 64 bit, but seems we've got lots of java core even occupied full hard space and cause application crash
system info :
JavaVersion is J2RE 1.4.2 IBM AIX 5L for PowerPC (64 bit JVM)
build caix64142ifx-20051115 (SR3 + 94164 + 97403 + 97482)
(note the 64 bit JVM).
-Xmx=1024 and -xms=512, free heap is 330,415,568 bytes, and allocated
is 1073,740,288 bytes.
some core dump :
WebContainer : 4
at java.io.ObjectInputStream.readString()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0() at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readArray()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readObject()
at java.util.HashMap.readObject()
at sun.reflect.GeneratedMethodAccessor399.invoke()
at sun.reflect.DelegatingMethodAccessorImpl.invoke()
at java.lang.reflect.Method.invoke()
at java.io.ObjectStreamClass.invokeReadObject()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.defaultReadFields()
at java.io.ObjectInputStream.readSerialData()
at java.io.ObjectInputStream.readOrdinaryObject()
at java.io.ObjectInputStream.readObject0()
at java.io.ObjectInputStream.readObject()
at com.ibm.ws.naming.util.Serialization$1.run()
at com.ibm.ws.security.util.AccessController.doPrivileged()
at com.ibm.ws.naming.util.Serialization.deserializeObject()
at com.ibm.ws.naming.util.Helpers
.processSerializedObjectForLookupExt()
at com.ibm.ws.naming.util.Helpers.processSerializedObjectForLookup()
at com.ibm.ws.naming.jndicos.CNContextImpl.processResolveResults()
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup()
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup()
at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt()
at com.ibm.ws.naming.jndicos.CNContextImpl.lookup()
at com.ibm.ws.naming.util.WsnInitCtx.lookup()
at javax.naming.InitialContext.lookup()
at com.ibm.wsspi.RegistryLoader.getPluginRegistry()
at com.ibm.wsspi.IPluginRegistryFactory.getPluginRegistry()
at ibmjsp.com2E_ibm_2E_ws_2E_console_2E_appmanagement
._collectionTableLayout._jspService()
at com.ibm.ws.jsp.runtime.HttpJspBase.service()
at javax.servlet.http.HttpServlet.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest()
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper
.handleRequest()
at com.ibm.ws.jsp.webcontainerext.JSPExtensionServletWrapper
.handleRequest()
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.include()
at org.apache.jasper.runtime.JspRuntimeLibrary.include()
at org.apache.jasper.runtime.PageContextImpl.include()
at org.apache.struts.tiles.TilesUtilImpl.doInclude()
at org.apache.struts.tiles.TilesUtil.doInclude()
at org.apache.struts.taglib.tiles.InsertTag.doInclude()
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag()
at org.apache.struts.taglib.tiles.InsertTag.doEndTag()
at ibmjsp.secure.layouts.vboxLayout._jspService()
at com.ibm.ws.jsp.runtime.HttpJspBase.service()
at javax.servlet.http.HttpServlet.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter()
... more stack lines not shown ===================================
"WebContainer: 2" shows much
the same stack trace, but extends to include
@com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState()
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter()
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter()
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter()
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest()
Through IBM, we got those answer :
Looking at the javacore files, we're seeing OOM errors. However,
looking at a heapdump, we're not seeing total heap space exhaustion -
around half of the heap is occupied, and occupancy by the vartious
objects seems, to me at least, to be normal.
The threads
seem to be using the compound class loader to load the customer's
"Hibernate" code - and in the largest of the threads in the heapdump -
(it consumes 258,720,168 bytes), the class org/hibernate/impl/Session
Impl and its children - consume 129,540,864 bytes.
The other 128,785,680 is consumed by org/hibernate/proxy/CGLIBLazyInitia
lizer. There are some other classes that make up the small difference
if you added these numbers.
The class com/softleader/hibernate/util/HibernateUtil and its children
consume 74,932,848 bytes.
I could go on through the heapdump, but there are very large classes
in here, and it may be more advantageous to get a verbosegc turned on
to show the patterns of JVM Heap requirements.
I suppose that the customer is developing this app for a 32bit
architecture market, since they are keeping their JVM Heap within the
32 bit limits. If this is the case, then Xmx=1280 and Xms=384 seems a
more reasonable JVM setting. If they are not, and they want to try
Xmx above 1280, then we have not yet had much experience with 64 bit
but I'd expect that the larger heapsize would benefit the application.
Either way, we need verbosegc turned on to monitor the heap requests.
That mean, i have to turn up JVM Heap size? or what can i do for it?
Thanks a lot.

Answer 1

I'd hook up a profiler to your app and monitor it to see what's happening and what objects aren't playing nice.