JVM Crashes With SIGSEGV Messages

Question

Hi All,
I thought that posting to this forum may be helpful as we have been trying to resolve a problem for some months that has become quite frustrating. We implemented an installation of Application Server in July last year, which was originally 9.0.3. Since then, we have been having stability problems with the OC4J application running on it, which crashes about once a week. These crashes have changed slightly in nature after application upgrades, and application server patches. We are now running the following specification:
p650 IBM Server
AIX 5.2 ML5
Oracle Application Server 9.0.4.1.0
IBM JDK 1.4.1 - version output:
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1)
Classic VM (build 1.4.1, J2RE 1.4.1 IBM AIX build ca1411ifx-20040810 (141SR3) (JIT enabled: jitc))
We recently symlinked in the native install of the JVM to the application server JDK directories, as we suspected a bug in the JVM was causing the problems.
The symptoms are that the JVM process is either restarted by opmn, or that it crashes, and is uncrecoverable. We have tried increasing the heap size (which is currently 1.5GB) but this has not seemed to help. In most crashes no out of memory errors are produced, although we have seen them in the past. Usually, opmn restarts the process itself and no manual intervention is required.
This problem could be to do with the Java code, which uses Toplink 9.0.4 to access the underlying database, however our supplier keeps proclaiming that the problem is in the IBM JDK, because their other customers aren't having the same problems on different platforms, (Windows, Linux and Solaris.)
The application server goes down for a cold backup every morning, and most days runs through without failure.
However, once or twice a week the JDK crashes and we have been unable to diagnose the fault.
The output of the ipm.log at the time looks like this:
05/04/01 15:01:56 [3] [libopmnoc4j] Could not connect to socket while pinging an OC4J process
05/04/01 15:01:56 [4] [libopmnoc4j] Process Ping Failed: OC4J~FWPROD~default_island~1 (1280573938:55396)
05/04/01 15:01:56 [4] Job 0 2683505 result:
fwprod.<edit>xxxxxxx</edit>~OC4J~FWPROD~default_island~1280573938:55396
Status: Alive
Operation: internal (Connection timed out)
ErrFile: /oracle/product/ias904/opmn/logs/OC4J~FWPROD~default_island~1
05/04/01 15:03:32 [3] [libopmnoc4j] Could not connect to socket while pinging an OC4J process
05/04/01 15:03:32 [4] [libopmnoc4j] Process Ping Failed: OC4J~FWPROD~default_island~1 (1280573938:55396)
05/04/01 15:03:32 [4] [libopmnoc4j] Process Unreachable: OC4J~FWPROD~default_island~1 (1280573938:55396)
05/04/01 15:03:32 [4] Job 0 2683505 result:
fwprod.fwprod.nottscc.gov.uk~OC4J~FWPROD~default_island~1280573938:55396
Status: Alive
Operation: internal (Connection timed out)
ErrFile: /oracle/product/ias904/opmn/logs/OC4J~FWPROD~default_island~1
05/04/01 15:03:32 [4] Restarting Process: OC4J~FWPROD~default_island~1 (1280573938:55396)
05/04/01 15:05:55 [4] Stopping Process: OC4J~FWPROD~default_island~1 (1280573938:55396)
05/04/01 15:05:55 [4] [libopmnoc4j] Forcefully Terminating Process: OC4J~FWPROD~default_island~1 (1280573938:55396)
The JVM sometimes produces thread dumps, or heap dumps, and we have added a script to do a 'kill -3' at every shutdown.
The thread dumps seem to contain SIGSEGV messages, which is what lead the developers to suggest a JDK problem.
We have been told that the JVM should not crash this way, regardless of what the code is like, but are not convinced that the problem lies in that area.
Does anyone here have exprience of running Java applications on the IBM JDK an Oracle Application Server with similar problems?

Answer 1

We seem to have tracked the problem down to inefficient use of large objects in the application. Excessive loading of large document objects was casuing lots of full garbage collects to be formed, which seemed to be destablising the JDK. The new release of the application stopped loading these large objects into memory except when they are actually being opened, which has kept our heap usage much more stable, and seems to have helped avert crashes.