How can a JVM terminate with an exit code of 141 and no other diagnostics?

Question

Hello,
We are encountering a JVM process that dies with little explanation other than an exit code of 141. No hotspot error file (hs_err_*) or crash dump.  To date, the process runs anywhere from 30 minutes to 8 days before the problem occurs. The last application log entry is always the report of a lost SSL connection, the result of an thrown SSLException.  (The exception itself is unavailable at this time – the JVM dies before it is logged -- working on that.)
How can a JVM produce an exit code of 141, and nothing else?  Can anyone suggest ideas for capturing additional diagnostic information?  Any help would be greatly appreciated!  Environment and efforts to date are described below.
Thanks,
-KK
Host machine: 8x Xeon server with 256GB memory, RHEL 6 (or RHEL 5.5) 64-bit
Java: Oracle Java SE 7u21 (or 6u26)
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
JVM arguments:
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+CMSClassUnloadingEnabled
-XX:MaxPermSize=256m
-XX:NewSize=64m
-Xms128m
-Xmx1037959168
-Djava.awt.headless=true
-Djava.security.egd=file:///dev/./urandom
Diagnostics attempted to date:
LD_PRELOAD=libjsig.so.   A modified version of libjsig.so was created to report all signal handler registrations and to report SIGPIPE signals received.  (Exit code 141 could be interpreted as 128+SIGPIPE(13).)  No JNI libraries are registering any signal handlers, and no SIGPIPE signal is reported by the library for the duration of the JVM run.  Calls to ::exit() are also intercepted and reported.  No call to exit() is reported.
Inspect /var/log/messages for any indication that the OS killed the process, e.g. via the Out Of Memory (OOM) Killer.  Nothing found.
Set ‘ulimit –c unlimited’, in case the default limit of 0 (zero) was preventing a core file from being written.  Still no core dump.
‘top’ reports the VIRT size of the process can grow to 20GB or more in a matter of hours, which is unusual compared to other JVM processes.  The RES (resident set size) does not grow beyond about 375MB, however, which is an considered normal.
This JVM process creates many short-lived Thread objects by way of a thread pool, averaging 1 thread every 2 seconds, and these objects end up referenced only by a Weak reference.   The CMS collector seems lazy about collecting these, and upwards of 2000 Thread objects have been seen (in heap dumps) held only by Weak references.  (The Java heap averages about 100MB, so the collector is not under any pressure.) However, a forced collection (via jconsole) cleans out the Thread objects as expected.  Any relationship of this to the VIRT size or the JVM disappearance, however, cannot be established.
The process also uses NIO and direct buffers, and maintains a DirectByteBuffer cache. There is some DirectByteBuffer churn. MBeans report stats like:
Direct buffer pool: allocated=669 (20,824,064 bytes), released=665 (20,725,760), active=4 (98,304)  [note: equals 2x 32K buffers and 2x 16K buffers]
java.nio.BufferPool > direct: Count=18, MemoryUsed=1343568, TotalCapacity=1343568
These numbers appear normal and also do not seem to correlate with the VIRT size or the JVM disappearance.

Answer 1

True, but the JNI call would still be reported by the LD_PRELOAD intercept, unless the native code could somehow circumvent that.  Using a test similar to GoodbyeWorld (shown below), I verified that the JNI call to exit() is reported.  In the failure case, no call to exit() is reported.
Can an OS (or a manual) 'kill' specify an exit code?  Where could "141" be coming from?
Thanks,
-K2
=== GoodbyeWorldFromJNI.java ===
package com.attachmate.test;
public class GoodbyeWorldFromJNI
    public static final String LIBRARY_NAME = "goodbye";
    static {
        try {
            System.loadLibrary(LIBRARY_NAME);
        } catch (UnsatisfiedLinkError error) {
            System.err.println("Failed to load " + System.mapLibraryName(LIBRARY_NAME));
    private static native void callExit(int exitCode);
    public static void main(String[] args) {
        callExit(141);
=== goodbye.c ===
#include <stdlib.h>
#include "goodbye.h"  // javah generated header file
JNIEXPORT void JNICALL Java_com_attachmate_test_GoodbyeWorldFromJNI_callExit
  (JNIEnv *env, jclass theClass, jint exitCode)
    exit(exitCode);
=== script.sh ===
#!/bin/bash -v
uname -a
export PATH=/opt/jre1.7.0_25/bin:$PATH
java -version
pwd
LD_PRELOAD=./lib/linux-amd64/libjsigdebug.so java -classpath classes -Djava.library.path=lib/linux-amd64 com.attachmate.test.GoodbyeWorldFromJNI > stdout.txt
echo $?
tail stdout.txt
=== script output ===
[keithk@keithk-RHEL5-dev goodbyeJNI]$ ./script.sh
#!/bin/bash -v
uname -a
Linux keithk-RHEL5-dev 2.6.18-164.2.1.el5 #1 SMP Mon Sep 21 04:37:42 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
export PATH=/opt/jre1.7.0_25/bin:$PATH
java -version
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
pwd
/tmp/goodbyeJNI
LD_PRELOAD=./lib/linux-amd64/libjsigdebug.so java -classpath classes -Djava.library.path=lib/linux-amd64 com.attachmate.test.GoodbyeWorldFromJNI > stdout.txt
echo $?
141
tail stdout.txt
JSIG: exit(141) called
JSIG: Call stack has 4 frames:
JSIG: ./lib/linux-amd64/libjsigdebug.so [0x2b07dc1bdc2f]
JSIG: ./lib/linux-amd64/libjsigdebug.so(exit+0x29) [0x2b07dc1bea41]
JSIG: /tmp/goodbyeJNI/lib/linux-amd64/libgoodbye.so [0x2aaab3e82547]
JSIG: [0x2aaaab366d8e]       
=== ===