This is to sum up the problem. Here are facts :

Question

This is to sum up the problem. Here are facts :
1. Java process size grows to ~ 3G and eventually exhaust native memory. Only
data segment of the process grows in size.
2. Java heap size remains stable, way under the max size. Perm generation
size is stable, way under the max size.
Here is a table showing evolution of Java process Virtual Memory
Start Monitoroing Java exited with OutOfMem
VmSize: 2764108 kB VmSize: 3094912 kB
VmLck: 0 kB VmLck: 0 kB
VmRSS: 1737364 kB VmRSS: 2020136 kB
VmData: 2461448 kB => VmData: 2792252 kB
VmStk: 216 kB VmStk: 216 kB
VmExe: 56 kB VmExe: 56 kB
VmLib: 301488 kB VmLib: 301488 kB
Plase note that Exe, Stack and Libs are the same at the beginning and the endof the period. Data segment increased for about 0.4G. The memory leak takes place in the native code.
Here is our log
     05/08/13 07:38:10 java.lang.OutOfMemoryError: unable to create new nativethread
     05/08/13 07:38:10 at java.lang.Thread.start(Native Method)
     05/08/13 07:38:10 at com.evermind.util.ReleasableResourcePooledExecutor.addThread(ReleasableResourcePooledExecutor.java:121)
     05/08/13 07:38:10 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(PooledExecutor.java:978)
     05/08/13 07:38:10 at com.evermind.util.ThreadPool.launch(ThreadPool.java:255)
     05/08/13 07:38:10 at om.evermind.server.http.AJPConnectionListener.run(AJPConnectionListener.java:74)
     05/08/13 07:38:10 at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:186)
     05/08/13 07:38:10 at java.lang.Thread.run(Thread.java:534)
     Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 612052 bytes for Chunk::new. Out of swap space?

Answer 1

100M doesn't seem too extreme ... I ran a test program that created 100 sleeping threads, and ps -l reported the process size as 77M. I don't know Linux well enough to know the threading model (I suspect it's LWP-per-thread, which means that ps -m will give correct output).
However, given what you've shown (heap doesn't grow, process map does, failure occurs in code that starts a new thread), it really seems that your problem has something to do with too many threads.
My first thought, since I've seen it before is that there may be fragmentation of the memory map, such that there's no available segment when the JVM tries to allocate the stack for a new thread. When I saw this problem, the JVM (1.3 Solaris) was using a pretty naive approach to mapping those blocks, such that a few long-lived threads would cause it to run out of space. This information is available in the /proc/maps directory (man proc for more info).
There could also be some limit to the number of threads you can create. I know that Solaris returns ENOMEM when it's unable to create a thread; not sure what Linux returns.
Finally, I'm going to suggest again that you put logging messages in the thread factory that you're giving to the pool. Since you control that code, it's an excellent way to watch what's really happening at the time you run out of memory.