OutOfMemory : Unable to new native thread

Question

Let me start off by giving some background.
We began experiencing this issue within the past month
in our production environment, but cannot reproduce
it in an isolated testing environment.
I have performed extensive tests in regards to Heap Sizes
and Thread Stack sizes in hopes of resolving this issue, but
none have worked.
Hardware/OS Settings
OS for platform is Linux
Distribution is RHAS 2.1
Kernel version is 2.4.9-e.30enterprise SMP
Box has 2 CPUs which are hyperthreaded
ULIMIT settings
core file size (blocks) 0
data seg size (kbytes) unlimited
file size (blocks) unlimited
max locked memory (kbytes) unlimited
max memory size (kbytes) unlimited
open files 1024
pipe size (512 bytes) 8
stack size (kbytes) 8192
cpu time (seconds) unlimited
max user processes 8191
virtual memory (kbytes) unlimited
/etc/sysctl.conf , contains (effectively overrides the ulimt setting)
fs.file-max = 65535
/proc/sys/kernel/threads-max
16383
glibc version is 2.2.4
Physical Mem : ~ 2 Gig
Swap : ~ 4 Gig
JVM Settings
java version "1.4.2_03"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02)
Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode)
-server -Djava.awt.headless=true -XX:+DisableExplicitGC -Xms1228m -Xmx1228m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -verbosegc -verbosegc -XX:+PrintGCDetails -Xloggc:/opt/logs/gc_log.txt
Tests
The way i understand it, the memory available for thread creation is:
Physical Memory - "-Xmx" = ~ 800 MB (in our case)
Also, the default stack size on linux is 512k, so a base line (with no other
phyical memory usage than the JVM) is:
800 * 1000 / 512 = ~ 1562 threads
however, i when the error recurrs, i can do a :
ps -ef | grep java | wc -l (each java thread is represented by a process on linux)
and that # is way below 1562 (approx 90 or so)
additionally, via java code at the time of the exception you can get
the Mother Thread group:
        private static ThreadGroup getMotherThreadGroup()
          ThreadGroup mother = Thread.currentThread().getThreadGroup();
          while(mother.getParent() != null)
               mother = mother.getParent();
          return mother;
     }and then get the # of threads in the group:
        Thread      currentThread = Thread.currentThread();
        ThreadGroup mother        = getMotherThreadGroup();       
     Thread[]    threads       = new Thread[mother.activeCount() + 5];          
        int threadCount;
        for (int count;;)
            count = mother.enumerate(threads,true);
            if (count < threads.length)
                threadCount = count;
                break;
            threads = new Thread[threads.length << 1];
        }  once again this # is way below 1562 (matches rougly to the 'ps' output)
Additionally, I have modified the JVM parameters to indepdently reduce
-Xms by 100 m and reduce the stack size to -Xss256k and neither of
these resolve the issue.
Questions
What other OS level settings could be affecting the # of threads that can be created?
What JVM settings could be affecting the # of threads that can be created?
Are there ever Java threads present that will no be represented by the
code in this post, or a 'ps' process listing?

Answer 1

Some additional information, in regards to Linux and Maximum # of Threads.
This bug entry in Redhat's Bug DB, provides some insight into the native
thread threshold:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=70058
From the Java Bug Database:
On Redhat 7.2, the pthread library has a hard-coded thread limit, which
is 1024. To increase this limit, you have to change PTHREAD_THREADS_MAX
and recompile pthread library. See LinuxThreads FAQ. Hitting an
OutOfMemoryError after 1014 threads on RH-7.2 is normal.
The error on RH-9 looks strange. But when I tested on my RH-9 box, things
work ok. The testcase runs until it has hit one of the system limits:
+ max user processes (use "ulimit -u" to adjust)
+ thread limit, this is calculated based on the amount of physical memory
(you can change it by editing /proc/sys/kernel/threads-max)
+ virtual memory. 32bit Linux by default allows 2GB virtual address space
in user application. Unless you specify a smaller stack size (-Xss96k
will do it), java will run out of virtual space in about 4000 threads
(default stack size is 512k, 2GB/512k = 4096, but notice that the
virtual memory is also used for C heap, library image, data segments, et al.)
Another Linux Thread explanation:
Threads
Limitations on threads are tightly tied to both file descriptor
limits, and process limits.
Under Linux, threads are counted as processes, so any limits to
the number of processes also applies to threads. In a heavily
threaded app like a threaded TCP engine, or a java server, you
can quickly run out of threads.
For starters, you want to get an idea how many threads you can
open. The `thread-limit` util mentioned in the Tuning Utilities
section is probabaly as good as any.
The first step to increasing the possible number of threads is
to make sure you have boosted any process limits as mentioned
before.
There are few things that can limit the number of threads,
including process limits, memory limits, mutex/semaphore/shm/ipc
limits, and compiled in thread limits. For most cases, the
process limit is the first one to run into, then the compiled in
thread limits, then the memory limits.
To increase the limits, you have to recompile glibc. Oh fun!.
And the patch is essentially two lines!. Woohoo!
--- ./linuxthreads/sysdeps/unix/sysv/linux/bits/local_lim.h.akl Mon Sep 4
19:37:42 2000
+++ ./linuxthreads/sysdeps/unix/sysv/linux/bits/local_lim.h Mon Sep 4
19:37:56 2000
@@ -64,7 +64,7 @@
/* The number of threads per process. */
#define POSIXTHREAD_THREADS_MAX 64
/* This is the value this implementation supports. */
-#define PTHREAD_THREADS_MAX 1024
+#define PTHREAD_THREADS_MAX 8192
/* Maximum amount by which a process can descrease its asynchronous I/O
priority level. */
--- ./linuxthreads/internals.h.akl Mon Sep 4 19:36:58 2000
+++ ./linuxthreads/internals.h Mon Sep 4 19:37:23 2000
@@ -330,7 +330,7 @@
THREAD_SELF implementation is used, this must be a power of two and
a multiple of PAGE_SIZE. */
#ifndef STACK_SIZE
-#define STACK_SIZE (2 * 1024 * 1024)
+#define STACK_SIZE (64 * PAGE_SIZE)
#endif
/* The initial size of the thread stack. Must be a multiple of PAGE_SIZE.
Now just patch glibc, rebuild, and install it. ;-> If you have a
package based system, I seriously suggest making a new package
and using it.
And two links that describe Linux/GLIBC changes and how they affect threads:
http://www.jlinux.org/server.html
http://www.volano.com/linux.html
C Code for testing Native Thread Creation:
/* compile with:   gcc -lpthread -o thread-limit thread-limit.c */
/* originally from: http://www.volano.com/linuxnotes.html */
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#define MAX_THREADS 10000
int i;
void run(void) {
  char c;
  if (i < 10)
    printf("Address of c = %u KB\n", (unsigned int) &c / 1024);
  sleep(60 * 60);
int main(int argc, char *argv[]) {
  int rc = 0;
  pthread_t thread[MAX_THREADS];
  printf("Creating threads ...\n");
  for (i = 0; i < MAX_THREADS && rc == 0; i++) {
    rc = pthread_create(&(thread), NULL, (void *) &run, NULL);
if (rc == 0) {
pthread_detach(thread[i]);
if ((i + 1) % 100 == 0)
     printf("%i threads so far ...\n", i + 1);
else
printf("Failed with return code %i creating thread %i.\n",
     rc, i + 1);
exit(0);