Java threads on machines with hundreds of cores

Question

I don't know if this is the right forum, but I try anyway.
My problem is with java on machines with many cores.
I have some large shared memory machines with 500 cores.
When a user run java programs on these machines java starts 300+ threads which to me seems way too much for the application as the same application runs faster on systems with less cores and memory and without creating 300+ threads.
I have found information about how to limit memory usage, but I have not been able to find any information about how to limit the number of threads java creates nor what criteria java uses to determine how many threads should be created.
In this context threads are the tasks reported under /proc/<PID of java binary>/task on a Linux system.
Thanks
John

Answer 1

Hi,
Thanks for the reply.
We did some more testing and found that java on startup looks in /sys/devices/system/cpu/, which holds information about the 512 CPU's on the system.
We then tried to lower number of allowed processes using ulimit -u 20 and got this error when running the program:
# A fatal error has been detected by the Java Runtime Environment:
# java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.
# Internal Error (gcTaskThread.cpp:38), pid=303212, tid=140353923254032
# Error: Cannot create GC thread. Out of system resources.
# JRE version: 6.0_21-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode linux-amd64 )
# An error report file with more information is saved as:
# /net/panfsdb0/panfsvol1/simon/projects/cge/test/hs_err_pid303212.log
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
Aborted
Now a Google search for gcTaskThread revealed that it is the Garbage Collector that creates the many threads. Google also told us that the number of GC threads can be controlled by the JVM option -XX:ParallelGCThreads=<number_of_GC_threads> . Specifying 8 GC threads the program now runs with 18 threads. Specifying 4 GC threads the program runs with 14 threads.
So it seems like the JVM by default creates as many Garbage Collectors threads as there are CPU's in the system.
This is a very bad practice as on such systems where programs are run within cpusets with just a small subset of the system wide CPU's.
Instead of searching /sys/devices/system/cpu/ for number of system wide CPU's JVM should read /dev/cpuset/<cpuset id>/cpus to see how many CPU's are actually available for the job.
Thanks
John
Edited by: 886656 on Sep 22, 2011 2:13 AM