Strange -Xmx behavior on Linux 64-bit

Question

Hi, I wanted to test the maximum memory I can allocate for a JVM on my system, and discovered that:
1. when I specify -Xmx not smaller enough (~700MB) than the reported available system memory, the JVM will crash when a java program tries to use up the allocated memory (instead of throwing an OutOfMemory error for the app to handle) -- this is bad, as without testing, I don't know my max -Xmx.
2. the JVM lets me specify much higher -Xmx that can possibly be allocated on the system, and when a java app tries to use more than is really available, this situation is then resolved by SIGKILLing by the system (instead of OutOfMemory error thrown by the JVM) -- this might be OK, but I remember that on 32-bit it at least didn't allow me to allocate more than was physically possible.
My system: AMD3500+, 4GB RAM, no SWAP space, kernel 2.6.22.9 x86_64 GNU/Linux
- I don't use disk swap space, but that should only mean that these problems are reproducible faster
- I use the new SLUB allocator in Linux kernel (instead of SLAB), but these things are usually heavily tested before allowed to mainstream.
How to reproduce:
1. go into single-user mode or boot into recovery console (this is recommended because when low on virtual memory, Linux randomly(?) kills user programs)
2. play with the program I attached - I post the log of my experience (the max I was able to get on a 4GB system).
What are your thoughts on this? Basically I want to give my app as much memory as possible, and then only keep checking for the OutOfMemoryError.
Should this be reported as bug/bugs?
George
root@test:~/test# free -b
             total       used       free     shared    buffers     cached
Mem:    4009447424   57647104 3951800320          0    1015808   14282752
-/+ buffers/cache:   42348544 3967098880
Swap:            0          0          0
root@test:~/test# java -Xmx3200000000 -cp . memtest
available JVM memory: 3,093,272,440
trying to use it up...
available JVM memory: 21,445,824
test done!----------------------------
here is the source code:
import java.text.NumberFormat;
import java.util.LinkedList;
import java.util.List;
public class memtest {
     static NumberFormat df;
     public static void main(String[] args) throws Exception {
          df = NumberFormat.getNumberInstance();
          df.setGroupingUsed(true);
          free_memory();
          p("trying to use it up...");
          List r = new LinkedList();
          try {
               while(true) {
                    Object oo[] = new Object[1000000];
                    System.out.print(".");
                    System.out.flush();
                    for(int i = 0; i < oo.length; i++)
                         oo[i] = new Object();
                    r.add(oo);
          catch(OutOfMemoryError o) {
               //ok
          System.out.println("");
          free_memory();
          p("test done!");
     static void free_memory() {
          Runtime r = Runtime.getRuntime();
          p("available JVM memory: " + df.format(r.freeMemory() + r.maxMemory() - r.totalMemory()));
     static void p(Object o) {
          System.out.println(o);
the crash information:
# An unexpected error has been detected by Java Runtime Environment:
# java.lang.OutOfMemoryError: requested 335544320 bytes for GrET in /BUILD_AREA/jdk6_03/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?
# Internal Error (414C4C4F434154494F4E0E494E4C494E450E4850500017), pid=4682, tid=1077070144
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05 mixed mode)
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
--------------- T H R E A D ---------------
Current thread (0x00002aab85440c00): VMThread [id=4684]
Stack: [0x000000004022c000,0x000000004032d000), sp=0x000000004032b240, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x61a4af]
V [libjvm.so+0x2407b2]
V [libjvm.so+0x2a9377]
V [libjvm.so+0x1a2389]
V [libjvm.so+0x4922e9]
V [libjvm.so+0x49203b]
V [libjvm.so+0x4c0378]
V [libjvm.so+0x4921ba]
V [libjvm.so+0x4925bc]
V [libjvm.so+0x4c49a7]
V [libjvm.so+0x4c481f]
V [libjvm.so+0x280c25]
V [libjvm.so+0x28136a]
V [libjvm.so+0x5de8ef]
V [libjvm.so+0x5e0d0b]
V [libjvm.so+0x530258]
V [libjvm.so+0x28c480]
V [libjvm.so+0x28f168]
V [libjvm.so+0x28e7b2]
V [libjvm.so+0x299938]
V [libjvm.so+0x5d939b]
V [libjvm.so+0x28c092]
V [libjvm.so+0x2054ea]
V [libjvm.so+0x28c40e]
V [libjvm.so+0x61b4ae]
V [libjvm.so+0x62d26a]
V [libjvm.so+0x62c852]
V [libjvm.so+0x62cac6]
V [libjvm.so+0x62c5ce]
V [libjvm.so+0x4cd71a]
VM_Operation (0x000000004022a630): generation collection for allocation, mode: safepoint, requested by thread 0x0000000040112000
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00002aab8546d800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=4690]
0x00002aab8546b800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=4689]
0x00002aab8546a000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=4688]
0x00002aab85468800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=4687]
0x00002aab85446400 JavaThread "Finalizer" daemon [_thread_blocked, id=4686]
0x00002aab85445800 JavaThread "Reference Handler" daemon [_thread_blocked, id=4685]
0x0000000040112000 JavaThread "main" [_thread_blocked, id=4683]
Other Threads:
=>0x00002aab85440c00 VMThread [id=4684]
0x00002aab8546f800 WatcherThread [id=4691]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000000004010edf0/0x000000004010ee40] Threads_lock - owner thread: 0x00002aab85440c00
[0x000000004010f590/0x000000004010f5c0] Heap_lock - owner thread: 0x0000000040112000
Heap
def new generation total 1025408K, used 911488K [0x00002aaaaea30000, 0x00002aaaf42d0000, 0x00002aaaf42d0000)
eden space 911488K, 100% used [0x00002aaaaea30000, 0x00002aaae6450000, 0x00002aaae6450000)
from space 113920K, 0% used [0x00002aaae6450000, 0x00002aaae6450000, 0x00002aaaed390000)
to space 113920K, 100% used [0x00002aaaed390000, 0x00002aaaf42d0000, 0x00002aaaf42d0000)
tenured generation total 2278784K, used 2278783K [0x00002aaaf42d0000, 0x00002aab7f430000, 0x00002aab7f430000)
the space 2278784K, 99% used [0x00002aaaf42d0000, 0x00002aab7f42fff8, 0x00002aab7f430000, 0x00002aab7f430000)
compacting perm gen total 21248K, used 2872K [0x00002aab7f430000, 0x00002aab808f0000, 0x00002aab84830000)
the space 21248K, 13% used [0x00002aab7f430000, 0x00002aab7f6fe108, 0x00002aab7f6fe200, 0x00002aab808f0000)
No shared spaces configured.
Dynamic libraries:
40000000-40009000 r-xp 00000000 09:00 321291 /opt/jdk1.6.0_03/bin/java
40108000-4010a000 rwxp 00008000 09:00 321291 /opt/jdk1.6.0_03/bin/java
4010a000-4012b000 rwxp 4010a000 00:00 0 [heap]
4012b000-4012e000 ---p 4012b000 00:00 0
4012e000-4022c000 rwxp 4012e000 00:00 0
4022c000-4022d000 ---p 4022c000 00:00 0
4022d000-4032d000 rwxp 4022d000 00:00 0
4032d000-40330000 ---p 4032d000 00:00 0
40330000-4042e000 rwxp 40330000 00:00 0
4042e000-40431000 ---p 4042e000 00:00 0
40431000-4052f000 rwxp 40431000 00:00 0
4052f000-40532000 ---p 4052f000 00:00 0
40532000-40630000 rwxp 40532000 00:00 0
40630000-40633000 ---p 40630000 00:00 0
40633000-40731000 rwxp 40633000 00:00 0
40731000-40734000 ---p 40731000 00:00 0
40734000-40832000 rwxp 40734000 00:00 0
40832000-40835000 ---p 40832000 00:00 0
40835000-40933000 rwxp 40835000 00:00 0
40933000-40934000 ---p 40933000 00:00 0
40934000-40a34000 rwxp 40934000 00:00 0
2aaaaaabc000-2aaaaaac4000 r-xp 00000000 09:00 10947 /lib/librt-2.5.so
2aaaaaac4000-2aaaaacc3000 ---p 00008000 09:00 10947 /lib/librt-2.5.so
2aaaaacc3000-2aaaaacc5000 rwxp 00007000 09:00 10947 /lib/librt-2.5.so
2aaaaacc5000-2aaaaacc6000 ---p 2aaaaacc5000 00:00 0
2aaaaacc6000-2aaaaacc7000 rwxp 2aaaaacc6000 00:00 0
2aaaaacc7000-2aaaaacce000 r-xp 00000000 09:00 321834 /opt/jdk1.6.0_03/jre/lib/amd64/native_threads/libhpi.so
2aaaaacce000-2aaaaadcf000 ---p 00007000 09:00 321834 /opt/jdk1.6.0_03/jre/lib/amd64/native_threads/libhpi.so
2aaaaadcf000-2aaaaadd1000 rwxp 00008000 09:00 321834 /opt/jdk1.6.0_03/jre/lib/amd64/native_threads/libhpi.so
2aaaaadd1000-2aaaaadd2000 rwxp 2aaaaadd1000 00:00 0
2aaaaadd2000-2aaaaadda000 rwxs 00000000 09:00 132592 /tmp/hsperfdata_root/4682
2aaaaade1000-2aaaaadf5000 r-xp 00000000 09:00 10937 /lib/libnsl-2.5.so
2aaaaadf5000-2aaaaaff5000 ---p 00014000 09:00 10937 /lib/libnsl-2.5.so
2aaaaaff5000-2aaaaaff7000 rwxp 00014000 09:00 10937 /lib/libnsl-2.5.so
2aaaaaff7000-2aaaaaff9000 rwxp 2aaaaaff7000 00:00 0
2aaaaaff9000-2aaaab000000 r-xp 00000000 09:00 10938 /lib/libnss_compat-2.5.so
2aaaab000000-2aaaab200000 ---p 00007000 09:00 10938 /lib/libnss_compat-2.5.so
2aaaab200000-2aaaab202000 rwxp 00007000 09:00 10938 /lib/libnss_compat-2.5.so
2aaaab202000-2aaaab20c000 r-xp 00000000 09:00 10942 /lib/libnss_nis-2.5.so
2aaaab20c000-2aaaab40b000 ---p 0000a000 09:00 10942 /lib/libnss_nis-2.5.so
2aaaab40b000-2aaaab40d000 rwxp 00009000 09:00 10942 /lib/libnss_nis-2.5.so
2aaaab40d000-2aaaab417000 r-xp 00000000 09:00 10940 /lib/libnss_files-2.5.so
2aaaab417000-2aaaab616000 ---p 0000a000 09:00 10940 /lib/libnss_files-2.5.so
2aaaab616000-2aaaab618000 rwxp 00009000 09:00 10940 /lib/libnss_files-2.5.so
2aaaab618000-2aaaab625000 r-xp 00000000 09:00 321827 /opt/jdk1.6.0_03/jre/lib/amd64/libverify.so
2aaaab625000-2aaaab724000 ---p 0000d000 09:00 321827 /opt/jdk1.6.0_03/jre/lib/amd64/libverify.so
2aaaab724000-2aaaab727000 rwxp 0000c000 09:00 321827 /opt/jdk1.6.0_03/jre/lib/amd64/libverify.so
2aaaab727000-2aaaab74f000 r-xp 00000000 09:00 321845 /opt/jdk1.6.0_03/jre/lib/amd64/libjava.so
2aaaab74f000-2aaaab84f000 ---p 00028000 09:00 321845 /opt/jdk1.6.0_03/jre/lib/amd64/libjava.so
2aaaab84f000-2aaaab856000 rwxp 00028000 09:00 321845 /opt/jdk1.6.0_03/jre/lib/amd64/libjava.so
2aaaab856000-2aaaab864000 r-xp 00000000 09:00 321832 /opt/jdk1.6.0_03/jre/lib/amd64/libzip.so
2aaaab864000-2aaaab966000 ---p 0000e000 09:00 321832 /opt/jdk1.6.0_03/jre/lib/amd64/libzip.so
2aaaab966000-2aaaab969000 rwxp 00010000 09:00 321832 /opt/jdk1.6.0_03/jre/lib/amd64/libzip.so
2aaaab969000-2aaaabbda000 rwxp 2aaaab969000 00:00 0
2aaaabbda000-2aaaae96a000 rwxp 2aaaabbda000 00:00 0
2aaaae96a000-2aaaae974000 rwxp 2aaaae96a000 00:00 0
2aaaae974000-2aaaaea2a000 rwxp 2aaaae974000 00:00 0
2aaaaea30000-2aaaf42d0000 rwxp 2aaaaea30000 00:00 0
2aaaf42d0000-2aab7f430000 rwxp 2aaaf42d0000 00:00 0
2aab7f430000-2aab808f0000 rwxp 2aab7f430000 00:00 0
2aab808f0000-2aab84830000 rwxp 2aab808f0000 00:00 0
2aab84830000-2aab84a5d000 rwxp 2aab84830000 00:00 0
2aab84a5d000-2aab84eb5000 rwxp 2aab84a5d000 00:00 0
2aab84eb5000-2aab84ec0000 rwxp 2aab84eb5000 00:00 0
2aab84ec0000-2aab84edf000 rwxp 2aab84ec0000 00:00 0
2aab84edf000-2aab85339000 rwxp 2aab84edf000 00:00 0
2aab85339000-2aab85344000 rwxp 2aab85339000 00:00 0
2aab85344000-2aab85364000 rwxp 2aab85344000 00:00 0
2aab85364000-2aab8548c000 rwxp 2aab85364000 00:00 0
2aab8548c000-2aab85608000 r-xs 02c8f000 09:00 126596 /opt/jdk1.6.0_03/jre/lib/rt.jar
2aab85608000-2aab95bf4000 rwxp 2aab85608000 00:00 0
2aab9abf7000-2aaba4bf8000 rwxp 2aab9abf7000 00:00 0
2aabd0000000-2aabd0021000 rwxp 2aabd0000000 00:00 0
2aabd0021000-2aabd4000000 ---p 2aabd0021000 00:00 0
2b14c3421000-2b14c343d000 r-xp 00000000 09:00 10928 /lib/ld-2.5.so
2b14c343d000-2b14c3440000 rwxp 2b14c343d000 00:00 0
2b14c363c000-2b14c363e000 rwxp 0001b000 09:00 10928 /lib/ld-2.5.so
2b14c363e000-2b14c3653000 r-xp 00000000 09:00 10945 /lib/libpthread-2.5.so
2b14c3653000-2b14c3853000 ---p 00015000 09:00 10945 /lib/libpthread-2.5.so
2b14c3853000-2b14c3855000 rwxp 00015000 09:00 10945 /lib/libpthread-2.5.so
2b14c3855000-2b14c3859000 rwxp 2b14c3855000 00:00 0
2b14c3859000-2b14c3860000 r-xp 00000000 09:00 321817 /opt/jdk1.6.0_03/jre/lib/amd64/jli/libjli.so
2b14c3860000-2b14c3961000 ---p 00007000 09:00 321817 /opt/jdk1.6.0_03/jre/lib/amd64/jli/libjli.so
2b14c3961000-2b14c3963000 rwxp 00008000 09:00 321817 /opt/jdk1.6.0_03/jre/lib/amd64/jli/libjli.so
2b14c3963000-2b14c3965000 r-xp 00000000 09:00 10934 /lib/libdl-2.5.so
2b14c3965000-2b14c3b65000 ---p 00002000 09:00 10934 /lib/libdl-2.5.so
2b14c3b65000-2b14c3b67000 rwxp 00002000 09:00 10934 /lib/libdl-2.5.so
2b14c3b67000-2b14c3b68000 rwxp 2b14c3b67000 00:00 0
2b14c3b68000-2b14c3caf000 r-xp 00000000 09:00 10931 /lib/libc-2.5.so
2b14c3caf000-2b14c3eaf000 ---p 00147000 09:00 10931 /lib/libc-2.5.so
2b14c3eaf000-2b14c3eb2000 r-xp 00147000 09:00 10931 /lib/libc-2.5.so
2b14c3eb2000-2b14c3eb4000 rwxp 0014a000 09:00 10931 /lib/libc-2.5.so
2b14c3eb4000-2b14c3eba000 rwxp 2b14c3eb4000 00:00 0
2b14c3eba000-2b14c4561000 r-xp 00000000 09:00 321851 /opt/jdk1.6.0_03/jre/lib/amd64/server/libjvm.so
2b14c4561000-2b14c4662000 ---p 006a7000 09:00 321851 /opt/jdk1.6.0_03/jre/lib/amd64/server/libjvm.so
2b14c4662000-2b14c479a000 rwxp 006a8000 09:00 321851 /opt/jdk1.6.0_03/jre/lib/amd64/server/libjvm.so
2b14c479a000-2b14c47d5000 rwxp 2b14c479a000 00:00 0
2b14c47e4000-2b14c4865000 r-xp 00000000 09:00 10935 /lib/libm-2.5.so
2b14c4865000-2b14c4a64000 ---p 00081000 09:00 10935 /lib/libm-2.5.so
2b14c4a64000-2b14c4a66000 rwxp 00080000 09:00 10935 /lib/libm-2.5.so
7fffe7674000-7fffe7689000 rwxp 7fffe7674000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vdso]
VM Arguments:
jvm_args: -Xmx3500000000
java_command: memtest
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
LD_LIBRARY_PATH=/opt/jdk1.6.0_03/jre/lib/amd64/server:/opt/jdk1.6.0_03/jre/lib/amd64:/opt/jdk1.6.0_03/jre/../lib/amd64
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x61b150], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x61b150], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x4cbdd0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x4cbdd0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x4cbdd0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x4cdff0], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x4cd930], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x4cd930], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x4cd930], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x4cd930], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR2: [libjvm.so+0x4cdff0], sa_mask[0]=0x00000000, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:4.0
uname:Linux 2.6.22.9 #1 Sun Oct 7 14:16:59 EDT 2007 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 8192k, CORE 0k, NPROC 40960, NOFILE 1024, AS infinity
load average:0.82 0.59 0.30
CPU:total 1 (1 cores per cpu, 1 threads per core) family 15 model 79 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext
Memory: 4k page, physical 3915476k(106744k free), swap 0k(0k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05) for linux-amd64, built on Sep 24 2007 23:09:21 by "java_re" with gcc 3.2.2 (SuSE Linux)

Answer 1

I actually use suggestion #1 on our webservers, setting both Xms and Xmx to the same value. This helps alleviate any surprises trying to allocate Java heap. Since we use the 32-bit JVM, we do have to strike a balance -- too large a value and the process runs out of native heap long before using up Java heap.
As for #2, I don't think you're going to get reliable results that way. Depending on when a GC occurs, the amount of free memory may be lower than what's really available. And the free memory reported will be Java heap, so while it may help with OutOfMemoryErrors, it probably won't help with native heap OOMs.
With x86-64, running out of address space for the Java process shouldn't be a problem (I don't run 64-bit unfortunately, so I can't say for sure). I think that if you allocate your Java heap at startup (using Xmx and Xms), and you have swap space available, it should be able to allocate native heap as needed. Running slowly is usually better than not running at all.