High Java memory consumption.

Question

Hello,
We are developing a solution using Servoy, which is a database server framework built in java. We have now 2 Servoy instances running on Debian linux servers, one with 64bit, both of them running java 1.6 update 26. We are having lot of crashes of the server with this error log:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32776 bytes for Chunk::new
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
# Out of Memory Error (allocation.cpp:317), pid=12359, tid=1777961872
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) Server VM (20.1-b02 mixed mode linux-x86 )
--------------- T H R E A D ---------------
Current thread (0x092c8400): JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=12368, stack(0x69f18000,0x69f99000)]
Stack: [0x69f18000,0x69f99000], sp=0x69f95fe0, free space=503k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x7248b0]
We checked both the heap memory and the memory used by java in this case and we notice a highly increase of the total java memory, while the heap memory is resonable:
JVM Information
java.vm.name=Java HotSpot(TM) Server VM
java.version=1.6.0_26
java.vm.info=mixed mode
java.vm.vendor=Sun Microsystems Inc.
Operating System Information
os.name=Linux
os.version=2.6.29-xs5.5.0.17
os.arch=i386
System Information
Heap memory: allocated=141440K, used=105555K, max=699072K
None Heap memory: allocated=49312K, used=49018K, max=180224K
root 16388 2.5 43.0 2829220 1811772 ? Sl 20:11 3:43 java -Djava.awt.headless=true -Xmx768m -Xms128m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.ja
Right now we are running it on the 64bit machine with -Xmx256m -Xms64m, and the memory is the same as on the 32bit one.
We also tried different memory configurations, but the result is the same. Java goes up to more the 2GB of memory used, while the heap is about 100MB - 400MB. In the output above you can see 1.8GB used, but this is at startup, after few hours it goes to more then 2GB - 2.5GB, then it crashes in at most few days, sometimes in few hours or even less.

Answer 1

Can the profiler see more then the heap? OK, I'll try to profile the server.
Servoy uses Tomcat 5.
The OS we run on is Debian. The 64bit machine is this:
JVM Information
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.version=1.6.0_26
java.vm.info=mixed mode
java.vm.vendor=Sun Microsystems Inc.
Operating System Information
os.name=Linux
os.version=2.6.32-5-amd64
os.arch=amd64
and the current memory usage is this:
System Information
Heap memory: allocated=253440K, used=228114K, max=253440K
None Heap memory: allocated=112512K, used=112223K, max=180224K
, while the java process memory is this:
root 11629 6.9 44.4 2105640 1829808 hvc0 Sl 11:42 14:18 java -Djava.awt.headless=true -Xmx256m -Xms64m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-codec.jar:lib/commons-httpclient.jar:lib/activation.jar:lib/antlr.jar:lib/commons-collections.jar:lib/commons-dbcp.jar:lib/commons-fileupload-1.2.1.jar:lib/commons-io-1.4.jar:lib/commons-logging.jar:lib/commons-pool.jar:lib/dom4j.jar:lib/help.jar:lib/jabsorb.jar:lib/hibernate3.jar:lib/j2db.jar:lib/j2dbdev.jar:lib/jdbc2_0-stdext.jar:lib/jmx.jar:lib/jndi.jar:lib/js.jar:lib/jta.jar:lib/BrowserLauncher2.jar:lib/jug.jar:lib/log4j.jar:lib/mail.jar:lib/ohj-jewt.jar:lib/oracle_ice.jar:lib/server-bootstrap.jar:lib/servlet-api.jar:lib/wicket-extentions.jar:lib/wicket.jar:lib/wicket-calendar.jar:lib/slf4j-api.jar:lib/slf4j-log4j.jar:lib/joda-time.jar:lib/rmitnl.jar:lib/networktnl.jar com.servoy.j2db.server.ApplicationServer
About another app, I'm confused, because we do have another java app that uses much less:
root      1149  0.3  2.1 843312 86988 ?        Sl   10:16   0:58 java -Xmx512m -Xms128m -cp noaa_server.zip server.WebServer
and for this one, the heap is this:
Memory: total: 129892352, free: 114666408, maximum: 518979584
So yes, this one looks better. It uses 15MB from heap and 80MB in total from ram.
But still, I don't get it, how can the process memory be affected while the heap is much low?
Edited by: 897090 on Nov 14, 2011 6:35 AM
Edited by: 897090 on Nov 14, 2011 6:39 AM
Edited by: 897090 on Nov 14, 2011 6:40 AM