Running out of memory despite having set je.maxMemory to a moderate value

Question

I have set je.maxMemory to 20MB (je.maxMemory=20000000) and allowed a max heap size of 512MB (-Xms256M -Xmx512M).
After two hours of running my web service, I'm running out of memory. After having profiled my service (using Yourkit Java Profiler 1.10.6), I can see the following:
Name                                               Objects ShallowSize  RetainedSize
byte[]                                               16711   124124880     124124880
com.sleepycat.je.tree.BIN                              181       24616     116254200
com.sleepycat.je.tree.Node[]                           187       98736     115743184
com.sleepycat.je.tree.LN                              7092      226944     115253600
java.util.concurrent.ConcurrentHashMap$HashEntry       554       17728      78328944
java.util.concurrent.ConcurrentHashMap$HashEntry[]    1053       34728      77489632
java.util.concurrent.ConcurrentHashMap                 117        5616      71812072
java.util.concurrent.ConcurrentHashMap$Segment[]       118       10304      71807912
java.util.concurrent.ConcurrentHashMap$Segment        1052       42080      71798808
com.sleepycat.je.tree.IN                                 6         672      45592352
java.lang.String                                    135888     4348416      14152664The memory profiler claims further, that com.sleepycat.je.tree.BIN is responsible for 71% of all heap memory.
In any case, com.sleepycat.je.tree.BIN claims ~ 116MB of heap memory, which is by any goodwill, exceeded the limit of 20MB.
How can this be?
How is JE ensuring that the limit is not exceeded? Is there a timer (thread) running which once a while checks the memory used and then cleans up ; or is memory usage checked creating a com.sleepycat.je.tree.BIN object?
My environment:
BDB JE 4.0.92 - used as cache loader within Jboss Cache (3.2.7.GA), running on a JBOSS Application Server, Java 1.6 (IBM) on Linux. Further details are listed in the system properties below (except some deleted security items).
System properties:
(java.lang.String, int, java.lang.StringBuffer, int)=contains
DestroyJavaVM helper thread=(java.lang.String, java.security.KeyStore$Entry, java.security.KeyStore$ProtectionParameter)
base.collection.name=CD2JAVA
bind.address=10.12.25.130
catalina.base=/work/ocrgws_test/server0
catalina.ext.dirs=/work/ocrgws_test/server0/lib
catalina.home=/work/ocrgws_test/server0
catalina.useNaming=false
com.arjuna.ats.arjuna.objectstore.objectStoreDir=/work/ocrgws_test/server0/data/tx-object-store
com.arjuna.ats.jta.lastResourceOptimisationInterface=org.jboss.tm.LastResource
com.arjuna.ats.tsmx.agentimpl=com.arjuna.ats.internal.jbossatx.agent.LocalJBossAgentImpl
com.arjuna.common.util.logger=log4j_releveler
com.arjuna.common.util.logging.DebugLevel=0x00000000
com.arjuna.common.util.logging.FacilityLevel=0xffffffff
com.arjuna.common.util.logging.VisibilityLevel=0xffffffff
com.ibm.cpu.endian=little
com.ibm.jcl.checkClassPath=
com.ibm.oti.configuration=scar
com.ibm.oti.jcl.build=20100326_1904
com.ibm.oti.shared.enabled=false
com.ibm.oti.vm.bootstrap.library.path=/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs:/opt/ibm/java-x86_64-60/jre/lib/amd64
com.ibm.oti.vm.library.version=24
com.ibm.util.extralibs.properties=
com.ibm.vm.bitmode=64
common.loader=${catalina.home}/lib,${catalina.home}/lib/*.jar
epo.jboss.deploymentscanner.extradirs=/work/ocrgws_test/app/
external.cert.ldap.* = ***************
file.encoding=UTF-8
file.separator=/
flipflop.activation.time=16:30
hibernate.bytecode.provider=javassist
ibm.signalhandling.rs=false
ibm.signalhandling.sigchain=true
ibm.signalhandling.sigint=true
ibm.system.encoding=UTF-8
jacorb.config.log.verbosity=0
java.assistive=ON
java.awt.fonts=
java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment
java.awt.printerjob=sun.print.PSPrinterJob
java.class.path=/work/ocrgws_test/config:/usr/local/jboss-eap-4.3-cp07/bin/run.jar:/opt/ibm/java-x86_64-60/lib/tools.jar
java.class.version=50.0
java.compiler=j9jit24
java.endorsed.dirs=/usr/local/jboss-eap-4.3-cp07/lib/endorsed
java.ext.dirs=/opt/ibm/java-x86_64-60/jre/lib/ext
java.fullversion=JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr8-20100401_55940 (JIT enabled, AOT enabled)
J9VM - 20100401_055940
JIT - r9_20100401_15339
GC - 20100308_AA_CMPRSS
java.home=/opt/ibm/java-x86_64-60/jre
java.io.tmpdir=/tmp
java.jcl.version=20100408_01
java.library.path=/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs:/opt/ibm/java-x86_64-60/jre/lib/amd64:/usr/lib64/mpi/gcc/openmpi/lib64:/usr/lib
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
java.net.preferIPv4Stack=true
java.protocol.handler.pkgs=org.jboss.net.protocol
java.rmi.server.codebase=http://10.12.25.130:8083/
java.rmi.server.hostname=10.12.25.130
java.rmi.server.randomIDs=true
java.runtime.name=Java(TM) SE Runtime Environment
java.runtime.version=pxa6460sr8-20100409_01 (SR8)
java.security.krb5.conf=/usr/local/jboss/etc/krb5.conf
java.specification.name=Java Platform API Specification
java.specification.vendor=Sun Microsystems Inc.
java.specification.version=1.6
java.util.prefs.PreferencesFactory=java.util.prefs.FileSystemPreferencesFactory
java.vendor.url=http://www.ibm.com/
java.vendor=IBM Corporation
java.version=1.6.0
java.vm.info=JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr8-20100401_55940 (JIT enabled, AOT enabled)
J9VM - 20100401_055940
JIT - r9_20100401_15339
GC - 20100308_AA_CMPRSS
java.vm.name=IBM J9 VM
java.vm.specification.name=Java Virtual Machine Specification
java.vm.specification.vendor=Sun Microsystems Inc.
java.vm.specification.version=1.0
java.vm.vendor=IBM Corporation
java.vm.version=2.4
javax.management.builder.initial=org.jboss.mx.server.MBeanServerBuilderImpl
javax.net.ssl.trustStore=/usr/local/jboss/etc/ldap.truststore
javax.net.ssl.trustStorePassword=password
jboss.bind.address=10.12.25.130
jboss.home.dir=/usr/local/jboss-eap-4.3-cp07
jboss.home.url=file:/usr/local/jboss-eap-4.3-cp07/
jboss.identity=30df88bc0a52e350x6e2ff59cx136c17794d5x-8000757
jboss.lib.url=file:/usr/local/jboss-eap-4.3-cp07/lib/
jboss.messaging.controlchanneludpaddress=239.1.200.4
jboss.messaging.datachanneludpaddress=239.1.200.4
jboss.partition.name=ocrgws_test_Partition
jboss.partition.udpGroup=239.1.200.4
jboss.remoting.domain=JBOSS
jboss.remoting.instanceid=30df88bc0a52e350x6e2ff59cx136c17794d5x-8000757
jboss.remoting.jmxid=luu002t.internal.epo.org_1334685694459
jboss.remoting.version=22
jboss.security.disable.secdomain.option=true
jboss.server.config.url=file:/work/ocrgws_test/server0/conf/
jboss.server.data.dir=/work/ocrgws_test/server0/data
jboss.server.home.dir=/work/ocrgws_test/server0
jboss.server.home.url=file:/work/ocrgws_test/server0/
jboss.server.lib.url=file:/work/ocrgws_test/server0/lib/
jboss.server.log.dir=/work/ocrgws_test/server0/log
jboss.server.name=luu002t_ocrgws_test_server0
jboss.server.temp.dir=/work/ocrgws_test/server0/tmp
jboss.tomcat.udpGroup=239.1.200.4
jbossmx.loader.repository.class=org.jboss.mx.loading.UnifiedLoaderRepository3
je.maxMemory=20000000
jgroups.bind_addr=10.12.25.130
jmx.console.bindcredential=3bpwdmpc
jmx.console.binddn=cn=jbossauth-ro,ou=accounts,ou=auth,dc=epo,dc=org
jmx.console.rolesctxdn=ou=roles-test,ou=jboss,ou=applications,ou=internal,dc=epo,dc=org
jndi.datasource.name=java:MainframeDS
jnp.disableDiscovery=true
jxe.current.romimage.version=15
jxe.lowest.romimage.version=15
line.separator=
mainframelogin.password=720652a1e842fc7f
mainframelogin.username=test_t
org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger
org.apache.tomcat.util.http.ServerCookie.VERSION_SWITCH=true
org.epo.jboss.application.home=/work/ocrgws_test
org.hyperic.sigar.path=/work/ocrgws_test/server0/./deploy/hyperic-hq.war/native-lib
org.jboss.ORBSingletonDelegate=org.jacorb.orb.ORBSingleton
org.omg.CORBA.ORBClass=org.jacorb.orb.ORB
org.omg.CORBA.ORBSingletonClass=org.jboss.system.ORBSingleton
org.w3c.dom.DOMImplementationSourceList=org.apache.xerces.dom.DOMXSImplementationSourceImpl
os.arch=amd64
os.name=Linux
os.version=2.6.32.46-0.3-xen
package.access=sun.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.,sun.beans.
package.definition=sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.
path.separator=:
poll.interval.milliseconds=300000
program.name=run.sh
server.loader=
shared.loader=
spnego.config=/usr/local/jboss/etc/spnego.properties
sun.arch.data.model=64
sun.boot.class.path=/usr/local/jboss-eap-4.3-cp07/lib/endorsed/xercesImpl.jar:/usr/local/jboss-eap-4.3-cp07/lib/endorsed/xalan.jar:/usr/local/jboss-eap-4.3-cp07/lib/endorsed/serializer.jar:/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs/jclSC160/vm.jar:/opt/ibm/java-x86_64-60/jre/lib/annotation.jar:/opt/ibm/java-x86_64-60/jre/lib/beans.jar:/opt/ibm/java-x86_64-60/jre/lib/java.util.jar:/opt/ibm/java-x86_64-60/jre/lib/jndi.jar:/opt/ibm/java-x86_64-60/jre/lib/logging.jar:/opt/ibm/java-x86_64-60/jre/lib/security.jar:/opt/ibm/java-x86_64-60/jre/lib/sql.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmorb.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmorbapi.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmcfw.jar:/opt/ibm/java-x86_64-60/jre/lib/rt.jar:/opt/ibm/java-x86_64-60/jre/lib/charsets.jar:/opt/ibm/java-x86_64-60/jre/lib/resources.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmpkcs.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmcertpathfw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjgssfw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjssefw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmsaslfw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjcefw.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjgssprovider.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmjsseprovider2.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmcertpathprovider.jar:/opt/ibm/java-x86_64-60/jre/lib/ibmxmlcrypto.jar:/opt/ibm/java-x86_64-60/jre/lib/management-agent.jar:/opt/ibm/java-x86_64-60/jre/lib/xml.jar:/opt/ibm/java-x86_64-60/jre/lib/jlm.jar:/opt/ibm/java-x86_64-60/jre/lib/javascript.jar:/tmp/yjp201202191932.jar
sun.boot.library.path=/opt/ibm/java-x86_64-60/jre/lib/amd64/compressedrefs:/opt/ibm/java-x86_64-60/jre/lib/amd64
sun.io.unicode.encoding=UnicodeLittle
sun.java.command=org.jboss.Main -b 10.12.25.130 -Djboss.server.home.dir=/work/ocrgws_test/server0 -Djboss.server.home.url=file:/work/ocrgws_test/server0 -Djboss.server.name=luu002t_ocrgws_test_server0 -Djboss.partition.name=ocrgws_test_Partition -Depo.jboss.deploymentscanner.extradirs=/work/ocrgws_test/app/ -Dorg.epo.jboss.application.home=/work/ocrgws_test
sun.java.launcher.pid=17781
sun.java.launcher=SUN_STANDARD
sun.java2d.fontpath=
sun.jnu.encoding=UTF-8
sun.rmi.dgc.client.gcInterval=3685000
sun.rmi.dgc.server.gcInterval=3685000
system=java.io.ObjectStreamField
tomcat.util.buf.StringCache.byte.enabled=true
user.country=US
user.dir=/work/ocrgws_test
user.home=*****************
user.language=en
user.name=***********
user.timezone=Europe/Berlin
user.variant=

Answer 1

The memory profiler claims further, that com.sleepycat.je.tree.BIN is responsible for 71% of all heap memory. In any case, com.sleepycat.je.tree.BIN claims ~ 116MB of heap memory, which is by any goodwill, exceeded the limit of 20MB. >
I'm not sure whether the profiler is reporting live objects only (referenced) or all objects (including those not yet reclaimed). If the latter, it isn't telling you how much memory is actually referenced by the JE cache.
Please look at the JE stats to see what the cache usage is, from JE's point of view.
If you believe there is a bug in JE cache management, you'll need to write a small standalone test to demonstrate it and submit it to us, since we don't know of any such bug. Also note that we'll have difficulty supporting JE 4.0 (without a support contract anyway). Please use JE 5.0, or at least 4.1.
Eviction occurs as objects are allocated, as well as in background threads. Eviction in background threads and concurrent eviction were greatly improved in JE 4.1.
--mark