OutOfMemoryError with many deployed apps

Question

We have 62 applications (EARs) deployed. 21 of these application contains a single stateless session bean and somewhere between 3 and 40 entity beans each. In total we currently have 466 entity beans. The rest of the applications contain only a single stateful session bean each.
When all these applications are enabled we can run for a few minutes before getting an OutOfMemoryError. When we disable 20 of the latter kind, the ones with stateful session beans, the server seems to work OK.
We have not yet, for obvious reasons, rolled the system out to the users so only a few developers are using it. The clients are fat Swing clients that talk IIOP with the server.
In one of the beans we have a method that does three consecutive System.gc() and then measures the heap usage. With all applications enabled only around 30% of the maximum heap space is used, even right before the server crashes. The heap max is set to 512MB and since the usage rate is so low it doesn't seem meaningful to increase it. Also, if there is a leak then just increasing the heap may give us a longer uptime without really solving the problem. We suspect that there is some other limit somewhere, but since the error message in the log doesn't have a stack trace it is very difficult to narrow it down.
Has anyone had a similar problem and maybe found a solution?
TIA,
/Gunnar

Answer 1

I've now tried switching off UseTLab, which had no effect. I've tried UseParallelGC earlier, also with no effect. When the server goes down there are only around 80 threads running, and I dont think that qualifies as a "large number of threads".
Some other info:
Hardware: Dell PowerEdge 2650, 4 x 2388 MHz CPU's, 4 GB RAM and a 6 GB paging file.
OS: Windows 2000 Server v 5.0.2195 SP4
JVM: Java HotSpot(TM) Server VM, Version 1.4.2_04
Instead of listing the JVM options, here is the complete server.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE server PUBLIC "-//Sun Microsystems Inc.//DTD Sun ONE Application Server 7.0//EN" "file:///C:/Sun/AppServer7/lib/dtds/sun-server_1_1.dtd">
<!--
Copyright (c) 2002 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
-->
<!-- server_7_0_dtd version: 1.1.2.39 -->
<server locale="en_US" name="server1" log-root="C:/Sun/AppServer7/domains/domain1/server1/logs" application-root="C:/Sun/AppServer7/domains/domain1/server1/applications" session-store="C:/Sun/AppServer7/domains/domain1/server1/session-store">
<http-service qos-enabled="false" qos-metrics-interval-in-seconds="30" qos-recompute-time-interval-in-millis="100">
<http-listener id="http-listener-1" address="0.0.0.0" port="81" acceptor-threads="1" blocking-enabled="false" security-enabled="false" default-virtual-server="server1" server-name="toker" enabled="true"/>
<mime id="mime1" file="mime.types"/>
<acl id="acl1" file="C:/Sun/AppServer7/domains/domain1/server1/config/generated.server1.acl"/>
<virtual-server-class id="defaultclass" config-file="obj.conf" default-object="default" accept-language="false" enabled="true">
<virtual-server id="server1" config-file="server1-obj.conf" http-listeners="http-listener-1" hosts="toker" mime="mime1" state="on" accept-language="false">
<http-qos enforce-bandwidth-limit="false" enforce-connection-limit="false"/>
<auth-db id="default" database="default"/>
<property name="dir" value=""/>
<property name="nice" value=""/>
<property name="user" value=""/>
<property name="group" value=""/>
<property name="chroot" value=""/>
<property name="docroot" value="D:/Project"/>
<property name="accesslog" value="C:/Sun/AppServer7/domains/domain1/server1/logs/access"/>
</virtual-server>
</virtual-server-class>
</http-service>
<iiop-service>
<orb message-fragment-size="1024" steady-thread-pool-size="10" max-thread-pool-size="200" idle-thread-timeout-in-seconds="300" max-connections="1024" monitoring-enabled="false"/>
<iiop-listener id="orb-listener-1" address="0.0.0.0" port="3700" enabled="true">
          </iiop-listener>
</iiop-service>
<web-container monitoring-enabled="false">
     </web-container>
<ejb-container steady-pool-size="16" pool-resize-quantity="8" max-pool-size="64" cache-resize-quantity="32" max-cache-size="256" pool-idle-timeout-in-seconds="600" cache-idle-timeout-in-seconds="600" removal-timeout-in-seconds="5400" victim-selection-policy="nru" commit-option="B" monitoring-enabled="false">
     </ejb-container>
<mdb-container steady-pool-size="10" pool-resize-quantity="2" max-pool-size="60" idle-timeout-in-seconds="600" monitoring-enabled="false">
     </mdb-container>
<jms-service port="7676" admin-user-name="admin" admin-password="admin" init-timeout-in-seconds="30" enabled="true">
     </jms-service>
<log-service file="C:/Sun/AppServer7/domains/domain1/server1/logs/server.log" level="INFO" log-stdout="true" log-stderr="true" echo-log-messages-to-stderr="true" create-console="true" log-virtual-server-id="false" use-system-logging="false">
     </log-service>
<security-service default-realm="ldap" anonymous-role="ANYONE" audit-enabled="false">
<auth-realm name="file" classname="com.iplanet.ias.security.auth.realm.file.FileRealm">
<property name="file" value="C:/Sun/AppServer7/domains/domain1/server1/config/keyfile"/>
<property name="jaas-context" value="fileRealm"/>
</auth-realm>
<auth-realm name="ldap" classname="com.iplanet.ias.security.auth.realm.ldap.LDAPRealm">
<property value="ldap://localhost:389" name="directory"/>
<property value="ou=People,dc=pliktverket,dc=local" name="base-dn"/>
<property value="ldapRealm" name="jaas-context"/>
<property value="ou=Groups,dc=pliktverket,dc=local" name="group-base-dn"/>
</auth-realm>
<auth-realm name="certificate" classname="com.iplanet.ias.security.auth.realm.certificate.CertificateRealm">
          </auth-realm>
<!--
          <auth-realm name="solaris" classname="com.iplanet.ias.security.auth.realm.solaris.SolarisRealm">
               <property name="jaas-context" value="solarisRealm"/>
          </auth-realm>
          -->
</security-service>
<transaction-service automatic-recovery="false" timeout-in-seconds="0" tx-log-dir="C:/Sun/AppServer7/domains/domain1/server1/logs" heuristic-decision="rollback" keypoint-interval="2048" monitoring-enabled="false">
     </transaction-service>
<java-config java-home="C:/Sun/AppServer7/jdk" server-classpath="/usr/share/lib/mps/secv1/jss3.jar;C:/Sun/AppServer7/lib/appserv-rt.jar;${java.home}/lib/tools.jar;C:/Sun/AppServer7/lib/appserv-ext.jar;C:/Sun/AppServer7/lib/appserv-cmp.jar;C:\Sun\AppServer7\share\lib/mail.jar;C:/Sun/AppServer7/lib/appserv-ideplugin.jar;C:\Sun\AppServer7\imq\lib/imq.jar;C:\Sun\AppServer7\imq\lib/jaxm-api.jar;C:\Sun\AppServer7\imq\lib/imqadmin.jar;C:\Sun\AppServer7\imq\lib/imqutil.jar;C:/Sun/AppServer7/lib/appserv-admin.jar;C:\Sun\AppServer7\share\lib/jaxrpc-impl.jar;C:\Sun\AppServer7\share\lib/jaxrpc-api.jar;C:\Sun\AppServer7\share\lib/jaxr-impl.jar;C:\Sun\AppServer7\share\lib/jaxr-api.jar;C:\Sun\AppServer7\share\lib/activation.jar;C:\Sun\AppServer7\share\lib/saaj-api.jar;C:\Sun\AppServer7\share\lib/saaj-impl.jar;C:\Sun\AppServer7\share\lib/commons-logging.jar;C:\Sun\AppServer7\imq\lib/fscontext.jar;C:\Sun\AppServer7\imq\lib/providerutil.jar;C:/Sun/AppServer7/lib/appserv-jstl.jar" classpath-suffix="C:/Sun/AppServer7/pointbase/client_tools/lib/pbclient42RE.jar" env-classpath-ignored="true" debug-options="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n" debug-enabled="false" javac-options="-g" rmic-options="-iiop -poa -alwaysgenerate -keepgenerated -g">
<profiler enabled="false" name="hprof">
<jvm-options>-Xrunhprof:file=profiler.log,thread=y</jvm-options>
</profiler>
<jvm-options>-Djava.security.policy=C:/Sun/AppServer7/domains/domain1/server1/config/server.policy</jvm-options>
<jvm-options>-Djava.security.auth.login.config=C:/Sun/AppServer7/domains/domain1/server1/config/login.conf</jvm-options>
<jvm-options>-Djava.util.logging.manager=com.iplanet.ias.server.logging.ServerLogManager</jvm-options>
<jvm-options>-Dcom.sun.aas.configRoot=C:/Sun/AppServer7\config</jvm-options>
<jvm-options>-Dcom.sun.aas.imqLib=C:\Sun\AppServer7\imq\lib</jvm-options>
<jvm-options>-Dcom.sun.aas.imqBin=C:\Sun\AppServer7\imq\bin</jvm-options>
<jvm-options>-Dcom.sun.aas.webServicesLib=C:\Sun\AppServer7\share\lib</jvm-options>
<jvm-options>-Dorg.xml.sax.parser=org.xml.sax.helpers.XMLReaderAdapter</jvm-options>
<jvm-options>-Dorg.xml.sax.driver=org.apache.crimson.parser.XMLReaderImpl</jvm-options>
<jvm-options>-Dcom.sun.jdo.api.persistence.model.multipleClassLoaders=reload</jvm-options>
<jvm-options>-Dbug.dependant.delete=true</jvm-options>
<jvm-options>-Dbug.many2many.add=true</jvm-options>
<jvm-options>-Dplis.docroot=D:/Project</jvm-options>
<jvm-options>-Djavax.rmi.CORBA.UtilClass=com.iplanet.ias.util.orbutil.IasUtilDelegate</jvm-options>
<jvm-options>-Xms1024m -Xmx1024m</jvm-options>
<jvm-options>-Dsun.rmi.dgc.server.gcInterval=3600000</jvm-options>
<jvm-options>-Xrs</jvm-options>
</java-config>
<resources>
<jdbc-resource enabled="true" pool-name="PlisPool" jndi-name="jdbc/PlisDS"/>
<jdbc-resource enabled="true" pool-name="PlisPool" jndi-name="jdo/PlisDBPM"/>
<persistence-manager-factory-resource factory-class="com.sun.jdo.spi.persistence.support.sqlstore.impl.PersistenceManagerFactoryImpl" enabled="true" jndi-name="jdo/PlisDB" jdbc-resource-jndi-name="jdo/PlisDBPM"/>
<jdbc-connection-pool steady-pool-size="1" max-pool-size="32" max-wait-time-in-millis="60000" pool-resize-quantity="2" idle-timeout-in-seconds="300" is-isolation-level-guaranteed="true" is-connection-validation-required="true" connection-validation-method="table" fail-all-connections="false" datasource-classname="com.informix.jdbcx.IfxDataSource" name="PlisPool" transaction-isolation-level="read-committed" validation-table-name="singleton">
<description>luna/plisutv/plis_db</description>
<property value="improved" name="user"/>
<property value="plis" name="password"/>
<property value="true" name="ifxDELIMIDENT"/>
<property value="1531" name="portNumber"/>
<property value="plis_db" name="databaseName"/>
<property value="plisutv" name="serverName"/>
<property value="luna" name="ifxIFXHOST"/>
</jdbc-connection-pool>
</resources>
<applications dynamic-reload-enabled="false" dynamic-reload-poll-interval-in-seconds="2">
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\CodeBook_2" name="CodeBook" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\UtbForband_1" name="UtbForband" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PlisServer_2" name="PlisServer" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\CivKpl_2" name="CivKpl" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Batch_2" name="Batch" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Faq_1" name="Faq" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Infra_4" name="Infra" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\InfraOvrigt_1" name="InfraOvrigt" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IntPerson_3" name="IntPerson" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\KoderBtkd_1" name="KoderBtkd" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\KoderOrg_1" name="KoderOrg" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\KoderOvrigt_1" name="KoderOvrigt" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Monstring_1" name="Monstring" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Person_4" name="Person" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PArende_1" name="PArende" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PArendeStod_1" name="PArendeStod" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Prov_1" name="Prov" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\ProvStod_1" name="ProvStod" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RegSlagning_1" name="RegSlagning" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Reseplanering_1" name="Reseplanering" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\Stodtabeller_1" name="Stodtabeller" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\BiBerUtb_1" name="BiBerUtb" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\BiPerson_3" name="BiPerson" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvApp_1" name="IvApp" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvCode_2" name="IvCode" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvEntity_2" name="IvEntity" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvFaq_1" name="IvFaq" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvMedd_1" name="IvMedd" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvRole_1" name="IvRole" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvSvc_1" name="IvSvc" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvUser_2" name="IvUser" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvVar_1" name="IvVar" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IpPlisNytt_1" name="IpPlisNytt" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IpServer_1" name="IpServer" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IpTjanst_2" name="IpTjanst" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\GuPer_1" name="GuPer" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\HvPerson_1" name="HvPerson" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PkPer_1" name="PkPer" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PpMed_1" name="PpMed" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PpNyPerson_3" name="PpNyPerson" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PpSyom_1" name="PpSyom" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkAns_1" name="RkAns" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkBsl_1" name="RkBsl" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkBtk_3" name="RkBtk" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkFar_2" name="RkFar" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkHor_2" name="RkHor" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkInr_1" name="RkInr" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkKom_1" name="RkKom" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkLkf_1" name="RkLkf" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkPar_1" name="RkPar" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkPko_1" name="RkPko" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkRer_1" name="RkRer" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkSlt_1" name="RkSlt" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkTil_1" name="RkTil" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\SaBmi_1" name="SaBmi" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\SaHan_1" name="SaHan" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\SaLnd_1" name="SaLnd" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\SaMas_1" name="SaMas" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\SaMon_1" name="SaMon" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\SaPup_1" name="SaPup" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\PlisRun_2" name="PlisRun" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IvProj_1" name="IvProj" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\RkOrg_3" name="RkOrg" virtual-servers="server1"/>
<j2ee-application enabled="true" location="C:\Sun\AppServer7\domains\domain1\server1\applications\j2ee-apps\IbSys_1" name="IbSys" virtual-servers="server1"/>
</applications>
</server>
/Gunnar