OutOfMemory error in Weblogic jvm even when enough heap is still available

Question

Hi Everyone,
We get a strange OutOfMemory error that happens only once in our Weblogic server in production environment.The Server immediately recovers and there is no issue as far as the application functionality is concerned.
This is the exact error as seen in weblogic console.log file :
caused by: java.lang.OutOfMemoryError: Java heap space
I enabled the verbose gc logging too with the jvm parameter "-XX:+PrintHeapAtGC".Am copying the gc.log file excerpt when we got the Out of Memory issue in console.log.
gc.log when error was seen in console.log:
172047.050: [GC {Heap before gc invocations=28392:
par new generation   total 49088K, used 49024K [0xb8000000, 0xbb000000, 0xbb000000)
  eden space 49024K, 100% used [0xb8000000, 0xbafe0000, 0xbafe0000)
  from space 64K,   0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
  to   space 64K,   0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
concurrent mark-sweep generation total 737280K, used 364191K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166556K [0xe8000000, 0xf8000000, 0xf8000000)
172047.050: [ParNew: 49024K->0K(49088K), 0.1014386 secs] 413215K->382087K(786368K)Heap after gc invocations=28393:
par new generation total 49088K, used 0K [0xb8000000, 0xbb000000, 0xbb000000)
eden space 49024K, 0% used [0xb8000000, 0xb8000000, 0xbafe0000)
from space 64K, 0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
to space 64K, 0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
concurrent mark-sweep generation total 737280K, used 382087K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166556K [0xe8000000, 0xf8000000, 0xf8000000)
, 0.1021810 secs]
172055.735: [GC {Heap before gc invocations=28393:
par new generation   total 49088K, used 49008K [0xb8000000, 0xbb000000, 0xbb000000)
  eden space 49024K,  99% used [0xb8000000, 0xbafdc128, 0xbafe0000)
  from space 64K,   0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
  to   space 64K,   0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
concurrent mark-sweep generation total 737280K, used 382087K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166616K [0xe8000000, 0xf8000000, 0xf8000000)
172055.735: [ParNew: 49008K->0K(49088K), 0.0536844 secs] 431095K->387656K(786368K)Heap after gc invocations=28394:
par new generation total 49088K, used 0K [0xb8000000, 0xbb000000, 0xbb000000)
eden space 49024K, 0% used [0xb8000000, 0xb8000000, 0xbafe0000)
from space 64K, 0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
to space 64K, 0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
concurrent mark-sweep generation total 737280K, used 387656K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166616K [0xe8000000, 0xf8000000, 0xf8000000)
, 0.0544288 secs]
172059.106: [Full GC {Heap before gc invocations=28394:
par new generation   total 49088K, used 22668K [0xb8000000, 0xbb000000, 0xbb000000)
  eden space 49024K,  46% used [0xb8000000, 0xb9623158, 0xbafe0000)
  from space 64K,   0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
  to   space 64K,   0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
concurrent mark-sweep generation total 737280K, used 387656K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166616K [0xe8000000, 0xf8000000, 0xf8000000)
172059.106: [CMS: 387656K->335512K(737280K), 6.1797673 secs] 410325K->335512K(786368K), [CMS Perm : 166616K->166119K(262144K)]Heap after gc invocations=28395:
par new generation total 49088K, used 0K [0xb8000000, 0xbb000000, 0xbb000000)
eden space 49024K, 0% used [0xb8000000, 0xb8000000, 0xbafe0000)
from space 64K, 0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
to space 64K, 0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
concurrent mark-sweep generation total 737280K, used 335512K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166119K [0xe8000000, 0xf8000000, 0xf8000000)
, 6.1804623 secs]
172073.913: [GC {Heap before gc invocations=28395:
par new generation   total 49088K, used 49024K [0xb8000000, 0xbb000000, 0xbb000000)
  eden space 49024K, 100% used [0xb8000000, 0xbafe0000, 0xbafe0000)
  from space 64K,   0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
  to   space 64K,   0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
concurrent mark-sweep generation total 737280K, used 335512K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166146K [0xe8000000, 0xf8000000, 0xf8000000)
172073.913: [ParNew: 49024K->0K(49088K), 0.0616737 secs] 384536K->341716K(786368K)Heap after gc invocations=28396:
par new generation total 49088K, used 0K [0xb8000000, 0xbb000000, 0xbb000000)
eden space 49024K, 0% used [0xb8000000, 0xb8000000, 0xbafe0000)
from space 64K, 0% used [0xbafe0000, 0xbafe0000, 0xbaff0000)
to space 64K, 0% used [0xbaff0000, 0xbaff0000, 0xbb000000)
concurrent mark-sweep generation total 737280K, used 341716K [0xbb000000, 0xe8000000, 0xe8000000)
concurrent-mark-sweep perm gen total 262144K, used 166146K [0xe8000000, 0xf8000000, 0xf8000000)
, 0.0623851 secs]
I read at some forums that Heap fragmentation could be the reason when you get the OutOfMemory error?Also the rate with which GC is running if it consumes more than 98% of time and releases less than 2% of heap,this error could be thrown.
Could any of the above be a cause for this?We dont have the flexibilty of increasing the Xmx anymore.
We are using a Concurrent Mark Sweep collector as can be seen from script below.
Here are the jvm parameters am using in Server start script:
JAVA_HOME/bin/java -server -Xms768m -Xmx768m -XX:MaxPermSize=256m -XX:MaxNewSize=64m -Xbootclasspath/a:$BOOTCLASSPATH -classpath $CLASSPATH -Dbea.home=$BEA_HOME -Dweblogic.Domain=$DOMAIN_NAME -Dweblogic.Name=$SERVER_NAME -Djava.security.auth.login.config=../properties/login.config -Dcom.sun.management.jmxremote.port=7003 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UsePerfData -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Xloggc:../log/gc.log -Djava.security.policy==$WL_HOME/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -Dweblogic.management.discover=false -XX:-TraceClassUnloading -Dweblogic.management.server=scc-prd-admin.XXX.com:7001 weblogic.Server
We are using jdk1.5.0_06 version of java with Weblogic.
Could someone please help resolve this issue?Any thoughts?

Answer 1

Are you using RMI? Have you adjusted sun.rmi.dgc.server.gcInterval and sun.rmi.dgc.client.gcInterval from their default values for calling System.gc()? I gather (from the format of your logs) that you are running an older JVM, where the default RMI GC intervals were 60000 milliseconds (1 minute). In later JVM's, the default RMI GC interval was increased to 1 hour. See the [JavaTM RMI Release Notes for JDK 6|http://java.sun.com/javase/6/docs/technotes/guides/rmi/relnotes.html] for some details on this.
Otherwise you might be using some third party code that calls System.gc() periodically. A later JVM will show "(System)" in the full GC output for collections that were caused by a call to System.gc(). You can effectively disable that behavior with the -XX:+DisableExplicitGC command line option, though that might have bad effects on your application if the third party code needed those full collections. But it would be diagnostic.