Repeated Full GC even though heap is empty

Question

Hi,
These are the options specified to the VM. The box has 16gigs of memory on it.
XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:MaxPermSize=500m -XX:PermSize=500m -XX:MaxNewSize=4120m -XX:NewSize=4120m -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8688 -Dcom.sun.management.jmxremote.authenticate=false
-server -verbosegc -Dsun.net.inetaddr.ttl=600 -Dsun.net.inetaddr.negative.ttl=1
our application is loads a lot of data into the server cache from a DB ( ~ 6.5 Gigs). I am seeing a lot of Full GC's happening even though none of the spaces are shown as full in the GC output. Below is the sample output that I have from the GC, and its outputted virtually all over the logs, including right after I start the server.
Can anyone recommend tweaking any settings that can help me avoid this?
Much appreciated.
[Full GC {Heap before gc invocations=786:
par new generation   total 4214784K, used 749836K [0x0000002a9c6f0000, 0x0000002b9def0000, 0x0000002b9def0000)
  eden space 4210688K,  17% used [0x0000002a9c6f0000, 0x0000002aca333278, 0x0000002b9d6f0000)
  from space 4096K,   0% used [0x0000002b9d6f0000, 0x0000002b9d6f0000, 0x0000002b9daf0000)
  to   space 4096K,   0% used [0x0000002b9daf0000, 0x0000002b9daf0000, 0x0000002b9def0000)
concurrent mark-sweep generation total 8167424K, used 116027K [0x0000002b9def0000, 0x0000002d906f0000, 0x0000002d906f0000)
concurrent-mark-sweep perm gen total 512000K, used 80871K [0x0000002d906f0000, 0x0000002dafaf0000, 0x0000002dafaf0000)
[CMS: 116027K->127094K(8167424K), 1.0926490 secs] 865864K->127094K(12382208K), [CMS Perm : 80871K->80871K(512000K)]Heap after gc invocations=787:
par new generation total 4214784K, used 0K [0x0000002a9c6f0000, 0x0000002b9def0000, 0x0000002b9def0000)
eden space 4210688K, 0% used [0x0000002a9c6f0000, 0x0000002a9c6f0000, 0x0000002b9d6f0000)
from space 4096K, 0% used [0x0000002b9d6f0000, 0x0000002b9d6f0000, 0x0000002b9daf0000)
to space 4096K, 0% used [0x0000002b9daf0000, 0x0000002b9daf0000, 0x0000002b9def0000)
concurrent mark-sweep generation total 8167424K, used 127094K [0x0000002b9def0000, 0x0000002d906f0000, 0x0000002d906f0000)
concurrent-mark-sweep perm gen total 512000K, used 80871K [0x0000002d906f0000, 0x0000002dafaf0000, 0x0000002dafaf0000)
, 1.0928710 secs]
[Full GC {Heap before gc invocations=787:
par new generation   total 4214784K, used 46964K [0x0000002a9c6f0000, 0x0000002b9def0000, 0x0000002b9def0000)
  eden space 4210688K,   1% used [0x0000002a9c6f0000, 0x0000002a9f4cd2a0, 0x0000002b9d6f0000)
  from space 4096K,   0% used [0x0000002b9d6f0000, 0x0000002b9d6f0000, 0x0000002b9daf0000)
  to   space 4096K,   0% used [0x0000002b9daf0000, 0x0000002b9daf0000, 0x0000002b9def0000)
concurrent mark-sweep generation total 8167424K, used 127094K [0x0000002b9def0000, 0x0000002d906f0000, 0x0000002d906f0000)
concurrent-mark-sweep perm gen total 512000K, used 80871K [0x0000002d906f0000, 0x0000002dafaf0000, 0x0000002dafaf0000)
[CMS: 127094K->127093K(8167424K), 0.8199480 secs] 174059K->127093K(12382208K), [CMS Perm : 80871K->80871K(512000K)]Heap after gc invocations=788:
par new generation total 4214784K, used 0K [0x0000002a9c6f0000, 0x0000002b9def0000, 0x0000002b9def0000)
eden space 4210688K, 0% used [0x0000002a9c6f0000, 0x0000002a9c6f0000, 0x0000002b9d6f0000)
from space 4096K, 0% used [0x0000002b9d6f0000, 0x0000002b9d6f0000, 0x0000002b9daf0000)
to space 4096K, 0% used [0x0000002b9daf0000, 0x0000002b9daf0000, 0x0000002b9def0000)
concurrent mark-sweep generation total 8167424K, used 127093K [0x0000002b9def0000, 0x0000002d906f0000, 0x0000002d906f0000)
concurrent-mark-sweep perm gen total 512000K, used 80871K [0x0000002d906f0000, 0x0000002dafaf0000, 0x0000002dafaf0000)
}

Answer 1

Someone is calling System.gc() for you. And ordinary CMS cycle looks like [GC [ParNew: 263K->107K(14784K), 0.0289244 secs] 263K->107K(63936K), 0.0292115 secs]
[GC [1 CMS-initial-mark: 0K(49152K)] 107K(63936K), 0.0047868 secs]
[CMS-concurrent-mark: 0.031/0.032 secs]
[CMS-concurrent-preclean: 0.002/0.002 secs]
[GC[YG occupancy: 107 K (14784 K)][Rescan (parallel) , 0.0004983 secs][weak refs processing, 0.0000093 secs] [1 CMS-remark: 0K(49152K)] 107K(63936K), 0.0065329 secs]
[CMS-concurrent-sweep: 0.000/0.000 secs]
[CMS-concurrent-reset: 0.002/0.002 secs]In contrast, you've shown what CMS prints out if someone calls System.gc(). An explicit call to System.gc() runs a serial (one-thread) stop-the-world mark-compact collector. Almost certainly not what you want if you are using the concurrent mark-sweep collector and tuning your heap to try to get short garbage collection pauses. The only good news is that your collections don't take that long.
You should track down whoever is calling System.gc() and tell them to stop. If you can't do that, there are two workarounds. If you are running JDK-1.6.0, you can use the -XX:+ExplicitGCInvokesConcurrent flag to have calls to System.gc() just start a regular CMS cycle. That will block the thread that calls System.gc() until the cycle is complete, but allow the other threads to run concurrently with the collection. The larger hammer is -XX:+DisableExplicitGC, which turns calls to System.gc() into a no-op. If someone thought they needed garbage collection to happen promptly (e.g., under the impression that they needed finalize() method to run, etc.), then using that flag will disappoint them.
Your log doesn't show how often these full collections are happening. If you add the -XX:+PrintGCTimeStamps flag you will be able to see when they happen, in seconds since the launch of the VM. That may help you coordinate these collections with actions in your program to identify actions that cause System.gc() calls.
For example, RMI uses full collections at regular intervals to implement their distributed garbage collection algorithm. In JDK-1.5.0 (and earlier) the default interval was once a minute. In JDK-1.6.0 (and later) the default interval has been changed to once an hour. You can change the RMI distributed garbage collection interval with the properties -Dsun.rmi.dgc.server.gcInterval= and -Dsun.rmi.dgc.client.gcInterval=. That's a third possible workaround.