Concurrent mode failure triggered by a full permanent generation?

Question

Hello
I've read various forum posts and blogs on concurrent mode failures and I'm aware why such failures occur (small old gen, promotion guarantee failure, fragmentation) in the old gen and some options (increase heap size/old gen, use CMSInitiatingFraction, explicit System.gc() at quiet hours for compaction, etc.) that may be used to alleviate the problem. In fact, thanks to these forums, I used some of these options to effectively deal with concurrent mode failures on a SPARC system (Solaris 9). However, with my current JVM tuning exercise, the following GC log snippet from a JVM 1.4.2_10 running on HP-UX 11.23 (ia64) leaves me in doubt.
346179.949: [GC 346179.950: [ParNew: 419022K->27014K(491520K), 0.1540543 secs] 976980K->585383K(1474560K), 0.1542645 secs]
346444.644: [GC 346444.645: [ParNew: 420230K->420230K(491520K), 0.0000416 secs]346444.645: [CMS (concurrent mode failure)[Unloading class sun.reflect.GeneratedMethodAccessor1107]
[Unloading class sun.reflect.GeneratedMethodAccessor1087]
[Unloading class org.springframework.ejb.support.AbstractJmsMessageDrivenBean]
[Unloading class sun.reflect.GeneratedConstructorAccessor552]
...#lots of classes unloaded here#
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor155]
: 558369K->317371K(983040K), 9.5144462 secs] 978599K->317371K(1474560K), 9.5147394 secs]
346677.529: [GC 346677.530: [ParNew: 393216K->13960K(491520K), 0.2163942 secs] 710587K->331331K(1474560K), 0.2166204 secs]
JVM 1.4.2_10 Flags: -Xms1536m -Xmx1536m -Xmn576m -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=31 -XX:TargetSurvivorRatio=80 -XX:SurvivorRatio=4 -XX:+TraceClassLoading -XX:+TraceClassUnloading -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xverbosegc:file=.....
Question: When using CMS, perm gen collection is turned off by default and a serial Full GC collects the perm gen. Could this have led to the concurrent mode failure seen above causing classes to be unloaded? i.e. when CMS collector needs to clean up perm gen via a serial STW GC, it generates a concurrent mode failure?
I doubt the concurrent mode failure could be caused by a full promotion guarantee failure or fragmentation given the available free heap and the fact that this was the first GC in the old generation since server startup.
Note: We do have a perm gen memory leak caused by spring/CGLIB. While we are fixing the class loading issue, we increased MaxPermSize from 256m to 512m to buy some time. With MaxPermSize=512m and -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled, we had very frequent CMS GCs (every 5 seconds) and I read that this happens because the CMSInitiatingthreshold is the same for both old gen and perm gen. So, we removed -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled and the CMS GCs are no longer frequent.
Thanks
Edited by: cybergavin on Dec 28, 2009 12:09 AM

Answer 1

Hmm..I think I was a bit hasty there...
[ParNew: 420230K->420230K(491520K), 0.0000416 secs]346444.645: [CMS (concurrent mode failure)[Unloading class........ indicates a parallel scavenge could not be performed most probably due to a "young generation guarantee" failure. Also, HPjmeter revealed no issues with perm gen growth. So, I have decreased the size of the young generation and triggered CMS earlier (changes in bold below):
*JVM 1.4.2_10 Flags:* -Xms1536m -Xmx1536m *-Xmn512m* -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=31 -XX:TargetSurvivorRatio=80 -XX:SurvivorRatio=4 -XX:+TraceClassLoading -XX:+TraceClassUnloading -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps *-XX:CMSInitiatingOccupancyFraction=50 -XX:+UseCMSInitiatingOccupancyOnly* -Xverbosegc:file=.....
Will post  back on results of the above....