Synchronisation of Garbage Collections in Coherence Storage Nodes

Question

Hi,
We have noticed an issue in Coherence 3.5 whereby the Garbage Collections on the storage nodes are run in parallel with one another.
Whilst we have not seen any major issues with this so far, it is of concern to us as having several storage nodes become unresponsive simultaneously may lead to cluster splits, service slowdown/unavailability, unnecessary rebalancing of data, and lost data.
Has anyone else experienced this issue before? If so, what approach, if any, did you take to resolve it? Is using data affinity the solution?
Cheers

Answer 1

Hi
As you probably know, Coherence has no control over garbage collection.
Here is a set of GC options for proper use of CMS on multi-core platform for a Hotspot 1.6.0_xx JVM (where xx must be >= 22 for some options to work properly). You may need to experiment with these values to derive the best performance for your application.
-server
-d64 (Solaris and HP-UX for 64-bit JVM; for 32-bit JVM -d32; other platforms have separate 32-bit and 64-bit installs)
-verbose:gc (writes GC log to stdout; use -Xloggc to direct it to some custom location)
-Xms<heapsize>m
-Xmx<heapsize>m
-XX:PermSize=<permsize>m
-XX:MaxPermSize=<permsize>m
-XX:NewSize=<up to heapsize/2>m
-XX:MaxNewSize=<up to heapsize/2>m
-XX:+HandlePromotionFailure
-XX:TargetSurvivorRatio=<nn> (somewhere between 60 to 80, decrease if you note direct Survivor space overflow)
-XX:MaxTenuringThreshold=8
-XX:SoftRefLRUPolicyMSPerMB=5000 ("a soft reference will survive (after the last strong reference to the object has been collected) for 5000ms times the number of megabytes of free space in the heap")
-XX:+AggressiveOpts
-XX:+DoEscapeAnalysis (from 1.6.0_22)
-XX:+UseCompressedOops (only on 64-bit JVM, from 1.6.0_22)
-XX:+DisableExplicitGC
-XX:+UseParNewGC
-XX:ParallelGCThreads=<between numcores und numcores/2 - e.g. numcores * 3/4>
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+ParallelRefProcEnabled
-XX:CMSInitiatingOccupancyFraction=<nn> (start with 75 and decrease in case CMS cannot keep up with the app)
-XX:+UseCMSInitiatingOccupancyOnly
-XX:-TraceClassUnloading
-XX:+PrintTenuringDistribution
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+HeapDumpOnOutOfMemoryError
Do not combine "ParallelGC" with CMS, you need to switch to "ParNewGC" to do so... (This is one of the more-or-less hidden secrets of Hotspot GC tuning.)
Paul