Potential Memory Leak in rmi

Question

We have a service that is being monitored via JMX. The JVM heap usage is growing and even major collections are not able to remove the garbage. Inspecting the heap shows garbage consisting of RMI related references (mostly, if not all, related class loaders). The only way to alleviate the issue is to issue explicit gc call through JMX (that removes all accumulated garbage). Our gc related options are:
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
And we have not touched either of: DisableExplicitGC or sun.rmi.dgc.server.gcInterval
I believe the problem is supposed to addressed by the code in sun.misc.GC.Daemon:
public void run() { for (;;) { long l; synchronized (lock) {  l = latencyTarget; if (l == NO_TARGET) { /* No latency target, so exit */ GC.daemon = null; return; }  long d = maxObjectInspectionAge(); if (d >= l) { /* Do a full collection. There is a remote possibility * that a full collection will occurr between the time * we sample the inspection age and the time the GC * actually starts, but this is sufficiently unlikely * that it doesn't seem worth the more expensive JVM * interface that would be required. */ System.gc(); d = 0; }  /* Wait for the latency period to expire, * or for notification that the period has changed */ try { lock.wait(l - d); } catch (InterruptedException x) { continue; } } } }
For some reason the above System.gc is not being invoked (that has been verified by looking at gc logs). Anyone has a suggestion as to how to address the issue?

Answer 1

Thanks for pointing to MOS notes, they were quite helpful. Though sometime on our system, ohasd.bin consumes more resources. Is it safe to kill it?
Also, we have observed that there are multiple oraagents belonging to different users such as root,grid and oracle.
grid 14620 1 0 20:32 ? 00:00:14 /u01/app/11.2.0/grid/bin/oraagent.bin
root 14625 1 0 20:32 ? 00:00:02 /u01/app/11.2.0/grid/bin/orarootagent.bin
root 14627 1 0 20:32 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdagent
grid 14803 1 0 20:32 ? 00:00:06 /u01/app/11.2.0/grid/bin/oraagent.bin
oracle 14807 1 0 20:32 ? 00:01:53 /u01/app/11.2.0/grid/bin/oraagent.bin
root 14811 1 0 20:32 ? 00:00:38 /u01/app/11.2.0/grid/bin/orarootagent.bin
When these are killed, not all are re-spawned automatically - typically oraagent belonging to "oracle" user is left out. Is this an expected behaviour or it will cause some instability in the clusterware?
Thanks