Concurrent mode failure- low memory -- OutOfMemory Exception

Question

WebLogic Web-tier instance frequently causing low memory. Where GC logs shows
Total time for which application threads were stopped: 0.7241688 seconds
46903.931: [GC [1 CMS-initial-mark: 375909K(524288K)] 441597K(983040K), 0.1330196 secs]
Total time for which application threads were stopped: 0.2807213 seconds
46904.064: [CMS-concurrent-mark-start]
46907.915: [CMS-concurrent-mark: 3.850/3.850 secs]
46907.915: [CMS-concurrent-preclean-start]
46907.952: [CMS-concurrent-preclean: 0.035/0.037 secs]
46907.952: [CMS-concurrent-abortable-preclean-start]
46938.527: [GC {Heap before gc invocations=218:
par new generation total 458752K, used 458752K [0xaf800000, 0xcf800000, 0xcf800000)
eden space 393216K, 100% used [0xaf800000, 0xc7800000, 0xc7800000)
from space 65536K, 100% used [0xc7800000, 0xcb800000, 0xcb800000)
to space 65536K, 0% used [0xcb800000, 0xcb800000, 0xcf800000)
concurrent mark-sweep generation total 524288K, used 375909K [0xcf800000, 0xef800000, 0xef800000)
concurrent-mark-sweep perm gen total 131072K, used 97681K [0xef800000, 0xf7800000, 0xf7800000)
46938.528: [ParNew (promotion failed): 458752K->458752K(458752K), 2.2293867 secs]46940.757: [CMS46941.097: [CMS-concurrent-abortable-preclean: 8.221/33.145 secs]
(concurrent mode failure): 500649K->524287K(524288K), 9.2616939 secs] 834661K->599632K(983040K)Heap after gc invocations=219:
par new generation total 458752K, used 75345K [0xaf800000, 0xcf800000, 0xcf800000)
eden space 393216K, 19% used [0xaf800000, 0xb4194420, 0xc7800000)
from space 65536K, 0% used [0xc7800000, 0xc7800000, 0xcb800000)
to space 65536K, 0% used [0xcb800000, 0xcb800000, 0xcf800000)
concurrent mark-sweep generation total 524288K, used 524287K [0xcf800000, 0xef800000, 0xef800000)
concurrent-mark-sweep perm gen total 131072K, used 96428K [0xef800000, 0xf7800000, 0xf7800000)
, 11.4919889 secs]
For your reference JVM arguments for these instances are as follows:
-Xms1024m -Xmx1024m
-XX:NewSize=512m
-XX:MaxNewSize=512m
-XX:SurvivorRatio=6
-Xnoclassgc -XX:PermSize=128m
-XX:MaxPermSize=128m
-XX:+HeapDumpOnOutOfMemoryError
-XX:+DisableExplicitGC
-XX:+UseConcMarkSweepGC
-verbose:gc -XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintHeapAtGC
-Xloggc:gc.log
-XX:MaxPermSize=128m
JDK: Sun JDK 1.5
Operating Environment: Solaris 5.10
WebLogic version 9.2 MP3
Thread dump shows:
"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x009679c0 nid=0x26 waiting for monitor entry [0xa3e7e000..0xa3e7f970]
at weblogic.utils.io.Chunk.<init>(Chunk.java:293)
at weblogic.utils.io.Chunk.getChunk(Chunk.java:141)
at weblogic.utils.io.ChunkedOutputStream.advance(ChunkedOutputStream.java:52)
at weblogic.utils.io.ChunkedOutputStream.write(ChunkedOutputStream.java:65)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1685)
We had applied patches as suggested by Oracle WebLogic. But, still issue persists :(
Any help on this issue appriciated.

Answer 1

1) Yes, a System.gc() will be reported by cms as a concurrent mode failure.
2) See if the release you're using has the flag -XX:+ExplicitGCInvokesConcurrent.
This will tell cms to start a concurrent collection when a System.gc() is involved.
It is in jdk6. It may still not do what you want if the the concurrent collections
are happening too frequently.
RMI schedules the full GC's because GC's are local to a JVM and
the collector on one JVM (call it JVM A) does not know about state of the heap
on another JVM (call it JVM B).
3) I would have expected setting the gcInterval to
a larger number would have had more of an affect. What was the value?
Post a fragment of a gc log (-XX:+PrintGCDetails) that shows the concurrent
mode failure. Maybe there's something else going on.