Unexpected Full GC's?

Question

I'm trying to track down a Java performance issue, and believe it may
be related to GC behavior. Quick background: j2se app server running
Caucho Resin 2.1.12, on RHEL3 with Sun JDK 1.4.2_04, modest number of
threads, serving approx 50-150 requests per second.
The system runs as expected until the first Full GC, which happens as
normal, when the heap has reached near full.
9235.733: [GC 777863K->768933K(780864K), 0.0228370 secs]
9236.850: [GC 779237K->779416K(780864K), 0.0259900 secs]
9236.876: [Full GC 779416K->168274K(780864K), 1.5076750 secs]
9241.554: [GC 217616K->212151K(780864K), 0.0139170 secs]
9241.677: [GC 221815K->172838K(780736K), 0.0124830 secs]
Then, it starts doing Full GC's much more frequently, every 30-300
seconds (notice that it was 9000 seconds before the first full GC),
and the heap isn't anywhere near full
9282.828: [GC 225917K->176877K(780992K), 0.0097920 secs]
9282.838: [Full GC 176877K->166557K(781120K), 1.0936260 secs]
9286.648: [GC 216409K->214382K(781248K), 0.0105110 secs]
9282.828: [GC 225917K->176877K(780992K), 0.0097920 secs]
9282.838: [Full GC 176877K->166557K(781120K), 1.0936260 secs]
9286.648: [GC 216409K->214382K(781248K), 0.0105110 secs]
This continues for the entire life of the application. As you'd
expect, this is significantly lowering throughput. I've tried all
sorts of jvm options, including DisableExplicitGC, even though I'm
pretty sure we aren't calling System.gc().
Any ideas??

Answer 1

If you look at your scavenge collections you're not actually removing a lot of data per scavenge. Therefore your new space is filling up with many objects that are long lived. The new space is filling up quickly and so results in a full collection where the majority of your objects are copied into the longer lived "old" space.
Basically your scavenge collections are inefficeient and so you are just filling up your old space until it becomes full and a full GC happens to clear it out. I'd suggest increasing your new space size using -Xmn, this would give your objects more time to potentially go out of scope and become elligible for collection.
You might also want to look at your code and see if you are holding on to any object references for too long which delays their collection.