Garbage Collector blocks - weird behaviour

Question

Hi!
After launching an application we ran into weird problems with the garbage collector. It's a combo of Jetty/JGroups/Helma-Application-Server and various other libraries on a RedHat 7.3 box. The problem occurs on a machine with relativly high load (~10 requests/sec) but doesn't come along with high load automatically.
After some hours uptime the garbage collector blocks for increasingly longer intervals until the application is reachable only for seconds every few minutes. In concerns exclusivly minor GC in the new space. Each stop lasts for just 1/10000 seconds but that in a loop of 1000s times back-to-back.
The logfile created by the -Xloggc option looks like this:
20085.784: [GC 20085.785: [ParNew
Desired survivor size 32768 bytes, new threshold 0 (max 0)
: 71552K->0K(71616K), 0.0634410 secs] 380477K->312523K(511936K), 0.0639970 secs]
Total time for which application threads were stopped: 0.0652310 seconds
Application time: 8.7885840 seconds
Total time for which application threads were stopped: 0.0005810 seconds
Application time: 0.0005620 seconds
Total time for which application threads were stopped: 0.0006080 seconds
Application time: 0.0002630 seconds
Total time for which application threads were stopped: 0.0004410 seconds
Application time: 0.0001790 seconds
Total time for which application threads were stopped: 0.0005480 seconds
Application time: 0.0001670 seconds
Total time for which application threads were stopped: 0.0003440 seconds
Application time: 0.0001210 seconds
Total time for which application threads were stopped: 0.0004450 seconds
Application time: 0.0001590 seconds
Total time for which application threads were stopped: 0.0004220 seconds
Application time: 0.0002180 seconds
20095.265: [GC 20095.265: [ParNew
Desired survivor size 32768 bytes, new threshold 0 (max 0)
: 71552K->0K(71616K), 0.0753260 secs] 384282K->317322K(511936K), 0.0759450 secs]
Total time for which application threads were stopped: 0.0767720 seconds
Application time: 0.3346050 seconds
While the "Total time.." lines were printed, the app was unreachable. "Application time" usually marks the time the app was running between two garbage collections. It seems as if the garbage collector tries to stop the app, can't do it for whatever reason and tries again a moment later.
We've tested any suitable garbage collector, we've tried out j2sdk1.4.2_02, j2sdk1.4.2_07, jdk1.5.0_01, jdk1.5.0_02, we've tried different machines to exclude hardware failure. It is hard to reproduce in a test environment but we've seen a few lines like above on a windows box too (no longer blocking times, tough). There aren't any OutOfMemoryErrors, the heap management looks fine. After GC about 3/4 of the heap are freed even while the above problem occurs, so we're ruling out a memory leak.
Maybe someone here has stumbled across that problem or has any ideas what could trigger such a behaviour? After two weeks of debugging I've run out of ideas on where to look for a bug.
Yours remotely,
Stefan

Answer 1

hi there
Can u add -XX:+PrintHeapAtGC -XX:+PrintGC and show the portion of the log files?
You are using the -XX:+ParNewGC? Any -XX:+UseConcMarkSweepGC?
Also, add -XX:+DisableExplicitGC
Hope this helps.