Long *DEADLY* GC pauses with parallel collector

Question

Hi everybody,
I am having a major problem with the Java garbage collector. My company runs a web server that processes an average of 17 requests per second (with peaks up to 50,000/sec). Normally this load is handled fine by the application. We have done a lot of work to make sure the memory used by our application does not grow over time (i.e. profiled for leaks), but we still have full GCs being done every 15 mins or so. --But all that is fine, and normal operation. Due to the nature of our server, responsiveness is important so we must keep the max pause and average pause as low as possible, and thus maximize throughput.
However, even though we're running the parallelGC, occasionally we run into a problem where the garbage collector pauses the whole JVM. This is deadly to our servers, and causes requests to queue up and the machine's system load to skyrocket. Sometimes this ends up taking down the whole machine. The strange part is that our reporting indicates that all other aspects of our systems (the Java app, the network, the database, etc.) are all running completely fine when these pauses happen --it seems that nothing triggers them.
Here is an illustration (taken straight from the GC viewer) of what happens:
http://www.darylbeattie.com/img/gc_high_load.jpg
And here are our JVM parameters (minus the GC log options, classpath and other common stuff):
-server -Xms1288m -Xmx1288m -Xss128k -XX:ThreadStackSize=128 -XX:+DisableExplicitGC -XX:+UseAdaptiveSizePolicy -XX:+UseParallelGC
We're using JDK 1.5.0_05, and we've tried to upgrade to 1.6 but found that the GC issues were still there. We don't want to upgrade until we're sure that we can live with the new GC.
We have fiddled with many different GC options over the years. We have tried the conc-mark-sweep GC and found it to be worse. We would be pleased with the parallel GC as long as we could get rid of these strange pauses.
My questions is: What can be done to alleviate these strange pauses?
Also, I didn't see a forum to post questions about tweaking the GC, so I'm posting to this one. Please direct me to the right forum if this is the wrong one. **X-post from "bigadmin" forum where I got no replies**

Answer 1

So with the Concurrent Low Pause collector things are far worse.
The server only runs for 45 minutes or so before there is a major collection that results in pausing our application for 18 seconds --this is almost 3 times worse than the deadly max pauses we were getting with the parallel GC, which were around 7 seconds at their worst.
The settings I was using for the GC were: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:+PrintGCDetails -XX:-TraceClassUnloading
What advice can you folks give me to help wrestle these max pauses down to a smaller amount? 7 seconds is enough to take down our server under high load... we'd like it to be around 1 second max.