ParNew long pauses in Tomcat 6 on T5120.

Question

I am running some simmulated user load for 50 concurent users on a Java application running inside Tomcat 6 on Sun's T5120 (8 processor 8G RAM) on Solaris 10. The utilization of the application seens to be ok (90-95%), but sometimes JVM goes into long GC pauses that are not acceptable for in my case. The longest pauses I can afford is about 10 seconds. The only other thing that's running on the machine is apache (mod_jk) that forwards the requests to the Tomcat process.
My GC Settings:
GC_VERBOSE_OPTS="-XX:+PrintTenuringDistribution -Xloggc:${CATALINA_HOME}/logs/gc_${TSTAMP}.log"
GC_VERBOSE="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime $GC_VERBOSE_OPTS"
MEMORY_SETTINGS="-Xms6912m -Xmx6912m -XX:NewSize=1728m -XX:MaxNewSize=1728m -XX:SurvivorRatio=8 -Xss256k -d64 -XX:PermSize=192M -XX:MaxPermSize=192M"
GC_SETTINGS=" -XX:+DisableExplicitGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -X:ParallelGCThreads=6
The output of verbose GC just before it goes into very long 55 second pause:
76824.094: [GC 76824.094: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age   1:   28658536 bytes,   28658536 total
- age   2:   78258856 bytes,  106917392 total
- age   3:   47935016 bytes,  154852408 total
: 1592576K->152602K(1592576K), 0.7859117 secs] 4605449K->3213877K(6900992K), 0.7864152 secs]
Total time for which application threads were stopped: 0.7876342 seconds
Application time: 17.9823540 seconds
Total time for which application threads were stopped: 0.0037981 seconds
Application time: 0.0002300 seconds
76842.868: [Full GC 76842.868: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age   1:   16586944 bytes,   16586944 total
- age   2:   24580024 bytes,   41166968 total
- age   3:   77453264 bytes,  118620232 total
- age   4:   41331496 bytes,  159951728 total
: 1568283K->157516K(1592576K), 0.5095165 secs] 4629559K->3218791K(6900992K), 0.5099791 secs]
Total time for which application threads were stopped: 0.5109060 seconds
Application time: 21.7442444 seconds
76865.132: [GC 76865.132: [ParNew
Desired survivor size 163027352 bytes, new threshold 5 (max 31)
- age   1:   70949544 bytes,   70949544 total
- age   2:     325496 bytes,   71275040 total
- age   3:   16995328 bytes,   88270368 total
- age   4:   51961024 bytes,  140231392 total
- age   5:   39539328 bytes,  179770720 total
: 1573196K->176896K(1592576K), 4.8568739 secs] 4634471K->3402338K(6900992K), 4.8573679 secs]
Total time for which application threads were stopped: 4.8667158 seconds
Application time: 0.0000997 seconds
76870.002: [GC [1 CMS-initial-mark: 3225442K(5308416K)] 3425594K(6900992K), 1.2304280 secs]
Total time for which application threads were stopped: 1.2421993 seconds
76871.233: [CMS-concurrent-mark-start]
Application time: 3.9732836 seconds
76875.212: [GC 76875.212: [ParNew
Desired survivor size 163027352 bytes, new threshold 5 (max 31)
- age   1:   89811184 bytes,   89811184 total
- age   2:   47431856 bytes,  137243040 total
- age   3:     229704 bytes,  137472744 total
- age   4:   10895976 bytes,  148368720 total
- age   5:   31371840 bytes,  179740560 total
: 1592576K->176896K(1592576K), 19.5701529 secs] 4818018K->3616356K(6900992K), 19.6394327 secs]
*Total time for which application threads were stopped: 19.6456021 seconds*
Application time: 5.3914509 seconds
76900.622: [GC 76900.622: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age   1:   73583152 bytes,   73583152 total
- age   2:   73057080 bytes,  146640232 total
- age   3:   32981616 bytes,  179621848 total
- age   4:      98336 bytes,  179720184 total
- age   5:      12880 bytes,  179733064 total
: 1592576K->176896K(1592576K), 54.7521974 secs] 5032036K->3921814K(6900992K), 54.7527833 secs]
*Total time for which application threads were stopped: 55.1387374 seconds*
Application time: 5.3171414 seconds
76960.700: [GC 76960.700: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age   1:   66359848 bytes,   66359848 total
- age   2:   62758584 bytes,  129118432 total
- age   3:   50477568 bytes,  179596000 total
: 1592576K->176896K(1592576K), 7.1058428 secs] 5337494K->4095887K(6900992K), 7.1063802 secs]
*Total time for which application threads were stopped: 7.1077796 seconds*
Application time: 12.7908379 seconds
Total time for which application threads were stopped: 0.0042312 seconds
Application time: 0.3358069 seconds
76980.938: [GC 76980.939: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age   1:   69012896 bytes,   69012896 total
- age   2:   56309432 bytes,  125322328 total
- age   3:   54304696 bytes,  179627024 total
: 1592576K->176896K(1592576K), 2.5164799 secs] 5512702K->4233158K(6900992K), 2.5169843 secs]
Total time for which application threads were stopped: 2.5178621 seconds
Application time: 14.8490191 seconds
Total time for which application threads were stopped: 0.0070005 seconds
Application time: 0.0003056 seconds
76998.312: [Full GC 76998.313: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age   1:   38875304 bytes,   38875304 total
- age   2:   63941104 bytes,  102816408 total
- age   3:   56090408 bytes,  158906816 total
: 1592576K->156275K(1592576K), 1.1885482 secs] 5648848K->4265573K(6900992K), 1.1891583 secs]
Total time for which application threads were stopped: 1.1899499 seconds
Application time: 8.8337996 seconds
I would appreciate some help tunning GC patameters to eliminate the long pauses.
Edited by: MikhailPDX on May 6, 2008 11:53 AM

Answer 1

I saw this posting before about pauses and I don't think I am having the same issue since I am using Java(TM) 2 Runtime Environment, Standard Edition 1.5.0_15-b04 Java HotSpot(TM) 64-Bit Server VM) where the mentioned pauses bug was supposedly fixed.
Edited by: MikhailPDX on May 7, 2008 10:30 AM