Help me stabilize this jRun configuration (CF9/Win2k3/IIS6)

Question

Hi everyone,
We've been struggling to keep our multi-server configuration stable for quite some time now.  At the end of last month we were running under CF 7.0.2 on a two servers setup (one instance each).  At that point we managed to get our uptime to around 1 week per instance before they would restart by themselves.  Since the beginning of the month we upgraded to CF 9 and we're back to square one with multi-restart a day.
Our current configuration is 2 Win2k3 servers, running a cluster of 4 instances, 2 instances per server. At this point we are pretty certain this is due to improper JVM settings.
We've been toying with them and while some are more stable than others we never quite got it right.
From the default:
java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/
To currently:
java.args=-server -Xmx896m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=512m -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/ -verbose:gc -Xloggc:c:/Jrun4/logs/gc/gcInstance1b.log
We have determined that we do need more than the default 512MB simply by monitoring with FusionReactor, on average our amount of memory consumed is hovering in the mid 300MB and can go up to low 700MB under heavy load.
Most of the crash will be logged in jrun4/bin/hs_err_pid*.log always an "Out of swap space"
The latest crash from yesterday generated the following log:
From it, I gather that its the PSPermGen that is full (most logs will show the same before a crash), which is why we increased MaxPermSize but the total still show as 107520K!??! I've attached the GC log after the HS_ERR.  No one here is a jRun expert, just developpers tasked with fixing this as best as we possibly can so any help would be greatly appreciated!!
# A fatal error has been detected by the Java Runtime Environment:
# java.lang.OutOfMemoryError: requested 2264248 bytes for Chunk::new. Out of swap space?
#  Internal Error (allocation.cpp:215), pid=15224, tid=4960
#  Error: Chunk::new
# JRE version: 6.0_14-b08
# Java VM: Java HotSpot(TM) Server VM (14.0-b16 mixed mode windows-x86 )
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
---------------  T H R E A D  ---------------
Current thread (0x5c69c800):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=4960, stack(0x5ce70000,0x5cf70000)]
Stack: [0x5ce70000,0x5cf70000],  sp=0x5cf6e90c,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x306157]
V  [jvm.dll+0xf7d2c]
V  [jvm.dll+0x3915c]
V  [jvm.dll+0x393f3]
V  [jvm.dll+0x16a470]
V  [jvm.dll+0xa9552]
V  [jvm.dll+0xd7843]
V  [jvm.dll+0xd7f6c]
V  [jvm.dll+0x9e082]
V  [jvm.dll+0xda0e0]
V  [jvm.dll+0xdac6f]
V  [jvm.dll+0x2e63f4]
V  [jvm.dll+0x267dac]
C  [MSVCR71.dll+0x9565]
C  [kernel32.dll+0x2482f]
Current CompileTask:
C2:3749  !   cfdspInit2ecfm988339296.runPage()Ljava/lang/Object; (4792 bytes)
---------------  P R O C E S S  ---------------
Java Threads: ( => current thread )
  0x6f812800 JavaThread "task" daemon [_thread_blocked, id=12580, stack(0x6acb0000,0x6adb0000)]
  0x79981800 JavaThread "task" daemon [_thread_blocked, id=14328, stack(0x6a940000,0x6aa40000)]
  0x7882a800 JavaThread "task" daemon [_thread_blocked, id=6132, stack(0x6a840000,0x6a940000)]
  0x6eb97800 JavaThread "task" daemon [_thread_blocked, id=1756, stack(0x6a740000,0x6a840000)]
  0x6f221800 JavaThread "task" daemon [_thread_blocked, id=14024, stack(0x6a640000,0x6a740000)]
  0x6f668800 JavaThread "task" daemon [_thread_blocked, id=15956, stack(0x69740000,0x69840000)]
  0x62887400 JavaThread "RMI TCP Connection(10140)-192.168.40.35" daemon [_thread_in_native, id=11024, stack(0x628d0000,0x629d0000)]
  0x61ec3800 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=5848, stack(0x60b80000,0x60c80000)]
  0x788d2c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=13348, stack(0x60460000,0x60560000)]
  0x7955e400 JavaThread "RMI TCP Connection(10136)-192.168.40.35" daemon [_thread_in_native, id=10912, stack(0x639d0000,0x63ad0000)]
  0x621c5400 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=3308, stack(0x64ed0000,0x64fd0000)]
  0x619fb400 JavaThread "RMI TCP Connection(10132)-192.168.40.34" daemon [_thread_in_native, id=7784, stack(0x642d0000,0x643d0000)]
  0x62719c00 JavaThread "jrpp-821" [_thread_in_native, id=5124, stack(0x60360000,0x60460000)]
  0x78751400 JavaThread "jrpp-820" [_thread_in_native, id=16240, stack(0x60180000,0x60280000)]
  0x7fa71400 JavaThread "jrpp-819" [_thread_in_native, id=15228, stack(0x5fb80000,0x5fc80000)]
  0x78ccb400 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=6224, stack(0x64ad0000,0x64bd0000)]
  0x6edc8c00 JavaThread "RMI TCP Connection(10123)-192.168.40.34" daemon [_thread_in_native, id=4592, stack(0x632d0000,0x633d0000)]
  0x6e37a400 JavaThread "jrpp-818" [_thread_in_native, id=2712, stack(0x617d0000,0x618d0000)]
  0x62816c00 JavaThread "jrpp-817" [_thread_in_native, id=11972, stack(0x60980000,0x60a80000)]
  0x6fab2400 JavaThread "RMI TCP Connection(10074)-192.168.40.34" daemon [_thread_in_native, id=8124, stack(0x64dd0000,0x64ed0000)]
  0x78769400 JavaThread "RMI TCP Connection(10050)-192.168.40.34" daemon [_thread_in_native, id=8720, stack(0x60a80000,0x60b80000)]
  0x5c82ac00 JavaThread "RMI TCP Connection(10030)-192.168.40.34" daemon [_thread_in_native, id=12620, stack(0x62cd0000,0x62dd0000)]
  0x61ee6800 JavaThread "task" daemon [_thread_in_native, id=5936, stack(0x60d80000,0x60e80000)]
  0x6e239c00 JavaThread "scheduler-28" [_thread_blocked, id=13084, stack(0x69a40000,0x69b40000)]
  0x61cc2400 JavaThread "RMI TCP Connection(10001)-192.168.40.34" daemon [_thread_in_native, id=3300, stack(0x5fe80000,0x5ff80000)]
  0x5da0a400 JavaThread "scheduler-27" [_thread_blocked, id=5764, stack(0x62fd0000,0x630d0000)]
  0x5c82a400 JavaThread "scheduler-26" [_thread_blocked, id=15592, stack(0x61290000,0x61390000)]
  0x7963bc00 JavaThread "scheduler-25" [_thread_blocked, id=1564, stack(0x60880000,0x60980000)]
  0x61f3a800 JavaThread "RMI RenewClean-[192.168.40.34:4835]" daemon [_thread_in_native, id=12824, stack(0x62bd0000,0x62cd0000)]
  0x6df88800 JavaThread "web-3" [_thread_in_native, id=10008, stack(0x667d0000,0x668d0000)]
  0x6e27dc00 JavaThread "scheduler-3" [_thread_blocked, id=7512, stack(0x63ad0000,0x63bd0000)]
  0x5dd3b400 JavaThread "scheduler-24" [_thread_blocked, id=7552, stack(0x629d0000,0x62ad0000)]
  0x6ebcd800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=8736, stack(0x60080000,0x60180000)]
  0x6f595400 JavaThread "FusionReactor Web Server (Server Thread Pool Member Thread-8617)" daemon [_thread_blocked, id=7408, stack(0x5fd80000,0x5fe80000)]
  0x5d9d3800 JavaThread "scheduler-2" [_thread_blocked, id=1884, stack(0x60e80000,0x60f80000)]
  0x619d4400 JavaThread "scheduler-23" [_thread_blocked, id=7984, stack(0x616d0000,0x617d0000)]
  0x62748800 JavaThread "RMI RenewClean-[192.168.40.34:4547]" daemon [_thread_blocked, id=12652, stack(0x60c80000,0x60d80000)]
  0x61a28400 JavaThread "MultiThreadedHttpConnectionManager cleanup" daemon [_thread_blocked, id=7212, stack(0x5fc80000,0x5fd80000)]
  0x00717400 JavaThread "DestroyJavaVM" [_thread_blocked, id=12764, stack(0x00030000,0x00130000)]
  0x6261dc00 JavaThread "scheduler-1" [_thread_blocked, id=9928, stack(0x64800000,0x64900000)]
  0x5d508400 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=11168, stack(0x65fd0000,0x660d0000)]
  0x62186800 JavaThread "worker #4" daemon [_thread_blocked, id=15864, stack(0x66cd0000,0x66dd0000)]
  0x6e827c00 JavaThread "worker #3" daemon [_thread_blocked, id=15700, stack(0x663d0000,0x664d0000)]
  0x625a4c00 JavaThread "worker #2" daemon [_thread_blocked, id=2940, stack(0x662d0000,0x663d0000)]
  0x622d5000 JavaThread "worker #1" daemon [_thread_blocked, id=12548, stack(0x661d0000,0x662d0000)]
  0x5d7ffc00 JavaThread "worker #0" daemon [_thread_blocked, id=9384, stack(0x660d0000,0x661d0000)]
  0x5d406800 JavaThread "Thread-49" [_thread_blocked, id=9248, stack(0x65c20000,0x65d20000)]
  0x622a8000 JavaThread "obj-skimmer" daemon [_thread_blocked, id=3680, stack(0x640d0000,0x641d0000)]
  0x620d7800 JavaThread "obj-skimmer" daemon [_thread_blocked, id=12312, stack(0x63fd0000,0x640d0000)]
  0x62351800 JavaThread "obj-skimmer" daemon [_thread_blocked, id=11708, stack(0x63dd0000,0x63ed0000)]
  0x5dedd800 JavaThread "obj-skimmer" daemon [_thread_blocked, id=8800, stack(0x5f9d0000,0x5fad0000)]
  0x5dec6400 JavaThread "obj-skimmer" daemon [_thread_blocked, id=14688, stack(0x5f8d0000,0x5f9d0000)]
  0x5d519c00 JavaThread "obj-skimmer" daemon [_thread_blocked, id=4868, stack(0x5f7d0000,0x5f8d0000)]
  0x5d7cec00 JavaThread "obj-skimmer" daemon [_thread_blocked, id=12844, stack(0x5f6d0000,0x5f7d0000)]
  0x62515800 JavaThread "obj-skimmer" daemon [_thread_blocked, id=7396, stack(0x5f5d0000,0x5f6d0000)]
  0x6283e400 JavaThread "obj-skimmer" daemon [_thread_blocked, id=6164, stack(0x5f4d0000,0x5f5d0000)]
  0x5c731c00 JavaThread "scheduler-0" [_thread_blocked, id=11724, stack(0x5f3d0000,0x5f4d0000)]
  0x621e9000 JavaThread "Thread-41" daemon [_thread_blocked, id=3196, stack(0x5ef90000,0x5f090000)]
  0x5d32f800 JavaThread "jndi-5" [_thread_blocked, id=2808, stack(0x63ed0000,0x63fd0000)]
  0x5da96c00 JavaThread "Timer-8" daemon [_thread_blocked, id=9864, stack(0x733a0000,0x734a0000)]
  0x61f0e000 JavaThread "Timer-7" [_thread_blocked, id=3116, stack(0x732a0000,0x733a0000)]
  0x623f2c00 JavaThread "Timer-6" [_thread_blocked, id=3528, stack(0x731a0000,0x732a0000)]
  0x5d327c00 JavaThread "scheduler-22" [_thread_blocked, id=3612, stack(0x730a0000,0x731a0000)]
  0x621bc400 JavaThread "FusionReactor Web Server (Server Thread Pool Member Thread-36)" daemon [_thread_blocked, id=13668, stack(0x72ea0000,0x72fa0000)]
  0x61981400 JavaThread "FusionReactor Web Server (Server Thread Pool Member Thread-35)" daemon [_thread_blocked, id=15540, stack(0x72da0000,0x72ea0000)]
  0x5d9dcc00 JavaThread "FusionReactor Request Monitor" [_thread_blocked, id=3464, stack(0x72ca0000,0x72da0000)]
  0x5ddeb400 JavaThread "FusionReactor FD/R Remote Data Collector for {CF_Instance2a.jrun4.lcbwebott02}" [_thread_blocked, id=9760, stack(0x72ba0000,0x72ca0000)]
  0x5dcf9400 JavaThread "FusionReactor Web Server" [_thread_in_native, id=12396, stack(0x72aa0000,0x72ba0000)]
  0x5df7a400 JavaThread "FusionReactor Enterprise Service - Local Metrics Cache" daemon [_thread_blocked, id=9780, stack(0x729a0000,0x72aa0000)]
  0x5def8400 JavaThread "Timer-5" daemon [_thread_blocked, id=1640, stack(0x728a0000,0x729a0000)]
  0x621a4c00 JavaThread "Timer-4" daemon [_thread_blocked, id=13496, stack(0x727a0000,0x728a0000)]
  0x5da2a400 JavaThread "Timer-3" daemon [_thread_blocked, id=5568, stack(0x726a0000,0x727a0000)]
  0x61b60c00 JavaThread "Timer-2" daemon [_thread_blocked, id=10724, stack(0x725a0000,0x726a0000)]
  0x622bbc00 JavaThread "Timer-1" daemon [_thread_blocked, id=8252, stack(0x724a0000,0x725a0000)]
  0x5d840c00 JavaThread "Thread-25" daemon [_thread_blocked, id=13204, stack(0x71fa0000,0x720a0000)]
  0x6e037c00 JavaThread "Timer-0" daemon [_thread_blocked, id=10944, stack(0x70670000,0x70770000)]
  0x5d815400 JavaThread "task" daemon [_thread_blocked, id=8680, stack(0x71da0000,0x71ea0000)]
  0x5dea9800 JavaThread "task" daemon [_thread_blocked, id=8052, stack(0x71570000,0x71670000)]
  0x6228c000 JavaThread "Thread-23" daemon [_thread_blocked, id=11888, stack(0x71170000,0x71270000)]
  0x6228c800 JavaThread "Thread-22" daemon [_thread_blocked, id=15232, stack(0x70f70000,0x71070000)]
  0x62289000 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=8972, stack(0x70e70000,0x70f70000)]
  0x5da2ac00 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=11732, stack(0x70d70000,0x70e70000)]
  0x6e181c00 JavaThread "task" daemon [_thread_blocked, id=15432, stack(0x70870000,0x70970000)]
  0x5db7e400 JavaThread "task" daemon [_thread_blocked, id=15828, stack(0x70770000,0x70870000)]
  0x61c3dc00 JavaThread "Thread-21" daemon [_thread_blocked, id=12384, stack(0x6d9b0000,0x6dab0000)]
  0x6e052800 JavaThread "Thread-20" daemon [_thread_blocked, id=1760, stack(0x6d500000,0x6d600000)]
  0x61af4400 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=8832, stack(0x6d400000,0x6d500000)]
  0x627dd400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=5612, stack(0x6d1b0000,0x6d2b0000)]
  0x5da64000 JavaThread "Thread-19" [_thread_blocked, id=15120, stack(0x6c6b0000,0x6c7b0000)]
  0x6229d800 JavaThread "task" daemon [_thread_blocked, id=6736, stack(0x6ccb0000,0x6cdb0000)]
  0x5de23400 JavaThread "task" daemon [_thread_blocked, id=15424, stack(0x6cbb0000,0x6ccb0000)]
  0x621f6c00 JavaThread "task" daemon [_thread_blocked, id=6324, stack(0x6b0b0000,0x6b1b0000)]
  0x5d3b5400 JavaThread "scheduler-21" [_thread_blocked, id=4284, stack(0x6aeb0000,0x6afb0000)]
  0x624c2800 JavaThread "Thread-17" daemon [_thread_blocked, id=6100, stack(0x6bdb0000,0x6beb0000)]
  0x624e6800 JavaThread "Thread-16" daemon [_thread_blocked, id=9080, stack(0x6bcb0000,0x6bdb0000)]
  0x621e5400 JavaThread "task" daemon [_thread_blocked, id=14136, stack(0x6bbb0000,0x6bcb0000)]
  0x6215e800 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=9112, stack(0x6b9b0000,0x6bab0000)]
  0x62522000 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=7220, stack(0x6b8b0000,0x6b9b0000)]
  0x622a8800 JavaThread "Thread-15" daemon [_thread_blocked, id=15848, stack(0x6afb0000,0x6b0b0000)]
  0x622b1400 JavaThread "task" daemon [_thread_blocked, id=4512, stack(0x6adb0000,0x6aeb0000)]
  0x62163400 JavaThread "Thread-14" daemon [_thread_blocked, id=3944, stack(0x6abb0000,0x6acb0000)]
  0x622b2c00 JavaThread "task" daemon [_thread_blocked, id=12724, stack(0x6aab0000,0x6abb0000)]
  0x623aa800 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=7736, stack(0x6a540000,0x6a640000)]
  0x623a9400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=13556, stack(0x6a440000,0x6a540000)]
  0x62010000 JavaThread "scheduler-20" [_thread_blocked, id=13884, stack(0x69d40000,0x69e40000)]
  0x5d4e0c00 JavaThread "Thread-13" daemon [_thread_blocked, id=14880, stack(0x69640000,0x69740000)]
  0x621a0000 JavaThread "Thread-12" daemon [_thread_blocked, id=10732, stack(0x69440000,0x69540000)]
  0x5d41e800 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=8964, stack(0x69340000,0x69440000)]
  0x62216c00 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=6644, stack(0x69240000,0x69340000)]
  0x61b5b800 JavaThread "scheduler-19" [_thread_blocked, id=10856, stack(0x68f40000,0x69040000)]
  0x62178400 JavaThread "scheduler-18" [_thread_blocked, id=10184, stack(0x68e40000,0x68f40000)]
  0x62177000 JavaThread "scheduler-17" [_thread_blocked, id=6816, stack(0x68d40000,0x68e40000)]
  0x62176000 JavaThread "scheduler-16" [_thread_blocked, id=7700, stack(0x68c40000,0x68d40000)]
  0x62174000 JavaThread "scheduler-15" [_thread_blocked, id=8296, stack(0x68b40000,0x68c40000)]
  0x61b60000 JavaThread "scheduler-14" [_thread_blocked, id=6212, stack(0x68a40000,0x68b40000)]
  0x61b5e800 JavaThread "scheduler-13" [_thread_blocked, id=6868, stack(0x68940000,0x68a40000)]
  0x61b5d800 JavaThread "scheduler-12" [_thread_blocked, id=12472, stack(0x68840000,0x68940000)]
  0x62168400 JavaThread "scheduler-11" [_thread_blocked, id=9468, stack(0x68740000,0x68840000)]
  0x62167000 JavaThread "scheduler-10" [_thread_blocked, id=7224, stack(0x68640000,0x68740000)]
  0x62165800 JavaThread "scheduler-9" [_thread_blocked, id=3520, stack(0x68540000,0x68640000)]
  0x5d41b400 JavaThread "scheduler-8" [_thread_blocked, id=11824, stack(0x68440000,0x68540000)]
  0x5d419c00 JavaThread "scheduler-7" [_thread_blocked, id=4956, stack(0x68340000,0x68440000)]
  0x620dc400 JavaThread "scheduler-6" [_thread_blocked, id=14988, stack(0x68240000,0x68340000)]
  0x620db400 JavaThread "scheduler-5" [_thread_blocked, id=9580, stack(0x68140000,0x68240000)]
  0x5d3cf400 JavaThread "scheduler-4" [_thread_blocked, id=10220, stack(0x68040000,0x68140000)]
  0x5dc40c00 JavaThread "scheduler-3" [_thread_blocked, id=6780, stack(0x67ed0000,0x67fd0000)]
  0x621a4400 JavaThread "scheduler-2" [_thread_blocked, id=9216, stack(0x67dd0000,0x67ed0000)]
  0x619d5400 JavaThread "scheduler-1" [_thread_blocked, id=7032, stack(0x67ad0000,0x67bd0000)]
  0x5dd32400 JavaThread "scheduler-0" [_thread_blocked, id=15420, stack(0x665d0000,0x666d0000)]
  0x5df2bc00 JavaThread "Transaction Domain PrivateJmsDomain" [_thread_blocked, id=15384, stack(0x67cd0000,0x67dd0000)]
  0x5d2f7800 JavaThread "clock" daemon [_thread_blocked, id=7236, stack(0x67bd0000,0x67cd0000)]
  0x5d41f400 JavaThread "Transaction Domain DefaultDomain" [_thread_blocked, id=3908, stack(0x679d0000,0x67ad0000)]
  0x6223bc00 JavaThread "tyrex.util.daemonMaster" daemon [_thread_blocked, id=15468, stack(0x677d0000,0x678d0000)]
  0x621bcc00 JavaThread "task" daemon [_thread_blocked, id=7364, stack(0x676d0000,0x677d0000)]
  0x5df2d400 JavaThread "task" daemon [_thread_blocked, id=6052, stack(0x670d0000,0x671d0000)]
  0x5d927800 JavaThread "Thread-10" daemon [_thread_blocked, id=8776, stack(0x66bd0000,0x66cd0000)]
  0x61944c00 JavaThread "Thread-9" daemon [_thread_blocked, id=12100, stack(0x66ad0000,0x66bd0000)]
  0x620be400 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=13248, stack(0x669d0000,0x66ad0000)]
  0x5d52f400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=15988, stack(0x668d0000,0x669d0000)]
  0x5d8dd400 JavaThread "task" daemon [_thread_blocked, id=12096, stack(0x664d0000,0x665d0000)]
  0x5d4cd400 JavaThread "task" daemon [_thread_blocked, id=12332, stack(0x65ed0000,0x65fd0000)]
  0x5dc4e400 JavaThread "Thread-8" daemon [_thread_blocked, id=11804, stack(0x659d0000,0x65ad0000)]
  0x5dce2400 JavaThread "Thread-7" daemon [_thread_blocked, id=13224, stack(0x658d0000,0x659d0000)]
  0x61ca1c00 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=992, stack(0x657d0000,0x658d0000)]
  0x61abb400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=8984, stack(0x656d0000,0x657d0000)]
  0x5d87ac00 JavaThread "jndi-3" [_thread_blocked, id=10016, stack(0x654d0000,0x655d0000)]
  0x5da66800 JavaThread "jndi-2" [_thread_in_native, id=15032, stack(0x653d0000,0x654d0000)]
  0x5de6bc00 JavaThread "jndi-1" [_thread_blocked, id=10964, stack(0x652d0000,0x653d0000)]
  0x61ca2400 JavaThread "jndi-0" [_thread_blocked, id=13736, stack(0x651d0000,0x652d0000)]
  0x61918000 JavaThread "task" daemon [_thread_blocked, id=1892, stack(0x64cd0000,0x64dd0000)]
  0x61917c00 JavaThread "task" daemon [_thread_blocked, id=2692, stack(0x64bd0000,0x64cd0000)]
  0x61a64400 JavaThread "Thread-6" daemon [_thread_blocked, id=3580, stack(0x644d0000,0x645d0000)]
  0x5db8d400 JavaThread "Thread-5" daemon [_thread_blocked, id=15852, stack(0x645d0000,0x646d0000)]
  0x5df10800 JavaThread "Thread-4" daemon [_thread_blocked, id=12984, stack(0x643d0000,0x644d0000)]
  0x5da1c000 JavaThread "RMI RenewClean-[192.168.40.35:1941]" daemon [_thread_blocked, id=9096, stack(0x641d0000,0x642d0000)]
  0x5df7b800 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=6672, stack(0x63cd0000,0x63dd0000)]
  0x5dbef800 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=13076, stack(0x63bd0000,0x63cd0000)]
  0x5d335c00 JavaThread "jms-fifo-5" [_thread_blocked, id=5108, stack(0x638d0000,0x639d0000)]
  0x5dc50400 JavaThread "jms-fifo-4" [_thread_blocked, id=588, stack(0x637d0000,0x638d0000)]
  0x5daaa000 JavaThread "jms-fifo-3" [_thread_blocked, id=9968, stack(0x636d0000,0x637d0000)]
  0x5d4eb800 JavaThread "jms-fifo-2" [_thread_blocked, id=5636, stack(0x635d0000,0x636d0000)]
  0x61a40400 JavaThread "jms-fifo-1" [_thread_blocked, id=15884, stack(0x634d0000,0x635d0000)]
  0x5dbf6800 JavaThread "RMI RenewClean-[192.168.40.35:2178]" daemon [_thread_blocked, id=8716, stack(0x631d0000,0x632d0000)]
  0x5d455c00 JavaThread "RMI RenewClean-[192.168.40.34:2233]" daemon [_thread_blocked, id=14028, stack(0x614d0000,0x615d0000)]
  0x5dc71400 JavaThread "RMI RenewClean-[192.168.40.34:3534]" daemon [_thread_blocked, id=7068, stack(0x613d0000,0x614d0000)]
  0x5d509000 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=10984, stack(0x5ed90000,0x5ee90000)]
  0x5d819c00 JavaThread "snapshot thread" daemon [_thread_blocked, id=10840, stack(0x5eb90000,0x5ec90000)]
  0x5d7b3400 JavaThread "discovery announcement" daemon [_thread_blocked, id=6580, stack(0x5ea90000,0x5eb90000)]
  0x5c810800 JavaThread "multicast request" daemon [_thread_in_native, id=11208, stack(0x5e990000,0x5ea90000)]
  0x5c74a400 JavaThread "unicast request" daemon [_thread_in_native, id=1328, stack(0x5e890000,0x5e990000)]
  0x5d303800 JavaThread "event expire" daemon [_thread_blocked, id=7796, stack(0x5e790000,0x5e890000)]
  0x5c7b8800 JavaThread "service expire" daemon [_thread_blocked, id=15348, stack(0x5e690000,0x5e790000)]
  0x5c7b8400 JavaThread "Thread-3" daemon [_thread_blocked, id=13080, stack(0x5e590000,0x5e690000)]
  0x5d3b6800 JavaThread "Thread-2" daemon [_thread_blocked, id=6772, stack(0x5e490000,0x5e590000)]
  0x5d3b6400 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=13304, stack(0x5e390000,0x5e490000)]
  0x5d4e1c00 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=10632, stack(0x5e290000,0x5e390000)]
  0x5d3d5400 JavaThread "GC Daemon" daemon [_thread_blocked, id=13068, stack(0x5e190000,0x5e290000)]
  0x5d492800 JavaThread "RMI Reaper" [_thread_blocked, id=6540, stack(0x5e090000,0x5e190000)]
  0x5c6cc000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=10780, stack(0x5df90000,0x5e090000)]
  0x5c69fc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=8196, stack(0x5cf70000,0x5d070000)]
=>0x5c69c800 JavaThread "CompilerThread1" daemon [_thread_in_native, id=4960, stack(0x5ce70000,0x5cf70000)]
  0x5c697c00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=5152, stack(0x5cd70000,0x5ce70000)]
  0x5c696400 JavaThread "Attach Listener" daemon [_thread_blocked, id=14576, stack(0x5cc70000,0x5cd70000)]
  0x5c695400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=16064, stack(0x5cb70000,0x5cc70000)]
  0x5bf5f800 JavaThread "Finalizer" daemon [_thread_blocked, id=16192, stack(0x5ca70000,0x5cb70000)]
  0x5bf5e400 JavaThread "Reference Handler" daemon [_thread_blocked, id=7228, stack(0x5c970000,0x5ca70000)]
Other Threads:
  0x5c691000 VMThread [stack: 0x5c870000,0x5c970000] [id=8468]
  0x5c6a1c00 WatcherThread [stack: 0x5d070000,0x5d170000] [id=10620]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGen      total 89856K, used 19025K [0x55490000, 0x5b6f0000, 0x5b810000)
  eden space 79232K, 16% used [0x55490000,0x561a64c0,0x5a1f0000)
  from space 10624K, 52% used [0x5ac90000,0x5b20e2f8,0x5b6f0000)
  to   space 10752K, 0% used [0x5a1f0000,0x5a1f0000,0x5ac70000)
PSOldGen        total 460416K, used 308422K [0x23810000, 0x3f9b0000, 0x55490000)
  object space 460416K, 66% used [0x23810000,0x36541bb8,0x3f9b0000)
PSPermGen       total 107520K, used 106079K [0x03810000, 0x0a110000, 0x23810000)
  object space 107520K, 98% used [0x03810000,0x09fa7e40,0x0a110000)
Dynamic libraries:
0x00400000 - 0x00411000     C:\JRun4\bin\jrun.exe
0x7c800000 - 0x7c8c2000     C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f42000     C:\WINDOWS\system32\kernel32.dll
0x7d1e0000 - 0x7d27c000     C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cef000     C:\WINDOWS\system32\RPCRT4.dll
0x76f50000 - 0x76f63000     C:\WINDOWS\system32\Secur32.dll
0x78520000 - 0x785c3000     C:\WINDOWS\WinSxS\x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_x-ww_D08D0375\MSVCR 90.dll
0x6db60000 - 0x6df6a000     C:\JRun4\jre\bin\server\jvm.dll
0x77380000 - 0x77411000     C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c49000     C:\WINDOWS\system32\GDI32.dll
0x76aa0000 - 0x76acd000     C:\WINDOWS\system32\WINMM.dll
0x7c340000 - 0x7c396000     C:\JRun4\bin\MSVCR71.dll
0x76290000 - 0x762ad000     C:\WINDOWS\system32\IMM32.DLL
0x7f000000 - 0x7f009000     C:\WINDOWS\system32\LPK.DLL
0x75490000 - 0x754f5000     C:\WINDOWS\system32\USP10.dll
0x6d860000 - 0x6d86c000     C:\JRun4\jre\bin\verify.dll
0x6d3e0000 - 0x6d3ff000     C:\JRun4\jre\bin\java.dll
0x6d340000 - 0x6d348000     C:\JRun4\jre\bin\hpi.dll
0x76b70000 - 0x76b7b000     C:\WINDOWS\system32\PSAPI.DLL
0x6d8a0000 - 0x6d8af000     C:\JRun4\jre\bin\zip.dll
0x6d6c0000 - 0x6d6d3000     C:\JRun4\jre\bin\net.dll
0x71c00000 - 0x71c17000     C:\WINDOWS\system32\WS2_32.dll
0x77ba0000 - 0x77bfa000     C:\WINDOWS\system32\msvcrt.dll
0x71bf0000 - 0x71bf8000     C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b61000     C:\WINDOWS\System32\mswsock.dll
0x76ed0000 - 0x76efa000     C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000     C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000     C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f85000     C:\WINDOWS\system32\rasadhlp.dll
0x5d630000 - 0x5d63c000     C:\JRun4\bin\portscan.dll
0x5f270000 - 0x5f2ca000     C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000     C:\WINDOWS\System32\wshtcpip.dll
0x68000000 - 0x68035000     C:\WINDOWS\system32\rsaenh.dll
0x76920000 - 0x769e2000     C:\WINDOWS\system32\USERENV.dll
0x71c40000 - 0x71c97000     C:\WINDOWS\system32\netapi32.dll
0x6d7b0000 - 0x6d7b6000     C:\JRun4\jre\bin\rmi.dll
0x6d6e0000 - 0x6d6e9000     C:\JRun4\jre\bin\nio.dll
0x5f190000 - 0x5f197000     C:\JRun4\bin\FusionReactor.dll
0x77d00000 - 0x77d8b000     C:\WINDOWS\system32\OLEAUT32.dll
0x77670000 - 0x777a9000     C:\WINDOWS\system32\ole32.dll
0x5f1a0000 - 0x5f1aa000     C:\WINDOWS\system32\perfos.dll
0x6d840000 - 0x6d848000     C:\JRun4\jre\bin\sunmscapi.dll
0x761b0000 - 0x76243000     C:\WINDOWS\system32\CRYPT32.dll
0x76190000 - 0x761a2000     C:\WINDOWS\system32\MSASN1.dll
0x5f090000 - 0x5f110000     C:\JRun4\servers\CF_Instance2a\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\izmjniado.dll
0x76cf0000 - 0x76d0a000     C:\WINDOWS\system32\iphlpapi.dll
0x76cd0000 - 0x76ce9000     C:\WINDOWS\system32\MPRAPI.dll
0x76df0000 - 0x76e24000     C:\WINDOWS\system32\ACTIVEDS.dll
0x76dc0000 - 0x76de8000     C:\WINDOWS\system32\adsldpc.dll
0x76b80000 - 0x76bae000     C:\WINDOWS\system32\credui.dll
0x7c8d0000 - 0x7d0cf000     C:\WINDOWS\system32\SHELL32.dll
0x7d180000 - 0x7d1d2000     C:\WINDOWS\system32\SHLWAPI.dll
0x76a80000 - 0x76a98000     C:\WINDOWS\system32\ATL.DLL
0x76e30000 - 0x76e3c000     C:\WINDOWS\system32\rtutils.dll
0x7e020000 - 0x7e02f000     C:\WINDOWS\system32\SAMLIB.dll
0x770e0000 - 0x771e8000     C:\WINDOWS\system32\SETUPAPI.dll
0x77420000 - 0x77523000     C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x- ww_D8713E55\comctl32.dll
0x5f200000 - 0x5f223000     C:\JRun4\servers\CF_Instance2a\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\CFXNeo.dll
0x5f230000 - 0x5f243000     C:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib\cfregistry.dll
0x647d0000 - 0x647e3000     C:\JRun4\servers\CF_Instance2a\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\cfregistry.dll
0x65ad0000 - 0x65c1a000     C:\JRun4\jre\bin\awt.dll
0x64a00000 - 0x64a27000     C:\WINDOWS\system32\WINSPOOL.DRV
0x77530000 - 0x775c7000     C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_5.82.3790.3959_x -ww_78FCF8D0\comctl32.dll
0x6d2e0000 - 0x6d334000     C:\JRun4\jre\bin\fontmanager.dll
0x64900000 - 0x6490e000     C:\JRun4\servers\CF_Instance2a\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\NeoUUID.dll
0x6d610000 - 0x6d619000     C:\JRun4\jre\bin\management.dll
0x5f2d0000 - 0x5f380000     K:\Instance\Core\WEB02_2a_cfx_image_mx.dll
0x73eb0000 - 0x73fd1000     C:\WINDOWS\system32\MFC42.DLL
0x60280000 - 0x60351000     C:\WINDOWS\system32\WININET.dll
0x5f380000 - 0x5f389000     C:\WINDOWS\system32\Normaliz.dll
0x5fad0000 - 0x5fb15000     C:\WINDOWS\system32\iertutil.dll
0x71bb0000 - 0x71bb9000     C:\WINDOWS\system32\WSOCK32.dll
0x5f390000 - 0x5f3cd000     C:\WINDOWS\system32\ODBC32.dll
0x762b0000 - 0x762f9000     C:\WINDOWS\system32\comdlg32.dll
0x5fb20000 - 0x5fb30000     C:\WINDOWS\system32\MSVCIRT.dll
0x607e0000 - 0x607f7000     C:\WINDOWS\system32\odbcint.dll
0x5ec90000 - 0x5ecb4000     C:\JRun4\jre\bin\jpeg.dll
0x5ecd0000 - 0x5ecdb000     C:\JRun4\servers\CF_Instance2a\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\clib_jiio_util .dll
0x61180000 - 0x6128c000     C:\JRun4\servers\CF_Instance2a\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\clib_jiio_sse2 .dll
0x5ed00000 - 0x5ed23000     C:\JRun4\jre\bin\dcpr.dll
VM Arguments:
jvm_args: -Xmx896m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=512m -XX:+UseParallelGC -Dcoldfusion.rootDir=C:\JRun4/ -verbose:gc -Xloggc:c:/Jrun4/logs/gc/gcInstance2a.log -Djmx.invoke.getters=true
java_command: <unknown>
Launcher Type: generic
Environment Variables:
PATH=C:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib;C:\JRun4\servers \cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\jintegra\bin;C:\JRun4\servers\cfusion\cfu sion-ear\cfusion-war\WEB-INF\cfusion\jintegra\bin\international;C:\JRun4\servers\cfusion\c fusion-ear\cfusion-war\WEB-INF\cfusion\lib\oosdk\classes\win;C:\Program Files\Windows Resource Kits\Tools\;C:\Program Files\Support Tools\;C:\Perl\bin\;C:\JRun4\verity\k2\_nti40\bin;C:\Program Files\EMC\PowerPath\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Dell\SysMgt\RAC4;C:\Program Files\Dell\SysMgt\oma\bin;C:\Program Files\Microsoft SQL Server\80\Tools\Binn\;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files\Microsoft SQL Server\90\DTS\Binn\;C:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\;C:\PROGRA~1\SAAZOD\zScripts\
USERNAME=lcbadmin
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 8, GenuineIntel
---------------  S Y S T E M  ---------------
OS: Windows Server 2003 family Build 3790 Service Pack 2
CPU:total 8 (2 cores per cpu, 2 threads per core) family 15 model 4 stepping 8, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ht
Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)
vm_info: Java HotSpot(TM) Server VM (14.0-b16) for windows-x86 JRE (1.6.0_14-b08), built on May 21 2009 07:57:57 by "java_re" with MS VC++ 7.1
time: Tue May 18 21:48:13 2010
elapsed time: 154379 seconds
0.381: [GC 3072K->717K(7680K), 0.0028050 secs]
0.578: [GC 3789K->923K(7680K), 0.0026605 secs]
2.581: [GC 3995K->1086K(7680K), 0.0028255 secs]
2.860: [GC 4158K->1283K(7680K), 0.0042886 secs]
3.056: [GC 4355K->1467K(7680K), 0.0028442 secs]
3.231: [GC 4537K->1648K(11008K), 0.0025800 secs]
3.713: [GC 7792K->2190K(11008K), 0.0040999 secs]
3.985: [GC 4856K->2263K(17408K), 0.0025673 secs]
3.988: [Full GC 2263K->2146K(17408K), 0.0487271 secs]
4.696: [GC 14434K->3018K(16832K), 0.0051747 secs]
5.486: [GC 14858K->3805K(16192K), 0.0045260 secs]
6.045: [GC 15197K->4709K(16000K), 0.0074686 secs]
6.053: [Full GC 4709K->4451K(22464K), 0.1103924 secs]
6.787: [GC 15039K->5937K(22528K), 0.0117950 secs]
7.218: [GC 16625K->7804K(22912K), 0.0099954 secs]
7.573: [GC 18492K->10050K(24704K), 0.0144209 secs]
8.256: [GC 21826K->10582K(25024K), 0.0079624 secs]
8.639: [GC 22358K->10701K(27776K), 0.0130334 secs]
8.652: [Full GC 10701K->7499K(36096K), 0.1193615 secs]
10.349: [GC 21067K->8260K(36160K), 0.0036341 secs]
14.083: [GC 21828K->10306K(37184K), 0.0113857 secs]
15.701: [GC 25090K->9822K(37696K), 0.0059270 secs]
16.614: [GC 24606K->11610K(38272K), 0.0164009 secs]
16.824: [GC 26842K->11008K(38976K), 0.0064352 secs]
17.759: [GC 26240K->11017K(40128K), 0.0056167 secs]
17.967: [GC 27657K->11008K(40384K), 0.0056861 secs]
18.194: [GC 27648K->11008K(42240K), 0.0062386 secs]
18.530: [GC 29504K->11009K(42176K), 0.0052893 secs]
19.610: [GC 29505K->11657K(43200K), 0.0098857 secs]
19.912: [GC 31561K->11633K(43712K), 0.0067032 secs]
20.364: [GC 31537K->12066K(45184K), 0.0081479 secs]
20.805: [GC 32470K->12014K(45440K), 0.0109562 secs]
22.058: [GC 33262K->17043K(46848K), 0.0195642 secs]
23.211: [GC 39571K->19376K(47424K), 0.0259335 secs]
24.402: [GC 41904K->20994K(50112K), 0.0253877 secs]
25.890: [GC 44930K->21307K(50624K), 0.0184082 secs]
26.371: [GC 45243K->22727K(52160K), 0.0178302 secs]
26.884: [GC 47687K->23246K(53120K), 0.0208570 secs]
27.995: [GC 48206K->25664K(54592K), 0.0241083 secs]
29.207: [GC 51298K->31755K(57792K), 0.0290945 secs]
29.236: [Full GC 31755K->27853K(74304K), 0.3884927 secs]
29.924: [GC 46184K->38399K(76288K), 0.0460258 secs]
30.401: [GC 62847K->55516K(80192K), 0.0404456 secs]
30.441: [Full GC 55516K->51591K(107648K), 0.3037522 secs]
30.873: [GC 62219K->59963K(115968K), 0.0441025 secs]
31.388: [GC 84732K->72340K(118976K), 0.0392704 secs]
31.756: [GC 97172K->90818K(126720K), 0.0627947 secs]
31.819: [Full GC 90818K->89854K(162432K), 0.3339941 secs]
32.169: [GC 92791K->90051K(162432K), 0.0469329 secs]
32.477: [GC 117269K->98790K(164992K), 0.0352808 secs]
33.032: [GC 128657K->112415K(164992K), 0.0569482 secs]
33.625: [GC 142303K->129501K(167872K), 0.0829067 secs]
33.708: [Full GC 129501K->128716K(214720K), 0.3245804 secs]
34.410: [GC 133817K->130014K(214720K), 0.0563804 secs]
47.633: [GC 162782K->137909K(216000K), 0.0515256 secs]
49.988: [GC 171957K->141420K(216000K), 0.0516951 secs]
51.694: [GC 175468K->145781K(216000K), 0.1004621 secs]
51.795: [Full GC 145781K->134519K(260224K), 0.5717566 secs]
53.350: [GC 168567K->136461K(260224K), 0.0109643 secs]
54.436: [GC 170509K->141065K(260224K), 0.0167210 secs]
55.290: [GC 175113K->145039K(260224K), 0.0231981 secs]
56.200: [GC 179087K->151640K(260224K), 0.0254553 secs]
57.069: [GC 185688K->159187K(260224K), 0.0374343 secs]
58.092: [GC 193235K->165658K(260224K), 0.0415577 secs]
58.735: [GC 199706K->171695K(260224K), 0.0381981 secs]
59.441: [GC 205743K->177923K(260224K), 0.0466761 secs]
60.138: [GC 211971K->184639K(260224K), 0.0398748 secs]
60.178: [Full GC 184639K->184545K(308672K), 0.6346645 secs]
61.473: [GC 218593K->191727K(308672K), 0.0228897 secs]
62.103: [GC 225775K->198230K(308672K), 0.0452323 secs]
62.593: [GC 232278K->204937K(308672K), 0.0441287 secs]
63.152: [GC 238985K->210234K(308672K), 0.0466643 secs]
63.631: [GC 244282K->214332K(308672K), 0.0308136 secs]
64.069: [GC 248380K->220520K(308672K), 0.0310264 secs]
64.490: [GC 254568K->224630K(307648K), 0.0338149 secs]
64.887: [GC 258678K->227375K(281664K), 0.0177588 secs]
65.336: [GC 261423K->232312K(307456K), 0.0275194 secs]
66.622: [GC 268984K->233248K(307584K), 0.0294454 secs]
68.689: [GC 269920K->233475K(307200K), 0.0303114 secs]
70.063: [GC 272323K->233989K(308224K), 0.0319816 secs]
70.948: [GC 272827K->236267K(309504K), 0.0381517 secs]
72.151: [GC 276971K->243396K(309760K), 0.0479739 secs]
72.810: [GC 284100K->256970K(305792K), 0.0670952 secs]
73.202: [GC 291018K->241617K(308672K), 0.0469614 secs]
73.250: [Full GC 241617K->241374K(365888K), 1.0516054 secs]
75.419: [GC 274751K->244364K(365888K), 0.0663203 secs]
76.501: [GC 278366K->244598K(365888K), 0.0116506 secs]
77.218: [GC 278646K->249688K(365376K), 0.0161159 secs]
77.761: [GC 283736K->249237K(340544K), 0.0173527 secs]
78.002: [GC 283285K->249652K(366336K), 0.1664675 secs]
78.802: [GC 286066K->250024K(364736K), 0.0352561 secs]
86.046: [GC 286428K->246641K(365952K), 0.0169778 secs]
89.807: [GC 286513K->247751K(366080K), 0.0128220 secs]
104.515: [GC 287623K->247447K(365824K), 0.0143550 secs]
105.456: [GC 290071K->247027K(367168K), 0.0130806 secs]
107.171: [GC 289651K->251837K(364800K), 0.0211884 secs]
107.490: [GC 296701K->253512K(366848K), 0.0189826 secs]
107.981: [GC 298376K->251486K(365760K), 0.0193742 secs]
109.163: [GC 298971K->250285K(367168K), 0.0158604 secs]
115.811: [GC 297773K->250474K(365760K), 0.0139898 secs]
118.378: [GC 299946K->254646K(367168K), 0.0217229 secs]
119.027: [GC 304118K->259176K(367296K), 0.0324158 secs]
119.615: [GC 308840K->262151K(368896K), 0.0358532 secs]
120.623: [GC 311815K->270706K(366784K), 0.0824720 secs]
121.470: [GC 316146K->279753K(371584K), 0.0766884 secs]
122.268: [GC 325193K->288492K(360192K), 0.1021346 secs]
122.929: [GC 322540K->294521K(365888K), 0.0907757 secs]
123.531: [GC 328569K->300541K(365888K), 0.0978875 secs]
124.140: [GC 334589K->305907K(365888K), 0.0988647 secs]
124.703: [GC 339955K->311069K(365888K), 0.0951130 secs]
125.315: [GC 345117K->312745K(365888K), 0.0801517 secs]
125.533: [GC 346793K->313321K(365888K), 0.0943218 secs]
125.627: [Full GC 313321K->279735K(412288K), 1.3003860 secs]
126.992: [GC 313783K->280075K(412288K), 0.0125182 secs]
127.076: [GC 314123K->280267K(412288K), 0.0065298 secs]
128.996: [GC 314315K->283281K(412288K), 0.0107040 secs]
129.398: [GC 317329K->283830K(412288K), 0.0095192 secs]
129.695: [GC 317878K->283843K(412288K), 0.0104606 secs]
134.828: [GC 317891K->284435K(412288K), 0.0091083 secs]
135.574: [GC 318483K->287393K(412288K), 0.0108201 secs]
135.749: [GC 321441K->286125K(412288K), 0.0087844 secs]
135.795: [GC 320173K->286269K(412288K), 0.0064130 secs]
135.841: [GC 320317K->286401K(412288K), 0.0059679 secs]
136.118: [GC 320449K->286916K(412288K), 0.0069953 secs]
136.761: [GC 320964K->288092K(412288K), 0.0079408 secs]
138.931: [GC 322140K->288414K(412288K), 0.0071464 secs]
155.109: [GC 322462K->290915K(412288K), 0.0073039 secs]
162.080: [GC 324963K->290026K(378752K), 0.0085490 secs]
166.335: [GC 324074K->292428K(409984K), 0.0075349 secs]
166.668: [GC 327820K->293701K(383808K), 0.0110931 secs]
167.470: [GC 329093K->295465K(406464K), 0.0431141 secs]
167.814: [GC 332777K->297419K(408256K), 0.0173732 secs]
168.042: [GC 334731K->295698K(406976K), 0.0249668 secs]
168.530: [GC 336018K->294099K(408128K), 0.0117701 secs]
186.485: [GC 334406K->294290K(405568K), 0.0099247 secs]
189.467: [GC 337042K->293803K(407552K), 0.0092321 secs]
192.025: [GC 336555K->294114K(405184K), 0.0144861 secs]
192.537: [GC 338658K->293848K(406848K), 0.0096140 secs]
198.694: [GC 338392K->294634K(404800K), 0.0275907 secs]
200.414: [GC 340714K->295225K(406208K), 0.0243144 secs]
201.828: [GC 341305K->294499K(404800K), 0.0191665 secs]
202.519: [GC 342051K->294865K(405824K), 0.0105044 secs]
207.017: [GC 342417K->295663K(404992K), 0.0190171 secs]
218.114: [GC 344495K->297980K(405760K), 0.0192997 secs]
218.433: [GC 346812K->296451K(407424K), 0.0127834 secs]
219.292: [GC 346819K->296114K(407168K), 0.0103118 secs]
227.933: [GC 346482K->298582K(407936K), 0.0205685 secs]
228.202: [GC 350358K->299398K(408192K), 0.0224917 secs]
228.984: [GC 351174K->297324K(410048K), 0.0120500 secs]
231.777: [GC 351722K->297050K(410816K), 0.0146803 secs]
236.016: [GC 351431K->297635K(411136K), 0.0126529 secs]
239.092: [GC 353955K->298634K(411840K), 0.0155092 secs]
239.552: [GC 354954K->299904K(413120K), 0.0215851 secs]
252.632: [GC 357566K->298439K(413440K), 0.0137175 secs]
269.175: [GC 356103K->298922K(414016K), 0.0170458 secs]
273.686: [GC 358166K->299811K(414592K), 0.0213339 secs]
275.229: [GC 359066K->301064K(415616K), 0.0209921 secs]
288.095: [GC 361412K->304481K(416000K), 0.0232993 secs]
288.392: [GC 364833K->303245K(418624K), 0.0226790 secs]
301.675: [GC 364685K->302009K(418816K), 0.0362261 secs]
307.093: [GC 363449K->300955K(419392K), 0.0126385 secs]
309.805: [GC 364430K->301618K(420160K), 0.0126741 secs]
325.062: [GC 365106K->302651K(420352K), 0.0151472 secs]
339.201: [GC 367545K->301935K(420928K), 0.0181489 secs]
346.999: [GC 366831K->301900K(420864K), 0.0125003 secs]
348.433: [GC 367756K->303969K(421312K), 0.0198808 secs]
357.384: [GC 369825K->302469K(421888K), 0.0198447 secs]
358.633: [GC 369349K->303806K(422272K), 0.3034111 secs]
359.588: [GC 370686K->306106K(427136K), 0.3337606 secs]
360.523: [GC 378170K->306180K(427392K), 0.0179965 secs]
361.119: [GC 378244K->304632K(434688K), 0.0256676 secs]
364.956: [GC 384248K->303336K(434880K), 0.0169394 secs]
365.935: [GC 382952K->305060K(434624K), 0.0194296 secs]
368.833: [GC 385060K->305320K(434944K), 0.0149410 secs]
370.448: [GC 385320K->304360K(435200K), 0.0216550 secs]
377.487: [GC 385185K->306224K(435456K), 0.0202275 secs]
385.574: [GC 387056K->305525K(435392K), 0.0180600 secs]
388.927: [GC 386805K->306564K(435648K), 0.0175783 secs]
391.674: [GC 387844K->309037K(434304K), 0.0254813 secs]
407.514: [GC 388909K->308487K(433600K), 0.0175757 secs]
409.194: [GC 388359K->311467K(432512K), 0.0209529 secs]
414.063: [GC 388395K->314784K(433728K), 0.0241777 secs]
415.431: [GC 391712K->313315K(430912K), 0.0223951 secs]
430.083: [GC 387427K->314564K(430592K), 0.0232336 secs]
439.450: [GC 388676K->315193K(430400K), 0.0247885 secs]
441.171: [GC 386745K->312782K(431040K), 0.0169260 secs]
452.250: [GC 384334K->313844K(430720K), 0.0202268 secs]
460.522: [GC 385844K->319503K(430976K), 0.2599954 secs]
468.974: [GC 391503K->322586K(424064K), 0.0422984 secs]
469.474: [GC 387095K->323360K(424512K), 0.0407064 secs]
477.136: [GC 387872K->322728K(425728K), 0.0311824 secs]
478.559: [GC 384552K->323311K(421760K), 0.0272584 secs]
484.216: [GC 385128K->322961K(425280K), 0.0298063 secs]
495.042: [GC 383377K->323348K(425472K), 0.0294568 secs]
507.955: [GC 383750K->323970K(424128K), 0.0292337 secs]
513.449: [GC 384514K->323429K(424832K), 0.0228552 secs]
522.316: [GC 383973K->324246K(424256K), 0.0451143 secs]
529.871: [GC 385935K->326326K(424896K), 0.0519706 secs]
530.225: [GC 388022K->324837K(426240K), 0.0314384 secs]
534.490: [GC 388133K->324714K(425792K), 0.0241508 secs]
563.372: [GC 388010K->324722K(427456K), 0.0246593 secs]
572.215: [GC 390130K->324496K(427200K), 0.0251314 secs]
573.388: [GC 389904K->324967K(427712K), 0.0333442 secs]
585.876: [GC 391143K->325337K(416128K), 0.0146273 secs]
588.092: [GC 391513K->326307K(427840K), 0.0150402 secs]
590.242: [GC 391587K->327561K(417344K), 0.0243195 secs]
597.280: [GC 392841K->326350K(427904K), 0.0166214 secs]
605.425: [GC 391822K->325727K(427776K), 0.0130987 secs]
621.244: [GC 391199K->326293K(427840K), 0.0137431 secs]
631.514: [GC 392213K->325980K(427968K), 0.0116427 secs]
635.911: [GC 391900K->326451K(427840K), 0.0144855 secs]
642.943: [GC 393075K->327191K(428096K), 0.0141549 secs]
650.865: [GC 393815K->328001K(427136K), 0.0262237 secs]
654.381: [GC 395201K->327424K(427712K), 0.0161351 secs]
654.834: [GC 394624K->327019K(428288K), 0.0150662 secs]
665.058: [GC 395243K->327139K(427904K), 0.0115758 secs]
673.535: [GC 395352K->326908K(427776K), 0.0198125 secs]
695.278: [GC 396284K->327746K(428352K), 0.0181528 secs]
704.497: [GC 397122K->328052K(427456K), 0.0190775 secs]
711.387: [GC 398196K->329218K(428096K), 0.0499620 secs]
719.837: [GC 399362K->328605K(427200K), 0.0170412 secs]
735.922: [GC 399517K->327069K(427776K), 0.0145175 secs]
736.804: [GC 397981K->327242K(428416K), 0.0110488 secs]
747.130: [GC 398986K->328367K(428224K), 0.0190119 secs]
749.678: [GC 400104K->327812K(428224K), 0.0143051 secs]
767.357: [GC 400452K->328673K(428416K), 0.0338326 secs]
772.008: [GC 401313K->330038K(428352K), 0.0205973 secs]
773.157: [GC 403446K->328909K(428544K), 0.0178709 secs]
773.757: [GC 402310K->328315K(429696K), 0.0140103 secs]
779.192: [GC 403187K->329304K(429440K), 0.0122486 secs]
784.114: [GC 404184K->329444K(429376K), 0.0136336 secs]
800.024: [GC 405604K->330673K(430016K), 0.0161595 secs]
810.128: [GC 406833K->329507K(430208K), 0.0155031 secs]
822.611: [GC 406518K->328618K(430656K), 0.0159114 secs]
830.465: [GC 405674K->329259K(430720K), 0.0134547 secs]
832.320: [GC 406955K->331849K(431040K), 0.0192622 secs]
839.589: [GC 409543K->330317K(431488K), 0.0152272 secs]
846.029: [GC 408841K->330677K(431936K), 0.0146644 secs]
855.691: [GC 409205K->330917K(432000K), 0.0147851 secs]
857.089: [GC 410085K->329282K(432256K), 0.0113335 secs]
862.172: [GC 408450K->330507K(433280K), 0.0744561 secs]
864.102: [GC 410827K->329939K(433152K), 0.0602524 secs]
871.764: [GC 410259K->337849K(432896K), 0.0464115 secs]
873.725: [GC 418041K->340453K(435264K), 0.0492662 secs]
875.230: [GC 420645K->343085K(429760K), 0.0488546 secs]
876.560: [GC 417581K->348660K(432512K), 0.0499122 secs]
881.246: [GC 423156K->354299K(420736K), 0.0444023 secs]
881.291: [Full GC 354299K->184625K(324608K), 1.7573846 secs]
885.122: [GC 247345K->189442K(330496K), 0.0344186 secs]
890.421: [GC 252162K->189400K(329792K), 0.0117897 secs]
892.905: [GC 252312K->191219K(330240K), 0.0194622 secs]
894.713: [GC 254126K->190575K(329984K), 0.0155164 secs]
896.839: [GC 254816K->190146K(330368K), 0.0139860 secs]
901.632: [GC 254400K->189745K(330304K), 0.0130588 secs]
909.589: [GC 255600K->191848K(331008K), 0.0234403 secs]
916.343: [GC 257704K->191739K(329920K), 0.0422734 secs]
921.147: [GC 258939K->193093K(330752K), 0.0297285 secs]
925.261: [GC 260293K->192604K(330368K), 0.0226062 secs]
929.613: [GC 261084K->191169K(331008K), 0.0156673 secs]
944.701: [GC 259649K->192220K(330240K), 0.0410416 secs]
949.753: [GC 261788K->192155K(331008K), 0.0406027 secs]
953.023: [GC 261717K->192548K(331008K), 0.0233991 secs]
956.654: [GC 263396K->192558K(331328K), 0.0204636 secs]
958.453: [GC 263401K->191823K(332352K), 0.0191734 secs]
962.069: [GC 264143K->192447K(332288K), 0.0143854 secs]
966.391: [GC 264767K->191895K(332928K), 0.0149928 secs]
986.458: [GC 265488K->194857K(333184K), 0.0228897 secs]
998.709: [GC 268457K->192952K(333568K), 0.0181598 secs]
1004.050: [GC 267448K->192375K(333952K), 0.0159302 secs]
1010.544: [GC 266871K->192397K(334144K), 0.0150398 secs]
1013.933: [GC 267661K->194230K(334400K), 0.0254282 secs]
1018.907: [GC 269494K->194579K(334784K), 0.0176419 secs]
1034.138: [GC 270803K->193420K(334976K), 0.0201663 secs]
1068.738: [GC 269638K->192210K(335360K), 0.0204423 secs]
1071.498: [GC 269128K->193375K(335488K), 0.0151018 secs]
1074.150: [GC 270303K->195766K(335936K), 0.0306906 secs]
1078.711: [GC 273334K->194600K(336128K), 0.0151313 secs]
1091.405: [GC 272168K->193745K(336192K), 0.0133192 secs]
1117.409: [GC 272273K->193573K(336576K), 0.0178818 secs]
1131.229: [GC 272101K->193468K(336960K), 0.0162684 secs]
1135.116: [GC 272700K->195603K(337152K), 0.0260614 secs]
1150.247: [GC 274835K->193585K(337536K), 0.0128450 secs]
1179.512: [GC 273585K->194432K(337792K), 0.0153779 secs]
1186.196: [GC 274432K->194550K(337728K), 0.0154796 secs]
1195.271: [GC 275190K->196552K(337984K), 0.0223260 secs]
1203.048: [GC 277192K->196154K(338816K), 0.0183032 secs]
1233.017: [GC 277370K->193641K(338880K), 0.0131341 secs]
1244.191: [GC 274857K->194742K(339008K), 0.0112069 secs]
1250.311: [GC 276399K->193065K(339200K), 0.0254185 secs]
1253.819: [GC 274729K->194369K(339840K), 0.0129195 secs]
1256.108: [GC 276737K->195947K(339904K), 0.0238248 secs]
1267.525: [GC 278315K->195033K(340352K), 0.0569615 secs]
1271.692: [GC 278425K->196492K(340480K), 0.0190379 secs]
1277.693: [GC 279852K->197949K(340480K), 0.0191917 secs]
1280.238: [GC 281405K->196790K(340864K), 0.0157921 secs]
1285.199: [GC 280246K->197847K(340608K), 0.0181497 secs]
1286.007: [GC 281175K->196440K(337472K), 0.0111939 secs]
1304.891: [GC 279768K->197845K(340608K), 0.0182715 secs]
1315.601: [GC 281237K->197856K(340736K), 0.0167214 secs]
1316.421: [GC 281248K->198430K(340608K), 0.0246224 secs]
1328.348: [GC 281758K->198595K(339136K), 0.0165731 secs]
1349.267: [GC 281915K->197864K(340928K), 0.0170268 secs]
1368.354: [GC 281192K->199896K(340288K), 0.0262476 secs]
1376.770: [GC 283224K->200903K(339136K), 0.0309814 secs]
1381.040: [GC 282183K->200807K(338496K), 0.0238972 secs]
1392.801: [GC 282087K->200812K(338880K), 0.0230000 secs]
1402.978: [GC 280812K->199419K(335424K), 0.0188024 secs]
1423.983: [GC 279419K->199784K(339264K), 0.0155648 secs]
1427.064: [GC 280360K->200673K(339072K), 0.0203093 secs]
1434.942: [GC 281249K->200341K(339584K), 0.0223844 secs]
1438.108: [GC 281621K->200922K(339584K), 0.0322821 secs]
1443.355: [GC 282202K->200559K(339840K), 0.0218897 secs]
1456.235: [GC 282351K->201313K(339712K), 0.0227011 secs]
1470.330: [GC 283105K->202311K(339840K), 0.0259252 secs]
1485.071: [GC 284423K->201711K(340032K), 0.0171362 secs]
1494.809: [GC 283823K->201909K(340160K), 0.0157363 secs]
1497.843: [GC 284469K->203516K(340224K), 0.0455427 secs]
1514.800: [GC 286076K->201843K(340224K), 0.0183698 secs]
1532.011: [GC 284019K->200888K(335296K), 0.0148204 secs]
1542.673: [GC 283064K->202547K(340032K), 0.0133714 secs]
1546.169: [GC 284275K->201936K(339968K), 0.0131052 secs]
1558.229: [GC 283664K->203817K(339840K), 0.0270803 secs]
1563.074: [GC 285609K->204088K(339904K), 0.0235084 secs]
1565.882: [GC 285880K->203251K(339968K), 1.4040290 secs]
1568.277: [GC 285427K->206133K(339904K), 1.4095963 secs]
1572.106: [GC 288309K->218429K(329664K), 1.3890185 secs]
1574.922: [GC 289719K->228796K(334784K), 1.5292558 secs]
1577.851: [GC 300091K->234048K(321472K), 0.0349367 secs]
1578.641: [GC 292032K->235261K(322496K), 1.4253011 secs]
1580.562: [GC 293231K->236958K(325248K), 0.0272196 secs]
1581.905: [GC 291230K->242541K(325376K), 1.4795405 secs]
1584.465: [GC 296813K->242311K(318848K), 0.0404727 secs]
1586.335: [GC 288263K->244486K(322112K), 0.0371221 secs]
1587.556: [GC 290426K->247801K(321856K), 0.0331232 secs]
1587.795: [GC 294007K->243424K(322112K), 0.0236656 secs]
1588.929: [GC 289632K->247212K(321472K), 0.0233302 secs]
1590.843: [GC 294188K->251257K(321792K), 1.5163833 secs]
1594.496: [GC 298233K->251601K(323072K), 0.0251024 secs]
1594.521: [Full GC 251601K->229607K(373056K), 1.2681137 secs]
1597.865: [GC 278951K->240314K(372800K), 0.0389260 secs]
1599.989: [GC 289658K->242870K(373952K), 0.0228907 secs]
1604.417: [GC 294768K->242335K(373760K), 0.0189292 secs]
1606.399: [GC 294239K->238333K(375168K), 0.0103706 secs]
1609.289: [GC 292093K->237476K(352576K), 0.0084749 secs]
1614.643: [GC 291229K->238324K(375552K), 0.0109861 secs]
1618.865: [GC 291316K->242479K(356800K), 0.0357355 secs]
1619.215: [GC 295471K->243649K(375872K), 0.0218387 secs]
1621.325: [GC 297793K->241168K(374784K), 0.0128466 secs]
1627.656: [GC 295312K->240383K(376512K), 0.0130406 secs]
1633.452: [GC 296831K->241343K(375936K), 0.1381361 secs]
1639.291: [GC 297731K->240512K(377664K), 0.0167985 secs]
1641.070: [GC 299836K->241890K(377408K), 0.0126484 secs]
1643.135: [GC 301218K->242024K(379200K), 0.0830174 secs]
1657.391: [GC 304808K->242766K(379008K), 0.0230685 secs]
1660.963: [GC 305550K->241718K(379200K), 0.0121118 secs]
1672.385: [GC 307247K->242373K(380032K), 0.0148897 secs]
1679.189: [GC 307909K->242663K(379456K), 0.0336012 secs]
1679.806: [GC 310311K->242331K(380544K), 0.0137211 secs]
1698.018: [GC 309972K->242272K(380288K), 0.0129662 secs]
1707.166: [GC 312209K->241933K(381504K), 0.0170909 secs]
1710.486: [GC 311885K->242797K(381568K), 0.0167249 secs]
1713.338: [GC 314555K->243384K(382272K), 0.0192025 secs]
1717.070: [GC 315192K->242810K(382144K), 0.0123115 secs]
1724.488: [GC 316282K->244688K(382912K), 0.0156554 secs]
1731.274: [GC 318160K->241808K(383744K), 0.0144968 secs]
1737.801: [GC 316560K->243104K(384064K), 0.0121330 secs]
1739.864: [GC 317856K->245407K(384896K), 0.0341079 secs]
1743.852: [GC 321311K->244312K(384896K), 0.0141990 secs]
1746.971: [GC 320216K->244438K(385344K), 0.0183935 secs]
1766.204: [GC 322134K->243594K(385792K), 0.0132516 secs]
1770.080: [GC 321290K->243829K(386240K), 0.0130751 secs]
1784.406: [GC 322869K->244602K(386624K), 0.0156597 secs]
1796.876: [GC 323597K->243561K(386816K), 0.0148723 secs]
1798.332: [GC 323561K->245704K(387136K), 0.0154095 secs]
1800.109: [GC 325704K->247336K(388032K), 0.0409818 secs]
1804.904: [GC 328168K->245655K(388736K), 0.0166685 secs]
1806.600: [GC 326487K->245580K(389760K), 0.0219245 secs]
1818.889: [GC 328204K->244470K(390144K), 0.0157939 secs]
1822.358: [GC 327094K->245905K(390336K), 0.0175462 secs]
1827.555: [GC 329224K->246088K(390592K), 0.0208042 secs]
1837.514: [GC 329416K->245605K(390336K), 0.0137855 secs]
1847.001: [GC 329445K->246131K(390720K), 0.0146275 secs]
1848.864: [GC 329971K->246494K(391296K), 0.0175766 secs]
1859.203: [GC 331036K->246937K(391360K), 0.0146327 secs]
1860.966: [GC 331481K->248414K(390208K), 0.0216726 secs]
1864.134: [GC 331742K->247451K(389056K), 0.0162971 secs]
1873.550: [GC 330779K->248640K(389760K), 0.0218835 secs]
1876.250: [GC 330803K->248184K(388224K), 0.0208859 secs]
1880.577: [GC 330360K->247325K(390336K), 0.0147365 secs]
1892.113: [GC 329565K->247859K(387456K), 0.0144426 secs]
1900.820: [GC 330099K->248412K(389952K), 0.0195921 secs]
1903.069: [GC 330716K->247796K(389888K), 0.0155545 secs]
1910.944: [GC 330100K->248981K(390336K), 0.0165833 secs]
1912.105: [GC 331910K->253094K(390400K), 0.0212416 secs]
1912.393: [GC 336038K->252108K(388096K), 0.0182545 secs]
1917.882: [GC 332364K->252928K(387456K), 0.0226416 secs]
1920.975: [GC 333184K->255435K(387264K), 0.0831686 secs]
1921.451: [GC 333515K->253724K(388160K), 0.0265402 secs]
1930.344: [GC 331804K->253979K(387904K), 0.0208346 secs]
1934.661: [GC 332059K->253520K(388032K), 0.0179034 secs]
1941.664: [GC 331600K->254297K(388096K), 0.0206944 secs]
1947.160: [GC 332761K->253651K(388096K), 0.0194694 secs]
1948.391: [GC 332115K->254824K(388288K), 0.0213165 secs]
1955.883: [GC 333798K->252957K(388288K), 0.0172918 secs]
1962.014: [GC 331933K->254893K(388864K), 0.0256110 secs]
1981.902: [GC 334509K->256567K(388352K), 0.0445018 secs]
1992.276: [GC 336183K->254832K(388736K), 0.0185104 secs]
1993.072: [GC 333808K->253054K(388480K), 0.0171761 secs]
2006.441: [GC 332030K->254222K(388160K), 0.0331208 secs]
2012.620: [GC 332686K->254008K(381568K), 0.0121851 secs]
2029.610: [GC 332472K->254865K(388224K), 0.0141391 secs]
2033.518: [GC 333009K->254448K(388160K), 0.0171172 secs]
2037.370: [GC 332592K->254761K(388032K), 0.0137820 secs]
2042.150: [GC 332962K->257542K(388096K), 0.0195266 secs]
2048.972: [GC 335750K->256642K(387520K), 0.0175746 secs]
2054.523: [GC 335234K->255301K(387904K), 0.0131941 secs]
2057.202: [GC 333884K->255117K(388800K), 0.0174713 secs]
2059.506: [GC 334861K->255720K(388864K), 0.0172076 secs]
2067.436: [GC 335464K->256201K(388800K), 0.0154266 secs]
2075.384: [GC 336457K->255145K(388864K), 0.0134541 secs]
2077.148: [GC 335401K->255820K(389312K), 0.0158148 secs]
2084.353: [GC 336972K->255983K(389440K), 0.0134647 secs]
2093.116: [GC 337135K->256751K(389440K), 0.0155239 secs]
2102.787: [GC 338658K->258431K(389632K), 0.0215284 secs]
2104.462: [GC 340351K->256945K(390144K), 0.0153015 secs]
2109.728: [GC 339505K->257115K(390208K), 0.0159563 secs]
2122.379: [GC 339675K->256748K(390400K), 0.0161834 secs]
2133.977: [GC 339948K->257442K(390400K), 0.0158151 secs]
2139.718: [GC 340642K->257842K(390912K), 0.0182630 secs]
2143.532: [GC 341810K->255648K(390784K), 0.0156949 secs]
2156.980: [GC 339610K->256127K(391296K), 0.0232994 secs]
2162.709: [GC 340479K->259011K(389952K), 0.0216918 secs]
2165.392: [GC 343363K->257891K(391296K), 0.0156548 secs]
2172.867: [GC 341987K->256830K(387328K), 0.1029279 secs]
2175.829: [GC 340926K->258830K(390848K), 0.0175429 secs]
2193.642: [GC 342860K->257384K(390912K), 0.0158681 secs]
2198.592: [GC 341416K->257722K(391040K), 0.0145218 secs]
2216.548: [GC 342330K->258612K(391168K), 0.0147463 secs]
2218.664: [GC 343218K->258786K(391360K), 0.0167079 secs]
2223.931: [GC 343970K->261044K(391552K), 0.0325592 secs]
2229.163: [GC 346228K->260589K(390656K), 0.0185048 secs]
2233.913: [GC 344557K->259905K(391104K), 0.0178199 secs]
2237.120: [GC 343873K->259165K(390720K), 0.0131946 secs]
2241.975: [GC 343261K->259505K(390976K), 0.0162827 secs]
2250.736: [GC 343601K->259841K(390912K), 0.0159241 secs]
2259.976: [GC 344513K->260319K(391168K), 0.0205832 secs]
2265.025: [GC 344956K->259597K(391488K), 0.0152182 secs]
2266.097: [GC 344973K->259896K(391680K), 0.0151583 secs]
2283.470: [GC 345272K->261445K(391808K), 0.0237256 secs]
2284.093: [GC 347077K->260412K(391936K), 0.0237551 secs]
2301.941: [GC 346044K->261961K(391616K), 0.0202204 secs]
2305.248: [GC 347337K->261329K(391808K), 0.0165993 secs]
2331.961: [GC 346705K->262034K(391488K), 0.0184896 secs]
2340.389: [GC 347154K->261811K(391680K), 0.0146796 secs]
2345.010: [GC 346931K->261477K(391360K), 0.0261079 secs]
2362.483: [GC 346725K->261484K(391552K), 0.0150812 secs]
2366.469: [GC 346732K->261782K(391616K), 0.0176710 secs]
2381.854: [GC 347478K->262227K(391808K), 0.0178963 secs]
2405.250: [GC 347923K->263473K(391552K), 0.0290353 secs]
2407.033: [GC 348977K->262593K(391872K), 0.0134900 secs]
2410.744: [GC 348097K->262613K(391680K), 0.0152754 secs]
2430.766: [GC 348181K->262504K(391808K), 0.0168945 secs]
2445.377: [GC 348072K->262613K(391872K), 0.0150553 secs]
2457.907: [GC 348373K->261561K(391936K), 0.0132542 secs]
2464.252: [GC 347321K->262665K(391936K), 0.0156359 secs]
2466.159: [GC 348425K->263117K(391936K), 0.0366962 secs]
2501.045: [GC 348877K->262979K(391936K), 0.0179926 secs]
2515.249: [GC 348995K->262886K(391936K), 0.0150738 secs]
2526.311: [GC 348902K->264968K(391488K), 0.0330411 secs]
2529.752: [GC 350582K->263343K(391936K), 0.0154606 secs]
2539.265: [GC 348975K->263855K(391552K), 0.0177899 secs]
2543.927: [GC 349614K->264173K(391808K), 0.0240386 secs]
2549.568: [GC 349933K->263423K(392000K), 0.0161444 secs]
2556.347: [GC 349628K->264100K(392128K), 0.0160603 secs]
2561.802: [GC 350308K->263122K(392256K), 0.0182325 secs]
2566.114: [GC 349586K->263404K(392320K), 0.0137574 secs]
2586.702: [GC 349853K->266723K(390848K), 0.0274725 secs]
2589.854: [GC 351715K->266060K(391616K), 0.0168321 secs]
2591.739: [GC 351052K->265623K(391232K), 0.0195907 secs]
2607.384: [GC 350743K->265415K(391488K), 0.0174390 secs]
2618.555: [GC 350535K->265563K(391616K), 0.0167459 secs]
2637.109: [GC 351195K->266551K(391808K), 0.0148909 secs]
2642.146: [GC 352183K->266672K(392000K), 0.0180908 secs]
2643.168: [GC 352624K->266330K(392064K), 0.0147989 secs]
2647.188: [GC 352282K->267880K(391616K), 0.0173880 secs]
2655.985: [GC 353512K->267766K(391296K), 0.0149468 secs]
2656.891: [GC 353398K->266275K(391744K), 0.0144024 secs]
2664.408: [GC 351444K->266017K(388928K), 0.0209675 secs]
2668.686: [GC 351201K->266578K(391680K), 0.0153836 secs]
2686.271: [GC 351826K->266792K(391616K), 0.0148674 secs]
2687.701: [GC 352040K->266975K(391744K), 0.0195823 secs]
2689.757: [GC 352543K->268009K(391680K), 0.0159841 secs]
2703.374: [GC 353573K->267804K(392000K), 0.0164064 secs]
2707.609: [GC 353867K->267629K(391936K), 0.0251624 secs]
2712.220: [GC 353709K->265927K(391936K), 0.0125782 secs]
2737.914: [GC 352071K->268752K(391168K), 0.0185614 secs]
2750.502: [GC 354890K->266908K(392064K), 0.0142372 secs]
2760.888: [GC 352796K->268633K(392064K), 0.0171518 secs]
2766.927: [GC 354521K->268366K(391808K), 0.0181752 secs]
2768.030: [GC 354254K->269350K(391936K), 0.0206453 secs]
2771.794: [GC 355238K->268665K(392064K), 0.0168314 secs]
2809.208: [GC 354553K->270422K(391872K), 0.0188068 secs]
2809.559: [GC 356294K->268022K(391616K), 0.0139145 secs]
2819.650: [GC 353270K->268398K(391744K), 0.0131524 secs]
2824.293: [GC 353646K->268658K(391424K), 0.0141748 secs]
2826.554: [GC 353962K->268920K(391616K), 0.0155995 secs]
2828.896: [GC 354232K->270468K(391488K), 0.0282409 secs]
2843.234: [GC 355844K->270455K(390848K), 0.0208270 secs]
2861.999: [GC 355831K->269224K(391744K), 0.1919214 secs]
2870.048: [GC 354344K->270798K(390720K), 0.0189311 secs]
2882.122: [GC 355918K->269501K(391680K), 0.0173705 secs]
2889.172: [GC 354493K->271061K(390656K), 0.0332939 secs]
2894.716: [GC 356053K->270154K(391680K), 0.0169549 secs]
2896.430: [GC 355082K->271813K(391104K), 0.0194223 secs]
2896.842: [GC 356741K->272049K(390656K), 0.0313532 secs]
2901.564: [GC 355761K->272403K(390272K), 0.0189123 secs]
2908.595: [GC 356115K->272029K(390336K), 0.0179135 secs]
2914.717: [GC 354717K->271988K(390464K), 0.0146900 secs]
2923.602: [GC 354672K->271800K(390208K), 0.0173560 secs]
2939.749: [GC 3

Answer 1

Hi! Thanks for the reply,
After much googling we've finally found a stable configuration.
java.args=-server -Xmx768m -Xss64k -Dsun.io.useCanonCaches=false -XX:MaxPermSize=512m -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/ -verbose:gc -Xloggc:c:/Jrun4/logs/gc/gcInstance1b.log
I tried playing with the survivor ratio but came back to the default when I didn't notice much improvement.  In the end what did it (pretty sure) is -Xss to reduce the requests size living on the stack.  Our large initial heap, perm + overhead combined with the default request size was causing the JVM to go over and blow up I think.
We've been stable for over a month now, the next step for us is to reduce the MaxPermSize a little at a time to find our optimal range.