Garbage collection pauses screw up my cluster. JVM tuning suggestions?

I am running Sun's 64 bit JVM 1.7.0. Everything hums along nicely until my storage enabled nodes all show GC log entries like these and then the cluster experiences timeouts. This happens like once a day at most. Suggestions? Thinking maybe add more storage enabled JVMs? Any JVM settings I should use to prevent this? What can determine the cause? I don't think the machine's CPU is maxed out at the time.
Thanks,
Andrew
2012-02-02T09:09:15.585-0600: 61965.351: [GC 61965.351: [ParNew: 915634K->5291K(1023552K), 0.0396299 secs] 1143514K->233236K(3299264K), 0.0406305 secs] [Times: user=1.26 sys=0.00, real=0.04 secs]
2012-02-02T09:09:22.816-0600: 61972.582: [GC 61972.583: [ParNew: 915115K->4454K(1023552K), 0.0497440 secs] 1143060K->233468K(3299264K), 0.0510981 secs] [Times: user=0.86 sys=0.02, real=0.05 secs]
2012-02-02T09:09:29.647-0600: 61979.413: [GC 61979.413: [ParNew: 914278K->2750K(1023552K), 0.0141793 secs] 1143292K->232548K(3299264K), 0.0152009 secs] [Times: user=0.44 sys=0.00, real=0.02 secs]
2012-02-02T09:09:37.432-0600: 61987.198: [GC 61987.199: [ParNew: 912574K->2926K(1023552K), 0.0145958 secs] 1142372K->232786K(3299264K), 0.0156003 secs] [Times: user=0.45 sys=0.00, real=0.02 secs]
2012-02-02T09:09:45.213-0600: 61994.979: [GC 61994.980: [ParNew: 912750K->3264K(1023552K), 0.0133675 secs] 1142610K->233225K(3299264K), 0.0144490 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2012-02-02T09:09:50.709-0600: 62000.474: [GC 62000.475: [ParNew: 913088K->2920K(1023552K), 0.0169685 secs] 1143049K->232971K(3299264K), 0.0179272 secs] [Times: user=0.31 sys=0.00, real=0.02 secs]
2012-02-02T09:09:57.938-0600: 62007.704: [GC 62007.705: [ParNew: 912744K->4014K(1023552K), 0.0367154 secs] 1142795K->234128K(3299264K), 0.0377584 secs] [Times: user=0.36 sys=0.00, real=0.04 secs]
2012-02-02T09:10:04.134-0600: 62013.899: [GC 62013.899: [ParNew: 913838K->4306K(1023552K), 0.0179481 secs] 1143952K->234465K(3299264K), 0.0188570 secs] [Times: user=0.28 sys=0.00, real=0.02 secs]
2012-02-02T09:10:10.223-0600: 62019.988: [GC 62019.989: [ParNew: 914130K->4890K(1023552K), 0.0258524 secs] 1144289K->235119K(3299264K), 0.0267208 secs] [Times: user=0.37 sys=0.00, real=0.03 secs]
2012-02-02T09:10:19.538-0600: 62029.304: [GC 62029.305: [ParNew: 914714K->106275K(1023552K), 0.4447269 secs] 1144943K->336555K(3299264K), 0.4456875 secs] [Times: user=3.46 sys=0.14, real=0.45 secs]
2012-02-02T09:10:28.095-0600: 62037.860: [GC 62037.861: [ParNew: 1016099K->113728K(1023552K), 2.3496060 secs] 1246379K->627933K(3299264K), 2.3506549 secs] [Times: user=11.48 sys=1.48, real=2.35 secs]
2012-02-02T09:10:40.675-0600: 62050.440: [GC 62050.441: [ParNew: 1023552K->113728K(1023552K), 1.6754442 secs] 1537757K->946051K(3299264K), 1.6765925 secs] [Times: user=36.85 sys=0.37, real=1.68 secs]
2012-02-02T09:10:52.540-0600: 62062.306: [GC 62062.306: [ParNew: 1023552K->113728K(1023552K), 1.6465925 secs] 1855875K->1262979K(3299264K), 1.6475185 secs] [Times: user=38.24 sys=0.61, real=1.65 secs]
2012-02-02T09:10:54.190-0600: 62063.955: [GC [1 CMS-initial-mark: 1149251K(2275712K)] 1276188K(3299264K), 0.2430027 secs] [Times: user=0.25 sys=0.00, real=0.24 secs]
2012-02-02T09:10:54.434-0600: 62064.199: [CMS-concurrent-mark-start]
2012-02-02T09:10:55.375-0600: 62065.140: [CMS-concurrent-mark: 0.941/0.941 secs] [Times: user=10.44 sys=0.25, real=0.94 secs]
2012-02-02T09:10:55.375-0600: 62065.140: [CMS-concurrent-preclean-start]
2012-02-02T09:10:56.231-0600: 62065.996: [CMS-concurrent-preclean: 0.836/0.856 secs] [Times: user=1.78 sys=0.00, real=0.86 secs]
2012-02-02T09:10:56.231-0600: 62065.996: [CMS-concurrent-abortable-preclean-start]
2012-02-02T09:11:00.905-0600: 62070.669: [CMS-concurrent-abortable-preclean: 4.665/4.673 secs] [Times: user=9.75 sys=0.03, real=4.67 secs]
2012-02-02T09:11:00.906-0600: 62070.671: [GC[YG occupancy: 626192 K (1023552 K)]62070.672: [Rescan (parallel) , 0.4593515 secs]62071.132: [weak refs processing, 0.8817154 secs]62072.014: [scrub string table, 0.0006451 secs] [1 CMS-remark: 1149251K(2275712K)] 1775444K(3299264K), 1.7319597 secs] [Times: user=15.60 sys=0.00, real=1.73 secs]
2012-02-02T09:11:02.639-0600: 62072.404: [CMS-concurrent-sweep-start]
2012-02-02T09:11:04.088-0600: 62073.853: [CMS-concurrent-sweep: 1.443/1.449 secs] [Times: user=4.63 sys=0.05, real=1.45 secs]
2012-02-02T09:11:04.088-0600: 62073.853: [CMS-concurrent-reset-start]
2012-02-02T09:11:04.101-0600: 62073.866: [CMS-concurrent-reset: 0.013/0.013 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2012-02-02T09:11:09.312-0600: 62079.077: [GC 62079.077: [ParNew: 1023552K->113728K(1023552K), 1.6178150 secs] 1587225K->979876K(3299264K), 1.6187922 secs] [Times: user=37.58 sys=0.11, real=1.62 secs]
2012-02-02T09:11:21.172-0600: 62090.937: [GC 62090.937: [ParNew: 1023552K->113728K(1023552K), 1.5327921 secs] 1889700K->1243425K(3299264K), 1.5335996 secs] [Times: user=34.29 sys=0.14, real=1.53 secs]
2012-02-02T09:11:33.305-0600: 62103.070: [GC 62103.070: [ParNew: 1023552K->113728K(1023552K), 1.6349678 secs] 2153249K->1475423K(3299264K), 1.6361176 secs] [Times: user=36.40 sys=0.17, real=1.64 secs]
2012-02-02T09:11:45.480-0600: 62115.245: [GC 62115.246: [ParNew: 1023552K->113728K(1023552K), 1.6566701 secs] 2385247K->1744596K(3299264K), 1.6579350 secs] [Times: user=37.52 sys=0.25, real=1.66 secs]
2012-02-02T09:11:47.140-0600: 62116.905: [GC [1 CMS-initial-mark: 1630868K(2275712K)] 1744677K(3299264K), 0.2085259 secs] [Times: user=0.22 sys=0.00, real=0.21 secs]
2012-02-02T09:11:47.350-0600: 62117.115: [CMS-concurrent-mark-start]
2012-02-02T09:11:48.481-0600: 62118.245: [CMS-concurrent-mark: 1.130/1.130 secs] [Times: user=12.03 sys=0.02, real=1.13 secs]
2012-02-02T09:11:48.481-0600: 62118.245: [CMS-concurrent-preclean-start]
2012-02-02T09:11:49.191-0600: 62118.955: [CMS-concurrent-preclean: 0.702/0.710 secs] [Times: user=1.51 sys=0.00, real=0.71 secs]
2012-02-02T09:11:49.191-0600: 62118.955: [CMS-concurrent-abortable-preclean-start]
2012-02-02T09:11:53.016-0600: 62122.780: [CMS-concurrent-abortable-preclean: 3.819/3.825 secs] [Times: user=8.44 sys=0.05, real=3.83 secs]
2012-02-02T09:11:53.017-0600: 62122.781: [GC[YG occupancy: 637511 K (1023552 K)]62122.782: [Rescan (parallel) , 0.6066033 secs]62123.389: [weak refs processing, 0.8883109 secs]62124.278: [scrub string table, 0.0005759 secs] [1 CMS-remark: 1630868K(2275712K)] 2268379K(3299264K), 1.8602409 secs] [Times: user=16.88 sys=0.02, real=1.86 secs]
2012-02-02T09:11:54.878-0600: 62124.643: [CMS-concurrent-sweep-start]
2012-02-02T09:11:57.491-0600: 62127.255: [CMS-concurrent-sweep: 2.610/2.613 secs] [Times: user=7.57 sys=0.16, real=2.61 secs]
2012-02-02T09:11:57.491-0600: 62127.256: [CMS-concurrent-reset-start]
2012-02-02T09:11:57.498-0600: 62127.262: [CMS-concurrent-reset: 0.007/0.007 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2012-02-02T09:12:01.137-0600: 62130.902: [GC 62130.902: [ParNew: 1023552K->113728K(1023552K), 1.4807261 secs] 1678232K->1015472K(3299264K), 1.4816808 secs] [Times: user=26.47 sys=0.00, real=1.48 secs]
2012-02-02T09:12:13.113-0600: 62142.877: [GC 62142.878: [ParNew: 1023552K->113728K(1023552K), 1.5627711 secs] 1925296K->1258829K(3299264K), 1.5635313 secs] [Times: user=34.26 sys=0.00, real=1.56 secs]
2012-02-02T09:12:25.167-0600: 62154.931: [GC 62154.932: [ParNew: 1023552K->113728K(1023552K), 1.5751999 secs] 2168653K->1510600K(3299264K), 1.5762551 secs] [Times: user=31.75 sys=0.00, real=1.58 secs]
2012-02-02T09:12:36.964-0600: 62166.729: [GC 62166.730: [ParNew: 1023552K->113728K(1023552K), 1.6056739 secs] 2420424K->1768193K(3299264K), 1.6065643 secs] [Times: user=36.02 sys=0.08, real=1.61 secs]
2012-02-02T09:12:38.573-0600: 62168.337: [GC [1 CMS-initial-mark: 1654465K(2275712K)] 1768195K(3299264K), 0.1967081 secs] [Times: user=0.20 sys=0.00, real=0.20 secs]
2012-02-02T09:12:38.770-0600: 62168.535: [CMS-concurrent-mark-start]
2012-02-02T09:12:39.611-0600: 62169.376: [CMS-concurrent-mark: 0.841/0.841 secs] [Times: user=9.24 sys=0.00, real=0.84 secs]
2012-02-02T09:12:39.611-0600: 62169.376: [CMS-concurrent-preclean-start]
2012-02-02T09:12:40.036-0600: 62169.801: [CMS-concurrent-preclean: 0.419/0.425 secs] [Times: user=0.84 sys=0.02, real=0.42 secs]
2012-02-02T09:12:40.036-0600: 62169.801: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2012-02-02T09:12:46.001-0600: 62175.765: [CMS-concurrent-abortable-preclean: 5.955/5.964 secs] [Times: user=12.00 sys=0.08, real=5.96 secs]
2012-02-02T09:12:46.003-0600: 62175.767: [GC[YG occupancy: 770945 K (1023552 K)]62175.767: [Rescan (parallel) , 0.7886971 secs]62176.557: [weak refs processing, 0.8179398 secs]62177.375: [scrub string table, 0.0008280 secs] [1 CMS-remark: 1654465K(2275712K)] 2425410K(3299264K), 1.8989263 secs] [Times: user=25.40 sys=0.02, real=1.90 secs]
2012-02-02T09:12:47.903-0600: 62177.667: [CMS-concurrent-sweep-start]
2012-02-02T09:12:50.676-0600: 62180.440: [CMS-concurrent-sweep: 2.772/2.774 secs] [Times: user=8.20 sys=0.06, real=2.77 secs]
2012-02-02T09:12:50.676-0600: 62180.441: [CMS-concurrent-reset-start]
2012-02-02T09:12:50.696-0600: 62180.460: [CMS-concurrent-reset: 0.019/0.019 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2012-02-02T09:12:52.272-0600: 62182.037: [GC 62182.037: [ParNew: 1023552K->113728K(1023552K), 1.5954584 secs] 1584704K->951316K(3299264K), 1.5963712 secs] [Times: user=34.26 sys=0.00, real=1.60 secs]
2012-02-02T09:13:34.402-0600: 62224.165: [GC 62224.166: [ParNew: 1023552K->113728K(1023552K), 0.9755846 secs] 1861140K->1084527K(3299264K), 0.9767018 secs] [Times: user=19.19 sys=0.02, real=0.98 secs]
2012-02-02T09:16:04.628-0600: 62374.391: [GC 62374.391: [ParNew: 1023552K->113517K(1023552K), 0.1955335 secs] 1994351K->1173747K(3299264K), 0.1966082 secs] [Times: user=2.12 sys=0.05, real=0.20 secs]
2012-02-02T09:18:44.395-0600: 62534.156: [GC 62534.157: [ParNew: 1023341K->19351K(1023552K), 0.0506139 secs] 2083571K->1079581K(3299264K), 0.0517891 secs] [Times: user=0.37 sys=0.00, real=0.05 secs]
2012-02-02T09:21:20.270-0600: 62690.030: [GC 62690.030: [ParNew: 929175K->8239K(1023552K), 0.0405564 secs] 1989405K->1068470K(3299264K), 0.0417057 secs] [Times: user=0.76 sys=0.00, real=0.04 secs]
2012-02-02T09:23:54.367-0600: 62844.125: [GC 62844.126: [ParNew: 918063K->5778K(1023552K), 0.0497713 secs] 1978294K->1066008K(3299264K), 0.0508845 secs] [Times: user=0.58 sys=0.00, real=0.05 secs]
2012-02-02T09:26:33.522-0600: 63003.279: [GC 63003.279: [ParNew: 915602K->5421K(1023552K), 0.0524503 secs] 1975832K->1065651K(3299264K), 0.0536713 secs] [Times: user=0.59 sys=0.02, real=0.06 secs]
2012-02-02T09:29:13.270-0600: 63163.025: [GC 63163.026: [ParNew: 915245K->5390K(1023552K), 0.0317213 secs] 1975475K->1065620K(3299264K), 0.0330578 secs] [Times: user=0.25 sys=0.00, real=0.03 secs]
2012-02-02T09:31:58.001-0600: 63327.754: [GC 63327.755: [ParNew: 915214K->5524K(1023552K), 0.0617393 secs] 1975444K->1065894K(3299264K), 0.0629008 secs] [Times: user=1.05 sys=0.00, real=0.06 secs]
2012-02-02T09:34:43.454-0600: 63493.206: [GC 63493.207: [ParNew: 915348K->5600K(1023552K), 0.0376009 secs] 1975718K->1066058K(3299264K), 0.0388287 secs] [Times: user=0.36 sys=0.00, real=0.04 secs]
2012-02-02T09:37:15.601-0600: 63645.352: [GC 63645.353: [ParNew: 915424K->6656K(1023552K), 0.0273727 secs] 1975882K->1067278K(3299264K), 0.0285435 secs] [Times: user=0.56 sys=0.00, real=0.03 secs]
2012-02-02T09:39:41.873-0600: 63791.623: [GC 63791.623: [ParNew: 916480K->8588K(1023552K), 0.0239936 secs] 1977102K->1069403K(3299264K), 0.0252618 secs] [Times: user=0.75 sys=0.00, real=0.03 secs]
2012-02-02T09:42:02.276-0600: 63932.025: [GC 63932.025: [ParNew: 918412K->8408K(1023552K), 0.0259162 secs] 1979227K->1069313K(3299264K), 0.0269622 secs] [Times: user=0.34 sys=0.02, real=0.03 secs]
2012-02-02T09:44:22.990-0600: 64072.737: [GC 64072.738: [ParNew: 918232K->8777K(1023552K), 0.0208236 secs] 1979137K->1069755K(3299264K), 0.0220036 secs] [Times: user=0.41 sys=0.00, real=0.02 secs]

Today i used
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:-CMSConcurrentMTEnabled
-XX:+CMSScavengeBeforeRemark
and had the same problem with all 3 storage enabled nodes simultaneously when I closed the Extend app. The GC logs from those storage nodes were nearly identical with each showing three "abort preclean" messages like below before eventually stabilizing again. CPU usage for the storage node JVMs was up a bit on the machine at the time then eventually came down. Extend proxy node CPU usage never goes up very high and GC times are always good there.
Could it be because the Extend app has many CQCs running and when the app disconnects without shutting them down properly their results queue up in the storage enabled nodes? Something the Extend proxy can't pass along is getting added to old space?
9:04:29.313-0600: 34201.081: [GC 34201.081: [ParNew: 914605K->4950K(1023552K), 0.0469807 secs] 1057777K->148216K(3299264K), 0.0480525 secs] [Times: user=0.78 sys=0.03, real=0.05 secs]
9:04:35.742-0600: 34207.509: [GC 34207.509: [ParNew: 914774K->4559K(1023552K), 0.0317232 secs] 1058040K->147868K(3299264K), 0.0324390 secs] [Times: user=0.48 sys=0.00, real=0.03 secs]
9:04:44.806-0600: 34216.573: [GC 34216.574: [ParNew: 914383K->113728K(1023552K), 1.4924395 secs] 1057692K->348748K(3299264K), 1.4934226 secs] [Times: user=7.52 sys=0.81, real=1.49 secs]
9:04:56.498-0600: 34228.265: [GC 34228.265: [ParNew: 1023552K->113728K(1023552K), 1.3332836 secs] 1258572K->634502K(3299264K), 1.3341394 secs] [Times: user=28.58 sys=0.44, real=1.34 secs]
9:05:09.845-0600: 34241.612: [GC 34241.612: [ParNew: 1023552K->113728K(1023552K), 1.5617413 secs] 1544326K->920121K(3299264K), 1.5628521 secs] [Times: user=33.43 sys=0.34, real=1.56 secs]
9:05:24.569-0600: 34256.335: [GC 34256.336: [ParNew: 1023552K->113728K(1023552K), 2.3253881 secs] 1829945K->1186221K(3299264K), 2.3263253 secs] [Times: user=33.95 sys=0.48, real=2.33 secs]
9:05:39.812-0600: 34271.579: [GC 34271.580: [ParNew: 1023552K->113728K(1023552K), 1.6817535 secs] 2096045K->1453026K(3299264K), 1.6827555 secs] [Times: user=33.60 sys=0.20, real=1.68 secs]
9:05:41.498-0600: 34273.264: [GC [1 CMS-initial-mark: 1339298K(2275712K)] 1453026K(3299264K), 0.2348391 secs] [Times: user=0.22 sys=0.02, real=0.24 secs]
9:05:41.733-0600: 34273.500: [CMS-concurrent-mark-start]
9:05:44.732-0600: 34276.498: [CMS-concurrent-mark: 2.999/2.999 secs] [Times: user=6.49 sys=0.05, real=3.00 secs]
9:05:44.732-0600: 34276.498: [CMS-concurrent-preclean-start]
9:05:45.839-0600: 34277.606: [CMS-concurrent-preclean: 1.095/1.107 secs] [Times: user=2.25 sys=0.00, real=1.11 secs]
9:05:45.839-0600: 34277.606: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 9:05:50.866-0600: 34282.633: [CMS-concurrent-abortable-preclean: 5.021/5.027 secs] [Times: user=10.14 sys=0.05, real=5.03 secs]
9:05:50.868-0600: 34282.634: [GC[YG occupancy: 809255 K (1023552 K)]9:05:50.868-0600: 34282.635: [GC 34282.635: [ParNew: 809255K->113728K(1023552K), 1.0828740 secs] 2148553K->1611793K(3299264K), 1.0835767 secs] [Times: user=19.41 sys=0.06, real=1.08 secs]
34283.719: [Rescan (parallel) , 0.1498045 secs]34283.869: [weak refs processing, 1.0258216 secs]34284.896: [scrub string table, 0.0006202 secs] [1 CMS-remark: 1498065K(2275712K)] 1611793K(3299264K), 2.7794183 secs] [Times: user=24.56 sys=0.06, real=2.78 secs]
9:05:53.648-0600: 34285.415: [CMS-concurrent-sweep-start]
9:05:56.112-0600: 34287.879: [CMS-concurrent-sweep: 2.464/2.464 secs] [Times: user=7.38 sys=0.03, real=2.46 secs]
9:05:56.112-0600: 34287.879: [CMS-concurrent-reset-start]
9:05:56.120-0600: 34287.887: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
9:06:06.204-0600: 34297.970: [GC 34297.971: [ParNew: 1023552K->113728K(1023552K), 1.5001175 secs] 1845197K->1139597K(3299264K), 1.5011371 secs] [Times: user=31.31 sys=0.00, real=1.50 secs]
9:06:20.044-0600: 34311.810: [GC 34311.811: [ParNew: 1023552K->113728K(1023552K), 1.6438234 secs] 2049421K->1389674K(3299264K), 1.6446616 secs] [Times: user=36.97 sys=0.03, real=1.64 secs]
9:06:34.063-0600: 34325.829: [GC 34325.830: [ParNew: 1023552K->113728K(1023552K), 2.0298132 secs] 2299498K->1648212K(3299264K), 2.0308665 secs] [Times: user=34.10 sys=0.03, real=2.03 secs]
9:06:36.096-0600: 34327.862: [GC [1 CMS-initial-mark: 1534484K(2275712K)] 1648282K(3299264K), 0.2109338 secs] [Times: user=0.22 sys=0.00, real=0.21 secs]
9:06:36.308-0600: 34328.074: [CMS-concurrent-mark-start]
9:06:39.105-0600: 34330.871: [CMS-concurrent-mark: 2.797/2.797 secs] [Times: user=6.02 sys=0.00, real=2.80 secs]
9:06:39.105-0600: 34330.871: [CMS-concurrent-preclean-start]
9:06:39.795-0600: 34331.561: [CMS-concurrent-preclean: 0.686/0.690 secs] [Times: user=1.40 sys=0.00, real=0.69 secs]
9:06:39.795-0600: 34331.561: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 9:06:46.623-0600: 34338.389: [CMS-concurrent-abortable-preclean: 6.820/6.828 secs] [Times: user=13.85 sys=0.03, real=6.83 secs]
9:06:46.624-0600: 34338.390: [GC[YG occupancy: 891686 K (1023552 K)]9:06:46.624-0600: 34338.391: [GC 34338.391: [ParNew: 891686K->113728K(1023552K), 1.2772255 secs] 2426171K->1846761K(3299264K), 1.2780695 secs] [Times: user=25.66 sys=0.11, real=1.28 secs]
34339.669: [Rescan (parallel) , 0.1456700 secs]34339.815: [weak refs processing, 0.6135547 secs]34340.431: [scrub string table, 0.0005247 secs] [1 CMS-remark: 1733033K(2275712K)] 1846761K(3299264K), 2.3580749 secs] [Times: user=30.44 sys=0.11, real=2.36 secs]
9:06:48.983-0600: 34340.749: [CMS-concurrent-sweep-start]
9:06:51.895-0600: 34343.660: [CMS-concurrent-sweep: 2.911/2.911 secs] [Times: user=7.71 sys=0.06, real=2.91 secs]
9:06:51.895-0600: 34343.660: [CMS-concurrent-reset-start]
9:06:51.941-0600: 34343.707: [CMS-concurrent-reset: 0.047/0.047 secs] [Times: user=0.09 sys=0.00, real=0.05 secs]
9:07:02.441-0600: 34354.207: [GC 34354.207: [ParNew: 1023552K->113728K(1023552K), 1.5969685 secs] 1805173K->1136718K(3299264K), 1.5980247 secs] [Times: user=33.85 sys=0.00, real=1.60 secs]
9:07:18.155-0600: 34369.921: [GC 34369.922: [ParNew: 1023552K->113728K(1023552K), 2.2864633 secs] 2046542K->1381538K(3299264K), 2.2874742 secs] [Times: user=33.74 sys=0.02, real=2.29 secs]
9:07:33.522-0600: 34385.288: [GC 34385.288: [ParNew: 1023552K->113728K(1023552K), 1.6890223 secs] 2291362K->1632772K(3299264K), 1.6898157 secs] [Times: user=35.18 sys=0.01, real=1.69 secs]
9:07:35.213-0600: 34386.979: [GC [1 CMS-initial-mark: 1519044K(2275712K)] 1632799K(3299264K), 0.2011283 secs] [Times: user=0.20 sys=0.00, real=0.20 secs]
9:07:35.415-0600: 34387.181: [CMS-concurrent-mark-start]
9:07:37.810-0600: 34389.576: [CMS-concurrent-mark: 2.395/2.395 secs] [Times: user=5.63 sys=0.00, real=2.39 secs]
9:07:37.810-0600: 34389.576: [CMS-concurrent-preclean-start]
9:07:38.623-0600: 34390.389: [CMS-concurrent-preclean: 0.808/0.813 secs] [Times: user=1.70 sys=0.05, real=0.81 secs]
9:07:38.623-0600: 34390.389: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 9:07:45.370-0600: 34397.135: [CMS-concurrent-abortable-preclean: 6.742/6.747 secs] [Times: user=13.76 sys=0.11, real=6.75 secs]
9:07:45.371-0600: 34397.136: [GC[YG occupancy: 950924 K (1023552 K)]9:07:45.372-0600: 34397.137: [GC 34397.137: [ParNew: 950924K->113728K(1023552K), 1.3829715 secs] 2469968K->1894060K(3299264K), 1.3840623 secs] [Times: user=31.28 sys=0.00, real=1.38 secs]
34398.522: [Rescan (parallel) , 0.1888848 secs]34398.711: [weak refs processing, 0.3561537 secs]34399.068: [scrub string table, 0.0004774 secs] [1 CMS-remark: 1780332K(2275712K)] 1894060K(3299264K), 2.1091730 secs] [Times: user=36.55 sys=0.00, real=2.11 secs]
9:07:47.482-0600: 34399.247: [CMS-concurrent-sweep-start]
9:07:50.500-0600: 34402.265: [CMS-concurrent-sweep: 3.017/3.018 secs] [Times: user=8.63 sys=0.33, real=3.02 secs]
9:07:50.500-0600: 34402.265: [CMS-concurrent-reset-start]
9:07:50.507-0600: 34402.273: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
9:09:58.395-0600: 34530.160: [GC 34530.160: [ParNew: 1023552K->113438K(1023552K), 0.4065570 secs] 1764220K->964374K(3299264K), 0.4077345 secs] [Times: user=4.63 sys=0.03, real=0.41 secs]
9:12:06.040-0600: 34657.802: [GC 34657.803: [ParNew: 1023262K->27778K(1023552K), 0.0334396 secs] 1874198K->878714K(3299264K), 0.0345056 secs]-Andrew

Similar Messages

Garbage Collection Pauses in a client application: Trashing.

Hi,
we have put in production a financial trading application written in Java. We are experimenting a strange behaviour of the garbage collector. We are experimenting GC pauses every 4-5 hours which raise CPU time to 100% for 30 seconds. These pauses are not-deterministic. Memory consumption of the entire application is very low (about 30MB).
This is a strange beheaviour we have detected in the log file of GC:
5020.027: [GC 27847K->14659K(63936K), 0.0086360 secs]
5020.036: [Inc GC 14659K->24546K(63936K), 0.0149416 secs]
5020.107: [GC 27842K->14658K(63936K), 0.0086947 secs]
5020.116: [Inc GC 14658K->24546K(63936K), 0.0094716 secs]
5020.181: [GC 27842K->14658K(63936K), 0.0086846 secs]
5020.190: [Inc GC 14658K->24546K(63936K), 0.0095778 secs]
5020.255: [GC 27842K->14658K(63936K), 0.0102155 secs]
5020.266: [Inc GC 14658K->24546K(63936K), 0.0084659 secs]
5020.335: [GC 27842K->14658K(63936K), 0.0088606 secs]
5020.344: [Inc GC 14658K->24554K(63936K), 0.0084514 secs]
as you can see in one second the GC and the IncGC are called many times. WHY?
We have read articles and tried all various known settings. But we didn't find any solution and at the moment we don't know how to move.
Thank you for your hint
Piero

Which settings have you tried? By the looks of things you've not set any of your JVM memory variables.
Try bumping up your maximum heap space using -Xmx.
If you are running on a nice server try using -Xmx256m -Xmx256m -Xmn64m, these should help for a start.
Also why are you using the "incremental garbage collector" when you have such a tiny application as you can see a full garbage collection is only taking 0.01 seconds.
Also try reading this http://java.sun.com/docs/hotspot/gc/

Simulate full garbage collection pauses

Hi,
I'm doing some performance tuning for our Java systems, and one thing I want to test is how the applications behave when "stop-the-world" Full GC pauses of various different durations occur in our JVMs. We have seen some problems in our systems where longer GC pauses have negative effects. However, it is difficult to recreate these conditions using the applications themselves. Does anyone know if it is possible to force a JVM to do a "stop-the-world" pause of a specified duration, that would simulate a GC?
Notes -
1) I'm not talking about forcing a GC - I know how to do this but this give no control on the duration of the GC. Appreciate might be able to work a solution by writing a program to use up X amount of memory followed by a forced GC, but hoping for something simpler!
2) We have already exhaustively tuned our JVMs to reduces frequency and duration of Full GC as much as possible but they do still need to occur occassionally. Have engaged with forums alredy on how best to do this, so no need for more info on this!
3) We are using multiple JVMs which communicate with each other (RMI, etc.), and a large part of what I'm testing is the effect on JVM A of a long GC pause in JVM B while A is communicating with B
4) JVM version is 1.5.0_u18, normally running on Solaris 9/10
Thanks in advance for any insights!
Regards,
Adrian

Hi,
The docs here:
[http://docs.sun.com/app/docs/doc/806-1367/6jalj6mv1?a=view|http://docs.sun.com/app/docs/doc/806-1367/6jalj6mv1?a=view]
seem to suggest something like what I want to do could have been achieved in Java 1.2, by sending SIGQUIT to the process. However, in Java 5, SIGQUIT just creates a thread dump. Anyone know if its possible to achive the behaviour from that link in Java 5 by any other means / signal?
Regards,
Adrian
Edited by: AdrianFitz on 05-Mar-2010 13:40

Garbage Collector pausing my Server for MINUTES at a time

Garbage Collection pauses are really crippling my server. Here is a grep of Full GCs from the log file. (The lines following each Full GC have a timestamp, yymmdd:hhmmss.sss)
[Full GC 876186K->169083K(222400K), 28.4152794 secs]
011120:004809.413:
[Full GC 883850K->214246K(259008K), 68.2294388 secs]
011120:033449.558: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 209.2 MB/900.0 MB (Free: 43.7 MB, Allocated: 252.9 MB)
[Full GC 881484K->300898K(347200K), 73.3787317 secs]
011120:045927.680: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 293.8 MB/900.0 MB (Free: 45.2 MB, Allocated: 339.1 MB)
[Full GC 886225K->269194K(317824K), 79.8212023 secs]
011120:055800.166: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 262.9 MB/900.0 MB (Free: 47.5 MB, Allocated: 310.4 MB)
[Full GC 890405K->261774K(338112K), 104.6192760 secs]
011120:064350.312: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 255.6 MB/900.0 MB (Free: 74.5 MB, Allocated: 330.2 MB)
[Full GC 893451K->306762K(361024K), 134.4052782 secs]
011120:073523.335: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 299.6 MB/900.0 MB (Free: 53.0 MB, Allocated: 352.6 MB)
[Full GC 896034K->286275K(350016K), 103.4689894 secs]
011120:081636.710: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 279.6 MB/900.0 MB (Free: 62.2 MB, Allocated: 341.8 MB)
[Full GC 882755K->260923K(310272K), 129.6787672 secs]
011120:093812.612: *** RECLAIMED MemoryManager( Idle ) - Memory Usage: 254.8 MB/900.0 MB (Free: 48.2 MB, Allocated: 303.0 MB)
You can see that between 12:48am and 9:38am, the Server is pausing for a combined total of TWELVE MINUTES! Often for 2 whole minutes at a time, during which the Server is completely unresponsive.
I have read Sun's GC tuning guide at http://java.sun.com/docs/hotspot/gc/index.html but it is not much help:
It says "Pauses can be minimized by using a small young generation and incremental collection, at the expense of throughput."
Well, I AM using incremental collection, and the young generation is the default size, which is quite small I believe. (32 MB max?)
It also says "Unless you have problems with pauses, try granting as much memory as possible to the JVM. The default size (64MB) is often too small."
Well I DO have problems with pauses, but I HAVE to allocate a ton of memory. The Server must support hundreds of simultaneous users, each with hundreds of K's of personal, dynamic, temporary data.
It also says "Unless you find problems with excessive major collection or pause times, grant plenty of memory to the young generation. The default MaxNewSize (32MB) is generally too small."
I AM having problems with excessive major collections and pause times, so according to this I should NOT grant more memory to young generation (eden).
What am I left with? Nothing! I am totally out to lunch here. The box this is running on is a dual P3-600 with a gig of RAM and Red Hat 6.2, with Sun JRE 1.3.1_02. My startup command line options are:
java -server -Xincgc -verbose:gc -Xms256M -Xmx900M
What should I do? The Tuning guide essentially says to screw around with generation sizes, but none of its specific tips seem to apply to me! This is a production server and I am worried about making things worse by playing around, so I was hoping someone out there might have some experience with this and concrete advice before I just start messing around with the memory map. Should I increase the young generation size? Decrease it? Something else altogether? HELP!

With default settings, I am getting horrible performance, under peak load, 25%-50% of server time is being spent with GC, approx. 11,000 minor GCs per hour! And a Full GC every hour or two which takes up several minutes. On average, I am losing 13.09 minutes per hour due to minor GC, and 54.5 seconds to major GC.
I tried your settings, Chuck, and they seemed to be working beautifully at first... minor GCs were now occuring once every few minutes for a couple seconds, instead of 3 times per second for a tenth of a second. Minor GC were now taking 1.06 minutes per hour instead of 13.09, more than a 1300% improvement!
HOWEVER...
After 7 hours, the Server started getting REALLY slow for no apparent reason. It seemed almost as if reading from sockets was suddenly becoming extremely time intensive. Then I got a HUGE minor GC that took 25.6 seconds. [GC 507410K->478034K(551296K), 25.6328459 secs], then a couple minutes later, the Server CRASHED with this output:
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x40573aa7
Function name=malloc
Library=/lib/libc.so.6
Current Java thread:
Dynamic libraries:
011121:231738.194: Search ( pizzala, lzh ) returned 50 results (28 ms)
08048000-0804c000 r-xp 00000000 08:06 376904 /usr/local/java/jre1.3.1_01/bin/i386/native_threads/java
0804c000-0804d000 rw-p 00003000 08:06 376904 /usr/local/java/jre1.3.1_01/bin/i386/native_threads/java
40000000-40013000 r-xp 00000000 08:06 311298 /lib/ld-2.1.3.so
40013000-40014000 rw-p 00012000 08:06 311298 /lib/ld-2.1.3.so
40015000-40016000 r--p 00000000 08:06 2424834 /usr/share/locale/en_US/LC_MESSAGES/SYS_LC_MESSAGES
40016000-40017000 r--p 00000000 08:06 2408452 /usr/share/locale/en_US/LC_MONETARY
40017000-40018000 r--p 00000000 08:06 2408454 /usr/share/locale/en_US/LC_TIME
40018000-40023000 r-xp 00000000 08:06 311344 /lib/libpthread-0.8.so
40023000-4002a000 rw-p 0000a000 08:06 311344 /lib/libpthread-0.8.so
4002b000-40034000 r-xp 00000000 08:06 1081477 /usr/local/java/jre1.3.1_01/lib/i386/native_threads/libhpi.so
40034000-40035000 rw-p 00008000 08:06 1081477 /usr/local/java/jre1.3.1_01/lib/i386/native_threads/libhpi.so
40035000-403b0000 r-xp 00000000 08:06 2769091 /usr/local/java/jre1.3.1_01/lib/i386/server/libjvm.so
403b0000-403b1000 ---p 0037b000 08:06 2769091 /usr/local/java/jre1.3.1_01/lib/i386/server/libjvm.so
403b1000-404fd000 rw-p 0037b000 08:06 2769091 /usr/local/java/jre1.3.1_01/lib/i386/server/libjvm.so
40515000-40517000 r-xp 00000000 08:06 311314 /lib/libdl-2.1.3.so
40517000-40519000 rw-p 00001000 08:06 311314 /lib/libdl-2.1.3.so
4051a000-40607000 r-xp 00000000 08:06 311305 /lib/libc-2.1.3.so
40607000-4060b000 rw-p 000ec000 08:06 311305 /lib/libc-2.1.3.so
4060f000-40621000 r-xp 00000000 08:06 311318 /lib/libnsl-2.1.3.so
40621000-40623000 rw-p 00011000 08:06 311318 /lib/libnsl-2.1.3.so
40625000-40641000 r-xp 00000000 08:06 311316 /lib/libm-2.1.3.so
40641000-40642000 rw-p 0001b000 08:06 311316 /lib/libm-2.1.3.so
40642000-40676000 r-xp 00000000 08:06 704575 /usr/lib/libstdc++-2-libc6.1-1-2.9.0.so
40676000-40682000 rw-p 00033000 08:06 704575 /usr/lib/libstdc++-2-libc6.1-1-2.9.0.so
40684000-40695000 r-xp 00000000 08:06 1081479 /usr/local/java/jre1.3.1_01/lib/i386/libverify.so
40695000-40697000 rw-p 00010000 08:06 1081479 /usr/local/java/jre1.3.1_01/lib/i386/libverify.so
40697000-406b8000 r-xp 00000000 08:06 1081480 /usr/local/java/jre1.3.1_01/lib/i386/libjava.so
406b8000-406ba000 rw-p 00020000 08:06 1081480 /usr/local/java/jre1.3.1_01/lib/i386/libjava.so
406bb000-406cf000 r-xp 00000000 08:06 1081481 /usr/local/java/jre1.3.1_01/lib/i386/libzip.so
406cf000-406d2000 rw-p 00013000 08:06 1081481 /usr/local/java/jre1.3.1_01/lib/i386/libzip.so
406d2000-41400000 r--s 00000000 08:06 1081510 /usr/local/java/jre1.3.1_01/lib/rt.jar
4142d000-416d2000 r--s 00000000 08:06 1081511 /usr/local/java/jre1.3.1_01/lib/i18n.jar
416d2000-416e8000 r--s 00000000 08:06 1081498 /usr/local/java/jre1.3.1_01/lib/sunrsasign.jar
77a9f000-77ab5000 r--p 00000000 08:06 2408451 /usr/share/locale/en_US/LC_CTYPE
77ab5000-77abd000 r--p 00000000 08:06 2408450 /usr/share/locale/en_US/LC_COLLATE
77abd000-77abe000 r--p 00000000 08:06 2408453 /usr/share/locale/en_US/LC_NUMERIC
77abe000-77abf000 r-xp 00000000 08:06 1622117 /usr/lib/gconv/ISO8859-1.so
77abf000-77ac0000 rw-p 00000000 08:06 1622117 /usr/lib/gconv/ISO8859-1.so
77ac1000-77ac9000 r-xp 00000000 08:06 311336 /lib/libnss_files-2.1.3.so
77ac9000-77aca000 rw-p 00007000 08:06 311336 /lib/libnss_files-2.1.3.so
77b37000-77b40000 r-xp 00000000 08:06 1081484 /usr/local/java/jre1.3.1_01/lib/i386/libnet.so
77b40000-77b41000 rw-p 00008000 08:06 1081484 /usr/local/java/jre1.3.1_01/lib/i386/libnet.so
77b41000-77b45000 r-xp 00000000 08:06 1081513 /usr/local/java/jre1.3.1_01/lib/i386/libNBIO.so
77b45000-77b46000 rw-p 00003000 08:06 1081513 /usr/local/java/jre1.3.1_01/lib/i386/libNBIO.so
77b46000-77b4f000 r-xp 00000000 08:06 311342 /lib/libnss_nisplus-2.1.3.so
77b4f000-77b51000 rw-p 00008000 08:06 311342 /lib/libnss_nisplus-2.1.3.so
77b51000-77b59000 r-xp 00000000 08:06 311340 /lib/libnss_nis-2.1.3.so
77b59000-77b5b000 rw-p 00007000 08:06 311340 /lib/libnss_nis-2.1.3.so
77b5b000-77b5e000 r-xp 00000000 08:06 311334 /lib/libnss_dns-2.1.3.so
77b5e000-77b5f000 rw-p 00002000 08:06 311334 /lib/libnss_dns-2.1.3.so
77b5f000-77b6b000 r-xp 00000000 08:06 311346 /lib/libresolv-2.1.3.so
77b6b000-77b6c000 rw-p 0000b000 08:06 311346 /lib/libresolv-2.1.3.so
Local Time = Wed Nov 21 23:17:38 2001
Elapsed Time = 26581
# The exception above was detected in native code outside the VM
# Java VM: Java HotSpot(TM) Server VM (1.3.1_01 mixed mode)
# An error report file has been saved as hs_err_pid20055.log.
# Please refer to the file for further information.
Yummy. Signal 11's really make my day fun :(. I tried restarting the Server, but it crashed again in the same way 15 minutes later. Then I switched back to default settings and it has run fine all weekend.
The Sig11 makes me think either our C lib is buggy (our Linux distro is getting a bit old), or we have a physical memory problem. Odd that it only shows up when I tried your new settings though...

Exorbitant Garbage Collection Times

Once a day, at around 4-4:30am, our application experiences a single Full GC with taking an exorbitant amount of time. A Full GC kicks in and pauses all threads for 5 to 30 minutes. Full GC's regularly occur at other times and complete in 5-20 seconds. Anyone seen this or any have any ideas about this?
Example of normal GC that occurs frequently throughout the day:
[Full GC[Unloading class $Proxy136]
4632722K->1616289K(6141952K), 14.9400110 secs]
Example of ridiculously long GC that occurs daily at 4-4:30am:
[Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor455]
5456231K->2017269K(6141952K), 492.0402160 secs]
Here's our VM params:
-server -XX:-OmitStackTraceInFastThrow -Xincgc -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:MaxGCPauseMillis=20000 -XX:+DisableExplicitGC -XX:NewSize=1500m -XX:SurvivorRatio=18 -verbose:gc
-Xms6000M -Xmx6000M
We're running Java 1.6 on Linux, Redhat 3.

Just experienced a 498 second (yikes!) garbage collection pause. The entry from the log is below (we added -XX:+PrintGCTimeStamps -XX:+PrintGCDetails and removed -verbose:gc). I will also send the entire log to the email address you have provided.
Note that we actually reduced our Xmx to 4000M (from 6000M) despite this machine having 8gig physical ram in the hopes that this would prevent any paging, and it seems to have drastically improved the situation - these exorbitant pauses now occur only very seldomly.
This is the full command we're justing to start the JBoss JVM:
/proj/bo/apps/jdk/production/bin/java -Dprogram.name=run.sh -server -XX:-OmitSta
ckTraceInFastThrow -Xincgc -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:MaxGCPauseMillis=20000 -XX:+DisableExplicitGC
-XX:NewSize=1500m -XX:SurvivorRatio=18 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Djava.endorsed.dirs=/var/local/jboss/jboss-c
urrent/lib/endorsed -Dsystem.state=production -Dcom.hb.common.api.util.Environment.app.name=Hermes -Djboss.partition.name=as20.
nyc.hcmny.com -Dorg.jboss.security.SecurityAssociation.ThreadLocal=true -Doracle.net.tns_admin=/var/opt/oracle/etc -Djnp.discov
eryPort=1103 -classpath /var/local/jboss/jboss-current/bin/run.jar:/proj/bo/apps/jdk/production/lib/tools.jar -Xms4000M -Xmx400
0M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8134 -Dcom.sun.management.jmxremote.authenticate=true -Dc
om.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/proj/bo/jmxremote.password org.jboss.Main -
cdefault
This is the exact JRE we're using:
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-b105, mixed mode)
Here's the 498 second GC pause:
103999.802: [GC 103999.802: [ParNew (promotion failed): 1459200K->1459200K(1459200K), 478.7897900 secs]104478.592: [CMS104483.552: [CMS-concurrent-mark: 5.170/491.280 secs]
(concurrent mode failure)[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor605]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor866]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor870]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor887]
[Unloading class sun.reflect.GeneratedMethodAccessor267]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor864]
[Unloading class sun.reflect.GeneratedConstructorAccessor148]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor861]
[Unloading class sun.reflect.GeneratedMethodAccessor268]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor606]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor862]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor886]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor706]
[Unloading class sun.reflect.GeneratedConstructorAccessor145]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor865]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor350]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor860]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor868]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor863]
[Unloading class sun.reflect.GeneratedConstructorAccessor147]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor867]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor869]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor185]
[Unloading class sun.reflect.GeneratedConstructorAccessor125]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor871]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor707]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor708]
: 2517456K->2073715K(2560000K), 19.7399910 secs] 3927185K->2073715K(4019200K) icms_dc=5 , 498.5297810 secs]
     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
     at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1838)
     at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1747)
     at java.io.ObjectOutputStream.<init>(ObjectOutputStream.java:226)
21:37:07,931 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=as20.nyc.hcmny.com/163214, BranchQual=, localId=163214] timed out. status=STATUS_ACTIVE
2007-09-12 21:37:08

Garbage collection Java Virtual Machine : Hewlett-Packard Hotspot release 1.3.1.01

"Hi,
I try and understand the mechanism of garbage collection of the Java Virtual Machine : Hewlett-Packard Hotspot release 1.3.1.01.
There is description of this mechanism in the pdf file : "memory management and garbage collection" available at the paragraph "Java performance tuning tutorial" at the page :
http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1607,00.html
Regarding my question :
Below is an extract of the log file of garbage collections. This extract has 2 consecutive garbage collections.
(each begins with "<GC:").
<GC: 1 387875.630047 554 1258496 1 161087488 0 161087488 20119552 0 20119552
334758064 238778016 335544320
46294096 46294096 46399488 5.319209 >
<GC: 5 387926.615209 555 1258496 1 161087488 0 161087488 0 0 20119552
240036512 242217264 335544320
46317184 46317184 46399488 5.206192 >
There are 2 "full garbage collections", one of reason "1" and one of reason "5".
For the first one "Old generation After " =238778016
For the second "Old generation After " =238778016
Thus, "Old generation Before garbage collection" of the second is higher than "Old generation After garbage collection". Why?
I expected all objects to be allocated in the "Eden" space. And therefore I did not expect to s

I agree but my current Hp support is not very good on JVM issues.
Rob Woollen <[email protected]> wrote:
You'd probably be better off asking this question to HP.
-- Rob
Martial wrote:
The object of this mail is the Hewlett-Packard 1.3.1.01 Hotspot JavaVirtual Machine
release and its garbage collection mechanism.
I am interested in the "-Xverbosegc" option for garbage collectionmonitoring.
I have been through the online document :
http://www.hp.com/products1/unix/java/infolibrary/prog_guide/java1_3/hotspot.html#-Xverbosegc
I would like to find out more about the garbage collection mechanismand need
further information to understand the result of the log file generatedwith the
"-Xverbosegc"
For example here is an extract of a garbage collection log file generatedwith
Hewlett-Packard Hotspot Java Virtual Machine. Release 1.3.1.01.
These are 2 consecutive rows of the files :
<GC: 5 385565.750251 543 48 1 161087488 0 161087488 0 0 20119552 264184480255179792
335544320 46118384 46118384 46137344 5.514721 >
<GC: 1 385876.530728 544 1258496 1 161087488 0 161087488 20119552 020119552 334969696
255530640 335544320 46121664 46106304 46137344 6.768760 >
We have 2 full garbage collections, one of Reason 5 and the next oneof Reason
1.
What happened between these 2 garbage collections as we got : "Oldgeneration
After" of row 2 is higher than "Old generation Before" of row 1? Iexpected Objects
to be initially allocated in eden and so we could not get "old generation2modified
between the end of one garbage collection and before the next one.
Could you please clarify this issue and/or give more information aboutgarbage
collection mechanisms with the Hewlett-Packard Hotspot Java VirtualMachine. Release
1.3.1.01.

Low Latency Garbage Collection

Hi,
What is the best garbage collector configuration for a low latency system?
Multi-threaded server process running on a multi-core box (say 8 cores). Reading and writing messages to the network, no disk. Want all messages to be processed in as timely a manner as possible wrt their individual latencies. Want to avoid some messages being delayed by a garbage collection pause. What will give me the 'fairest' latency time amongst the messages?
I should add, Java SE 1.6 recent version.
Thanks in advance for you suggestions.
Rupert Smith
Edited by: rupertlssmith on Dec 2, 2009 10:17 AM

The simplest approach is don't create any objects in your critical path. (Or keep them to a minimum) Javolution and Joda-time have a libraries which helps.
You can try the concurrent mark sweep or G1 collectors to see if they help.
You can get the GC low enough that you will see other delays such as the SMC or network jitter or memory bank effects instead.
i.e. If your server is simple enough, you can see more jitter from the hardware/OS than Java itself.
Edited by: Peter__Lawrey on 02-Dec-2009 20:45

Improving Latency - Garbage Collection

Hi there,
Are there any suggestions as to what you can do to reduce latency in real time applications. Any suggestions as to improving performance times or garbage collection?
Thanks,

Hi there,
Are there any suggestions as to what you can do to
reduce latency in real time applications. High octane Java!
I would have suggested you search for a specific forum on this site that features such topics but alas, google is not for you, so....

[JS] $.gc() - garbage collection, etc.

Hello everyone, bear with me for all the questions.
$.gc() - Initiates garbage collection in the JavaScript engine.
Ok, maybe I have a fundamental misunderstanding of things here, but I am wanting to understand a few things:
I was wanting to know if $.gc() can be called and executed outside of running scripts directly from with in the ESTK toolkit? The reason I ask is that its listed in the tools guide but not the javascript reference. I know things like $.writeln() only seem to apply/display from with in the ETSK toolkit, is it still executed otherwise silently when it appears in a script, $.sleep() works fine when used. So what about $.gc(), can I call it and have it execute from within my scripts outside of running them via the ETSK toolkit?
Is there a way to test if its in fact being called, executed, working? I have seen people talking about calling it twice and doubling up, $.gc() $.gc(), thus does it even work, or is it just a fabled mythical thing.
Do codes when executed outside the ETSK toolkit get funneled through the same javascript runtime engine that they do when executing from with in the ETSK toolkit? Is it all the same and the ETSK toolkit is just Adobe's GUI for writing, debugging, targeting different api's for the various programs with in the same underlying engine?
Does ScriptBay tap into and use this same underlying framework? Does it do anything inherently on its own for garbage collection? (since it can run scripts sequentially, repetitively, etc..)
Does anyone know what embedded javascript runtime/engine Adobe is using? This would be beneficial for any specific optimizations that can be considered or kept in mind in general based on the specific JS engine thats implemented when coding.
So aside from the mythical $.gc() , what other best practices for garbage collection should be considered? I know people suggest wrapping your codes in a function, talk of #targetengine (saw in InDesign forum), etc… are there any other common do's and dont's, best practices pertaining specifically to illustrator? Granted I can find a lot about Javascript in general across the web but what about how it applies to Illustrator, its api, limitations, performance, garbage collection, etc..
Thanks. ;-)
Again, sorry for all the questions, I am still trying to continue to wrap my head around different aspects of this whole thing. Thanks everyone for any feedback your able to offer and provide. Many thanks in advance, its greatly appreciated. Thanks again.

I can't answer or help with all of those… Here are my thoughts… The dollar object properties and functions can be called on by script from AI, BR, ID & PS… They are NOT restricted for use in the ESTK tookit. Yeah $.writeln() would write to the console if the toolkit were open if NOT then it will probably launch it… So remove or comment these out. As you can see from $.sleep() it does what it says when run in the host app. #targetengine is exclusive to ID this app can have multi-instances ( or something like that ). The other apps have 1 engine and thats it. AI's engine is persistent so wrapping your code is good practice I do this for BR & PS too… See…
http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/devnet/pdf/illustrator/scrip ting/readme_cs5.txt
As for garbage collection I've never fould the need to use it… I see some like to set variables back to null but again I don't do this…
You would have been better off asking this question in the ID scripting forum… There are several people there who are better qualified to answer some of this…

Hotspot core dumping during JVM garbage collection ?

We have an application which calls a 3rd party supplied server API which has recently been upgraded to use Java 1.5
We are getting the following error reported by our client application. The application is also now running Java 1.5 but references many classes in jar files which would have quite old code in.
The supplier of the API has stated that the problem requires us to recompile all our jar files using v 1.5 ( including things like jconnect and jms ?!?!? ). This sounds like a bit of a cop-out to me, not to mention being impossible since we don't have the source for things like jconnect.
I suspect that there is a garbage collection problem at the bottom of all this, but I'm not sure how I can "prove" this, nor do I currently have any real clue as to how to fix any GC problem that may exist.
The application is supposed to wait for a message on a MQSeries queue and then transforms it via Xalan XSLT and sends it to a Server application. I've tried playing around with heap sizes etc but that just seems to make it worse. If I leave it at the settings that the previous version used then the client at least manages to process a couple of messages before core dumping. There doesn't seem to be a consistent trigger event to cause the core dump ( it's not a message arriving on a queue for example ) but it does seem to be fairly consistent timewise, i.e. after
Any ideas gratefully accepted.
Here's a logfile excerpt from my applications showing the Hotspot error message :
=====================================================================================
08-Jul-2008 10:01:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:02:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:03:05 Waiting for messages from COLT.BBFS
405.815: [GC [PSYoungGen: 17331K->9244K(37632K)] 111702K->103615K(192128K), 0.1615910 secs]
405.977: [Full GC#
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
# An error report file with more information is saved as hs_err_pid2600.log
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
=====================================================================================
The logfile referred to in the error message contains the following.
=====================================================================================
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
--------------- T H R E A D ---------------
Current thread (0x001484d8): VMThread [id=8]
siginfo:si_signo=10, si_errno=0, si_code=1, si_addr=0x0000080f
Registers:
O0=0x00487588 O1=0xfe7d6454 O2=0x000079b0 O3=0x00007800
O4=0x00008868 O5=0x00147d48 O6=0xf8781460 O7=0xfe0f7938
G1=0xe52aaae8 G2=0x00000003 G3=0x00000003 G4=0x001484d8
G5=0xf8781d98 G6=0x00000002 G7=0xf8781d98 Y=0x805683e2
PC=0xfe141348 nPC=0xfe14134c
Top of Stack: (sp=0xf8781460)
0xf8781460: fe786000 00c6ba20 fe10013c e54bab68
0xf8781470: 0000080f e54bab6c 0000080f 00000004
0xf8781480: 00487588 00000134 e54bab6c 00000004
0xf8781490: 00000001 00000000 f87814c0 fe17aef4
0xf87814a0: fe6485b4 fe7d899c 001484d8 0011da40
0xf87814b0: 00148988 00148c10 00148d7c f8781880
0xf87814c0: 007c3389 007c3c0e 00000f87 00008868
0xf87814d0: 00008800 00487588 fe141310 fe7d6454
Instructions: (pc=0xfe141348)
0xfe141338: ec 06 c0 1a 80 a5 a0 00 22 40 00 0a ba 07 60 01
0xfe141348: f2 05 a0 00 ae 0e 60 03 80 a5 e0 03 22 40 00 05
Stack: [0xf8702000,0xf8781d98), sp=0xf8781460, free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x141348]
V [libjvm.so+0x17aefc]
V [libjvm.so+0x2d557c]
V [libjvm.so+0x300ef8]
V [libjvm.so+0x301e84]
V [libjvm.so+0x2ff950]
V [libjvm.so+0x29df30]
V [libjvm.so+0x362b44]
V [libjvm.so+0x6436f0]
VM_Operation (0xe03012b0): parallel gc system gc, mode: safepoint, requested by thread 0x0031bca0
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00b2c028 JavaThread "Thread-4" [_thread_in_native, id=85]
0x007f5048 JavaThread "Thread-0" [_thread_blocked, id=84]
0x00c27cf0 JavaThread "Notification Delivery" [_thread_blocked, id=81]
0x0026fa08 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=73]
0x00821048 JavaThread "RMI RenewClean-[162.11.2.32:44425]" daemon [_thread_blocked, id=70]
0x0031bca0 JavaThread "GC Daemon" daemon [_thread_blocked, id=67]
0x00cd5d28 JavaThread "RMI Reaper" [_thread_blocked, id=66]
0x003c9300 JavaThread "Timer-0" daemon [_thread_blocked, id=65]
0x00929fe0 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=64]
0x0089bf18 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=42]
0x00c47248 JavaThread "Pool thread #7" daemon [_thread_blocked, id=38]
0x00c466a0 JavaThread "Pool thread #6" daemon [_thread_blocked, id=37]
0x00311850 JavaThread "Pool thread #5" daemon [_thread_blocked, id=36]
0x00287a40 JavaThread "Pool thread #4" daemon [_thread_blocked, id=35]
0x00286e98 JavaThread "Pool thread #3" daemon [_thread_blocked, id=34]
0x00c134b0 JavaThread "Pool thread #2" daemon [_thread_blocked, id=33]
0x00ad09e0 JavaThread "Pool thread #1" daemon [_thread_blocked, id=32]
0x00286cd8 JavaThread "PoolThreadManager" daemon [_thread_blocked, id=31]
0x00c129e0 JavaThread "Channel Reaper" daemon [_thread_blocked, id=30]
0x00c669e8 JavaThread "ORB Daemon Thread" daemon [_thread_blocked, id=29]
0x00b10170 JavaThread "Worker for ServerProtocol: (iiop) /0.0.0.0:20168" daemon [_thread_blocked, id=22]
0x008a17e0 JavaThread "Syn~ Client" daemon [_thread_blocked, id=21]
0x003dc378 JavaThread "PoolScavenger0" daemon [_thread_blocked, id=20]
0x0015a928 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=15]
0x00159880 JavaThread "CompilerThread1" daemon [_thread_blocked, id=14]
0x00158a18 JavaThread "CompilerThread0" daemon [_thread_blocked, id=13]
0x00157b98 JavaThread "AdapterThread" daemon [_thread_blocked, id=12]
0x00156dc8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11]
0x0014ccd8 JavaThread "Finalizer" daemon [_thread_blocked, id=10]
0x0014ad90 JavaThread "Reference Handler" daemon [_thread_blocked, id=9]
0x00038238 JavaThread "main" [_thread_in_native, id=1]
Other Threads:
=>0x001484d8 VMThread [id=8]
0x0015c3b0 WatcherThread [id=16]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00037728/0x00037758] Threads_lock - owner thread: 0x001484d8
[0x00033650/0x00037ba8] Heap_lock - owner thread: 0x0031bca0
Heap
PSYoungGen total 37632K, used 9244K [0xf2eb0000, 0xf7ba0000, 0xf8400000)
eden space 28352K, 0% used [0xf2eb0000,0xf2eb0000,0xf4a60000)
from space 9280K, 99% used [0xf4a60000,0xf5367080,0xf5370000)
to space 25216K, 0% used [0xf6300000,0xf6300000,0xf7ba0000)
PSOldGen total 154496K, used 94371K [0xe8400000, 0xf1ae0000, 0xf2eb0000)
object space 154496K, 61% used [0xe8400000,0xee028e78,0xf1ae0000)
PSPermGen total 35584K, used 18260K [0xe4400000, 0xe66c0000, 0xe8400000)
object space 35584K, 51% used [0xe4400000,0xe55d5158,0xe66c0000)
Dynamic libraries:
0x00010000      /dsdvlp/java/jvm/jdk1.5.0_03/bin/java
0xff350000      /usr/lib/libthread.so.1
0xff340000      /usr/lib/libdl.so.1
0xff200000      /usr/lib/libc.so.1
0xff390000      /usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1
0xfe000000      /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server/libjvm.so
0xff1e0000      /usr/lib/libsocket.so.1
0xff2d0000      /usr/lib/libsched.so.1
0xff1b0000      /usr/lib/libCrun.so.1
0xff160000      /usr/lib/libm.so.1
0xff080000      /usr/lib/libnsl.so.1
0xff060000      /usr/lib/libmp.so.2
0xff030000      /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/native_threads/libhpi.so
0xfdfc0000      /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libverify.so
0xfdf80000      /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libjava.so
0xfdf50000      /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libzip.so
0xfb7e0000      /usr/lib/locale/en_GB.ISO8859-1/en_GB.ISO8859-1.so.2
0xe4190000      /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libnet.so
0xe3bd0000      /dsdvlp/lib/5/libSolarisNatives.so
0xe3e90000      /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/librmi.so
VM Arguments:
jvm_args: -Djava.ext.dirs=/dsdvlp/java/tmijar/firs7/lib/cli:/dsdvlp/java/tmijar/firs7/lib/cli/ext:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB/ext:/dsdvlp/java/tmijar/firs7/lib/cmn:/dsdvlp/java/tmijar/firs7/lib/cmn/ext:/dsdvlp/java/tmijar/firs7/daemonlib -Duser.dir=/dsdvlp/java/tmijar/firs7 -Dopenorb.config=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB/config/SynOpenORB.xml -Dopenorb.home=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB -Dcom.coexis.syn.general.orbbinding=com.coexis.syn.general.orbbinding.openorb.OpenORBBinding_1_4 -Dsun.rmi.dgc.client.gcInterval=360000 -Dsun.rmi.dgc.server.gcInterval=360000000 -Xms32m -Xmx256m -Dcom.coexis.syn.clientcommandsconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/clientcommands.xml -Dcom.coexis.syn.clientconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/fsbbtd_client.xml -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
java_command: com.coexis.syn.mqmessaging.daemon.RunDaemon -p /dsdvlp/bin/5/lndsfsd_fsbbtd.properties start
Environment Variables:
JAVA_HOME=/dsdvlp/java/jvm/jdk150
CLASSPATH=.:/dsdvlp/java/jar/jconnect520.jar:/dsdvlp/java/jar/vbjapp340.jar:/dsdvlp/java/jar/vbjorb340.jar:/dsdvlp/java/jar/javax_jndi120.jar
PATH=/usr/local/etc:/usr/lang:/usr/openwin/bin:/usr/ucb:/bin:/usr/etc:/usr/local/5/bin:/dsdvlp/bin/5:/dsdvlp/bin/4:/home/app/sybase/5/bin:/home/app/sybase/5/localscripts:/home/app/sybase/5/sqr:/home/app/lang:/home/app/lang/SC2.0.1:/usr/ccs/bin:/usr/local/opt/Acrobat3/bin:/dsdvlp/bin:.
LD_LIBRARY_PATH=/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server:/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc:/dsdvlp/java/jvm/jdk1.5.0_03/jre/../lib/sparc:/usr/lib:/usr/openwin/lib:/usr/local/5/lib:/dsdvlp/lib/5:/dstest/lib/5:/home/app/sybase/5/lib:/dstest/cats/sun4/lib:/tmitest/Opus/opus/lib
SHELL=/bin/csh
DISPLAY=CLI00184.mfil.local:1.0
OS=5
--------------- S Y S T E M ---------------
OS: Solaris 8 2/02 s28s_u7wos_08a SPARC
Copyright 2002 Sun Microsystems, Inc. All Rights Reserved.
Assembled 18 December 2001
uname:SunOS 5.8 Generic_117350-20 sun4u (T1 libthread)
rlimit: STACK 8192k, CORE 9216k, NOFILE 4096, AS infinity
load average:2.24 2.67 2.68
CPU:total 4 has_v8, has_v9, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 8388608k(166384k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_03-b07) for solaris-sparc, built on Apr 13 2005 03:31:26 by unknown with unknown Workshop:0x550

The very first suggestion I have is to move your VM to a more recent update of 1.5.0.
It looks like you are crashing with 5.0u3, and I'm pretty sure 5.0u16 is available. You
don't want to waste your time chasing a bug that's already been fixed.

Long pauses during ParNew garbage collection Please Help !

Hi,
We are running a server application on an large machine (~120 CPU, ~380 GB Memory).
After running 1 or 2 hours we suddenly get exorbitant application pause times during garbage collection and a massive cpu usage from the java vm
We are running on Java 6 (64Bit) with 6GB Heap.
Concurrent garbage collection is turned on using the parameters:
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:CMSInitiatingOccupancyFraction=80
-XX:+DisableExplicitGC
We turned on verbose garbage collection and are getting the following output:
1. Normal operation:
Application time: 217.4656792 seconds
3180.905: [GC 3180.906: [ParNew
Desired survivor size 20119552 bytes, new threshold 4 (max 4)
- age   1:    2843824 bytes,    2843824 total
- age   2:    2577128 bytes,    5420952 total
- age   3:    5742024 bytes,   11162976 total
- age   4:     625672 bytes,   11788648 total
: 329531K->15764K(353920K), 0.1484379 secs] 2435799K->2122105K(3392144K), 0.1492386 secs]
Total time for which application threads were stopped: 0.1886810 seconds
2. The Problem:
Application time: 2.8858445 seconds
5008.433: [GC 5008.434: [ParNew
Desired survivor size 20119552 bytes, new threshold 2 (max 4)
- age   1:   15837712 bytes,   15837712 total
- age   2:   12284416 bytes,   28122128 total
: 348338K->39296K(353920K), 138.5317715 secs] 2487779K->2192551K(3392144K), 138.5327383 secs]
Total time for which application threads were stopped: 138.5778558 seconds
Application time: 2.9764564 seconds
5149.957: [GC 5149.957: [ParNew
Desired survivor size 20119552 bytes, new threshold 2 (max 4)
- age   1:    9483176 bytes,    9483176 total
- age   2:   14499344 bytes,   23982520 total
: 353920K->39296K(353920K), 231.5110574 secs] 2507175K->2204546K(3392144K), 231.5121011 secs]
Total time for which application threads were stopped: 231.5257754 seconds
Application time: 2.7932907 seconds
5384.277: [GC 5384.278: [ParNew
Desired survivor size 20119552 bytes, new threshold 4 (max 4)
- age   1:   10756376 bytes,   10756376 total
- age   2:    9135888 bytes,   19892264 total
: 353920K->28449K(353920K), 256.2065591 secs] 2519170K->2207651K(3392144K), 256.2076388 secs]
Total time for which application threads were stopped: 256.2221463 seconds
I can't find any significant differences in the log between fast and long running garbage collections.
I urgently need help in solving this problem !
What can I do ?

After the exciting reply I did get on my question, we did some more investigations on the problem and it seems that we finally found the solution to our problem.
The number of garbage collection threads used by the virtual machine defaults to the number of cpus of the machine.
This is ok for small machines or machines where the main load is produced by the java application itself.
In our environment the main load is not produced by the java application but oracle database processes.
When java tries to do it's garbage collection using 120 threads (# CPU) on the machine which is already overloaded by non java processes, the thread synchronization seems to produce an exorbitant overhead.
My theory is that spin locking is used on memory access, causing threads to spin while waiting for other blocking threads not getting cpu because of the heavy load on the system.
The solution is now to limit the number of garbage collection threads.
We did that on the first try by setting -XX:ParallelGCThreads=8
For over one day with heavy load no long GC pauses were experienced any more.

How to encourage jvm to garbage collect?

Hello,
I am working with an application that would benefit from more frequent garbage collection. It is running on a hefty machine, with multiple processors and more than 4 gigs of memory available for the JVM. Right now, we are setting the max heap size to 3 gigs.
The problem is that objects are accumulating, but are not being collected. During a particular operation that I am profiling, over 1 gig of objects are created, but are never collected. If I do a manual garbage collect (using jprofiler), they are all collected. There is no good way to load these objects in another way, or to create fewer objects.
I have spent a few days playing with -XX:MaxNewSize, -XX:NewSize, -XX:MaxHeapFreeRatio, -XX:TargetSurvivorRatio, and even -XX:+UseParNewGC. Unfortunately, I am unable to encourage the JVM to collect these objects automatically.
Are there any tricks that I am missing? Does anyone know any good way to encourage the JVM to collect garbage after a certain threshhold in the new generation?
Thanks,
--Jeff

You can try this!
How do I set the JVM's heap size?
On machines with limited memory (less than 384MB), it is recommended that you set the initial heap size lower than the default. Open the configuration file etc/netbeans.confin a text editor and modify the options in netbeans_default_optionssetting. Decrease the option -J-Xmx128mto -J-Xmx96m and the option -J-XX:MaxPermSize=96mto -J-XX:MaxPermSize=64m. Save the file and restart the IDE.
Bear in mind that UI responsiveness may be affected when the heap utilization gets close to its limit. Should you encounter an OutOfMemoryError, you need to increase Xmx or XX:MaxPermSize back to the default, or even higher.
Similarly when running on a machine with more memory it might be useful to increase the maximum size of the heap, especially when working with larger projects. Use the page linked below to get more details on this topic.
Applies to: NetBeans 4.x, 5.0
Platforms: All

Oracle JVM Garbage Collection not executing

I am experiencing a problem with garbage collection when running a java stored procedure in an 9.2.0.3.0 Oracle database running on a Windows 2k computer.
I have created a simple java class that represents a tree structure. Each instance of the class has a reference to it's parent and a Vecotor of all it's children. Each instance also has a Vector to hold name value pairs. I ran a program that creates 30 instances with 50 children each for a total of 1500 instances. Each instance has 30 properties(name value pairs).
By using the debugger in JDeveloper and breakpoints I can watch the javaw.exe process's use of RAM.
Start up: 6,788mb
After creation of 1500 instances: 17,720
After releasing all objects and calling System.gc(): 7,156
I deploy the package to my Oracle database and perform the exact same routine using breakpoints and test procedures in plsql Developer. Using the breakpoints and watching the oracle.exe process's use of RAM I see:
Start up: 81,116mb
After creation of 1500 instances: 94,952
After releasing all objects and calling System.gc(): 95,036
When run in Oracle the resources are not released. Is there somthing special that needs to be called to get the garbage collector to do it's job when running in an Oracle database?
Execution Instructions and source:
run app.main from jdeveloper and app.main2 as a plsql stored procedure. I have included my test procedure at the end and the plsql stored procedure package declarations. I used debuggers and breakpoints to watch the process, there is probably a way to do this by outputting the current memory being used by I didn't know how. If you use debuggers and breakpoints then when running in JDeveloper put your breakpoints on the 3 assignments of breakpointVar in SimpleClass.main(...). When running the stored procedure test block put them on the 3 assignments of breakpoint_var in there.
package mypackage2;
public class app {
private static SimpleClass topObject;
public static void main(String[] args) {
int breakpointVar;
breakpointVar = 0;
main2(30, 50);
breakpointVar = 1;
release();
breakpointVar = 2;
public static void main2(int numParents, int numChildren){
SimpleClass temp;
SimpleClass child;
topObject = new SimpleClass();
for(int i = 0; i < numParents; i ++){
temp = new SimpleClass();
addProperties(temp, 30);
topObject.addChild(temp);
for(int j = 0; j < numChildren; j++){
child = new SimpleClass();
addProperties(child, 30);
temp.addChild(child);
public static void release(){
topObject.releaseAllDecendents();
topObject.release();
System.gc();
private static void addProperties(SimpleClass toAddTo, int numProps){
cimxProperty toAdd;
for(int i = 0; i < numProps; i ++){
toAdd = new cimxProperty("prop "+i,"value "+i);
toAddTo.addProperty(toAdd);
package mypackage2;
import java.util.Vector;
public class SimpleClass {
private Vector _children;
private Vector _props;
private SimpleClass _parent;
public SimpleClass() {
_children = new Vector(10, 5);
_props = new Vector(10, 5);
_parent = null;
public void addChild(SimpleClass toAdd) {
toAdd._parent = this;
_children.add(toAdd);
public SimpleClass getChild(int index){
return (SimpleClass)_children.get(index);
public void addProperty(cimxProperty toAdd){
_props.add(toAdd);
public cimxProperty getProperty(int index) {
return (cimxProperty)_props.get(index);
public void releaseAllDecendents()
SimpleClass temp;
//remove all references to the the parent object from the children.
for(int i = 0; i < _children.size(); i ++)
temp = (SimpleClass)this._children.get(i);
temp.releaseAllDecendents();
temp._parent = null;
temp._props.clear();
temp._children.clear();
temp = null;
this._children.clear();
public void release() {
this._parent = null;
this._children.clear();
this._props.clear();
package mypackage2;
public class cimxProperty
private static String _dateFormat = "MM/DD/YYYY HH:MI:SS";
private String name;
private String value;
private String dataType;
public cimxProperty()
name = "";
value = "";
dataType = "";
public cimxProperty(String Name, String Value)
name = Name;
value = Value;
dataType = "UNKNOWN";
public cimxProperty(String Name, String Value, String DataType)
name = Name;
value = Value;
dataType = DataType;
public String toString()
return name + "["+dataType + "]: " + value;
public void setName(String name)
this.name = name;
public String getName()
return name;
public void setValue(String value)
this.value = value;
public String getValue()
return value;
public void setDataType(String DataType)
this.dataType = DataType;
public String getDataType()
return dataType;
CREATE OR REPLACE PACKAGE JAVA_MEMORY_TEST AUTHID CURRENT_USER AS PROCEDURE release; PROCEDURE main2(p_num_parents IN NUMBER, p_num_children IN NUMBER); END JAVA_MEMORY_TEST;
CREATE OR REPLACE PACKAGE BODY JAVA_MEMORY_TEST AS PROCEDURE release AS LANGUAGE JAVA NAME 'mypackage2.app.release()'; PROCEDURE main2(p_num_parents IN NUMBER, p_num_children IN NUMBER) AS LANGUAGE JAVA NAME 'mypackage2.app.main2(int, int)'; END JAVA_MEMORY_TEST;
declare
number breakpoint_var
begin
-- Call the procedure
breakpoint_var := 0;
java_memory_test.main2(30, 50);
breakpoint_var := 1;
java_memory_test.release();
breakpoint_var := 2;
end;

When run in Oracle the resources are not released. Is there somthing special that needs to
be called to get the garbage collector to do it's job when running in an Oracle database?Curious - I was always under the impression that you can not force garbage collection. All you could do was request it, and the JVM was free to do that when it was "good 'n ready", usually in a separate thread.
I gather this from the API docs for system.gc() where it says "Calling this method suggests that the Java virtual machine expend effort ..." and the word "suggests" imlies that it is not required to run.

Complete Garbage Collection JVM Options

Could anybody tell me where can I find the complete list and description of the garbage collector options?
Thanks
Edited by: dimonv on Mar 9, 2009 4:48 AM

dimonv wrote:
Could anybody tell me where can I find the complete list and description of the garbage collector options?Depends on the JVM vendor and version.
Did you mean to ask for this for the [Sun JVM|http://blogs.sun.com/watt/resource/jvm-options-list.html] ?

Manual garbage collection

Im running a huge combinatorial generation algorithm and i'm starting to run into memory allocation errors on larger problem instances. I am using the -Xmx256m run time option on the JVM to give it more memory but im generating huge masses off stuff here.
The question I have is, if I know an object isnt going to be used again is it worth destroying it in memory yourself; i assume if you have some undesired object you can do something like this;
myObject = null;
and then call the garbage collector to free up the memory..firstly will this work, secondly what is there performance tradeoff for calling the garbage collector more often and lastly is the garbage collector allready efficient enough that doing things like this is probably not worth it.
Regards,
Dave

Setting references to null MAY help get immediate relief, but its not guarunteed to help. Same with the System.gc() call.
Doing both those thing will not enable the system to operate with less memory than before, as variables that are out of scope will be discovered during garbage collection, and garbage collection WILL run when the heap is fully utilized.
Doing the above may make the GC pause lower at the expense of worse performance.
Instead of doing the above, I'd recommand using the Incremental GC by specifying -Xincgc on the command line. This will run GC more often in the background, leading to shorter GC pauses, but about 10% performance hit in general.
If OTOH you are running into OutOfMemoryErrors and don't think you should be, you probably have a memory leak. Most likely you are allocating objects and storing them somewhere then forgetting about them. For example, if you stick an object in a HashMap, and then forget about it, that object will not be GC'd, unless the HashMap can be GC'd.

Garbage collection pauses screw up my cluster. JVM tuning suggestions?

Similar Messages

Maybe you are looking for