Full GCs with long pauses
Long delays are being encountered when Full GCs kick in. Here are the statistics relating
to the a sample Full GC, with heap readings from before and after.
[Wed Dec 06 12:06:49 MET 2006] stdout: {Heap before GC invocations=1923:
[Wed Dec 06 12:06:49 MET 2006] stdout: Heap
[Wed Dec 06 12:06:49 MET 2006] stdout: PSYoungGen total 393344K, used 8322K [0x35400000, 0x50b00000, 0x50b00000)
[Wed Dec 06 12:06:49 MET 2006] stdout: eden space 337152K, 0% used [0x35400000,0x35400000,0x49d40000)
[Wed Dec 06 12:06:49 MET 2006] stdout: from space 56192K, 14% used [0x49d40000,0x4a560ad8,0x4d420000)
[Wed Dec 06 12:06:49 MET 2006] stdout: to space 56192K, 0% used [0x4d420000,0x4d420000,0x50b00000)
[Wed Dec 06 12:06:49 MET 2006] stdout: PSOldGen total 1214968K, used 1078564K [0x50b00000, 0x9ad7e000, 0xf54000
00)
[Wed Dec 06 12:06:49 MET 2006] stdout: object space 1214968K, 88% used [0x50b00000,0x92849260,0x9ad7e000)
[Wed Dec 06 12:06:49 MET 2006] stdout: PSPermGen total 50688K, used 39889K [0xf5400000, 0xf8580000, 0xf9400000)
[Wed Dec 06 12:06:49 MET 2006] stdout: object space 50688K, 78% used [0xf5400000,0xf7af4640,0xf8580000)
[Wed Dec 06 12:12:11 MET 2006] stdout: 67063.273: [Full GC[Unloading class sun.reflect.GeneratedMethodAccessor378]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor186]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor120]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor352]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor168]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor351]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor66]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor177]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor164]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor376]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor185]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor78]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor381]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor180]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor162]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor347]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor183]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor382]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor167]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor350]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor179]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor181]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedMethodAccessor349]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor184]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor174]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor69]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor173]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor76]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor182]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor44]
[Wed Dec 06 12:12:11 MET 2006] stdout: [Unloading class sun.reflect.GeneratedConstructorAccessor169]
[Wed Dec 06 12:12:48 MET 2006] stdout: 1086887K->1023366K(1608312K), 358.9035143 secs]
[Wed Dec 06 12:12:48 MET 2006] stdout: Heap after GC invocations=1923:
[Wed Dec 06 12:12:48 MET 2006] stdout: Heap
[Wed Dec 06 12:12:48 MET 2006] stdout: PSYoungGen total 393344K, used 0K [0x35400000, 0x50b00000, 0x50b00000)
[Wed Dec 06 12:12:48 MET 2006] stdout: eden space 337152K, 0% used [0x35400000,0x35400000,0x49d40000)
[Wed Dec 06 12:12:48 MET 2006] stdout: from space 56192K, 0% used [0x49d40000,0x49d40000,0x4d420000)
[Wed Dec 06 12:12:48 MET 2006] stdout: to space 56192K, 0% used [0x4d420000,0x4d420000,0x50b00000)
[Wed Dec 06 12:12:48 MET 2006] stdout: PSOldGen total 1214968K, used 1023366K [0x50b00000, 0x9ad7e000, 0xf5400000)
[Wed Dec 06 12:12:48 MET 2006] stdout: object space 1214968K, 84% used [0x50b00000,0x8f261810,0x9ad7e000)
[Wed Dec 06 12:12:48 MET 2006] stdout: PSPermGen total 49152K, used 39189K [0xf5400000, 0xf8400000, 0xf9400000)
[Wed Dec 06 12:12:48 MET 2006] stdout: object space 49152K, 79% used [0xf5400000,0xf7a456b8,0xf8400000)
[Wed Dec 06 12:12:48 MET 2006] stdout: } Total time for which application threads were stopped: 358.9442028 seconds
The original JVM paramaters were
vmparam -Xms512m # JVM Minimum Heap Size
vmparam -Xmx3072m # JVM Maximum Heap Size.
vmparam -Xmn439m # JVM New Objects Heap Size
vmparam -XX:MaxPermSize=64m # JVM Max Permanent Heap Size
vmparam -Xss256k # Stack size of threads
The system stopages due to Full GC with class unloading, ranged from 5 minutes to 25 minutes.
1)
This was occurring with the default collectors. The Throughput Collectors (Parallel Scavenge Collector)
was then trialled with the following settings
-XX:+UseParallelGC
-XX:-UseAdaptiveSizePolicy
UseAdaptiveSizePolicy was disabled due to Java Bug ID: 6314461
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6314461
This did not stop the excessive Full GC pauses from occurring.
2)
We then increased the PermSize and MaxPermSize and still use the Parallel GC
for eden collections.
vmparam -XX:PermSize=128m # Added new parameter
vmparam -XX:MaxPermSize=128m # Increased from 64m
Since the longest pauses with Full GC seemed to directly correspond with the amount
of reflected classes being unloaded, the Permanent space was increased, but this had
no effect. Permanent generation only filled up but only to around 80% and did not reduce
following Full GC and unloading of classes.
3)
The Low Pause Collector for young space was then implemented using
-XX:+UseParNewGC
PermSize and MaxPermSize were kept at 128m but obviously the Throughput Collector
parameter was removed.
This did not have the desired effect.
[Thu Dec 14 03:28:11 MET 2006] stdout: : 1819171K->1021636K(1819264K), 691.9769140 secs] 2247043K->1021636K(2255616K) Heap after GC invocations=1226:
[Thu Dec 14 03:28:11 MET 2006] stdout: Total time for which application threads were stopped: 692.2252804 seconds
4)
The Low Pause Collector for Eden and Permanent are now being trialled
using
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
To see does Full GC performance improve or can the application continue while
Full GC be in operation.
Should the Low Pause Collector for Young and Permanent exhibit the same system
delay as has been encountered with the other collectors, does it point specifically
at a potential code issue.
Or can this outage be linked with the Class unloading that is taking place
occasionally as part of Full GC.
System Info
The java version is java version "1.4.2_03"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02)
Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode)
It is not an option to upgrade to a later full version of java.
OS is SunOS 5.8 Generic_108528-26 sun4u sparc SUNW,Sun-Fire-880
8 cpu machine.
Heap Size seems to be adequate and they appears to be no swapping to disk
while Full GC is occurring to indicate lack of memory on the box.
Message was edited by:
shaneorourke
Just to point out that a similar post was raised prevoiusly by another
user, that seemed to witness the same pauses, see "PermGen not full
but full GC triggerred and classes unloaded"
http://forum.java.sun.com/thread.jspa?threadID=725550&messageID=4186298
Replies to that thread state that it is unlikely that class unloading
would be the source of the delay.
Regardless of how the Full GC is initiated, we have the scenario that
Full GCs collect the same amount of space take a normal duration,
but Full GCs with unloading invariably take longer.
Message was edited by:
shaneorourke
Similar Messages
-
Problem with long pauses in Tenured collection
We have a Swing application running on Windows XP, which gets long pauses when Tenured objects are collected after a running couple of days.
System:
Windows XP
Intel Pentium M 1.8 ghz
512 MB ram
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_10-b03)
JVM Arguments:
-XX:+ForceTimeHighResolution -Xloggc:gc.log -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xms16m -Xmx384m
The first Tenured collections after 1 hour:
3520.317: [GC 3520.317: [DefNew: 2414K->242K(2432K), 0.0035340 secs]3520.320: [Tenured: 32111K->15691K(32140K), 0.3034778 secs] 34405K->15691K(34572K), 0.3072704 secs]
3673.867: [GC 3673.868: [DefNew: 2335K->125K(2432K), 0.0028552 secs]3673.870: [Tenured: 32042K->18786K(32140K), 0.2668128 secs] 34312K->18786K(34572K), 0.2699225 secs]
3839.103: [GC 3839.103: [DefNew: 2423K->156K(2432K), 0.0028918 secs]3839.106: [Tenured: 32126K->22544K(32140K), 0.3026440 secs] 34341K->22544K(34572K), 0.3058143 secs]Pretty harmless so far.
After 1 day:
85332.923: [GC 85332.924: [DefNew: 6528K->703K(6528K), 0.0122671 secs]85332.936: [Tenured: 86929K->48880K(87032K), 0.6243351 secs] 92687K->48880K(93560K), 0.6369188 secs]
97305.405: [GC 97305.405: [DefNew: 6493K->91K(6528K), 0.0041855 secs]97305.410: [Tenured: 86755K->55828K(86776K), 0.5189188 secs] 92662K->55828K(93304K), 0.5233905 secs]
97844.653: [GC 97844.653: [DefNew: 7040K->768K(7040K), 0.0142342 secs]97844.668: [Tenured: 93393K->57002K(93432K), 0.5293686 secs] 99611K->57002K(100472K), 0.5438841 secs]The time between two Tenured collection just took a jump now to about 3 hours. The gc length is about half a second.
After 3 days:
256894.994: [GC 256894.994: [DefNew: 13938K->1166K(14720K), 0.0080440 secs]256895.002: [Tenured: 195455K->114421K(195696K), 0.8638032 secs] 208915K->114421K(210416K), 0.8721174 secs]
258670.232: [GC 258670.232: [DefNew: 14327K->1600K(14720K), 0.0273586 secs]258670.259: [Tenured: 196045K->117861K(196104K), 0.9146393 secs] 209062K->117861K(210824K), 0.9422603 secs]
261577.464: [GC 261577.464: [DefNew: 14144K->584K(14848K), 0.0078192 secs]261577.472: [Tenured: 196705K->121486K(196820K), 0.8976985 secs] 210412K->121486K(211668K), 0.9058215 secs]The Tenured collection interval is about 2-3 hours now. Each collection is about 1 second.
Then we get some peaks after 4 days:
413096.848: [GC 413096.848: [DefNew: 20126K->997K(21504K), 0.0067539 secs]413096.855: [Tenured: 285819K->160167K(285828K), 2.9034254 secs] 305888K->160167K(307332K), 2.9103817 secs]
472214.414: [GC 472214.414: [DefNew: 22388K->1227K(23616K), 0.0095498 secs]472214.424: [Tenured: 313891K->178781K(313992K), 22.2893423 secs] 336110K->178781K(337608K), 22.3386254 secs]
485465.045: [GC 485465.045: [DefNew: 23616K->2423K(23616K), 0.0404118 secs]485465.086: [Tenured: 315925K->203606K(316040K), 9.0069023 secs] 336994K->203606K(339656K), 9.0477205 secs]
492248.783: [GC 492248.783: [DefNew: 24461K->24461K(25536K), 0.0000111 secs]492248.783: [Tenured: 338665K->201218K(339344K), 6.5526445 secs] 363126K->201218K(364880K), 6.5985010 secs]
533898.023: [GC 533898.023: [DefNew: 25913K->25913K(27200K), 0.0000150 secs]533898.024: [Tenured: 337387K->205940K(363008K), 3.2169493 secs] 363301K->205940K(390208K), 3.2172333 secs]
546566.675: [GC 546566.675: [DefNew: 27199K->27199K(27200K), 0.0000103 secs]546566.675: [Tenured: 338319K->227086K(363008K), 26.2682151 secs] 365519K->227086K(390208K), 26.2684060 secs]
562738.779: [GC 562738.780: [DefNew: 26081K->26081K(27200K), 0.0000104 secs]562738.780: [Tenured: 336929K->230971K(363008K), 7.9569240 secs] 363010K->230971K(390208K), 7.9571524 secs]The average time between each young collection is approx. 20 seconds.
7 full gc's have been performed:
3.877: [Full GC 3.877: [Tenured: 1759K->1831K(15168K), 0.0801147 secs] 2351K->1831K(16320K), [Perm : 12287K->12287K(12288K)], 0.0802154 secs]
6.281: [Full GC 6.281: [Tenured: 5071K->4056K(15168K), 0.1245083 secs] 6018K->4056K(16320K), [Perm : 16383K->16383K(16384K)], 0.1245978 secs]
184.854: [Full GC 184.854: [Tenured: 12628K->10437K(15168K), 0.2133769 secs] 12995K->10437K(16320K), [Perm : 20480K->20480K(20480K)], 0.2135845 secs]
5216.782: [Full GC 5216.782: [DefNew: 3008K->101K(3008K), 0.0044835 secs] 35049K->32886K(42340K), 0.0046464 secs]
198022.892: [Full GC 198022.892: [DefNew: 11520K->790K(11520K), 0.0110744 secs] 117670K->108219K(164672K), 0.0112750 secs]
254208.982: [Full GC 254208.982: [DefNew: 13765K->844K(14720K), 0.0059109 secs] 151946K->139025K(210056K), 0.0061115 secs]
566774.751: [Full GC 566774.751: [DefNew: 25680K->1611K(27200K), 0.0103055 secs] 274385K->250385K(390208K), 0.0105140 secs]The long pauses are not acceptable at all. Only pauses under 1 second are tolerable.
I know there is at least one memory leak in the logged software, where probably 50.000-100.000 ThreadGroups are produced each day and not collected (but only with some configurations).
So, after browsing around, I have found out that UseConcMarkSweepGC could reduce the long pauses. I tried it together with CMSIncrementalMode, but the JVM running at one of our customers crashed after 2 hours while using both options together. Apparently there where some issues with the incremental collector, which was fixed in 1.6. I was forced to remove the VM arguments immediately, but will probably use UseConcMarkSweepGC later after some thorough testing.
I have tried find out which other VM options I could use to tune the application, but I'm quite new to this and the amount of options are a bit overwhelming at the moment. I thought I could ask here if someone know some obvious option values (like NewRatio,SurvivorRatio etc.)>
>
Doesn't a "Full Collection" always collect all generations (i.e. also the perm generation)? It's always been my understanding, that a "Full Collection" is some kind of "irregular" collection which only occurs if the JVM has "really bad need" for memory (e.g. when the concurrent phase of a CMS collection fails, when perm runs full or if someone calls System.gc()). So, as you write earlier, the first 3 collections labeld "Full GC" are real full GCs since they collect all generations (including perm). The latter four collections shouldn't be labeld "Full GC" (as you write). Yes, a full collection collects all the generations including the
perm generation.
In a stop-the-world collector such as the one you're using, a full
collection is not an unusual collection. They just happen less
frequenctly. Objects that survive a minor collection (collection
of the young gen) a few times will be promoted (copied to) the
tenured generation. The idea is that such objects may be
long lived so should be promoted to save the cost of handling
them during a minor collection. Eventually the tenured
generation gets full and then it gets collected.
And yes and yes. The first 3 are full collections and the
latter 4 are labelled incorrectly as full collections.
>
However, there are a lot of collections with "DefNew" followed by "Tenured" (the first three sections in the original post) which are regular collections where the promotion of objects from new to old caused the old generation to run full which triggers a "Tenured collection". But these collections (according to my understanding) are not full collections, but only regular tenured collections. (Sorry, I don't want to be picky about this. I guess that's even what you meant in your post...)I can see how this can be unclear. A "Tenured" collection
is a full collection. It also collects the perm gen. And
in fact it also collects the young gen again. This is how it
is implemented in the hotspot JVM. It can be implemented
differently in other JVM's.
The situation where a "DefNew" is followed by a "Tenured"
as in your logs is a bit unusual I think. The minor collection
is reducing the amount of used space in young generation.
DefNew: 13938K->1166K(14720K)
The decrease from 13938K to 1166K. There is then a collection
of the tenured gen.
Tenured: 195455K->114421K(195696K)
I've seen this when large objects (a significant fraction of
the young gen) are being allocated. The allocation fails in
the young gen, a minor collection is done but the space freed
in the young gen is still not large enough to allocate the object
so a collection of the tenured generation is done and the large
object is allocated directly out of the tenured gen. In the numbers
above after the minor collection about 13M of free space is
available in the young gen. Are you allocating objects larger
that 13M? -
WebLogic JMS high memory usage and frequent full GCs
Hi Team,
We have below set up in production (and all the lower environment).
1. WLS: 10.3.6 (64 bit)
2. Java: HP JDK 1.7.0.06 (64 bit)
3. Two Manage server instances in cluster with one Migratable target
4. one JMS server with Persistence store targeted to Migratable target
5. Messages are persisted in DB
6. about 37 distributed queues/topics are hosted on one JMS server
7. .Net JMS clients and WebLogic JMS clients connect to respective JMS distributed Queues/topics to either produce message or consume.
Since last month, we see that WebLogic Manage server hosting JMS server keeps on chewing memory "even with small (of the order of 28 messages in 4/5 days) load", GC indicates that objects are shifted from PSYoung space to ParOldGen during minor GCs but ParOldGen space keeps on filling up in smaller chunks. After x number of days (where x ranges from 5-7 days), even Full GC is not able to bring down memory significantly and that's when frequent full GCs start with long pauses making JMS highly unstable.
We don't have JMS Quotas configured but I think something else going on as it is not high on load. Any additional hints or troubleshooting steps is much appreciated?
Here is first and last GC log just to give an idea...
Heap after GC invocations=661 (full 0):
{Heap before GC invocations=1 (full 0):
PSYoungGen total 1225728K, used 1050624K [9fffffff09c00000, 9fffffff5f400000, 9fffffff5f400000)
eden space 1050624K, 100% used [0x9fffffff09c00000,0x9fffffff49e00000,0x9fffffff49e00000)
from space 175104K, 0% used [0x9fffffff54900000,0x9fffffff54900000,0x9fffffff5f400000)
to space 175104K, 0% used [0x9fffffff49e00000,0x9fffffff49e00000,0x9fffffff54900000)
ParOldGen total 2797568K, used 0K [0x9ffffffe5f000000, 0x9fffffff09c00000, 0x9fffffff09c00000)
object space 2797568K, 0% used [0x9ffffffe5f000000,0x9ffffffe5f000000,0x9fffffff09c00000)
PSPermGen total 49152K, used 47453K [0x9ffffffe1f000000, 0x9ffffffe22000000, 0x9ffffffe5f000000)
object space 49152K, 96% used [0x9ffffffe1f000000,0x9ffffffe21e57630,0x9ffffffe22000000)
2014-12-24T09:04:45.118-0700: 15.651: [GC [PSYoungGen: 1050624K->118221K(1225728K)] 1050624K->118293K(4023296K), 0.32600
15 secs] [Times: user=0.07 sys=0.00, real=0.33 secs]
{Heap before GC invocations=661 (full 0):
PSYoungGen total 1367168K, used 1364256K [9fffffff09c00000, 9fffffff5f400000, 9fffffff5f400000)
eden space 1333376K, 100% used [0x9fffffff09c00000,0x9fffffff5b220000,0x9fffffff5b220000)
from space 33792K, 91% used [0x9fffffff5d300000,0x9fffffff5f128030,0x9fffffff5f400000)
to space 33664K, 0% used [0x9fffffff5b220000,0x9fffffff5b220000,0x9fffffff5d300000)
ParOldGen total 2797568K, used 2654825K [0x9ffffffe5f000000, 0x9fffffff09c00000, 0x9fffffff09c00000)
object space 2797568K, 94% used [0x9ffffffe5f000000,0x9fffffff0109a5c0,0x9fffffff09c00000)
PSPermGen total 151552K, used 147591K [0x9ffffffe1f000000, 0x9ffffffe28400000, 0x9ffffffe5f000000)
object space 151552K, 97% used [0x9ffffffe1f000000,0x9ffffffe28021cd8,0x9ffffffe28400000)
2014-12-29T08:18:08.798-0700: 429216.418: [GC [PSYoungGen: 1364256K->33632K(1329792K)] 4019081K->2752791K(4127360K), 0.8
254362 secs] [Times: user=0.10 sys=0.00, real=0.83 secs]
Regards,Thanks Luz. I have been collecting gclogs and heap dumps (whenever app server is high on memory and frequent GCs with long pauses are seen). So far heap dumps haven't revealed anything specific (may be I am not able to read heap dumps properly ) but looking at GC confirms that app server is consuming allocated heap.I was following Oracle WebLogic server support Patter as you have indicated. I thought about opening a ticket with HP but on high level JVM is doing what is suppose to do i.e allocating memory space as needed and performing GC therefore didn't go that path.
Any other suggestions that might be useful. One of the things I forgot to mention ( and I will add it to original message), that one of the .Net consumers is not asynchronously connecting to its queue (to consume messages) rather it is synchronously connects and then if it doesn't find messages it disconnects and process goes on every few seconds. I was reading somewhere that frequent JNDI look ups should be avoided. Can this cause memory issue if so then it is strange as this same set up has been there over a year and we never has problem then?
Regards, -
Frequent expensive full GCs (after the first full GC) with JDK_1.5.0_06
On load testing the application we see that
- everything works fine for the first 35 minutes till we hit the first full GC
after the first full GC, which happened after 35 minutes (after about 500 minor GCs), strangely the full GCs happen very frequently, once about every 2-3 minutes (once every 5-6 minor GCs).This degrades the performance significantly
- The frequency of minor GCs remains the same, about 1 every 15-30 seconds.
- Monitoring using visualGC and JConsole shows that till the first full GC, the minor GC was collecting a majority of the newly allocated objects with only a few spilling over to the Survivors and subsequently to the old gen. Hence the old gen was filling up very slowly, and it took 35 minutes for it to fill up and cause the first full GC.
- However, immediately after this first full GC (Note that the load exposed to the servers remained the same), it seems that the minor GCs started collecting very less objects and a huge number of objects were spilled over to the Survivors (which got completely filled up) and subsequently to the old gen which also got filled up quickly within 5-6 minor GCs, leading to frequenty major GCs, which were very expensive (took ~20 seconds)
Here is our JVM parameters:
-XX:+PrintGCDetails -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -
XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -Xms2560m -Xmx2560m -XX:NewSize=896m -XX:MaxNewSize=896m
-XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 -XX:InitialSurvivorRatio=8 -XX:MinSurvivorRatio=8 -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=512m
This is an urgent issue and help is very much appreciatedDid you try the -XX:+UseConcMarkSweepGC collector?
One possibility regarding the frequent full gc's after the first
is the following somewhat contrived explanation. Until the first
full gc, most objects die in the survivor spaces and very few
get promoted (your observations). At the first full gc, the
entire heap is collected and all survivors end up, irrespective of
age, in the tenured (old) generation. This could include objects
that are relatively quite young. The application may store references
to newer younger objects in these relatively young but already
tenured objects. Suddenly there are many more references from
old to young gen, which keeps these referenced objects artifically
"alive" : the tenured objects may have already become unreachable
but will not be collected and will keep the younger gen objects
from being collected. So now you will have artificially longer object
lifetimes, resulting in a larger volume of survivors. If the survivor
spaces overflow, this can become a self-perpetuating
cycle as more young objects get tenured prematurely.
So I suggest two independent tacks here:
(1) enable adaptive sizing of generation (i noticed you had turned this
off; why?). This might allow GC to resize the generations and
survivor spaces to avoid (or mitigate) this badness. You might also
consider experimenting with -XX:+UseParallelOldGC which might
make the full gc's quicker.
(2) try the concurrent collector -XX:+UseConcMarkSweepGC
If you happen to do any of these experiments, I'd be interested to
know how the results turn out.
All the best! -
URGENT: Frequent expensive full GCs(after first full GC)with JDK_1.5.0_06
On load testing the application we see that
- everything works fine for the first 35 minutes till we hit the first full GC
after the first full GC, which happened after 35 minutes (after about 500 minor GCs), strangely the full GCs happen very frequently, once about every 2-3 minutes (once every 5-6 minor GCs).This degrades the performance significantly
- The frequency of minor GCs remains the same, about 1 every 15-30 seconds.
- Monitoring using visualGC and JConsole shows that till the first full GC, the minor GC was collecting a majority of the newly allocated objects with only a few spilling over to the Survivors and subsequently to the old gen. Hence the old gen was filling up very slowly, and it took 35 minutes for it to fill up and cause the first full GC.
- However, immediately after this first full GC (Note that the load exposed to the servers remained the same), it seems that the minor GCs started collecting very less objects and a huge number of objects were spilled over to the Survivors (which got completely filled up) and subsequently to the old gen which also got filled up quickly within 5-6 minor GCs, leading to frequenty major GCs, which were very expensive (took ~20 seconds)
Here is our JVM parameters:
-XX:+PrintGCDetails -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -
XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -Xms2560m -Xmx2560m -XX:NewSize=896m -XX:MaxNewSize=896m
-XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 -XX:InitialSurvivorRatio=8 -XX:MinSurvivorRatio=8 -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=512m
This is an urgent issue and help is very much appreciatedCross-posted; see a suggestion made at:
http://forum.java.sun.com/thread.jspa?threadID=5175759&messageID=9862052#9862052 -
I have a 7th generation ipod, i have a problem with the pause/play button, it no longer works.
I have a 7th generation ipod, i have a problem with the pause/play button, it no longer works.
What does it take to make sure it works ????
ThanksIf you successfullt restore to factory defaults/new iPod and still have the problem then it is likely a hardware problem and an appointment at the Genius Bar of an Apple store is in order.
-
Hello,
Since the last two weeks, we're experiencing Full GCs that take long time and block our application.
Our platform is:
Windows 2003 Server 64 bit
Tomcat 6.0.18
Java 1.6.0_u13.
Total memory: 8Gb.
-Xms4096m
-Xmx6144m
-Xss256k
Old generation: min 2,66Gb max 4,0G
Eden generation: min 1,5Gb max 2,0G
Perm generation: min 57Mb max 84Mb
Here a piece of our gc.log:
21601.266: [GC [PSYoungGen: 2009225K->6206K(1942528K)] 3913618K->1913365K(4738752K), 0.0992347 secs] [Times: user=0.16 sys=0.03, real=0.11 secs]
21611.268: [GC [PSYoungGen: 1942485K->5485K(1879040K)] 3849645K->1914490K(4675264K), 0.0986129 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]
21618.350: [GC [PSYoungGen: 1879021K->12392K(2060928K)] 3788026K->1930794K(4857152K), 0.2036000 secs] [Times: user=0.14 sys=0.02, real=0.20 secs]
21624.869: [GC [PSYoungGen: 1935116K->8545K(2065600K)] 3853519K->1930702K(4861824K), 0.0924182 secs] [Times: user=0.05 sys=0.00, real=0.09 secs]
21624.961: [Full GC (System) [PSYoungGen: 8545K->0K(2065600K)] [PSOldGen: 1922156K->635750K(2796224K)] 1930702K->635750K(4861824K) [PSPermGen: 38643K->38643K(60992K)], 1.3719842 secs] [Times: user=1.09 sys=0.00, real=1.36 secs]
21634.887: [GC [PSYoungGen: 2048500K->10395K(2078912K)] 2684250K->646145K(4875136K), 0.0180670 secs] [Times: user=0.02 sys=0.05, real=0.01 secs]
21645.165: [GC [PSYoungGen: 2074007K->8120K(2003392K)] 2709758K->648930K(4799616K), 0.0176943 secs] [Times: user=0.08 sys=0.00, real=0.03 secs]
25209.938: [GC [PSYoungGen: 1760190K->7344K(1707072K)] 4127764K->2376892K(4503296K), 0.0760053 secs] [Times: user=0.11 sys=0.03, real=0.06 secs]
25218.120: [GC [PSYoungGen: 1707056K->10511K(1657728K)] 4076604K->2382964K(4453952K), 0.0704081 secs] [Times: user=0.13 sys=0.00, real=0.08 secs]
25221.680: [GC [PSYoungGen: 1657679K->5731K(1982400K)] 4030132K->2385544K(4778624K), 0.0985384 secs] [Times: user=0.20 sys=0.00, real=0.09 secs]
25226.968: [GC [PSYoungGen: 993426K->6083K(1982784K)] 3373239K->2389543K(4779008K), 0.5147901 secs] [Times: user=0.08 sys=0.00, real=0.52 secs]
25227.483: [Full GC (System) [PSYoungGen: 6083K->0K(1982784K)] [PSOldGen: 2383460K->536034K(2796224K)] 2389543K->536034K(4779008K) [PSPermGen: 39765K->38756K(58816K)], 155.4030025 secs] [Times: user=1.34 sys=0.64, real=155.39 secs]
25389.833: [GC [PSYoungGen: 1976624K->12915K(2023616K)] 2512658K->566671K(4819840K), 13.1263397 secs] [Times: user=0.11 sys=0.03, real=13.13 secs]
25405.871: [GC [PSYoungGen: 2023579K->13857K(2031552K)] 2577335K->583736K(4827776K), 3.5796784 secs] [Times: user=0.03 sys=0.00, real=3.58 secs]
25416.490: [GC [PSYoungGen: 2024544K->9059K(2074944K)] 2594423K->587581K(4871168K), 0.1184355 secs] [Times: user=0.06 sys=0.00, real=0.13 secs]
25425.393: [GC [PSYoungGen: 2064144K->7415K(2075456K)] 2642665K->590222K(4871680K), 0.0441002 secs] [Times: user=0.02 sys=0.00, real=0.05 secs]
25438.547: [GC [PSYoungGen: 2062519K->8857K(2075264K)] 2645326K->594735K(4871488K), 0.0301620 secs] [Times: user=0.01 sys=0.00, real=0.03 secs]
The minor collections go perfect.
The first full GC in this piece of log takes 1,36s, and the old generation passes from 1922156K->635750K
But the second full GC (1 hour later) blocks the server for 155.39s, and the old generation pases from 2383460K->536034K
The server load between execution of full GC was similar. Any differences I saw in this two moments:
first full gc: number of Tomcat http threads : 230. The full gc provokes that Tomcat start new threads until get 500 (maxThreads)
second full gc: number of Tomcat http threads : 500.
I'm not an expert in tunning jvm. If somebody could help me, these pauses are killing the app and it's unacceptable for my client (for 2 min. the app does not work).
If you need more information (threads stack, histograms, ... I have ...)
Maybe I need to reduce the size of the old generation? Or change the garbage collector for old generation? The server cpu is going well, I prefer to consume more cpu in gc and avoid this pauses.
Any help will be very very appreciated.
Thanks in advance,
Joan.Hello,
Sorry for my continuous replies. Another full gc has occurred. This is the trace:
262631.531: [GC [PSYoungGen: 2071685K->10889K(2078464K)] 4711353K->2652961K(4874688K), 0.3392037 secs] [Times: user=0.08 sys=0.00, real=0.34 secs]
262640.608: [GC [PSYoungGen: 2071978K->11055K(2078784K)] 4714051K->2656136K(4875008K), 1.3267436 secs] [Times: user=0.06 sys=0.00, real=1.33 secs]
262649.880: [GC [PSYoungGen: 2072165K->17337K(2071872K)] 4717246K->2669454K(4868096K), 3.0493937 secs] [Times: user=0.16 sys=0.00, real=3.05 secs]
262658.069: [GC [PSYoungGen: 2071865K->14678K(2075840K)] 4723982K->2676078K(4872064K), 5.4598301 secs] [Times: user=0.08 sys=0.03, real=5.45 secs]
262666.592: [GC [PSYoungGen: 2069183K->15835K(2075776K)] 4730583K->2682691K(4872000K), 1.6894894 secs] [Times: user=0.11 sys=0.05, real=1.70 secs]
262673.352: [GC [PSYoungGen: 2070620K->14122K(2068992K)] 4737476K->2687230K(4865216K), 2.9711981 secs] [Times: user=0.06 sys=0.03, real=2.97 secs]
262682.891: [GC [PSYoungGen: 2068909K->10709K(2075712K)] 4742017K->2686673K(4871936K), 1.1160323 secs] [Times: user=0.19 sys=0.00, real=1.13 secs]
262691.733: [GC [PSYoungGen: 2065941K->10683K(2075584K)] 4741905K->2689212K(4871808K), 1.4898942 secs] [Times: user=0.05 sys=0.00, real=1.50 secs]
262701.512: [GC [PSYoungGen: 2065888K->9341K(2076480K)] 4744418K->2690073K(4872704K), 1.3835846 secs] [Times: user=0.13 sys=0.00, real=1.39 secs]
262709.896: [GC [PSYoungGen: 2065601K->10718K(2076480K)] 4746334K->2693212K(4872704K), 0.7815897 secs] [Times: user=0.05 sys=0.01, real=0.78 secs]
262718.681: [GC [PSYoungGen: 2067028K->10199K(2076672K)] 4749523K->2695508K(4872896K), 1.2951014 secs] [Times: user=0.06 sys=0.00, real=1.30 secs]
262727.940: [GC [PSYoungGen: 2067345K->10112K(2076608K)] 4752654K->2698129K(4872832K), 1.6785345 secs] [Times: user=0.08 sys=0.00, real=1.69 secs]
262736.092: [GC [PSYoungGen: 2067264K->10886K(2077184K)] 4755281K->2702237K(4873408K), 1.1985764 secs] [Times: user=0.06 sys=0.02, real=1.20 secs]
262744.626: [GC [PSYoungGen: 2069382K->11366K(2077248K)] 4760733K->2705795K(4873472K), 1.4323653 secs] [Times: user=0.13 sys=0.00, real=1.44 secs]
262754.913: [GC [PSYoungGen: 2069912K->11945K(2077824K)] 4764342K->2708689K(4874048K), 0.8340946 secs] [Times: user=0.03 sys=0.00, real=0.84 secs]
262765.038: [GC [PSYoungGen: 2072283K->11669K(2077888K)] 4769028K->2711204K(4874112K), 1.3040060 secs] [Times: user=0.09 sys=0.02, real=1.31 secs]
262776.885: [GC [PSYoungGen: 2072021K->9940K(2079360K)] 4771556K->2711919K(4875584K), 0.8646772 secs] [Times: user=0.09 sys=0.00, real=0.88 secs]
262783.713: [GC [PSYoungGen: 1173115K->8659K(2079168K)] 3875094K->2713127K(4875392K), 1.4434465 secs] [Times: user=0.09 sys=0.00, real=1.45 secs]
262785.157: [Full GC (System) [PSYoungGen: 8659K->0K(2079168K)] [PSOldGen: 2704467K->655075K(2796224K)] 2713127K->655075K(4875392K) [PSPermGen: 41376K->40988K(42048K)], 132.6088125 secs] [Times: user=1.47 sys=0.39, real=132.58 secs]
262925.100: [GC [PSYoungGen: 2062280K->16892K(2068992K)] 2717355K->683188K(4865216K), 8.9491611 secs] [Times: user=0.13 sys=0.00, real=8.94 secs]
262971.183: [GC [PSYoungGen: 2068978K->8052K(2074624K)] 2735273K->692990K(4870848K), 7.7833800 secs]
When a "bad" full GC occurs, I can see that just before it the minor collections start to perform bad. Normal values for minor collections are 0,0x seconds, and before a "bad" full GC these minor collections take seconds (1, 2, ... up to 5 seconds).
I can show an histogram of the first 25 entries before and after the full GC:
BEFORE
num #instances #bytes class name
1: 226135 718455272 [I
2: 1905686 637239584 [C
3: 122775 621626112 [B
4: 8218 538714736 [Lcom.sun.org.apache.xpath.internal.objects.XObject;
5: 160974 445403288 [Ljava.lang.Object;
6: 1759928 70397120 java.lang.String
7: 334672 42838016 com.sun.org.apache.xerces.internal.dom.DeferredElementNSImpl
8: 103592 42699248 [[I
9: 227966 41945744 java.net.SocksSocketImpl
10: 629974 35278544 com.vpfw.proxy.cache.data.B
11: 636515 30552720 java.util.concurrent.ConcurrentHashMap$HashEntry
12: 555354 26656992 java.util.HashMap$Entry
13: 89059 18034280 [Ljava.util.HashMap$Entry;
14: 230432 16591104 com.sun.org.apache.xerces.internal.dom.DeferredTextImpl
15: 225401 14425664 java.lang.ref.Finalizer
16: 222588 14245632 java.net.SocketInputStream
17: 334644 13385760 com.sun.org.apache.xerces.internal.dom.AttributeMap
18: 386837 12378784 java.util.concurrent.ConcurrentLinkedQueue$Node
19: 40226 11263504 [[Ljava.lang.Object;
20: 21646 11246112 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
21: 684613 10953808 java.lang.Object
22: 227958 9118320 java.net.Socket
23: 218591 8743640 java.net.Inet4Address
24: 57810 8730528 <constMethodKlass>
25: 230658 7381056 java.io.FileDescriptor
AFTER
num #instances #bytes class name
1: 91917 650028776 [B
2: 1767088 612264368 [C
3: 176313 581838552 [I
4: 141278 322132688 [Ljava.lang.Object;
5: 1762956 70518240 java.lang.String
6: 367122 46991616 com.sun.org.apache.xerces.internal.dom.DeferredElementNSImpl
7: 239506 44069104 java.net.SocksSocketImpl
8: 633882 35497392 com.vpfw.proxy.cache.data.B
9: 639797 30710256 java.util.concurrent.ConcurrentHashMap$HashEntry
10: 606366 29105568 java.util.HashMap$Entry
11: 72730 20521256 [[I
12: 89136 18497696 [Ljava.util.HashMap$Entry;
13: 252946 18212112 com.sun.org.apache.xerces.internal.dom.DeferredTextImpl
14: 236339 15125696 java.lang.ref.Finalizer
15: 233891 14969024 java.net.SocketInputStream
16: 366906 14676240 com.sun.org.apache.xerces.internal.dom.AttributeMap
17: 429913 13757216 java.util.concurrent.ConcurrentLinkedQueue$Node
18: 43319 12129544 [[Ljava.lang.Object;
19: 719233 11507728 java.lang.Object
20: 21649 11246408 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
21: 239498 9579920 java.net.Socket
22: 143 9317736 [Lcom.sun.org.apache.xpath.internal.objects.XObject;
23: 229531 9181240 java.net.Inet4Address
24: 57817 8735944 <constMethodKlass>
25: 242443 7758176 java.io.FileDescriptor
The big differencies is the "com.sun.org.apache.xpath.internal.objects.XObject;".
Before and after, the values are:
4: 8218 538714736 [Lcom.sun.org.apache.xpath.internal.objects.XObject;
22: 143 9317736 [Lcom.sun.org.apache.xpath.internal.objects.XObject;
Could this the cause of the full GC that takes long time?
Thanks,
Joan. -
Long pause of ParNew without Promotion failed.
I encounter a same problem like " [Long pauses |http://forums.sun.com/thread.jspa?threadID=5240437] ", no "promotion failed" tip but ParNew cost more than 20s, the following is the gc log's snapshot:
my jdk version is the latest JDK6u21 on RHEL5.1/IBM x3850 with 16core.
I try to use -X::+UseMemBar, but no effect. anybody could help me?
1913.579: [GC 1913.579: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15459032 bytes, 15459032 total
- age 2: 10545904 bytes, 26004936 total
: 433430K->35393K(458752K), 0.1056850 secs] 6509320K->6122752K(10420224K), 0.1061990 secs] [Times: user=0.83 sys=0.01, real=0.11 secs]
Heap after GC invocations=3850 (full 10):
par new generation total 458752K, used 35393K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.1082700 seconds
CMSCollector shouldConcurrentCollect: 1913.685
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.3237750 seconds
CMSCollector shouldConcurrentCollect: 1935.690
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
CMSCollector shouldConcurrentCollect: 1937.692
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Total time for which application threads were stopped: 23.8250250 seconds
CMSCollector shouldConcurrentCollect: 1937.834
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000880 seconds
Total time for which application threads were stopped: 0.0016130 seconds
CMSCollector shouldConcurrentCollect: 1937.836
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000580 seconds
Total time for which application threads were stopped: 0.0004200 seconds
CMSCollector shouldConcurrentCollect: 1937.837
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0019560 seconds
{Heap before GC invocations=3850 (full 10):
par new generation total 458752K, used 428609K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
1937.839: [GC 1937.839: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15389248 bytes, 15389248 total
- age 2: 10280752 bytes, 25670000 total
: 428609K->33832K(458752K), 0.0985530 secs] 6515968K->6131457K(10420224K), 0.0990430 secs] [Times: user=0.77 sys=0.01, real=0.10 secs]
Heap after GC invocations=3851 (full 10):
par new generation total 458752K, used 33832K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.0999980 seconds
CMSCollector shouldConcurrentCollect: 1937.939
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0446810 seconds
Total time for which application threads were stopped: 0.0048440 seconds
CMSCollector shouldConcurrentCollect: 1937.988
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.1273720 seconds
{Heap before GC invocations=3851 (full 10):
{Heap before GC invocations=3851 (full 10):
par new generation total 458752K, used 427048K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37808K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)Same problems occur with some of our server instances of a large server backend with huge heapsizes. Hardware is 16 x86 cores on Windows at one site. Although we dont't see "promotion failed" messages.
-Djava.awt.headless=true
-Djava.rmi.dgc.leaseValue=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dfile.encoding=UTF-8
-Xmx11000m
-Xms7000m
-Xmn3500m
-XX:PermSize=768m
-XX:MaxPermSize=768m
-XX:+UseParNewGC
-XX:+DisableExplicitGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:-UseLargePages
-XX:SoftRefLRUPolicyMSPerMB=1
-XX:SurvivorRatio=1
-XX:InitialTenuringThreshold=15
-Xshare:off
-verbose:class
-verbose:gc
-Xloggc:gc.log
-XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintCMSInitiationStatistics
-XX:PrintCMSStatistics=1
-XX:+PrintGCTaskTimeStamps
-XX:+PrintTenuringDistribution
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=1234
The following two options are currently disabled to exclude problems with concurrent CMS perm size collections:
-XX:+CMSClassUnloadingEnabled
-XX:+CMSIncrementalMode
This extract of the GC log shows one pause of 40 seconds at 2010/11/19 08:35:30 during the ParNew phase which normally runs for only 0.05 to 0.1 seconds.
INFO | jvm 1 | 2010/11/19 08:34:45 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:34:47 | CMSCollector shouldConcurrentCollect: 143805.306
INFO | jvm 1 | 2010/11/19 08:34:47 | time_until_cms_gen_full 1857957.8130999
INFO | jvm 1 | 2010/11/19 08:34:47 | free=425866008
INFO | jvm 1 | 2010/11/19 08:34:47 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:34:47 | promotion_rate=203.795
INFO | jvm 1 | 2010/11/19 08:34:47 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:34:47 | occupancy=0.8942350
INFO | jvm 1 | 2010/11/19 08:34:47 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:34:47 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:34:49 | 2010-11-19T08:34:49.332+0100: 143807.135: [GC 143807.135: [ParNew
INFO | jvm 1 | 2010/11/19 08:35:30 | Desired survivor size 671088640 bytes, new threshold 15 (max 15)
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 1: 102644048 bytes, 102644048 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 2: 30772776 bytes, 133416824 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 3: 27784600 bytes, 161201424 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 4: 10277736 bytes, 171479160 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 5: 1696 bytes, 171480856 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 6: 60584 bytes, 171541440 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 7: 1968 bytes, 171543408 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 8: 2200 bytes, 171545608 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 9: 81752 bytes, 171627360 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 10: 153928 bytes, 171781288 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 11: 48 bytes, 171781336 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 12: 136 bytes, 171781472 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 13: 10104 bytes, 171791576 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 14: 41344 bytes, 171832920 total
INFO | jvm 1 | 2010/11/19 08:35:30 | - age 15: 96 bytes, 171833016 total
INFO | jvm 1 | 2010/11/19 08:35:30 | : 1410396K->197458K(2621440K), 40.7184593 secs] 4926671K->3713735K(6553600K), 40.7185248 secs]CMSCollector shouldConcurrentCollect: 143847.854
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | [Times: user=1.25 sys=0.36, real=40.71 secs]
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.854
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | WrapperManager: The timer fell behind the system clock by 40700ms.
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.865
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.865
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.866
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620
INFO | jvm 1 | 2010/11/19 08:35:30 | free=425864592
INFO | jvm 1 | 2010/11/19 08:35:30 | contiguous_available=2684354560
INFO | jvm 1 | 2010/11/19 08:35:30 | promotion_rate=195.733
INFO | jvm 1 | 2010/11/19 08:35:30 | cms_allocation_rate=0
INFO | jvm 1 | 2010/11/19 08:35:30 | occupancy=0.8942354
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | initiatingPermOccupancy=0.9200000
INFO | jvm 1 | 2010/11/19 08:35:30 | CMSCollector shouldConcurrentCollect: 143847.871
INFO | jvm 1 | 2010/11/19 08:35:30 | time_until_cms_gen_full 1939631.4029620 -
Hi,
I'm trying to tune the JVM options for a JAVA app that requires low pauses. After playing around with several diferent combinations I still can not get a setup that will not give an occasional very long pause.
Here's an example of a long pause from the stdout log:
2007-11-24 07:44:29| occupancy=0.6494621
2007-11-24 07:44:29| initiatingOccupancy=0.6800000
2007-11-24 07:44:32| CMSCollector shouldConcurrentCollect: 64459.941
2007-11-24 07:44:32| time_until_cms_gen_full 1986.2853957
2007-11-24 07:44:32| free=4410787008
2007-11-24 07:44:32| contiguous_available=1888036816
2007-11-24 07:44:32| promotion_rate=1.98556e+06
2007-11-24 07:44:32| cms_allocation_rate=0
2007-11-24 07:44:32| occupancy=0.6494621
2007-11-24 07:44:32| initiatingOccupancy=0.6800000
2007-11-24 07:44:32| Total time for which application threads were stopped: *46.1601130 seconds*
2007-11-24 07:44:32| CMSCollector shouldConcurrentCollect: 64461.180
2007-11-24 07:44:32| time_until_cms_gen_full 1986.2853957
2007-11-24 07:44:32| free=4410787008
2007-11-24 07:44:32| contiguous_available=1888036816
2007-11-24 07:44:32| promotion_rate=1.98556e+06
2007-11-24 07:44:32| cms_allocation_rate=0
2007-11-24 07:44:32| occupancy=0.6494621
2007-11-24 07:44:32| initiatingOccupancy=0.6800000
2007-11-24 07:44:32| CMSCollector shouldConcurrentCollect: Total time for which application threads were stopped: 0.0048
4
The pause of 46 seconds is not good for the application. It would be OK to have an occasional 5 sec pause but not 46 seconds.
I've been through likely all JVM tunning papers, blogs and previus postings to this forum.
BTW I know that some of the JVM options I'm setting are already the default but it helps me keep track of which options I tried.
Can anyone help?
Here are some details:
Platform: Linux 2.6.9-42.0.10.ELsmp #1 SMP Tue Feb 27 09:40:21 EST 2007 x86_64 x86_64 x86_64 GNU/Linux
16GB of physical memory, 2 x dual core CPUs AMD 1.86GHz
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
JVM options:
-Xbatch -XX:-DontCompileHugeMethods -XX:CMSMaxAbortablePrecleanTime=10000
-XX:+UseConcMarkSweepGC
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=68
-XX:+UseParNewGC -XX:SurvivorRatio=15
-XX:+CMSConcurrentMTEnabled -XX:ParallelGCThreads=4 -XX:+CMSParallelSurvivorRemarkEnabled -XX:InitialTenuringThreshold=1 -XX:MaxTenuringThreshold=8 -XX:TargetSurvivorRatio=90 -XX:ThreadPriorityPolicy=0 -XX:+AggressiveOpts -XX:CMSTriggerRatio=65 -XX:+AlwaysActAsServerClassMachine -XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:PrintCMSStatistics=1 -XX:+PrintCMSInitiationStatistics
-XX:+PrintGCApplicationStoppedTime -verbose:gc -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+UseTLAB
-Xms13000m -Xmx13000m -XX:MaxNewSize=1000m -XX:NewSize=1000m -XX:ThreadStackSize=128I encounter same problem, no "promotion failed" tip but ParNew cost more than 20s, the following is the gc log's snapshot:
my jdk version is the latest JDK6u21 on RHEL5.1/IBM x3850 with 16core.
I try to use -X::+UseMemBar, but no effect.
1913.579: [GC 1913.579: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15459032 bytes, 15459032 total
- age 2: 10545904 bytes, 26004936 total
: 433430K->35393K(458752K), 0.1056850 secs] 6509320K->6122752K(10420224K), 0.1061990 secs] [Times: user=0.83 sys=0.01, real=0.11 secs]
Heap after GC invocations=3850 (full 10):
par new generation total 458752K, used 35393K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.1082700 seconds
CMSCollector shouldConcurrentCollect: 1913.685
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.3237750 seconds
CMSCollector shouldConcurrentCollect: 1935.690
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
CMSCollector shouldConcurrentCollect: 1937.692
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Total time for which application threads were stopped: 23.8250250 seconds
CMSCollector shouldConcurrentCollect: 1937.834
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000880 seconds
Total time for which application threads were stopped: 0.0016130 seconds
CMSCollector shouldConcurrentCollect: 1937.836
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0000580 seconds
Total time for which application threads were stopped: 0.0004200 seconds
CMSCollector shouldConcurrentCollect: 1937.837
time_until_cms_gen_full 144.6930408
free=3967091720
contiguous_available=2076692424
promotion_rate=2.45972e+07
cms_allocation_rate=0
occupancy=0.6110903
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0019560 seconds
{Heap before GC invocations=3850 (full 10):
par new generation total 458752K, used 428609K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 54% used [0x0000000596000000, 0x00000005982905c8, 0x000000059a000000)
to space 65536K, 0% used [0x0000000592000000, 0x0000000592000000, 0x0000000596000000)
concurrent mark-sweep generation total 9961472K, used 6087358K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
1937.839: [GC 1937.839: [ParNew
Desired survivor size 33554432 bytes, new threshold 2 (max 2)
- age 1: 15389248 bytes, 15389248 total
- age 2: 10280752 bytes, 25670000 total
: 428609K->33832K(458752K), 0.0985530 secs] 6515968K->6131457K(10420224K), 0.0990430 secs] [Times: user=0.77 sys=0.01, real=0.10 secs]
Heap after GC invocations=3851 (full 10):
par new generation total 458752K, used 33832K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 0% used [0x000000057a000000, 0x000000057a000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37802K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000)
Total time for which application threads were stopped: 0.0999980 seconds
CMSCollector shouldConcurrentCollect: 1937.939
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.0446810 seconds
Total time for which application threads were stopped: 0.0048440 seconds
CMSCollector shouldConcurrentCollect: 1937.988
time_until_cms_gen_full 4007.5147760
free=3956579920
contiguous_available=2070817568
promotion_rate=885817
cms_allocation_rate=0
occupancy=0.6121208
initiatingOccupancy=0.6500000
initiatingPermOccupancy=0.9200000
Application time: 0.1273720 seconds
{Heap before GC invocations=3851 (full 10):
{Heap before GC invocations=3851 (full 10):
par new generation total 458752K, used 427048K [0x000000057a000000, 0x000000059a000000, 0x000000059a000000)
eden space 393216K, 100% used [0x000000057a000000, 0x0000000592000000, 0x0000000592000000)
from space 65536K, 51% used [0x0000000592000000, 0x000000059410a2c8, 0x0000000596000000)
to space 65536K, 0% used [0x0000000596000000, 0x0000000596000000, 0x000000059a000000)
concurrent mark-sweep generation total 9961472K, used 6097624K [0x000000059a000000, 0x00000007fa000000, 0x00000007fa000000)
concurrent-mark-sweep perm gen total 63128K, used 37808K [0x00000007fa000000, 0x00000007fdda6000, 0x0000000800000000) -
my audio is being funny on my mac. It is brand new, and hasn't had trouble since this update. It gets stuck at full blast with a / though it. then no audio comes out. Once I turn it off it fixes it self thouhgh, or if I wait a while it finally fixes it's self. If this happening to anyone else?
I don't think that is normal behavior. The long press is possibly causing the system to force power down, which would then allow it to power back up.
Just to make sure, all drivers are up-to-date? Are there pending BIOS updates? If so, you'll want to make sure it can restart properly before applying the BIOS update
Try an Ubuntu LiveUSB or LiveCD to rule out software issues.
If that doesn't work, try taking out the battery and AC, pressing the power button ten times, then holding it for ten. This is usually the remedy for another issue, but try it anyway.
If that doesn't work, then try resetting the BIOS settings. Then try unplugging the CMOS battery overnight. Otherwise, might have to call for warranty.
W520: i7-2720QM, Q2000M at 1080/688/1376, 21GB RAM, 500GB + 750GB HDD, FHD screen
X61T: L7500, 3GB RAM, 500GB HDD, XGA screen, Ultrabase
Y3P: 5Y70, 8GB RAM, 256GB SSD, QHD+ screen -
ParNew long pauses in Tomcat 6 on T5120.
I am running some simmulated user load for 50 concurent users on a Java application running inside Tomcat 6 on Sun's T5120 (8 processor 8G RAM) on Solaris 10. The utilization of the application seens to be ok (90-95%), but sometimes JVM goes into long GC pauses that are not acceptable for in my case. The longest pauses I can afford is about 10 seconds. The only other thing that's running on the machine is apache (mod_jk) that forwards the requests to the Tomcat process.
My GC Settings:
GC_VERBOSE_OPTS="-XX:+PrintTenuringDistribution -Xloggc:${CATALINA_HOME}/logs/gc_${TSTAMP}.log"
GC_VERBOSE="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime $GC_VERBOSE_OPTS"
MEMORY_SETTINGS="-Xms6912m -Xmx6912m -XX:NewSize=1728m -XX:MaxNewSize=1728m -XX:SurvivorRatio=8 -Xss256k -d64 -XX:PermSize=192M -XX:MaxPermSize=192M"
GC_SETTINGS=" -XX:+DisableExplicitGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -X:ParallelGCThreads=6
The output of verbose GC just before it goes into very long 55 second pause:
76824.094: [GC 76824.094: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age 1: 28658536 bytes, 28658536 total
- age 2: 78258856 bytes, 106917392 total
- age 3: 47935016 bytes, 154852408 total
: 1592576K->152602K(1592576K), 0.7859117 secs] 4605449K->3213877K(6900992K), 0.7864152 secs]
Total time for which application threads were stopped: 0.7876342 seconds
Application time: 17.9823540 seconds
Total time for which application threads were stopped: 0.0037981 seconds
Application time: 0.0002300 seconds
76842.868: [Full GC 76842.868: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age 1: 16586944 bytes, 16586944 total
- age 2: 24580024 bytes, 41166968 total
- age 3: 77453264 bytes, 118620232 total
- age 4: 41331496 bytes, 159951728 total
: 1568283K->157516K(1592576K), 0.5095165 secs] 4629559K->3218791K(6900992K), 0.5099791 secs]
Total time for which application threads were stopped: 0.5109060 seconds
Application time: 21.7442444 seconds
76865.132: [GC 76865.132: [ParNew
Desired survivor size 163027352 bytes, new threshold 5 (max 31)
- age 1: 70949544 bytes, 70949544 total
- age 2: 325496 bytes, 71275040 total
- age 3: 16995328 bytes, 88270368 total
- age 4: 51961024 bytes, 140231392 total
- age 5: 39539328 bytes, 179770720 total
: 1573196K->176896K(1592576K), 4.8568739 secs] 4634471K->3402338K(6900992K), 4.8573679 secs]
Total time for which application threads were stopped: 4.8667158 seconds
Application time: 0.0000997 seconds
76870.002: [GC [1 CMS-initial-mark: 3225442K(5308416K)] 3425594K(6900992K), 1.2304280 secs]
Total time for which application threads were stopped: 1.2421993 seconds
76871.233: [CMS-concurrent-mark-start]
Application time: 3.9732836 seconds
76875.212: [GC 76875.212: [ParNew
Desired survivor size 163027352 bytes, new threshold 5 (max 31)
- age 1: 89811184 bytes, 89811184 total
- age 2: 47431856 bytes, 137243040 total
- age 3: 229704 bytes, 137472744 total
- age 4: 10895976 bytes, 148368720 total
- age 5: 31371840 bytes, 179740560 total
: 1592576K->176896K(1592576K), 19.5701529 secs] 4818018K->3616356K(6900992K), 19.6394327 secs]
*Total time for which application threads were stopped: 19.6456021 seconds*
Application time: 5.3914509 seconds
76900.622: [GC 76900.622: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age 1: 73583152 bytes, 73583152 total
- age 2: 73057080 bytes, 146640232 total
- age 3: 32981616 bytes, 179621848 total
- age 4: 98336 bytes, 179720184 total
- age 5: 12880 bytes, 179733064 total
: 1592576K->176896K(1592576K), 54.7521974 secs] 5032036K->3921814K(6900992K), 54.7527833 secs]
*Total time for which application threads were stopped: 55.1387374 seconds*
Application time: 5.3171414 seconds
76960.700: [GC 76960.700: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age 1: 66359848 bytes, 66359848 total
- age 2: 62758584 bytes, 129118432 total
- age 3: 50477568 bytes, 179596000 total
: 1592576K->176896K(1592576K), 7.1058428 secs] 5337494K->4095887K(6900992K), 7.1063802 secs]
*Total time for which application threads were stopped: 7.1077796 seconds*
Application time: 12.7908379 seconds
Total time for which application threads were stopped: 0.0042312 seconds
Application time: 0.3358069 seconds
76980.938: [GC 76980.939: [ParNew
Desired survivor size 163027352 bytes, new threshold 3 (max 31)
- age 1: 69012896 bytes, 69012896 total
- age 2: 56309432 bytes, 125322328 total
- age 3: 54304696 bytes, 179627024 total
: 1592576K->176896K(1592576K), 2.5164799 secs] 5512702K->4233158K(6900992K), 2.5169843 secs]
Total time for which application threads were stopped: 2.5178621 seconds
Application time: 14.8490191 seconds
Total time for which application threads were stopped: 0.0070005 seconds
Application time: 0.0003056 seconds
76998.312: [Full GC 76998.313: [ParNew
Desired survivor size 163027352 bytes, new threshold 16 (max 31)
- age 1: 38875304 bytes, 38875304 total
- age 2: 63941104 bytes, 102816408 total
- age 3: 56090408 bytes, 158906816 total
: 1592576K->156275K(1592576K), 1.1885482 secs] 5648848K->4265573K(6900992K), 1.1891583 secs]
Total time for which application threads were stopped: 1.1899499 seconds
Application time: 8.8337996 seconds
I would appreciate some help tunning GC patameters to eliminate the long pauses.
Edited by: MikhailPDX on May 6, 2008 11:53 AMI saw this posting before about pauses and I don't think I am having the same issue since I am using Java(TM) 2 Runtime Environment, Standard Edition 1.5.0_15-b04 Java HotSpot(TM) 64-Bit Server VM) where the mentioned pauses bug was supposedly fixed.
Edited by: MikhailPDX on May 7, 2008 10:30 AM -
HOW to view SWF full screen with controls
I have a SWF file. Is there a player that can allow me to
play it full screen with basic controls at the bottom? Controls
that I can choose play, stop, pause, fast fwd, rewind and a
progression bar. Thanks!
Glennon(release) {
fscommand("fullscreen", true);
glennchung wrote:
> I have a SWF file. Is there a player that can allow me
to play it full screen
> with basic controls at the bottom? Controls that I can
choose play, stop,
> pause, fast fwd, rewind and a progression bar. Thanks!
>
> Glenn
> -
Long pauses in podcasts feature
I have itunes 11.1.4. I download a lot of podcasts. Now when I try to go to podcasts, and refresh, or click on any podcast or its info, and sync to my ipod or iphone, there is a long pause for each of these steps, about 1-2 minutes each (spinning pinwheel). I had no problem for years. What's different?
Some options you might find useful are
-Xloggc:gc.log -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime
This will show you when the application actually stops the world. Note: there are other reasons an application can stop and it shows you them all not just GCs. -
Toggle Full Screen with lingo?
Hi there,
I have just got a Mac mini 10.6.2 and I'm testing exporting mac projectors from Dir11.5 on XP.
What I'd like to achieve is to swap between two states whilst playing the projector.
State 1: Small draggable Windowed state with title bar and Mac Dock / desktop visible.
State 2: Pure 800x600 Full Screen with No Dock or title bar visible.
I am using BuddyAPI successfully to switch monitor screen modes to 800x600 and back again to native res.
My problem is, if I want to hide the dock and title bar, I seem to have to publish with the "Full Screen" checkbox ticked in the publish settings. This means when I switch back to native res windowed mode there is the solid movie background colour hiding the mac desktop and dock (as you'd expect).
Is there a way to turn full screen (black out the desktop and dock) on and off during runtime with lingo?
I know its a long shot, but though I'd check with the pros before assuming it's not possible.
Cheers
Richard SmithMy understanding is that Flash will not allow you to do that.
Full screen mode is invoked only if explicitly requested from
within the flash on mouse events. -
Problems after doing full restore with Time Machine
If I reinstall Leopard and and do a complete restore with time Machine, I have to erase my Time Machine backup drive and do a complete TM backup from scratch because TM will not pick up where it left off doing incremental backups. It wants to start over like it's never done any backups at all. It's a pain because it take me 4 or 5 hours to back up 270 GB. I've had to do this twice so far because once I had a problem with my startup drive and just today I made an external hard drive with leopard and restored all my files via TM.
Is there anyway to get out of starting over from scratch when I do a full restore with TM ?The following procedure works for me if your restore happens to be from the last backup that Time Machine completed. (I'm not sure if it would work just as well if you restore from an older point in the Time Machine history -- but it certainly should!):
After you complete the full restore (which will force a Restart when it completes), log in and then immediately go to System Preferences / Time Machine and Turn Off Time Machine to prevent any premature additional backups.
Now go to System Preferences / Spotlight / Privacy and drag the icons for all your hard drives (including your main hard drive and the Time Machine hard drive) from the Desktop into the Privacy list. This will stop Spotlight indexing of all these drives to speed things up for what follows. You will do the indexing later (see below).
If you have any sort of automatic virus protection active, disable it at this point to speed up what follows.
Then Restart again (to get things into a fresh state), and then immediately do a Repair Permissions for your main hard drive (using Applications / Utilities / Disk Utility). Be patient. This will take 30 minutes or more and the progress bar may not advance until the very end. Do not be alarmed when several hundred notifications come up, as most of them are minor tidying up items, but some are significant. For example, you will see the ownership get adjusted for every help file in every language for every Lexar printer the system knows about (minor). You will also see the permissions adjusted for the root directory of your main hard drive (significant).
And all of these permission and ownership repairs will happen EVEN THOUGH the files you backed up into your Time Machine may have had no such problems. They are, apparently, a result of the method that Time Machine uses to rebuild your file system during the restore.
When the permission repair eventually completes, Quit Disk Utility and Shut Down the computer.
Now reset the Parameter memory (PRAM). PRAM holds copies of certain system settings for rapid access. To do this, hold down the 4 keys Apple-Option-P-R continuously and press and release the power button. When you hear the SECOND startup chime, release those 4 keys. The system will continue to boot up normally. This makes sure the system's Parameter memory is in sync with the System Preferences resulting from the restore you just completed. It probably would have been anyway, but this makes sure. Among other things, this makes sure the system takes proper note of your "computer name" (System Preferences / Sharing) which is crucial to Time Machine's ability to recognize and use your previous backup database on the Time Machine hard drive.
Now log in and fire up Mail to let it automatically finish the restore of its mailboxes by importing the necessary mail lists. Quit Mail when it finishes.
If you have any other, application specific tasks to perform to complete the restore for any other applications, now is the time to do them.
Finally, go back into System Preferences / Spotlight / Privacy, select the line showing your main hard drive in the list, and click the "-" on the bottom to remove it from the list. Repeat this for every other hard drive EXCEPT for your Time Machine hard drive. Exit System Preferences. Spotlight will now begin to re-index those hard drives from scratch. Watch this by clicking on the Spotlight icon in the menu bar. Wait for indexing to finish.
Your restore is now at the point where you can let Time Machine do a new backup.
I suggest you Restart again to get things into a fresh state (not truly necessary, but it is what I do). Then go into System Preferences / Time Machine and, at long last, Turn On Time Machine again. Then do a Back Up Now (right click on the Time Machine icon in the dock and select Back Up Now from the pop up menu).
Because of the restore, Time Machine will now do a Deep Traversal of your entire file system looking for EVERYTHING that has changed compared to the last backup on its hard drive (rather than depending on the file system transaction logs as it normally does to make incremental backups happen much faster). The "Preparing" stage for this will take a long time -- about as long as a Repair Permissions pass in Disk Utility. Eventually Time Machine will start transferring files. This will be a backup of significant size because all the permissions repairs you did above, etc., count as changes as far as Time Machine is concerned, not to mention that certain portions of the file system are rebuilt during the restore. But it should be WELL SHORT of actually doing a complete backup of everything on your system. I.e., it is just a particularly large, but nevertheless incremental, backup added on to the previous stuff on your Time Machine disk.
Crucial to this is that Time Machine recognizes the prior database on its hard drive as applying to your computer. Thus the permissions repair and PRAM resetting steps above.
When that backup eventually completes, go into System Preferences / Spotlight and remove your Time Machine drive from the Privacy list. Exit System Preferences and wait for Spotlight to finish re-indexing your Time Machine drive.
Restart once again, just to get things into a fresh state, and then re-enable any antivirus "live protection" stuff you disabled above.
You are done.
From this point on, Time Machine should do "normal" incremental backups, and the previous history of Time Machine backups should be accessible and used by Time Machine just as before.
--Bob
Message was edited by: BobP1776
Maybe you are looking for
-
I recently purchased an ipod nano 3rd generation. I installed itunes7.6 with quicktime the installation went fine. When I try to open itunes I get a blue screen error saying my computer needs to restart. I've uninstalled and reinstalled several times
-
To run and work on AIX server installed ODI
Hi all, As I was working on ODI till the date in window o/s of my desktop. But now I have to work on AIX server I need to know * Is there any difference to work on ODI while I am using AIX plz provide me necessary information.. * Will working methodo
-
Is mountain Lion compatible with Windows Small Business Server 2003
Hi, Can anyone tell me if Mac OS 10.8 is compatible with Microsoft Small Business Server 2003. We connected Mac Os 10.7 machines to our MS server but we have no results with Mac Os 10.8. Our Windows server is an HP proliant with SBS 2003 SP2. Thx P
-
Motion 2.1 & Genarts Sapphire Plug ins
These beautiful plug ins now seem to be incompatible with Motion 2.1 (UB). After emailing Genarts they say its how Apple has re written Motion and that it will no longer support AE plug ins! Has Apple gone mad!? Does anyone else have any information
-
LR4 to CS5 edited images saved to psd not visible in LR4.
Upgraded to LR4 and using CS5. If I edit an image in Photoshop then save it as a psd file to the origional location that the image was imported into LR4 the psd image-file does not show up in LR4. If I search for the file using finder (mac) I find t