Interpreting Concurrent Mark Sweep GC-Log

Hi.
I am trying to extract some metrics from the garbage collector log of the concurrent mark sweep garbage collector. I have found a nice walkthrough a this site:
[http://www.sun.com/bigadmin/content/submitted/cms_gc_logs.html |http://www.sun.com/bigadmin/content/submitted/cms_gc_logs.html ]
I'm using CMS in incremental mode and that's where I am having interpretation problems. The site above states that the incremental mode takes over the job of the "concurrent mark"-phase. If this is correct (is it?) that means that I have two different statistics that state how long the "concurrent mark"-phase took. First the incremental CMS statistics and second the ordinary CMS statistics.
The first incremental stats look like this:
2803.125: [GC 2803.125: [ParNew: 408832K->0K(409216K), 0.5371950 secs]
           611130K->206985K(1048192K) icms_dc=4 , 0.5373720 secs]
2824.209: [GC 2824.209: [ParNew: 408832K->0K(409216K), 0.6755540 secs]
           615806K->211897K(1048192K) icms_dc=4 , 0.6757740 secs] The second ordinary statistics look like this:
40.683: [CMS-concurrent-mark: 0.521/0.529 secs]I'm wondering wether the second (ordinary) statistics are still valid when using incremental mode.
Kind regards
Frank

40.683: [CMS-concurrent-mark: 0.521/0.529 secs]
Yes, these logs are still valid. icms_dc indicates the time in percentage that the concurrent work took between two young GCs.
e.g. for the following logs:
-->1.289: [GC 1.289: [DefNew: 1984K->191K(1984K), 0.1952985 secs] 5695K->5714K(7648K) icms_dc=55 , 0.1956749 secs] [Times: user=0.19 sys=0.00, real=0.20 secs]
1.488: [CMS-concurrent-reset-start]
1.494: [CMS-concurrent-reset: 0.006/0.006 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
1.503: [GC [1 CMS-initial-mark: 5522K(9208K)] 6715K(11192K), 0.0191550 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
1.522: [CMS-concurrent-mark-start]
-->1.531: [GC 1.531: [DefNew: 1710K->192K(1984K), 0.1260654 secs] 7232K->7220K(11192K) icms_dc=70 , 0.1265609 secs] [Times: user=0.10 sys=0.02, real=0.13 secs]
2.051: [CMS-concurrent-mark: 0.401/0.528 secs] [Times: user=0.50 sys=0.03, real=0.53 secs]
Concurrent work took around 70% of the time between the two young GCs at 1.531 and 1.289. Here, some part of CMS-concurrent-mark work was done before the second young GC and rest after that. And overall CMS-concurrent-mark phase ran for 0.401s out of total (2.051-1.522=) 0.529s.

Similar Messages

  • GC taking long time when using Concurrent Mark Sweep GC with Sun JDK 150_12

    We are having problem of Garbage collection taking too long. We are using Weblogic 9.2 and Sun JDK 150_12
    Below are the memory arguments we are using. We are using Concurrent Mark Sweep GC. What we are observing is Young Generation is getting filled up and consequent tenured generation also hangs with long pauses.
    Below are the JVM arguments we are using
    -Xms2560M -Xmx2560M -Xloggc:${LOGDIR}/gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MaxPermSize=256m
    I have seen many forums where there are many reported issues with Concurrent Mark Sweep garbage collection with Sun JDK, but with different recommendations. But did not find any defnite recommendation. Please advice.
    - - Tarun

    We are having problem of Garbage collection taking too long. We are using Weblogic 9.2 and Sun JDK 150_12
    Below are the memory arguments we are using. We are using Concurrent Mark Sweep GC. What we are observing is Young Generation is getting filled up and consequent tenured generation also hangs with long pauses.
    Below are the JVM arguments we are using
    -Xms2560M -Xmx2560M -Xloggc:${LOGDIR}/gc.log -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=128 -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MaxPermSize=256m
    I have seen many forums where there are many reported issues with Concurrent Mark Sweep garbage collection with Sun JDK, but with different recommendations. But did not find any defnite recommendation. Please advice.
    - - Tarun

  • Concurrent Mark-Sweep Problems in 1.4.2 VM?

    Can someone please tell me why the 1.4.2 JVM seems to ignore VM tuning flags as soon as the -XX:+UseConcMarkSweepGC flag is used?
    For example, please run this program (It is intentionally written to run the VM out of memory):
    ================START================================
    import java.util.*;
    public class UseMemory {
    private ArrayList m_ref = new ArrayList();
    public static void main(String args[]) {
    UseMemory um = new UseMemory();
    int times = Integer.parseInt(args[0]);
    int delay = Integer.parseInt(args[1]);
    int size = Integer.parseInt(args[2]);
    for ( int i = 0; i < times; i++ ) {
    byte[] memory = new byte[size];
    um.m_ref.add(memory);
    try {
    Thread.currentThread().sleep(delay);
    System.out.println(Runtime.getRuntime().freeMemory());
    } catch ( InterruptedException ie ) {
    ======================END============================
    First, run it under the 1.4.1 VM like this:
    /jdk1.4.1_05/bin/java -Xmx128m -Xms128m -Xnoclassgc -verbose:gc -XX:NewSize=32M -XX:MaxNewSize=32M -XX:PermSize=32M -XX:MaxPermSize=32M -XX:SurvivorRatio=4 -XX:+DisableExplicitGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=90 -classpath "." UseMemory 1200 100 1024000
    The output from the heap printing follows:
    =============BEGIN HEAP PRINT====================
    Heap
    par new generation total 27328K, used 20253K [0x02a60000, 0x04a60000, 0x04a60000)
    eden space 21888K, 92% used [0x02a60000, 0x03e27408, 0x03fc0000)
    from space 5440K, 0% used [0x03fc0000, 0x03fc0000, 0x04510000)
    to space 5440K, 0% used [0x04510000, 0x04510000, 0x04a60000)
    concurrent mark-sweep generation total 98304K, used 90000K [0x04a60000, 0x0aa60000, 0x0aa60000)
    CompactibleFreeListSpace space 98304K, 91% used [0x04a60000, 0x0aa60000)
    concurrent-mark-sweep perm gen total 32768K, used 1129K [0x0aa60000, 0x0ca60000, 0x0ca60000)
    CompactibleFreeListSpace space 32768K, 3% used [0x0aa60000, 0x0ca60000)
    0.001: [ParNew: 20253K->20253K(27328K), 0.0000643 secs]0.002: [CMS0.010: [dirty card accumulation, 0.0003718 secs]0.011: [dirty card rescan, 0.0026278 secs]0.013: [remark from roots, 0.0003656 secs]0.014: [weak refs processing, 0.0000028 secs]: 90000K->90000K(98304K), 0.0153770 secs] 110253K->110253K(125632K) Heap after GC invocations=1:
    Heap
    par new generation total 27328K, used 20253K [0x02a60000, 0x04a60000, 0x04a60000)
    eden space 21888K, 92% used [0x02a60000, 0x03e27408, 0x03fc0000)
    from space 5440K, 0% used [0x03fc0000, 0x03fc0000, 0x04510000)
    to space 5440K, 0% used [0x04510000, 0x04510000, 0x04a60000)
    concurrent mark-sweep generation total 98304K, used 90000K [0x04a60000, 0x0aa60000, 0x0aa60000)
    CompactibleFreeListSpace space 98304K, 91% used [0x04a60000, 0x0aa60000)
    concurrent-mark-sweep perm gen total 32768K, used 1129K [0x0aa60000, 0x0ca60000, 0x0ca60000)
    CompactibleFreeListSpace space 32768K, 3% used [0x0aa60000, 0x0ca60000)
    } , 0.0182148 secs]
    =============END HEAP PRINT====================
    Notice that the survivor ratio (4) is being used correctly, and that the default max tenuring level (31) is set correctly.
    Now let's run this with a 1.4.2_02 VM:
    /jdk1.4.2_04/bin/java -Xmx128m -Xms128m -Xnoclassgc -verbose:gc -XX:NewSize=32M -XX:MaxNewSize=32M -XX:PermSize=32M -XX:MaxPermSize=32M -XX:SurvivorRatio=4 -XX:+DisableExplicitGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=90 -classpath "." UseMemory 1200 100 1024000
    Here is the output from a verbose GC:
    =============BEGIN HEAP PRINT====================
    0.766: [GC  {Heap before GC invocations=2:
    Heap
    par new generation   total 32704K, used 30000K [0x10010000, 0x12010000, 0x12010000)
      eden space 32640K,  91% used [0x10010000, 0x11d5c1f8, 0x11ff0000)
      from space 64K,   0% used [0x11ff0000, 0x11ff0000, 0x12000000)
      to   space 64K,   0% used [0x12000000, 0x12000000, 0x12010000)
    concurrent mark-sweep generation total 98304K, used 60089K [0x12010000, 0x18010000, 0x18010000)
    concurrent-mark-sweep perm gen total 32768K, used 1008K [0x18010000, 0x1a010000, 0x1a010000)
    0.766: [ParNew
    Desired survivor size 32768 bytes, new threshold 0 (max 0)
    : 30000K->0K(32704K), 0.0701463 secs] 90090K->90089K(131008K) Heap after GC invocations=3:
    Heap
    par new generation total 32704K, used 0K [0x10010000, 0x12010000, 0x12010000)
    eden space 32640K, 0% used [0x10010000, 0x10010000, 0x11ff0000)
    from space 64K, 0% used [0x12000000, 0x12000000, 0x12010000)
    to space 64K, 0% used [0x11ff0000, 0x11ff0000, 0x12000000)
    concurrent mark-sweep generation total 98304K, used 90089K [0x12010000, 0x18010000, 0x18010000)
    concurrent-mark-sweep perm gen total 32768K, used 1008K [0x18010000, 0x1a010000, 0x1a010000)
    } , 0.0719439 secs]
    =============END HEAP PRINT====================
    So what in Sam Hain is going on here? Why has my survivor ration been set to be 511? Why has the max tenuring threshold been set to be 0? Who made the decision that I want to run with the 'promote all' JVM GC settings for me? I don't want this. It is killing me.
    Oh, one more thing, if I take out the -XX:+UseConcMarkSweepGC flag, then the GC output looks normal with the 1.4.2_04 VM.
    Is this a known bug, am I missing something obvious here?
    Please help.
    Thanks
    Andy

    We are using
    java -version -serverjava version "1.4.2_05"
    Java(TM) 2 Runtime Environment, Standard Edition
    (build 1.4.2_05-b04)
    Java HotSpot(TM) Server VM (build 1.4.2_05-b04, mixed
    mode)
    >
    and the issue is still there
    Bug 4874782 says that it was fixed for "tiger".
    Do we have any options except upgrading to 1.5?
    Any plans to fix it for 1.4.2 ?
    As I stated earlier, we believe this is fixed in 1.4.2_05.
    [For reasons that I do not understand and which are out-of-scope
    for me,  the bug-paraed database just shows that this is
    fixed in 1.5"tiger". However, I assure you that it's been
    fixed in 1.4.2_05.)
    i also just verified that it's fixed in 1.4.2_05 using a variety of tests
    that vary the heap settings via the command-line options.
    If you believe a bug still exists in 1.4.2_<n>, n >= 5,
    or in JDK 5.0, please file a bug against HotSpot (hotspot/garbage_collector)
    via appropriate channels, including specific details (command-lines
    used, behaviour seen) for the problem you are seeing.
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       

  • Concurrent Mark Sweep stops working

    Machine: Intel(R) Pentium(R) 4 CPU 2.80GHz, 512Mb
    OS: RedHat 9
    Java Version: 1.5.0_06
    Hi all,
    I have tuned GC for my java application and used the following options:
    -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=50 -Xms384m -Xmx384m -XX:MaxNewSize=64m -XX:MaxPermSize=128m
    -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps EVerything runs fine for the fist few days. When tenured generation reaches the 50% (approximately 184Mb) the CMS triggers and after a while it frees memory from the Tenured.
    This is the gc log when things run as expected:
    524578.375: [GC 524578.376: [DefNew: 24448K->0K(24512K), 0.0903230 secs] 208489K->184127K(393152K), 0.0909130 secs]
    524668.158: [GC 524668.158: [DefNew: 24448K->0K(24512K), 0.0752630 secs] 208575K->184207K(393152K), 0.0758560 secs]
    524768.230: [GC 524768.230: [DefNew: 24448K->0K(24512K), 0.0755960 secs] 208655K->184332K(393152K), 0.0762210 secs]
    524768.319: [GC [1 CMS-initial-mark: 184332K(368640K)] 184671K(393152K), 0.1741830 secs]
    524768.493: [CMS-concurrent-mark-start]
    524803.272: [CMS-concurrent-mark: 34.778/34.778 secs]
    524804.417: [CMS-concurrent-preclean-start]
    524804.417: [CMS-concurrent-preclean: 0.000/0.000 secs]
    524804.633: [CMS-concurrent-abortable-preclean-start]
    524804.633: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs]
    524858.404: [GC 524858.405: [DefNew: 24448K->0K(24512K), 0.1104150 secs] 208780K->184425K(393152K), 0.1110120 secs]
    524898.444: [GC[YG occupancy: 12329 K (24512 K)]524898.444: [Rescan (non-parallel) 524898.444: [grey object rescan, 0.0111740 secs]524898.455: [root rescan, 0.2529400 secs], 0.26
    43540 secs]524898.708: [weak refs processing, 0.4018890 secs] [1 CMS-remark: 184425K(368640K)] 196754K(393152K), 0.6813530 secs]
    524899.192: [CMS-concurrent-sweep-start]
    524902.081: [CMS-concurrent-sweep: 2.806/2.888 secs]
    524902.081: [CMS-concurrent-reset-start]
    524902.158: [CMS-concurrent-reset: 0.077/0.077 secs]
    524938.155: [GC 524938.156: [DefNew: 24448K->0K(24512K), 0.0961550 secs] 165882K->141600K(393152K), 0.0967840 secs]
    525034.493: [GC 525034.494: [DefNew: 24448K->0K(24512K), 0.0176080 secs] 166048K->141675K(393152K), 0.0181800 secs]Memory went from 184Mb to 141Mb used.
    Few days later, this happens:
    568948.375: [GC 568948.375: [DefNew: 24448K->0K(24512K), 0.1304260 secs] 208432K->184077K(393152K), 0.1310100 secs]
    569040.677: [GC 569040.677: [DefNew: 24448K->0K(24512K), 0.1944620 secs] 208525K->184146K(393152K), 0.1950090 secs]
    569063.364: [GC 569063.364: [DefNew: 24448K->0K(24512K), 0.1599060 secs] 208594K->184203K(393152K), 0.1604460 secs]
    569086.304: [GC 569086.304: [DefNew: 24448K->0K(24512K), 0.0976120 secs] 208651K->184253K(393152K), 0.0981370 secs]
    569178.155: [GC 569178.155: [DefNew: 24448K->0K(24512K), 0.4068420 secs] 208701K->184343K(393152K), 0.4074150 secs]
    569178.610: [GC [1 CMS-initial-mark: 184343K(368640K)] 185071K(393152K), 3.0703340 secs]
    569181.688: [CMS-concurrent-mark-start]
    569236.219: [CMS-concurrent-mark: 54.530/54.530 secs]
    569236.219: [CMS-concurrent-preclean-start]
    569236.219: [CMS-concurrent-preclean: 0.000/0.000 secs]
    569236.794: [CMS-concurrent-abortable-preclean-start]
    569236.794: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs]
    569258.316: [GC 569258.317: [DefNew: 24448K->0K(24512K), 0.1555370 secs] 208791K->184468K(393152K), 0.1561600 secs]
    569358.165: [GC 569358.166: [DefNew: 24446K->0K(24512K), 0.1217950 secs] 208914K->184547K(393152K), 0.1223810 secs]
    569468.210: [GC 569468.210: [DefNew: 24448K->0K(24512K), 0.1310600 secs] 208995K->184673K(393152K), 0.1316840 secs]
    569558.340: [GC 569558.340: [DefNew: 24448K->0K(24512K), 0.0756040 secs] 209121K->184766K(393152K), 0.0762220 secs]
    569674.515: [GC 569674.515: [DefNew: 24448K->0K(24512K), 0.2373030 secs] 209214K->184840K(393152K), 0.2378880 secs]
    569738.230: [GC 569738.230: [DefNew: 24447K->0K(24512K), 0.1289770 secs] 209287K->184945K(393152K), 0.1296230 secs]
    569842.966: [GC 569842.967: [DefNew: 24448K->0K(24512K), 0.0838030 secs] 209393K->185028K(393152K), 0.0843720 secs]
    569948.176: [GC 569948.176: [DefNew: 24448K->0K(24512K), 0.0946610 secs] 209476K->185120K(393152K), 0.0953410 secs]
    570026.855: [GC 570026.855: [DefNew: 24448K->0K(24512K), 0.0624600 secs] 209568K->185206K(393152K), 0.0630520 secs]
    570068.376: [GC 570068.376: [DefNew: 24448K->0K(24512K), 0.0897000 secs] 209654K->185280K(393152K), 0.0902590 secs]The memory kept growing for and entire day and no activity from CMS was ever detected. It seems that CMS simply crashed at this
    point. Although nothing is displayed.
    Can someone help me?
    Thanx in advance.

    Hi there,
    I've seen bug report: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4885046
    "...NPTL is the default library on RH-9..."
    "... The problem only happens when running a gcc-3.2-built JDK with NPTL..." ...well I think that's my case.
    They propose a workaround: LD_ASSUME_KERNEL to 2.4.1 but then again this will use LinuxThreads model, which you were saying that has bugs...
    The fact is that I'm experiencing many other problems that happen few days after app start up:
    - notify doesn't wake up a waiting thread
    - scheduled threads don't trigger
    BTW, a full GC dump after the CMS crash (I was not able to use pstack because guys from maintenance rebooted the app few hours later):
    133607.258: [GC 133607.259: [DefNew (promotion failed): 24447K->24447K(24512K), 1.2416730 secs]133608.500: [CMS (concurrent mode failure)[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1122]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor921]
    [Unloading class sun.reflect.GeneratedConstructorAccessor2]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor282]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor831]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor422]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor83]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1082]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor778]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor417]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1093]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1074]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1066]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor158]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor151]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor654]
    [Unloading class $Proxy86]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1061]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor565]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor722]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor759]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor984]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor758]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1]
    [Unloading class sun.reflect.GeneratedMethodAccessor248]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor309]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor864]
    [Unloading class $Proxy80]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1079]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1033]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor999]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor176]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1168]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor43]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor711]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor924]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor909]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor710]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor595]
    [Unloading class sun.reflect.GeneratedConstructorAccessor61]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor473]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1167]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor300]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor225]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor753]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor104]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1010]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor611]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor643]
    [Unloading class sun.reflect.GeneratedMethodAccessor243]
    [Unloading class sun.reflect.GeneratedMethodAccessor240]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor56]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor322]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor807]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor359]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor444]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor304]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor305]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1006]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor944]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor162]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor60]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor502]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor106]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor936]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor424]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1196]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1112]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1096]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor392]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor823]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor25]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor468]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor145]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor931]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor59]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1146]
    [Unloading class sun.reflect.GeneratedMethodAccessor210]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor993]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor876]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor547]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1028]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor601]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor602]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1152]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor234]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor209]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor249]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor129]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor554]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1086]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor277]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1088]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor723]
    [Unloading class sun.reflect.GeneratedMethodAccessor36]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor615]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor429]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor109]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor186]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor388]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor257]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1016]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor345]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor619]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor561]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor959]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor699]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor259]
    [Unloading class $Proxy82]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1155]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor132]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor693]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor477]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor347]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor870]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor126]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor210]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor720]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor237]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor973]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor771]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor598]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor302]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1043]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor355]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor532]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor69]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1176]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor437]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor957]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor862]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor497]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor281]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor100]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor166]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor223]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor583]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor613]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor378]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor670]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1166]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor838]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor57]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor452]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor498]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor219]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor409]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor262]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor80]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor556]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor544]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor123]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor354]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor802]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor605]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1170]
    [Unloading class sun.reflect.GeneratedMethodAccessor40]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor939]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor157]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor963]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1125]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor303]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor40]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor297]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor953]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor213]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor516]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor212]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1215]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor971]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor832]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor314]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor133]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor55]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1058]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor630]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor588]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor983]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor637]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor311]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor266]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1049]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor366]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor52]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor661]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor385]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor508]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor751]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor459]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1056]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor985]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor275]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor178]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor885]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor26]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor893]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor208]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor806]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor781]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor653]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1128]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor644]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1174]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor917]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor996]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1013]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor351]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor642]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor493]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor433]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor776]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor180]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor167]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor980]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor790]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor62]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor155]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1206]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1151]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor686]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1189]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1169]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1149]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1207]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor428]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor403]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1007]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor955]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor951]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor948]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor584]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor72]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor412]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor313]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor455]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor898]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor892]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor221]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1127]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor998]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor658]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1134]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1101]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor34]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor421]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor231]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor700]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1209]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1069]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor255]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1114]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor143]
    [Unloading class sun.reflect.GeneratedMethodAccessor45]
    [Unloading class sun.reflect.GeneratedMethodAccessor24]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1094]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor997]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor128]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor105]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor967]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor976]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor724]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor214]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor95]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor901]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1054]
    [Unloading class sun.reflect.GeneratedMethodAccessor245]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1036]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor271]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1062]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor152]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor600]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor623]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1158]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1140]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1198]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor387]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor107]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor375]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor715]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor566]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor694]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1015]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor258]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor820]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor292]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor48]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor679]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor672]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor702]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor763]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1059]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor264]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor995]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor181]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor391]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor371]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1035]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor390]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor261]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor416]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor346]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor549]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1083]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1085]
    [Unloading class sun.reflect.GeneratedMethodAccessor241]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor319]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor572]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor293]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor244]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor318]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor470]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor39]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor903]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1211]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor896]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor483]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor317]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor296]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor218]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor463]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor363]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1102]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor882]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor189]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor904]
    [Unloading class sun.reflect.GeneratedMethodAccessor38]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor772]
    [Unloading class sun.reflect.GeneratedMethodAccessor32]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor280]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1025]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor51]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor364]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor326]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor620]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor27]
    [Unloading class sun.reflect.GeneratedMethodAccessor242]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor501]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor975]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor425]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1071]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor617]
    [Unloading class sun.reflect.GeneratedMethodAccessor46]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1182]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor531]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor965]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor161]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor35]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor706]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor139]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1078]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor357]
    [Unloading class $Proxy89]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor942]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor682]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor130]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor114]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor895]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1030]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor906]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor438]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor142]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1092]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor988]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor575]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor274]
    [Unloading class sun.reflect.GeneratedMethodAccessor49]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor365]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor179]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor131]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor731]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor156]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1095]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1034]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor335]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor507]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor93]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor150]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor638]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor478]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor905]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor511]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor402]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor736]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1077]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor979]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor794]
    [Unloading class $Proxy96]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor191]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor689]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1123]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor675]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor464]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1205]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor992]
    [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor805]
    [Unloading class sun.reflect.Gener                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          

  • Understand Concurrent mark and sweep output

    Hello There,
    We are working on production issue where different processes stopped almost 5 mins and I thing GC stop the world occurred. But when I analysed/parsed gc logs I haven't seen much interesting. I analysed gc logs and found user: fields showing aroud 3 mins time. I don't understand what Times: field explaining.
    Times: user=0.45 sys=0.00, real=0.09 secs
    what USER/SYS/REAL indicates please?
    --Venkat                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       

    This are the typical OS timings (you also know from the times tool) and measure time in seconds for the GC threads. user is the accumulated CPU time spent in user mode, sys is the time spent in kernel mode and real is the wall clock time.
    Note that usr+sys is measured in CPU seconds, so if you have 8 cores and 6 are doing 1s long calculating something (like memory reachability) and 2 are in kernel page handlers the real time will be 1s and the user time will be 6s.
    Typically for GC you should not see large sys times at all and the real time should be small and the usr time should be parallelity-times more than real clock.
    Bernd

  • Canon 5d Mark ii not logging any footage

    Howdy-
    I have a Canon 5d Mark ii using FCP7. I always use the Log and Transfer plugin (The E1 that you obtain from Canon's website) to import footage, however when I went to use it the other day, and I clicked it, it did not read any files like it normally should. I click the external card that it is located on and nothing happens. Please help! I have a large project due soon! Thanks!

    Reading at your first post it sounds like you are logging and transferring direct from the card in the camera. If so, whilst it can be done, this is wrong. You need to make a copy of the entire contents to a hard drive on or connect to your computer.
    From the message you are getting you don't have all the data needed for log and transfer to import, its some how corrupted, or do you have still images on the card as well? Sometimes this causes a problem.
    For correct procedures with tapeless media look here; http://library.creativecow.net/rossshane/tapeless-workflowfcp-7/1
    It's a complete and excellent tutorial on the workflow you should follow by Shane Ross.

  • [solved] remove --MARK-- from everyting.log

    I use conky to print everything.log on my desktop. Every 20 minutes -- MARK -- appears. I don't know why, and there is probably a good reason for it. After a while, the only lines that appear are these lines. Can I remove this?
    Last edited by anadyr (2009-12-29 18:34:36)

    here's your solution
    http://bbs.archlinux.org/viewtopic.php?id=68092

  • Interpreting my iMac Kernel Panic Log

    Hello, I recently purchased a new iMac (mid 2010) Q core i7 OSX10.6.8
    I got two kernel panic in three days. I have on HD Cinema display connected and wacom tablet and a couple of hard drives. I also upgraded my RAM.  Below is my kernel panic log which is way to complex for me to interpret. I will appreciate all help with interpreting what it says and what maybe be my issue.
    Thanks!
    Interval Since Last Panic Report:  93274 sec
    Panics Since Last Report:          1
    Anonymous UUID:                    B0227A3D-879D-456F-BB6A-914E4E1FB7CC
    Tue Oct 25 07:56:18 2011
    panic(cpu 4 caller 0x2a8ad9): Kernel trap at 0x009990c2, type 14=page fault, registers:
    CR0: 0x80010033, CR2: 0x00000016, CR3: 0x00100000, CR4: 0x00000660
    EAX: 0x00300000, EBX: 0x00000016, ECX: 0x00000016, EDX: 0x10000000
    CR2: 0x00000016, EBP: 0x00e0a888, ESI: 0x0000001a, EDI: 0x00003fff
    EFL: 0x00010202, EIP: 0x009990c2, CS:  0x00000008, DS:  0x04400010
    Error code: 0x00000000
    Backtrace (CPU 4), Frame : Return Address (4 potential args on stack)
    0xe0a678 : 0x21b455 (0x5d09dc 0xe0a6ac 0x2238b1 0x0)
    0xe0a6c8 : 0x2a8ad9 (0x5927fc 0x9990c2 0xe 0x5929c6)
    0xe0a7a8 : 0x29e9a8 (0xe0a7c0 0x0 0xe0a888 0x9990c2)
    0xe0a7b8 : 0x9990c2 (0xe 0x48 0xe00010 0x230010)
    0xe0a888 : 0x9cc56a (0x17409c20 0x18abb26c 0x300000 0x4000)
    0xe0a8c8 : 0x9cc176 (0x173cf940 0x18abb26c 0xe0a97c 0xe0a978)
    0xe0a928 : 0x9a19a8 (0x173cf940 0x18abb26c 0xe0a97c 0xe0a978)
    0xe0a9a8 : 0x9c4936 (0x764d7000 0x18abb200 0x1 0x0)
    0xe0a9f8 : 0x9aef0e (0x22c3df00 0x18abb200 0x1cfc4000 0x766cf000)
    0xe0aa58 : 0x987a09 (0x1cfc4000 0x22c92800 0xe0aa88 0x17c)
    0xe0ab08 : 0x98453f (0x1cfc4000 0x0 0xe0ab6c 0xe0ab68)
    0xe0ab88 : 0x563bfe (0x1cfc4000 0x0 0xe0addc 0xe0ac14)
    0xe0abd8 : 0x56449d (0xa713c0 0x1cfc4000 0x221afe88 0x1)
    0xe0ac28 : 0x564d51 (0x1cfc4000 0x10 0xe0ac70 0x0)
    0xe0ad48 : 0x284a7f (0x1cfc4000 0x10 0x221afe88 0x1)
    0xe0bdf8 : 0x21d7f7 (0x221afe60 0x18bf35a0 0x1fa5b0 0xe407)
    0xe0be38 : 0x210983 (0x221afe00 0x0 0x18aec0e0 0x16b19730)
    0xe0be98 : 0x216be6 (0x221afe00 0x0 0x0 0x0)
    0xe0bf18 : 0x293b71 (0xe0bf44 0x0 0x0 0x0)
    0xe0bfc8 : 0x29f018 (0x1c7e9d40 0x0 0x10 0x225847e4)
          Kernel Extensions in backtrace (with dependencies):
             com.apple.ATIRadeonX3000(6.1.8)@0x97b000->0xb13fff
                dependency: com.apple.iokit.IOPCIFamily(2.6)@0x93b000
                dependency: com.apple.iokit.IONDRVSupport(2.2)@0x96e000
                dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x94c000
    BSD process name corresponding to current thread: Adobe Bridge CS5
    Mac OS version:
    10F2117
    Kernel version:
    Darwin Kernel Version 10.4.1: Fri Jul 16 23:04:20 PDT 2010; root:xnu-1504.7.51~1/RELEASE_I386
    System model name: iMac11,3 (Mac-F2238BAE)
    System uptime in nanoseconds: 97477722728500
    unloaded kexts:
    com.apple.driver.IOFireWireSerialBusProtocolSansPhysicalUnit          2.5.4 (addr 0x8541b000, size 0x8192) - last unloaded 10602028665899
    loaded kexts:
    com.aladdin.kext.aksfridge          1.0.2 - last loaded 10540314941428
    MonitorControl.drv          1.0d3
    at.obdev.nke.LittleSnitch          2.4.4
    com.apple.filesystems.autofs          2.1.0
    com.apple.driver.AppleBluetoothMultitouch          54
    com.apple.driver.AppleHWSensor          1.9.3d0
    com.apple.driver.AppleMikeyHIDDriver          1.2.0
    com.apple.driver.AGPM          100.12.12
    com.apple.driver.AppleUpstreamUserClient          3.4.0
    com.apple.driver.AppleHDA          1.9.3f5
    com.apple.driver.AppleMCCSControl          1.0.17
    com.apple.driver.AppleMikeyDriver          1.9.3f5
    com.apple.driver.AudioAUUC          1.9
    com.apple.kext.ATIFramebuffer          6.1.8
    com.apple.driver.AppleUSBDisplays          282
    com.apple.Dont_Steal_Mac_OS_X          7.0.0
    com.apple.driver.AudioIPCDriver          1.1.5
    com.apple.driver.AirPort.Atheros21          423.9.9
    com.apple.driver.ACPI_SMC_PlatformPlugin          4.3.0d3
    com.apple.ATIRadeonX3000          6.1.8
    com.apple.driver.AppleLPC          1.4.12
    com.apple.driver.AppleBacklight          170.0.30
    com.apple.driver.AppleIRController          303.8
    com.apple.driver.AppleUSBCardReader          2.5.4
    com.apple.driver.Oxford_Semi          2.5.4
    com.apple.iokit.SCSITaskUserClient          2.6.5
    com.apple.BootCache          31
    com.apple.AppleFSCompression.AppleFSCompressionTypeZlib          1.0.0d1
    com.apple.iokit.IOAHCIBlockStorage          1.6.2
    com.apple.driver.AppleFWOHCI          4.7.1
    com.apple.driver.AppleUSBHub          4.0.0
    com.apple.iokit.AppleBCM5701Ethernet          2.3.9b3
    com.apple.driver.AppleEFINVRAM          1.3.0
    com.apple.driver.AppleUSBEHCI          4.0.2
    com.apple.driver.AppleUSBUHCI          4.0.2
    com.apple.driver.AppleAHCIPort          2.1.2
    com.apple.driver.AppleACPIButtons          1.3.3
    com.apple.driver.AppleRTC          1.3.1
    com.apple.driver.AppleHPET          1.5
    com.apple.driver.AppleSMBIOS          1.6
    com.apple.driver.AppleACPIEC          1.3.3
    com.apple.driver.AppleAPIC          1.4
    com.apple.driver.AppleIntelCPUPowerManagementClient          105.11.0
    com.apple.security.sandbox          0
    com.apple.security.quarantine          0
    com.apple.nke.applicationfirewall          2.1.11
    com.apple.driver.AppleIntelCPUPowerManagement          105.11.0
    com.apple.driver.AppleBluetoothHIDKeyboard          141
    com.apple.driver.AppleHIDKeyboard          141
    com.apple.driver.IOBluetoothHIDDriver          2.3.7f1
    com.apple.driver.AppleMultitouchDriver          205.34
    com.apple.driver.AppleProfileReadCounterAction          17
    com.apple.driver.DspFuncLib          1.9.3f5
    com.apple.driver.AppleProfileTimestampAction          10
    com.apple.driver.AppleProfileThreadInfoAction          14
    com.apple.driver.AppleProfileRegisterStateAction          10
    com.apple.driver.AppleProfileKEventAction          10
    com.apple.driver.AppleProfileCallstackAction          20
    com.apple.driver.AppleSMBusController          1.0.8d0
    com.apple.iokit.IOFireWireIP          2.0.3
    com.apple.iokit.IOSurface          74.0
    com.apple.iokit.IOBluetoothSerialManager          2.3.7f1
    com.apple.iokit.IOSerialFamily          10.0.3
    com.apple.iokit.IOAudioFamily          1.7.6fc2
    com.apple.kext.OSvKernDSPLib          1.3
    com.apple.driver.AppleHDAController          1.9.3f5
    com.apple.iokit.IOHDAFamily          1.9.3f5
    com.apple.iokit.IO80211Family          311.1
    com.apple.iokit.AppleProfileFamily          41
    com.apple.driver.AppleSMC          3.0.1d2
    com.apple.driver.IOPlatformPluginFamily          4.3.0d3
    com.apple.driver.AppleSMBusPCI          1.0.8d0
    com.apple.iokit.IONDRVSupport          2.2
    com.apple.kext.ATI5000Controller          6.1.8
    com.apple.kext.ATISupport          6.1.8
    com.apple.iokit.IOGraphicsFamily          2.2
    com.apple.driver.BroadcomUSBBluetoothHCIController          2.3.7f1
    com.apple.driver.AppleUSBBluetoothHCIController          2.3.7f1
    com.apple.iokit.IOBluetoothFamily          2.3.7f1
    com.apple.iokit.IOUSBHIDDriver          4.0.2
    com.apple.iokit.IOUSBMassStorageClass          2.6.1
    com.apple.iokit.IOSCSIBlockCommandsDevice          2.6.5
    com.apple.iokit.IOFireWireSerialBusProtocolTransport          2.0.1
    com.apple.iokit.IOFireWireSBP2          4.0.6
    com.apple.driver.AppleUSBMergeNub          4.1.0
    com.apple.driver.AppleUSBComposite          3.9.0
    com.apple.iokit.IOSCSIMultimediaCommandsDevice          2.6.5
    com.apple.iokit.IOBDStorageFamily          1.6
    com.apple.iokit.IODVDStorageFamily          1.6
    com.apple.iokit.IOCDStorageFamily          1.6
    com.apple.driver.XsanFilter          402.1
    com.apple.iokit.IOAHCISerialATAPI          1.2.4
    com.apple.iokit.IOSCSIArchitectureModelFamily          2.6.5
    com.apple.iokit.IOFireWireFamily          4.2.6
    com.apple.iokit.IOUSBUserClient          4.0.0
    com.apple.iokit.IONetworkingFamily          1.9
    com.apple.iokit.IOUSBFamily          4.1.0
    com.apple.iokit.IOAHCIFamily          2.0.4
    com.apple.driver.AppleEFIRuntime          1.3.0
    com.apple.iokit.IOHIDFamily          1.6.5
    com.apple.iokit.IOSMBusFamily          1.1
    com.apple.kext.AppleMatch          1.0.0d1
    com.apple.security.TMSafetyNet          6
    com.apple.driver.DiskImages          286
    com.apple.iokit.IOStorageFamily          1.6.2
    com.apple.driver.AppleACPIPlatform          1.3.3
    com.apple.iokit.IOPCIFamily          2.6
    com.apple.iokit.IOACPIFamily          1.3.0
    Model: iMac11,3, BootROM IM112.0057.B00, 4 processors, Intel Core i7, 2.93 GHz, 16 GB, SMC 1.59f2
    Graphics: ATI Radeon HD 5750, ATI Radeon HD 5750, PCIe, 1024 MB
    Memory Module: global_name
    AirPort: spairport_wireless_card_type_airport_extreme (0x168C, 0x8F), Atheros 9280: 2.1.9.8.1
    Bluetooth: Version 2.3.7f1, 2 service, 19 devices, 1 incoming serial ports
    Network Service: AirPort, AirPort, en1
    Serial ATA Device: ST31000528AS, 931.51 GB
    Serial ATA Device: HL-DT-STDVDRW  GA32N
    USB Device: Hub, 0x0424  (SMSC), 0x2514, 0xfd100000
    USB Device: Built-in iSight, 0x05ac  (Apple Inc.), 0x8502, 0xfd110000
    USB Device: IR Receiver, 0x05ac  (Apple Inc.), 0x8242, 0xfd120000
    USB Device: Hub, 0x0424  (SMSC), 0x2514, 0xfa100000
    USB Device: Hub, 0x05ac  (Apple Inc.), 0x9131, 0xfa140000
    USB Device: Hub, 0x050d  (Belkin Corporation), 0x0234, 0xfa143000
    USB Device: Apple Cinema HD Display, 0x05ac  (Apple Inc.), 0x9223, 0xfa142000
    USB Device: PTK-640, 0x056a  (WACOM Co., Ltd.), 0x00b9, 0xfa130000
    USB Device: Internal Memory Card Reader, 0x05ac  (Apple Inc.), 0x8403, 0xfa120000
    USB Device: BRCM2046 Hub, 0x0a5c  (Broadcom Corp.), 0x4500, 0xfa110000
    USB Device: Bluetooth USB Host Controller, 0x05ac  (Apple Inc.), 0x8215, 0xfa111000

    I suspect one of two things, either com.aladdin.kext.aksfridge is causing it, I did a quick search and found this to be some type of MS Windows applicaiton. The next thing was at.obdev.nke.LittleSnitch which should be OK unless you are running a very old version.
    I'd recommend unistalling the Aladdin software first per the manufacturers instructions, restarting in Safe Mode to clear some caches then restarting normally and finally test.  If that does not do it then do the same with Little Snitch.

  • Forcing �Mark Sweep� Garbage Collection?

    I have a fairly complex Java server-based application which is experiencing some memory management problems. As the application runs, it will continue to grow over time. If 5 or 6 instances of the application are running concurrently, eventually the server performance is significantly degraded and the application will crash due to a lack of system memory or system swap space.
    When monitoring the application with JConsole, I have noticed that a large number of �Scavenge� garbage collections normally occur. These do not consume much CPU time, but they also do not seem to free much memory. When I manually click the �Perform GC� button, however, that seems to trigger a �MarkSweep� type of garbage collection. This will normally reclaim a huge amount of memory (although it takes a relatively large amount of CPU to perform.)
    Is there a way to cause this �MarkSweep� to occur more frequently? I have read that explicitly calling system.gc() in my application is discouraged, and may not even cause any garbage collection to occur. (Although it appears that JConsole is somehow forcing it to occur.) I have also seen it suggested that I could tweak some of the JVM parameters for heap size and ratios, but this approach seems difficult and problematic for my application.
    I realize that explicitly forcing MarkSweep collections may cause noticeable pauses in my application. Although that behavior is not ideal, it is tolerable � especially if I can control when the pauses occur. Any suggestions or insight would be appreciated.
    Thanks,
    Mike H.

    Seeing the date of this question my answer will not be in use by the guy who asked but for some one else could be.
    I had similar problem - I wanted to release unused memory from the JVM to the OS.
    After some research i found that calling several consequent times System.gc() helps. I do the calls with some pause between them (1-2 seconds). This makes the app to be slower at this particular moment but releases hundreds of Megabytes in my case.
    I tested with this approach and found that it works only when Serial GC is running, but fortunately this is the GC in most of cases that JVM uses.

  • How to use hs_err_pid pid .log file?

    I have some hs_err_pid<pid>.log files. I want to trace out the root cause of application crash. How can I use the log file to do this? Contents of a log file are given below:
    # An unexpected error has been detected by Java Runtime Environment:
    # SIGSEGV (0xb) at pc=0x062e73bc, pid=32733, tid=2849041328
    # Java VM: Java HotSpot(TM) Client VM (1.6.0_02-ea-b02 mixed mode)
    # Problematic frame:
    # V [libjvm.so+0x2e73bc]
    # If you would like to submit a bug report, please visit:
    # http://java.sun.com/webapps/bugreport/crash.jsp
    --------------- T H R E A D ---------------
    Current thread (0x0811a000): VMThread [id=32749]
    siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000005c
    Registers:
    EAX=0x00000000, EBX=0xa2b02830, ECX=0x92720ad1, EDX=0x927224a0
    ESP=0xa9d0e0a0, EBP=0xa9d0e0b8, ESI=0x9d3aaaac, EDI=0x0643b5f4
    EIP=0x062e73bc, CR2=0x0000005c, EFLAGS=0x00010292
    Top of Stack: (sp=0xa9d0e0a0)
    0xa9d0e0a0: 92720ad1 927224a0 a9d0e0c8 00000003
    0xa9d0e0b0: 06426340 06426340 a9d0e0e8 0632df77
    0xa9d0e0c0: 0643b5b4 9d3aaaac 0000005c 0000005c
    0xa9d0e0d0: 0643b5f4 57a9f5e8 00a9f568 081187ac
    0xa9d0e0e0: 08118758 00000001 a9d0e128 0632e1bd
    0xa9d0e0f0: 08118758 081187a8 00000000 0643b5f0
    0xa9d0e100: 0643b5f4 0643b5b4 00000001 00000001
    0xa9d0e110: 0643b5f4 00000000 001187a8 081187a8
    Instructions: (pc=0x062e73bc)
    0x062e73ac: 89 11 8b 14 90 8b 42 04 8d 48 08 8b 40 08 52 51
    0x062e73bc: ff 50 5c 8b 0d c8 62 42 06 83 c4 10 8b 01 85 c0
    Stack: [0xa9c8e000,0xa9d0f000), sp=0xa9d0e0a0, free space=512k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V [libjvm.so+0x2e73bc]
    V [libjvm.so+0x32df77]
    V [libjvm.so+0x32e1bd]
    V [libjvm.so+0x32db80]
    V [libjvm.so+0x1d4817]
    V [libjvm.so+0x1d3fc2]
    V [libjvm.so+0x19c112]
    V [libjvm.so+0x19bb7e]
    V [libjvm.so+0x19b895]
    V [libjvm.so+0x1d1c74]
    V [libjvm.so+0x186065]
    V [libjvm.so+0x1d1ffc]
    V [libjvm.so+0x3b2a0e]
    V [libjvm.so+0x3c4d87]
    V [libjvm.so+0x3c440c]
    V [libjvm.so+0x3c45da]
    V [libjvm.so+0x3c41bf]
    V [libjvm.so+0x30b599]
    C [libpthread.so.0+0x59ed]
    VM_Operation (0xb1262d20): generation collection for allocation, mode: safepoint, requested by thread 0x0876b800
    --------------- P R O C E S S ---------------
    Java Threads: ( => current thread )
    0x0876b800 JavaThread "Thread-127961" daemon [_thread_blocked, id=20969]
    0x08456800 JavaThread "Thread-127960" daemon [_thread_blocked, id=20968]
    0xae485c00 JavaThread "RMI TCP Connection(21182)-172.16.1.58" daemon [_thread_blocked, id=20967]
    0x0853f000 JavaThread "RMI RenewClean-[cryoserverlondon:40523,ci.cryoserver.rmi.ssl.RMISSLClientSocketFactory@a2a054]" daemon [_thread_blocked, id=20966]
    0x0848f800 JavaThread "RMI TCP Connection(21183)-172.16.1.58" daemon [_thread_blocked, id=20965]
    0x085b2400 JavaThread "RMI RenewClean-[cryoserverlondon:40520,ci.cryoserver.rmi.ssl.RMISSLClientSocketFactory@fecc3a]" daemon [_thread_blocked, id=20964]
    0xae484c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20963]
    0x086f6000 JavaThread "RMI RenewClean-[cryoserverlondon:40521,ci.cryoserver.rmi.ssl.RMISSLClientSocketFactory@a1e2cd]" daemon [_thread_blocked, id=20962]
    0x081d9800 JavaThread "Thread-127958" daemon [_thread_blocked, id=20959]
    0x083c4800 JavaThread "Thread-127957" daemon [_thread_blocked, id=20958]
    0xae483c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20957]
    0xae469800 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20956]
    0xae468c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20955]
    0x0860bc00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20954]
    0xae465c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20953]
    0x08525000 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20952]
    0xac80d000 JavaThread "RMI TCP Connection(21171)-172.16.1.58" daemon [_thread_blocked, id=20951]
    0xae467400 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20950]
    0xae46a400 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20948]
    0xac80c400 JavaThread "RMI TCP Connection(21169)-172.16.1.58" daemon [_thread_blocked, id=20947]
    0xae45c400 JavaThread "RMI TCP Connection(21168)-127.0.0.1" daemon [_thread_blocked, id=20946]
    0xac802800 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20945]
    0xae466800 JavaThread "RMI TCP Connection(21167)-127.0.0.1" daemon [_thread_blocked, id=20943]
    0xae46ac00 JavaThread "RMI TCP Connection(21157)-127.0.0.1" daemon [_thread_blocked, id=20886]
    0x081cfc00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20885]
    0xac80b400 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20864]
    0xac806c00 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20863]
    0xac803800 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20862]
    0x0831b400 JavaThread "RMI RenewClean-[cryoserverlondon:40523,ci.cryoserver.rmi.ssl.RMISSLClientSocketFactory@37497e]" daemon [_thread_blocked, id=20783]
    0x08737c00 JavaThread "RMI RenewClean-[cryoserverlondon:40520,ci.cryoserver.rmi.ssl.RMISSLClientSocketFactory@61eb66]" daemon [_thread_blocked, id=20781]
    0x08206400 JavaThread "RMI RenewClean-[cryoserverlondon:40521,ci.cryoserver.rmi.ssl.RMISSLClientSocketFactory@3f355]" daemon [_thread_blocked, id=20779]
    0xac803000 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20777]
    0xae401400 JavaThread "RMI TCP Connection(idle)" daemon [_thread_blocked, id=20763]
    0xae400800 JavaThread "Thread-127942" daemon [_thread_blocked, id=20385]
    0x08127400 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=2974]
    0x579a6c00 JavaThread "cache-expiry" [_thread_blocked, id=417]
    0x579a5c00 JavaThread "cache-expiry" [_thread_blocked, id=416]
    0x0818d000 JavaThread "cache-expiry" [_thread_blocked, id=415]
    0x0818ac00 JavaThread "cache-expiry" [_thread_blocked, id=414]
    0x57968800 JavaThread "cache-expiry" [_thread_blocked, id=412]
    0x57967c00 JavaThread "cache-expiry" [_thread_blocked, id=411]
    0x08443000 JavaThread "cache-expiry" [_thread_blocked, id=352]
    0x08442400 JavaThread "cache-expiry" [_thread_blocked, id=351]
    0x5798c800 JavaThread "Spool manager for /opt/cryoserver/cryoserver/data/uk-ln-sp-001/spool" [_thread_blocked, id=339]
    0x5798bc00 JavaThread "Spool manager agent 3" [_thread_blocked, id=338]
    0x5798b000 JavaThread "Spool manager agent 2" [_thread_blocked, id=336]
    0x57989800 JavaThread "Spool manager agent 1" [_thread_blocked, id=334]
    0x57988c00 JavaThread "Spool manager agent 0" [_thread_blocked, id=332]
    0x57987c00 JavaThread "Thread-3" daemon [_thread_blocked, id=330]
    0x57958400 JavaThread "Thread-2" [_thread_blocked, id=328]
    0x57957400 JavaThread "Thread-1" [_thread_blocked, id=327]
    0x57956000 JavaThread "uk-ln-sp-001 queue stats." [_thread_blocked, id=326]
    0x57955400 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=325]
    0x57954000 JavaThread "cache-expiry" [_thread_blocked, id=324]
    0x57953000 JavaThread "cache-expiry" [_thread_blocked, id=323]
    0x57952000 JavaThread "RMI TCP Accept-0" daemon [_thread_blocked, id=322]
    0x57951000 JavaThread "cache-expiry" [_thread_blocked, id=319]
    0x5794c400 JavaThread "cache-expiry" [_thread_blocked, id=318]
    0x5794b000 JavaThread "RMI TCP Accept-0" daemon [_thread_blocked, id=317]
    0x5794a400 JavaThread "Optimizer for /opt/cryoserver/cryoserver/data/uk-ln-im-002" [_thread_blocked, id=316]
    0x5797a000 JavaThread "cache-expiry" [_thread_blocked, id=315]
    0x57979000 JavaThread "cache-expiry" [_thread_blocked, id=314]
    0x5793e000 JavaThread "RMI TCP Accept-0" daemon [_thread_blocked, id=313]
    0x5793d800 JavaThread "cache-expiry" [_thread_blocked, id=312]
    0x57930c00 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=311]
    0x57975c00 JavaThread "cache-expiry" [_thread_blocked, id=310]
    0x57975000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=309]
    0x5797d000 JavaThread "cache-expiry" [_thread_blocked, id=308]
    0x57948000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=307]
    0x5792d400 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=306]
    0x5792e800 JavaThread "RMI TCP Accept-0" daemon [_thread_blocked, id=304]
    0x57925400 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=302]
    0x57944800 JavaThread "RMI TCP Accept-1099" daemon [_thread_blocked, id=301]
    0x57927000 JavaThread "GC Daemon" daemon [_thread_blocked, id=300]
    0x57926400 JavaThread "RMI Reaper" [_thread_blocked, id=32767]
    0x5791b800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=32766]
    0x0813dc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=32759]
    0x0813c400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=32758]
    0x0813b000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=32757]
    0x08139c00 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=32756]
    0x0811e400 JavaThread "Finalizer" daemon [_thread_blocked, id=32751]
    0x0811d000 JavaThread "Reference Handler" daemon [_thread_blocked, id=32750]
    0x08057c00 JavaThread "main" [_thread_blocked, id=32736]
    Other Threads:
    =>0x0811a000 VMThread [id=32749]
    0x0813f400 WatcherThread [id=32760]
    VM state:at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
    [0x08056e68/0x08056e90] Threads_lock - owner thread: 0x0811a000
    [0x080573a8/0x080573c0] Heap_lock - owner thread: 0x0876b800
    Heap
    par new generation total 29504K, used 29504K [0x57a90000, 0x59a90000, 0x59a90000)
    eden space 26240K, 100% used [0x57a90000, 0x59430000, 0x59430000)
    from space 3264K, 100% used [0x59430000, 0x59760000, 0x59760000)
    to space 3264K, 0% used [0x59760000, 0x59760000, 0x59a90000)
    concurrent mark-sweep generation total 1196032K, used 1036441K [0x59a90000, 0xa2a90000, 0xa2a90000)
    concurrent-mark-sweep perm gen total 17932K, used 10690K [0xa2a90000, 0xa3c13000, 0xa6a90000)
    Dynamic libraries:
    06000000-06417000 r-xp 00000000 68:01 89212 /opt/java/jdk1.6.0_02/jre/lib/i386/client/libjvm.so
    06417000-06430000 rwxp 00417000 68:01 89212 /opt/java/jdk1.6.0_02/jre/lib/i386/client/libjvm.so
    06430000-0684f000 rwxp 06430000 00:00 0
    08048000-08052000 r-xp 00000000 68:01 89972 /opt/java/jdk1.6.0_02/bin/java
    08052000-08053000 rwxp 00009000 68:01 89972 /opt/java/jdk1.6.0_02/bin/java
    08053000-08c80000 rwxp 08053000 00:00 0
    55555000-5556b000 r-xp 00000000 68:01 17445 /lib/ld-2.3.3.so
    5556b000-5556c000 rwxp 00016000 68:01 17445 /lib/ld-2.3.3.so
    5556c000-5556d000 rwxp 5556c000 00:00 0
    55582000-5558f000 r-xp 00000000 68:01 17472 /lib/tls/libpthread.so.0
    5558f000-55590000 rwxp 0000d000 68:01 17472 /lib/tls/libpthread.so.0
    55590000-55592000 rwxp 55590000 00:00 0
    55592000-55599000 r-xp 00000000 68:01 89221 /opt/java/jdk1.6.0_02/jre/lib/i386/jli/libjli.so
    55599000-5559b000 rwxp 00006000 68:01 89221 /opt/java/jdk1.6.0_02/jre/lib/i386/jli/libjli.so
    5559b000-5559d000 r-xp 00000000 68:01 17454 /lib/libdl.so.2
    5559d000-5559e000 rwxp 00002000 68:01 17454 /lib/libdl.so.2
    5559e000-556a7000 r-xp 00000000 68:01 17470 /lib/tls/libc.so.6
    556a7000-556af000 rwxp 00109000 68:01 17470 /lib/tls/libc.so.6
    556af000-556b3000 rwxp 556af000 00:00 0
    556b3000-556d4000 r-xp 00000000 68:01 17471 /lib/tls/libm.so.6
    556d4000-556d5000 rwxp 00020000 68:01 17471 /lib/tls/libm.so.6
    556d5000-556d8000 ---p 556d5000 00:00 0
    556d8000-55726000 rwxp 556d8000 00:00 0
    55726000-55727000 ---p 55726000 00:00 0
    55727000-55728000 rwxp 55727000 00:00 0
    55728000-5572e000 r-xp 00000000 68:01 89205 /opt/java/jdk1.6.0_02/jre/lib/i386/native_threads/libhpi.so
    5572e000-5572f000 rwxp 00006000 68:01 89205 /opt/java/jdk1.6.0_02/jre/lib/i386/native_threads/libhpi.so
    5572f000-5573a000 r-xp 00000000 68:01 89218 /opt/java/jdk1.6.0_02/jre/lib/i386/libverify.so
    5573a000-5573b000 rwxp 0000b000 68:01 89218 /opt/java/jdk1.6.0_02/jre/lib/i386/libverify.so
    5573b000-55742000 r-xp 00000000 68:01 17474 /lib/tls/librtkaio.so.1
    55742000-55743000 rwxp 00006000 68:01 17474 /lib/tls/librtkaio.so.1
    55743000-5574b000 rwxs 00000000 68:01 92381 /tmp/hsperfdata_cryouser/32733
    55758000-5576a000 r-xp 00000000 68:01 17457 /lib/libnsl.so.1
    5576a000-5576b000 rwxp 00011000 68:01 17457 /lib/libnsl.so.1
    5576b000-5576d000 rwxp 5576b000 00:00 0
    5576d000-55790000 r-xp 00000000 68:01 89219 /opt/java/jdk1.6.0_02/jre/lib/i386/libjava.so
    55790000-55792000 rwxp 00023000 68:01 89219 /opt/java/jdk1.6.0_02/jre/lib/i386/libjava.so
    55792000-557a1000 r-xp 00000000 68:01 89223 /opt/java/jdk1.6.0_02/jre/lib/i386/libzip.so
    557a1000-557a3000 rwxp 0000e000 68:01 89223 /opt/java/jdk1.6.0_02/jre/lib/i386/libzip.so
    557a3000-55eab000 rwxp 557a3000 00:00 0
    55eab000-577a3000 rwxp 55eab000 00:00 0
    577a3000-577c0000 rwxp 577a3000 00:00 0
    577c0000-57823000 rwxp 577c0000 00:00 0
    57823000-57824000 ---p 57823000 00:00 0
    57824000-578a4000 rwxp 57824000 00:00 0
    578a4000-578a7000 ---p 578a4000 00:00 0
    578a7000-578f5000 rwxp 578a7000 00:00 0
    578f5000-578fc000 r-xs 0004f000 68:09 23 /opt/cryoserver/cryoserver/lib/log4j-1.2.6.jar
    578fc000-578ff000 r-xs 00013000 68:01 89270 /opt/java/jdk1.6.0_02/jre/lib/jce.jar
    57900000-579f2000 rwxp 57900000 00:00 0
    579f2000-57a00000 ---p 579f2000 00:00 0
    57a00000-57a01000 ---p 57a00000 00:00 0
    57a01000-57a81000 rwxp 57a01000 00:00 0
    57a81000-57a87000 r-xs 00049000 68:09 29 /opt/cryoserver/cryoserver/lib/lucene-1.4-final.jar
    57a87000-57a89000 r-xs 00006000 68:09 27 /opt/cryoserver/cryoserver/lib/lucene-highlighter.jar
    57a89000-57a8e000 r-xs 00040000 68:09 30 /opt/cryoserver/cryoserver/lib/mail_1_2.jar
    57a8e000-57a8f000 r-xp 00000000 68:01 89250 /opt/java/jdk1.6.0_02/jre/lib/i386/librmi.so
    57a8f000-57a90000 rwxp 00000000 68:01 89250 /opt/java/jdk1.6.0_02/jre/lib/i386/librmi.so
    57a90000-a3c13000 rwxp 57a90000 00:00 0
    a3c13000-a6a90000 rwxp a3c13000 00:00 0
    a6a90000-a6cf1000 rwxp a6a90000 00:00 0
    a6cf1000-a6d08000 rwxp a6cf1000 00:00 0
    a6d08000-a6f5b000 rwxp a6d08000 00:00 0
    a6f5b000-a6f73000 rwxp a6f5b000 00:00 0
    a6f73000-a9642000 rwxp a6f73000 00:00 0
    a9642000-a9650000 r-xs 00656000 68:01 89944 /opt/java/jdk1.6.0_02/jre/lib/charsets.jar
    a9662000-a9a62000 rwxp a9662000 00:00 0
    a9a62000-a9a63000 ---p a9a62000 00:00 0
    a9a63000-a9ae3000 rwxp a9a63000 00:00 0
    a9ae3000-a9c5e000 r-xs 02c7f000 68:01 89177 /opt/java/jdk1.6.0_02/jre/lib/rt.jar
    a9c5e000-a9c8e000 rwxp a9c5e000 00:00 0
    a9c8e000-a9c8f000 ---p a9c8e000 00:00 0
    a9c8f000-a9d0f000 rwxp a9c8f000 00:00 0
    a9d0f000-a9d12000 ---p a9d0f000 00:00 0
    a9d12000-a9d60000 rwxp a9d12000 00:00 0
    a9d60000-a9d93000 r-xp 00000000 68:01 13874 /usr/lib/locale/en_GB.utf8/LC_CTYPE
    a9d93000-a9d99000 r-xs 00000000 68:01 19748 /usr/lib/gconv/gconv-modules.cache
    a9d99000-a9d9c000 ---p a9d99000 00:00 0
    a9d9c000-a9dea000 rwxp a9d9c000 00:00 0
    a9dea000-a9ded000 ---p a9dea000 00:00 0
    a9ded000-a9e3b000 rwxp a9ded000 00:00 0
    a9e3b000-a9e3e000 ---p a9e3b000 00:00 0
    a9e3e000-a9ebc000 rwxp a9e3e000 00:00 0
    a9ebc000-a9ebf000 ---p a9ebc000 00:00 0
    a9ebf000-a9f0d000 rwxp a9ebf000 00:00 0
    a9f0d000-a9f0e000 ---p a9f0d000 00:00 0
    a9f0e000-a9f8e000 rwxp a9f0e000 00:00 0
    a9f8e000-a9f90000 r-xs 0000a000 68:09 19 /opt/cryoserver/cryoserver/lib/activation.jar
    a9f90000-a9f9e000 r-xs 000a1000 68:09 22 /opt/cryoserver/cryoserver/lib/castor-0.9.3.21-xml.jar
    a9f9e000-a9fa1000 r-xs 00027000 68:09 270211 /opt/cryoserver/cryoserver/lib/commons-lang-2.0.jar
    a9fa1000-a9faa000 r-xs 0009f000 68:09 21 /opt/cryoserver/cryoserver/lib/cryoserver.jar
    a9faa000-aa017000 r-xs 0061c000 68:09 24 /opt/cryoserver/cryoserver/lib/j2ee.jar
    aa017000-aa019000 r-xs 00006000 68:09 25 /opt/cryoserver/cryoserver/lib/jakarta-regexp-1.2.jar
    aa019000-aa01e000 r-xs 0004d000 68:09 270219 /opt/cryoserver/cryoserver/lib/jcifs-1.2.11.jar
    aa01e000-aa021000 r-xs 0001b000 68:09 28 /opt/cryoserver/cryoserver/lib/junit.jar
    aa021000-aa02b000 r-xs 00055000 68:09 18 /opt/cryoserver/cryoserver/lib/ldap.jar
    aa02b000-aa031000 r-xs 00091000 68:01 89262 /opt/java/jdk1.6.0_02/jre/lib/jsse.jar
    aa031000-aa034000 r-xs 00035000 68:09 26 /opt/cryoserver/cryoserver/lib/mysql-connector-java-3.0.9-stable-bin.jar
    aa034000-aa03b000 r-xs 001ab000 68:09 20 /opt/cryoserver/cryoserver/lib/PDFBox-0.6.5.jar
    aa03b000-aa04a000 r-xs 001ac000 68:09 17 /opt/cryoserver/cryoserver/lib/xerces.jar
    aa04a000-aa05d000 r-xp 00000000 68:01 89227 /opt/java/jdk1.6.0_02/jre/lib/i386/libnet.so
    aa05d000-aa05e000 rwxp 00013000 68:01 89227 /opt/java/jdk1.6.0_02/jre/lib/i386/libnet.so
    aa05e000-aa148000 rwxp aa05e000 00:00 0
    aa148000-aa14b000 r-xs 00027000 68:01 89198 /opt/java/jdk1.6.0_02/jre/lib/ext/sunjce_provider.jar
    aa14b000-aa152000 r-xs 00106000 68:01 89316 /opt/java/jdk1.6.0_02/jre/lib/resources.jar
    aa167000-aa16f000 r-xp 00000000 68:01 17460 /lib/libnss_files.so.2
    aa16f000-aa170000 rwxp 00008000 68:01 17460 /lib/libnss_files.so.2
    aa170000-aa173000 ---p aa170000 00:00 0
    aa173000-aa1c1000 rwxp aa173000 00:00 0
    aa1c1000-aa1c4000 ---p aa1c1000 00:00 0
    aa1c4000-aa212000 rwxp aa1c4000 00:00 0
    aa212000-aa215000 ---p aa212000 00:00 0
    aa215000-aa263000 rwxp aa215000 00:00 0
    aa263000-aa266000 ---p aa263000 00:00 0
    aa266000-aa2b4000 rwxp aa266000 00:00 0
    aa2b4000-aa2b7000 ---p aa2b4000 00:00 0
    aa2b7000-aa305000 rwxp aa2b7000 00:00 0
    aa305000-aa308000 ---p aa305000 00:00 0
    aa308000-aa356000 rwxp aa308000 00:00 0
    aa356000-aa359000 ---p aa356000 00:00 0
    aa359000-aa3a7000 rwxp aa359000 00:00 0
    aa3a7000-aa3aa000 ---p aa3a7000 00:00 0
    aa3aa000-aa3f8000 rwxp aa3aa000 00:00 0
    aa3f8000-aa3fb000 ---p aa3f8000 00:00 0
    aa3fb000-aa449000 rwxp aa3fb000 00:00 0
    aa449000-aa44c000 ---p aa449000 00:00 0
    aa44c000-aa49a000 rwxp aa44c000 00:00 0
    aa49a000-aa49d000 ---p aa49a000 00:00 0
    aa49d000-aa4eb000 rwxp aa49d000 00:00 0
    aa4eb000-aa4ee000 ---p aa4eb000 00:00 0
    aa4ee000-aa53c000 rwxp aa4ee000 00:00 0
    aa53c000-aa53f000 ---p aa53c000 00:00 0
    aa53f000-aa58d000 rwxp aa53f000 00:00 0
    aa58d000-aa590000 ---p aa58d000 00:00 0
    aa590000-aa5de000 rwxp aa590000 00:00 0
    aa5de000-aa5e1000 ---p aa5de000 00:00 0
    aa5e1000-aa62f000 rwxp aa5e1000 00:00 0
    aa62f000-aa632000 ---p aa62f000 00:00 0
    aa632000-aa680000 rwxp aa632000 00:00 0
    aa680000-aa683000 ---p aa680000 00:00 0
    aa683000-aa6d1000 rwxp aa683000 00:00 0
    aa6d1000-aa6d4000 ---p aa6d1000 00:00 0
    aa6d4000-aa722000 rwxp aa6d4000 00:00 0
    aa722000-aa725000 ---p aa722000 00:00 0
    aa725000-aa773000 rwxp aa725000 00:00 0
    aa773000-aa776000 ---p aa773000 00:00 0
    aa776000-aa7c4000 rwxp aa776000 00:00 0
    aa7c4000-aa7c7000 ---p aa7c4000 00:00 0
    aa7c7000-aa815000 rwxp aa7c7000 00:00 0
    aa815000-aa818000 ---p aa815000 00:00 0
    aa818000-aa866000 rwxp aa818000 00:00 0
    aa866000-aa869000 ---p aa866000 00:00 0
    aa869000-aa8b7000 rwxp aa869000 00:00 0
    aa8b7000-aa8ba000 ---p aa8b7000 00:00 0
    aa8ba000-aa908000 rwxp aa8ba000 00:00 0
    aa908000-aa90b000 ---p aa908000 00:00 0
    aa90b000-aa959000 rwxp aa90b000 00:00 0
    aa959000-aa95c000 ---p aa959000 00:00 0
    aa95c000-aa9aa000 rwxp aa95c000 00:00 0
    aa9aa000-aa9ad000 ---p aa9aa000 00:00 0
    aa9ad000-aa9fb000 rwxp aa9ad000 00:00 0
    aa9fb000-aa9fe000 ---p aa9fb000 00:00 0
    aa9fe000-aaa4c000 rwxp aa9fe000 00:00 0
    aaa4c000-aaa4f000 ---p aaa4c000 00:00 0
    aaa4f000-aaa9d000 rwxp aaa4f000 00:00 0
    aaa9d000-aaaa0000 ---p aaa9d000 00:00 0
    aaaa0000-aaaee000 rwxp aaaa0000 00:00 0
    aaaee000-aaaf1000 ---p aaaee000 00:00 0
    aaaf1000-aab3f000 rwxp aaaf1000 00:00 0
    aab3f000-aab42000 ---p aab3f000 00:00 0
    aab42000-aab90000 rwxp aab42000 00:00 0
    aab90000-aab93000 ---p aab90000 00:00 0
    aab93000-aabe1000 rwxp aab93000 00:00 0
    aac32000-aac35000 ---p aac32000 00:00 0
    aac35000-aac83000 rwxp aac35000 00:00 0
    aac83000-aac86000 ---p aac83000 00:00 0
    aac86000-aacd4000 rwxp aac86000 00:00 0
    aacd4000-aacd7000 ---p aacd4000 00:00 0
    aacd7000-aad25000 rwxp aacd7000 00:00 0
    aad76000-aad79000 rwxp aad76000 00:00 0
    aad79000-aadc7000 rwxp aad79000 00:00 0
    aadc7000-aadca000 ---p aadc7000 00:00 0
    aadca000-aae18000 rwxp aadca000 00:00 0
    aae18000-aae1b000 ---p aae18000 00:00 0
    aae1b000-aae69000 rwxp aae1b000 00:00 0
    aae69000-aae6c000 ---p aae69000 00:00 0
    aae6c000-aaeba000 rwxp aae6c000 00:00 0
    aaeba000-aaebd000 ---p aaeba000 00:00 0
    aaebd000-aaf0b000 rwxp aaebd000 00:00 0
    aaf0b000-aaf0e000 ---p aaf0b000 00:00 0
    aaf0e000-aaf5c000 rwxp aaf0e000 00:00 0
    aaf5c000-aaf5f000 ---p aaf5c000 00:00 0
    aaf5f000-aafad000 rwxp aaf5f000 00:00 0
    aafad000-aafb0000 ---p aafad000 00:00 0
    aafb0000-aaffe000 rwxp aafb0000 00:00 0
    ab0a0000-ab0a3000 rwxp ab0a0000 00:00 0
    ab0a3000-ab0f1000 rwxp ab0a3000 00:00 0
    ab193000-ab196000 ---p ab193000 00:00 0
    ab196000-ab1e4000 rwxp ab196000 00:00 0
    ab200000-ab2fb000 rwxp ab200000 00:00 0
    ab2fb000-ab300000 ---p ab2fb000 00:00 0
    ab300000-ab600000 rwxp ab300000 00:00 0
    ab600000-ab603000 ---p ab600000 00:00 0
    ab603000-ab651000 rwxp ab603000 00:00 0
    ab651000-ab654000 ---p ab651000 00:00 0
    ab654000-ab6a2000 rwxp ab654000 00:00 0
    ab6a2000-ab6a5000 ---p ab6a2000 00:00 0
    ab6a5000-ab6f3000 rwxp ab6a5000 00:00 0
    ab6f3000-ab6f6000 ---p ab6f3000 00:00 0
    ab6f6000-ab744000 rwxp ab6f6000 00:00 0
    ab744000-ab747000 ---p ab744000 00:00 0
    ab747000-ab795000 rwxp ab747000 00:00 0
    ab837000-ab83a000 rwxp ab837000 00:00 0
    ab83a000-ab888000 rwxp ab83a000 00:00 0
    ab888000-ab88b000 rwxp ab888000 00:00 0
    ab88b000-ab8d9000 rwxp ab88b000 00:00 0
    ab97b000-ab97e000 ---p ab97b000 00:00 0
    ab97e000-ab9cc000 rwxp ab97e000 00:00 0
    ab9cc000-ab9cf000 rwxp ab9cc000 00:00 0
    ab9cf000-aba1d000 rwxp ab9cf000 00:00 0
    aba1d000-aba20000 rwxp aba1d000 00:00 0
    aba20000-aba6e000 rwxp aba20000 00:00 0
    aba6e000-aba71000 rwxp aba6e000 00:00 0
    aba71000-ababf000 rwxp aba71000 00:00 0
    ababf000-abac2000 ---p ababf000 00:00 0
    abac2000-abb10000 rwxp abac2000 00:00 0
    abb10000-abb13000 rwxp abb10000 00:00 0
    abb13000-abb61000 rwxp abb13000 00:00 0
    abb61000-abb64000 rwxp abb61000 00:00 0
    abb64000-abbb2000 rwxp abb64000 00:00 0
    abbb2000-abbb5000 rwxp abbb2000 00:00 0
    abbb5000-abc03000 rwxp abbb5000 00:00 0
    abd98000-abd9b000 rwxp abd98000 00:00 0
    abd9b000-abde9000 rwxp abd9b000 00:00 0
    abfcf000-abfd2000 rwxp abfcf000 00:00 0
    abfd2000-ac020000 rwxp abfd2000 00:00 0
    ac1b5000-ac1b8000 rwxp ac1b5000 00:00 0
    ac1b8000-ac206000 rwxp ac1b8000 00:00 0
    ac206000-ac209000 rwxp ac206000 00:00 0
    ac209000-ac257000 rwxp ac209000 00:00 0
    ac257000-ac25a000 rwxp ac257000 00:00 0
    ac25a000-ac2a8000 rwxp ac25a000 00:00 0
    ac2a8000-ac2ab000 rwxp ac2a8000 00:00 0
    ac2ab000-ac2f9000 rwxp ac2ab000 00:00 0
    ac2f9000-ac2fc000 rwxp ac2f9000 00:00 0
    ac2fc000-ac34a000 rwxp ac2fc000 00:00 0
    ac43d000-ac440000 rwxp ac43d000 00:00 0
    ac440000-ac48e000 rwxp ac440000 00:00 0
    ac4df000-ac4e2000 rwxp ac4df000 00:00 0
    ac4e2000-ac530000 rwxp ac4e2000 00:00 0
    ac6c5000-ac6c8000 ---p ac6c5000 00:00 0
    ac6c8000-ac716000 rwxp ac6c8000 00:00 0
    ac800000-ac82f000 rwxp ac800000 00:00 0
    ac82f000-ac900000 ---p ac82f000 00:00 0
    aca40000-aca43000 ---p aca40000 00:00 0
    aca43000-aca91000 rwxp aca43000 00:00 0
    acb33000-acb36000 ---p acb33000 00:00 0
    acb36000-acb84000 rwxp acb36000 00:00 0
    acb84000-acb87000 rwxp acb84000 00:00 0
    acb87000-acbd5000 rwxp acb87000 00:00 0
    acc26000-ad026000 rwxp acc26000 00:00 0
    ad1bb000-ad1be000 rwxp ad1bb000 00:00 0
    ad1be000-ad20c000 rwxp ad1be000 00:00 0
    ad36d000-ad370000 rwxp ad36d000 00:00 0
    ad370000-ad3be000 rwxp ad370000 00:00 0
    ad553000-ad556000 ---p ad553000 00:00 0
    ad556000-ad5a4000 rwxp ad556000 00:00 0
    ad697000-ad69a000 rwxp ad697000 00:00 0
    ad69a000-ad6e8000 rwxp ad69a000 00:00 0
    ad87d000-ad880000 rwxp ad87d000 00:00 0
    ad880000-ad8ce000 rwxp ad880000 00:00 0
    ad8ce000-ad8d1000 rwxp ad8ce000 00:00 0
    ad8d1000-ad91f000 rwxp ad8d1000 00:00 0
    ad970000-ad973000 rwxp ad970000 00:00 0
    ad973000-ad9c1000 rwxp ad973000 00:00 0
    adab4000-adab7000 rwxp adab4000 00:00 0
    adab7000-adb05000 rwxp adab7000 00:00 0
    adc00000-adc28000 rwxp adc00000 00:00 0
    adc28000-add00000 ---p adc28000 00:00 0
    ade00000-ade24000 rwxp ade00000 00:00 0
    ade24000-adf00000 ---p ade24000 00:00 0
    ae015000-ae018000 rwxp ae015000 00:00 0
    ae018000-ae066000 rwxp ae018000 00:00 0
    ae24c000-ae24f000 rwxp ae24c000 00:00 0
    ae24f000-ae29d000 rwxp ae24f000 00:00 0
    ae2ee000-ae2f1000 rwxp ae2ee000 00:00 0
    ae2f1000-ae33f000 rwxp ae2f1000 00:00 0
    ae400000-ae4a7000 rwxp ae400000 00:00 0
    ae4a7000-ae500000 ---p ae4a7000 00:00 0
    ae669000-ae66c000 ---p ae669000 00:00 0
    ae66c000-ae6ba000 rwxp ae66c000 00:00 0
    ae7fe000-ae801000 rwxp ae7fe000 00:00 0
    ae801000-ae84f000 rwxp ae801000 00:00 0
    ae8a0000-ae8a3000 rwxp ae8a0000 00:00 0
    ae8a3000-ae8f1000 rwxp ae8a3000 00:00 0
    ae942000-ae945000 ---p ae942000 00:00 0
    ae945000-ae993000 rwxp ae945000 00:00 0
    aead7000-aeada000 ---p aead7000 00:00 0
    aeada000-aeb28000 rwxp aeada000 00:00 0
    aeb28000-aeb2b000 rwxp aeb28000 00:00 0
    aeb2b000-aeb79000 rwxp aeb2b000 00:00 0
    aebca000-aebcd000 rwxp aebca000 00:00 0
    aebcd000-aec1b000 rwxp aebcd000 00:00 0
    aec1b000-aec1e000 rwxp aec1b000 00:00 0
    aec1e000-aec6c000 rwxp aec1e000 00:00 0
    aed00000-aed22000 rwxp aed00000 00:00 0
    aed22000-aee00000 ---p aed22000 00:00 0
    af0da000-af0dd000 rwxp af0da000 00:00 0
    af0dd000-af12b000 rwxp af0dd000 00:00 0
    af21e000-af221000 rwxp af21e000 00:00 0
    af221000-af26f000 rwxp af221000 00:00 0
    af26f000-af272000 rwxp af26f000 00:00 0
    af272000-af2c0000 rwxp af272000 00:00 0
    af5ea000-af5ed000 ---p af5ea000 00:00 0
    af5ed000-af63b000 rwxp af5ed000 00:00 0
    af72e000-af731000 rwxp af72e000 00:00 0
    af731000-af77f000 rwxp af731000 00:00 0
    af77f000-af782000 ---p af77f000 00:00 0
    af782000-af7d0000 rwxp af782000 00:00 0
    af7d0000-af7d3000 rwxp af7d0000 00:00 0
    af7d3000-af821000 rwxp af7d3000 00:00 0
    af872000-af875000 ---p af872000 00:00 0
    af875000-af8c3000 rwxp af875000 00:00 0
    afafa000-afafd000 rwxp afafa000 00:00 0
    afafd000-afb4b000 rwxp afafd000 00:00 0
    afb4b000-afb4e000 rwxp afb4b000 00:00 0
    afb4e000-afb9c000 rwxp afb4e000 00:00 0
    afc3e000-afc41000 rwxp afc3e000 00:00 0
    afc41000-afc8f000 rwxp afc41000 00:00 0
    afce0000-afce3000 rwxp afce0000 00:00 0
    afce3000-afd31000 rwxp afce3000 00:00 0
    afd82000-afd85000 rwxp afd82000 00:00 0
    afd85000-afdd3000 rwxp afd85000 00:00 0
    afe75000-afe78000 rwxp afe75000 00:00 0
    afe78000-afec6000 rwxp afe78000 00:00 0
    afec6000-afec9000 rwxp afec6000 00:00 0
    afec9000-aff17000 rwxp afec9000 00:00 0
    aff17000-aff1a000 rwxp aff17000 00:00 0
    aff1a000-aff68000 rwxp aff1a000 00:00 0
    affb9000-affbc000 ---p affb9000 00:00 0
    affbc000-b000a000 rwxp affbc000 00:00 0
    b014e000-b0151000 rwxp b014e000 00:00 0
    b0151000-b019f000 rwxp b0151000 00:00 0
    b019f000-b01a2000 rwxp b019f000 00:00 0
    b01a2000-b01f0000 rwxp b01a2000 00:00 0
    b0292000-b0295000 rwxp b0292000 00:00 0
    b0295000-b02e3000 rwxp b0295000 00:00 0
    b02e3000-b02e6000 rwxp b02e3000 00:00 0
    b02e6000-b0334000 rwxp b02e6000 00:00 0
    b0334000-b0337000 rwxp b0334000 00:00 0
    b0337000-b0385000 rwxp b0337000 00:00 0
    b0385000-b0388000 rwxp b0385000 00:00 0
    b0388000-b03d6000 rwxp b0388000 00:00 0
    b051a000-b051d000 ---p b051a000 00:00 0
    b051d000-b056b000 rwxp b051d000 00:00 0
    b060d000-b0610000 ---p b060d000 00:00 0
    b0610000-b065e000 rwxp b0610000 00:00 0
    b065e000-b0661000 rwxp b065e000 00:00 0
    b0661000-b06af000 rwxp b0661000 00:00 0
    b0700000-b0703000 rwxp b0700000 00:00 0
    b0703000-b0751000 rwxp b0703000 00:00 0
    b0844000-b0847000 rwxp b0844000 00:00 0
    b0847000-b0895000 rwxp b0847000 00:00 0
    b0895000-b0898000 rwxp b0895000 00:00 0
    b0898000-b08e6000 rwxp b0898000 00:00 0
    b0988000-b098b000 rwxp b0988000 00:00 0
    b098b000-b09d9000 rwxp b098b000 00:00 0
    b09d9000-b09dc000 rwxp b09d9000 00:00 0
    b09dc000-b0a2a000 rwxp b09dc000 00:00 0
    b0a2a000-b0a2d000 rwxp b0a2a000 00:00 0
    b0a2d000-b0a7b000 rwxp b0a2d000 00:00 0
    b0a7b000-b0a7e000 rwxp b0a7b000 00:00 0
    b0a7e000-b0acc000 rwxp b0a7e000 00:00 0
    b0b1d000-b0b20000 ---p b0b1d000 00:00 0
    b0b20000-b0b6e000 rwxp b0b20000 00:00 0
    b0bbf000-b0bc2000 rwxp b0bbf000 00:00 0
    b0bc2000-b0c10000 rwxp b0bc2000 00:00 0
    b0c10000-b0c13000 rwxp b0c10000 00:00 0
    b0c13000-b0c61000 rwxp b0c13000 00:00 0
    b0d03000-b0d06000 ---p b0d03000 00:00 0
    b0d06000-b0d54000 rwxp b0d06000 00:00 0
    b0d54000-b0d57000 ---p b0d54000 00:00 0
    b0d57000-b0da5000 rwxp b0d57000 00:00 0
    b0df6000-b0df9000 rwxp b0df6000 00:00

    Try this link to decode hs_err log file.
    http://openjdk.neojava.org/cgi-bin/hs_err
    Or download hs_err.pl perl script from the location mentioned in
    this blog.
    http://blogs.sun.com/dave/entry/a_tool_to_decode_hs

  • 64bit JVM Crash: interaction between Concurrent GC and HotSpot Compiler?

    During a test of a standalone server application, the application experienced a JVM crash for with no readily identifiable cause. We would like to at least come up with a theory to explain the crash; right now we have no idea what happened. We haven't found anything similar on the web so I am posting this to see if anybody has come across a similar problem.
    The OS was 64bit Windows Server 2003 family on an AMD machine with 8 hyperthreaded CPUs (4 dual-core, I believe). The JVM was a 1.5.0 Server VM. An excerpt of the crash log is below.
    The closest report we can find is Bug ID 5048441 which identified intermittent crashes with the Concurrent MarkSweep GC enabled; somebody claimed to have seen the problem under 1.5 but didn't have much more to add.
    We haven't started heavily testing under 64bit just yet but we will be moving soon, so we would really appreciate hearing about similar experiences.
    # An unexpected error has been detected by HotSpot Virtual Machine:
    #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000805dd33, pid=5100,
    tid=5636
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_04-b05 mixed mode)
    # Problematic frame:
    # V  [jvm.dll+0x5dd33]
    ---------------  T H R E A D  ---------------
    Current thread (0x00000000004ab8e0):  JavaThread "CompilerThread1" daemon
    [_thread_in_native, id=5636]
    siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000000
    VM state:not at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread: None
    Heap
    par new generation   total 42496K, used 27395K [0x000000007fff0000,
    0x0000000082980000, 0x0000000082980000)
      eden space 42432K,  64% used [0x000000007fff0000, 0x0000000081ab0de8,
    0x0000000082960000)
      from space 64K,   0% used [0x0000000082960000, 0x0000000082960000,
    0x0000000082970000)
      to   space 64K,   0% used [0x0000000082970000, 0x0000000082970000,
    0x0000000082980000)
    concurrent mark-sweep generation total 30677440K, used 4502193K
    [0x0000000082980000, 0x00000007d2ff0000, 0x00000007d2ff0000)
    concurrent-mark-sweep perm gen total 21248K, used 12956K [0x00000007d2ff0000,
    0x00000007d44b0000, 0x00000007d6ff0000)
    VM Arguments:
    jvm_args: -Xms30000M -Xmx30000M -enableassertions -XX:+PrintGCDetails
    -XX:MaxPermSize=64M -XX:+PrintGCTimeStamps -XX:+UseParNewGC
    -XX:+UseConcMarkSweepGC -Xdebug
    -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5432 -verbose:gc
    ---------------  S Y S T E M  ---------------
    OS: Windows Server 2003 family Build 3790 Service Pack 1
    CPU:total 8 em64t ht
    Memory: 4k page, physical 33549364k(27243692k free), swap 45158008k(12023400k free)
    vm_info: Java HotSpot(TM) 64-Bit Server VM (1.5.0_04-b05) for windows-amd64,
    built on Jun  3 2005 01:43:58 by "java_re" with unknown MS VC++:1400

    I had a very similar experience on yesterday. It accepted my username and password, went to the next screen and when I clicked on transaction management, the screen went back to my desktop and and a notebook appeared (hs_err_pid3960). This is what the top portion said:
    An unexpected error has been detected by HotSpot Virtual Machine:
    Exception_Access_Violation (0xc0000005) at pc=0x6d700e9d, pid=3960, tid=968
    Java VM: Java HotSpot (TM) Client VM (1.5.0_04-b05 mixed mode, sharing)
    Problematic frame:
    V [jvm.dll=0xc0e9d]
    THREAD
    Current thread (0x029ec868): VMThread [id=968]
    siginfo: ExceptionCode=0xc0000005, reading address
    This report is 4 pages long but it includes info on Registers, Top of Stack (sp=0x066cfba0), Instructions (pc=0x6d700e9d), Stack [0x065d0000, 0x066d0000), sp=0x066cfbao, free space=1022k. There were several Java Threads listed but they all said "thread blocked" and then listed an id=------. VM state: at safepoint (normal execution). It had Libraries, VM Arguments -xbootclasspath/a:c:\, and a few others. Environmental variables Path=C:
    Processor_Identifier=x86 Family 15 Model 4 Stepping 1, GenuineIntel
    OS: Windows XP Build 2600 Service Pack 2
    CPU: total 2 family 15
    Memory: 4k page,
    vm_info: Java HotSpot (TM) Client VM (1.5.0_04-b05) for windows-x86, built on June 3, 2005 by "java_re" with MS VC++ 6.0
    Now that I have given this info to you, how do I eleviate the problem so that I will be able to do my job. Without this function, I cannot perform an important part of my job. I am hoping that this info will help you all find out the problem and prevent it from continuing to happen.

  • How to read the memory/memdbg log with regards to an OOM?

    We're trying to migrate from Sun JVM 1.5.0_14 to JRockit R27.5.0 running WebLogic 9.2.2 on Sun Solaris 10 on T2000 boxes (8 cores and 32G of memory). When specifying an pausetime target of 200ms (-server -Xms1g -Xmx1g -XgcPrio:pausetime -Xpausetarget=200ms -XXgcThreads:4 -XXoptThreads:4 -XXnoSystemGC -XXallocPrefetch) we're running in OOMs like
    <17.03.2008 18.17 Uhr CET> <Notice> <Log Management> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
    Exception in thread "Dispatcher-Thread-19" java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 7057136, Num elements: 3528560
         at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:393)
         at java.lang.StringBuilder.append(StringBuilder.java:120)
         at java.lang.Thread.run(Thread.java:595)
    <17.03.2008 18.17 Uhr CET> <Notice> <Cluster> <BEA-000138> <Listening for announcements from cluster PT_VK_Cluster on 239.255.255.22:7001.> From the corresponding memory/memdbg log:
    [memory ][Mon Mar 17 17:45:30 2008][15174] Running with 32 bit heap and compressed references.
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Memory layout after heap allocation:
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] ' ' - free, '-' - OS reserved range, 'r' - reserved,  'x' - committed,
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] '+' - committed/read, 'e' - committed/executable
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 'J' - java heap, 'j' - java heap data structures
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 00 0.00Gb -jjjjjjjj+jJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 10 0.25Gb JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 20 0.50Gb JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 30 0.75Gb JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 40 1.00Gb JJJJJJJJJJJ                                                    
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 50 1.25Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 60 1.50Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 70 1.75Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 80 2.00Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] 90 2.25Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] a0 2.50Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] b0 2.75Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] c0 3.00Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] d0 3.25Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] e0 3.50Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] f0 3.75Gb                                                                
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Minimum TLA size is 2048 bytes
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Preferred TLA size is 40960 bytes
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Large object limit is 2048 bytes
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Minimal blocksize on the freelist is 2048 bytes
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Initial and maximum number of gc threads: 4, of which 4 parallel threads, 4 concurrent threads, and 4 yc threads.
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Maximum nursery percentage of free heap is: 95.
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Prefetch distance in balanced tree: 4
    [compact][Mon Mar 17 17:45:30 2008][15174] Compactset limit: 299900, Using matrixes: 1, Static: 0
    [memory ][Mon Mar 17 17:45:30 2008][15174] GC mode: Garbage collection optimized for short pausetimes, initial strategy: Concurrent Mark & Sweep
    [memory ][Mon Mar 17 17:45:30 2008][15174] heap size: 1048576K, maximal heap size: 1048576K
    [memory ][Mon Mar 17 17:45:30 2008][15174] < s>-<end>: GC <before>K-><after>K (<heap>K), <pause> ms
    [memory ][Mon Mar 17 17:45:30 2008][15174] <s/start> - start time of collection (seconds since jvm start)
    [memory ][Mon Mar 17 17:45:30 2008][15174] <end>     - end time of collection (seconds since jvm start)
    [memory ][Mon Mar 17 17:45:30 2008][15174] <before>  - memory used by objects before collection (KB)
    [memory ][Mon Mar 17 17:45:30 2008][15174] <after>   - memory used by objects after collection (KB)
    [memory ][Mon Mar 17 17:45:30 2008][15174] <heap>    - size of heap after collection (KB)
    [memory ][Mon Mar 17 17:45:30 2008][15174] <pause>   - total sum of pauses during collection (milliseconds)
    [memory ][Mon Mar 17 17:45:30 2008][15174]             run with -Xverbose:gcpause to see individual pauses
    [memdbg ][Mon Mar 17 17:45:30 2008][15174] Using prefetch linesize: 16 bytes  chunks: 512 bytes pf_dist: 128 bytes
    [gcpause][Mon Mar 17 18:17:38 2008][15174] Thread "Dispatcher-Thread-19" id=79 idx=0x138 tid=78 was in object alloc 3524.102 ms from 1924.890 s
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] GC reason: Large object allocation failed (7057136 bytes), cause: Alloc Queue
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] Stopping of javathreads took 19.615 ms
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] old collection 181 started
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] Alloc Queue size before GC: 7057136, tlas: 0, oldest: 0
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] Compacting 8 heap parts at index 24 (type external) (exceptional false)
    [compact][Mon Mar 17 18:17:38 2008][15174] OC 181: 8 parts (max 512), index 24. Type external, (exceptional false)
    [compact][Mon Mar 17 18:17:38 2008][15174] Area start: 0x5b00000, end: 0x6b00000
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] Starting initial marking phase (OC1).
    [nursery][Mon Mar 17 18:17:38 2008][15174] KeepAreaStart: 0x150a5378 KeepAreaEnd: 0x1aeb8918
    [nursery][Mon Mar 17 18:17:38 2008][15174] Young collection started. This YC is a part of OC#181 initial marking.
    [nursery][Mon Mar 17 18:17:38 2008][15174] Setting mmNurseryMarker[0] to 0xf803880
    [nursery][Mon Mar 17 18:17:38 2008][15174] Setting mmNurseryMarker[1] to 0x150a5378
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] Initial marking phase promoted 2 objects (0K)
    [gcpause][Mon Mar 17 18:17:38 2008][15174] old collection phase 1 pause time: 124.798376 ms, (start time: 1928.495 s)
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] Restarting of javathreads took 0.643 ms
    [memdbg ][Mon Mar 17 18:17:38 2008][15174] Starting concurrent marking phase (OC2).
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Starting precleaning phase (OC3).
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Stopping of javathreads took 19.442 ms
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Starting final marking phase (OC4).
    [nursery][Mon Mar 17 18:17:41 2008][15174] KeepAreaStart: 0x0 KeepAreaEnd: 0x0
    [nursery][Mon Mar 17 18:17:41 2008][15174] Young collection started. This YC is a part of OC#181 final marking.
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Final marking phase promoted 75597 objects (2829K)
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Removing 20 permanent work packets from pool, now 376 packets
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] total concurrent mark time: 3261.573 ms
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] ending marking phase
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] current error: -236.220931
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Requesting to run parallel sweep since Alloc Queue is non-empty.
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Will use parallel sweep instead of concurrent sweep due to request.
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] starting parallel sweeping phase
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 1473472 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 318008 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 60896 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 129984 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 1230896 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 1881216 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 262360 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 1293112 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 1031088 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 3392344 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 520800 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 94160 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 391624 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 2152 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 13144 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 1560688 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 3968 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 280952 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 232952 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 838880 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 142616 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 18656 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 103312 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 35496 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 374760 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 4216 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation added 1081256 bytes to the freelist
    [compact][Mon Mar 17 18:17:41 2008][15174] Evacuation found 9621 objects and moved 9587 objects
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] total sweep time: 57.433 ms
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] ending sweeping phase
    [nursery][Mon Mar 17 18:17:41 2008][15174] Setting mmNurseryMarker[0] to 0xf77a8c8
    [nursery][Mon Mar 17 18:17:41 2008][15174] Setting mmNurseryMarker[1] to 0x1501a790
    [nursery][Mon Mar 17 18:17:41 2008][15174] Nursery size increased from 0kb to 327680kb. Parts: 1177
    [gcpause][Mon Mar 17 18:17:41 2008][15174] Threads waited for memory 3398.495 ms starting at 1928.426 s
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Alloc Queue size after GC: 7057136, tlas: 0, oldest: 1
    [compact][Mon Mar 17 18:17:41 2008][15174] Average compact time ratio: 0.953058
    [compact][Mon Mar 17 18:17:41 2008][15174] Compaction pause: 36.552 (target 127.074), update ref pause: 2.080 (target 6.259)
    [compact][Mon Mar 17 18:17:41 2008][15174] Updated 16423 refs (limit: 54054).
    [compact][Mon Mar 17 18:17:41 2008][15174] Compaction ended at index 31, object end address 0x69f8058.
    [compact][Mon Mar 17 18:17:41 2008][15174] Summary: 181;24;31;8;2;0;36.552;127.074;2.080;6.259;16423;54054
    [compact][Mon Mar 17 18:17:41 2008][15174] External default parts to compact set to 16
    [compact][Mon Mar 17 18:17:41 2008][15174] Compaction too short, increasing compact ratio.
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] gc-trigger is 12.339 %
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Nursery size after OC: 335544320
    [gcpause][Mon Mar 17 18:17:41 2008][15174] old collection phase 4-0 pause time: 250.697868 ms, (start time: 1931.576 s)
    [gcpause][Mon Mar 17 18:17:41 2008][15174] (pause includes compaction: 36.552 ms (external), update ref: 2.080 ms)
    [memory ][Mon Mar 17 18:17:41 2008][15174] 1928.495-1931.826: GC 169061K->164649K (1048576K), sum of pauses 375.496 ms
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Page faults before GC: 4527, page faults after GC: 4527, pages in heap: 131072
    [finaliz][Mon Mar 17 18:17:41 2008][15174] (OC) Pending finalizers 0->2
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Restarting of javathreads took 0.726 ms
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] GC reason: Large object allocation failed (7057136 bytes), cause: Alloc Queue
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Stopping of javathreads took 19.403 ms
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] old collection 182 started
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Alloc Queue size before GC: 7057136, tlas: 0, oldest: 1
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Exceptional compaction at end of heap ordered. Requesting to run parallel sweep.
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Compacting 16 heap parts at index 496 (type external) (exceptional true)
    [compact][Mon Mar 17 18:17:41 2008][15174] OC 182: 16 parts (max 512), index 496. Type external, (exceptional true)
    [compact][Mon Mar 17 18:17:41 2008][15174] Area start: 0x40b00000, end: 0x42b00000
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Starting initial marking phase (OC1).
    [nursery][Mon Mar 17 18:17:41 2008][15174] KeepAreaStart: 0x1501a790 KeepAreaEnd: 0x1aabc2d0
    [nursery][Mon Mar 17 18:17:41 2008][15174] Young collection started. This YC is a part of OC#182 initial marking.
    [nursery][Mon Mar 17 18:17:41 2008][15174] Setting mmNurseryMarker[0] to 0xf77a8c8
    [nursery][Mon Mar 17 18:17:41 2008][15174] Setting mmNurseryMarker[1] to 0x1501a790
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Initial marking phase promoted 2351 objects (94K)
    [gcpause][Mon Mar 17 18:17:41 2008][15174] old collection phase 1 pause time: 126.584439 ms, (start time: 1931.874 s)
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Restarting of javathreads took 0.547 ms
    [memdbg ][Mon Mar 17 18:17:41 2008][15174] Starting concurrent marking phase (OC2).
    [compact][Mon Mar 17 18:17:42 2008][15174] Pointermatrix for thread '(GC Main Thread)' failed to extend beyond 43378 elements.
    [memdbg ][Mon Mar 17 18:17:42 2008][15174] Too many pointers, compaction aborted
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Starting precleaning phase (OC3).
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Stopping of javathreads took 19.425 ms
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Starting final marking phase (OC4).
    [nursery][Mon Mar 17 18:17:44 2008][15174] KeepAreaStart: 0x0 KeepAreaEnd: 0x0
    [nursery][Mon Mar 17 18:17:44 2008][15174] Young collection started. This YC is a part of OC#182 final marking.
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Final marking phase promoted 72379 objects (2561K)
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Removing 19 permanent work packets from pool, now 357 packets
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] total concurrent mark time: 3161.521 ms
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] ending marking phase
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] current error: -175.496244
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Requesting to run parallel sweep since Alloc Queue is non-empty.
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Will use parallel sweep instead of concurrent sweep due to request.
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] starting parallel sweeping phase
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] total sweep time: 18.785 ms
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] ending sweeping phase
    [nursery][Mon Mar 17 18:17:44 2008][15174] Setting mmNurseryMarker[0] to 0xf77aa98
    [nursery][Mon Mar 17 18:17:44 2008][15174] Setting mmNurseryMarker[1] to 0x14e0c8f0
    [nursery][Mon Mar 17 18:17:44 2008][15174] Nursery size increased from 0kb to 327680kb. Parts: 1263
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Alloc Queue size after GC: 7057136, tlas: 0, oldest: 2
    [compact][Mon Mar 17 18:17:44 2008][15174] Compaction pause: 0.000 (target 127.074), update ref pause: 0.000 (target 6.259)
    [compact][Mon Mar 17 18:17:44 2008][15174] Updated 43378 refs (limit: 54054).
    [compact][Mon Mar 17 18:17:44 2008][15174] Compaction ended at index 31, object end address 0x69f8058.
    [compact][Mon Mar 17 18:17:44 2008][15174] Summary: 182;24;31;16;0;1;0.000;127.074;0.000;6.259;43378;54054
    [compact][Mon Mar 17 18:17:44 2008][15174] Exceptional compaction size requested by allocator, not adjusting compact ratio.
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] gc-trigger is 12.339 %
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Nursery size after OC: 335544320
    [gcpause][Mon Mar 17 18:17:44 2008][15174] old collection phase 4-0 pause time: 235.501725 ms, (start time: 1934.832 s)
    [gcpause][Mon Mar 17 18:17:44 2008][15174] (pause includes compaction: 0.000 ms (no compaction), update ref: 0.000 ms)
    [memory ][Mon Mar 17 18:17:44 2008][15174] 1931.874-1935.068: GC 168649K->163056K (1048576K), sum of pauses 362.086 ms
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Page faults before GC: 4527, page faults after GC: 4527, pages in heap: 131072
    [finaliz][Mon Mar 17 18:17:44 2008][15174] (OC) Pending finalizers 0->2
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Restarting of javathreads took 0.667 ms
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] GC reason: Large object allocation failed (7057136 bytes), cause: Alloc Queue
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Stopping of javathreads took 20.308 ms
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] old collection 183 started
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Alloc Queue size before GC: 7057136, tlas: 0, oldest: 2
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Compacting 512 heap parts at index 0 (type internal) (exceptional true)
    [compact][Mon Mar 17 18:17:44 2008][15174] OC 183: 512 parts (max 512), index 0. Type internal, (exceptional true)
    [compact][Mon Mar 17 18:17:44 2008][15174] Area start: 0x2b00000, end: 0x42b00000
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Starting initial marking phase (OC1).
    [nursery][Mon Mar 17 18:17:44 2008][15174] KeepAreaStart: 0x14e0c8f0 KeepAreaEnd: 0x1a85bc48
    [nursery][Mon Mar 17 18:17:44 2008][15174] Young collection started. This YC is a part of OC#183 initial marking.
    [nursery][Mon Mar 17 18:17:44 2008][15174] Setting mmNurseryMarker[0] to 0xf77aa98
    [nursery][Mon Mar 17 18:17:44 2008][15174] Setting mmNurseryMarker[1] to 0x14e0c8f0
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Initial marking phase promoted 632 objects (18K)
    [gcpause][Mon Mar 17 18:17:44 2008][15174] old collection phase 1 pause time: 133.589550 ms, (start time: 1935.151 s)
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Restarting of javathreads took 0.465 ms
    [memdbg ][Mon Mar 17 18:17:44 2008][15174] Starting concurrent marking phase (OC2).
    [memdbg ][Mon Mar 17 18:17:48 2008][15174] Starting precleaning phase (OC3).
    [memdbg ][Mon Mar 17 18:17:48 2008][15174] Stopping of javathreads took 19.590 ms
    [memdbg ][Mon Mar 17 18:17:48 2008][15174] Starting final marking phase (OC4).
    [nursery][Mon Mar 17 18:17:48 2008][15174] KeepAreaStart: 0x0 KeepAreaEnd: 0x0
    [nursery][Mon Mar 17 18:17:48 2008][15174] Young collection started. This YC is a part of OC#183 final marking.
    [compact][Mon Mar 17 18:17:48 2008][15174] Pointermatrix for thread '(GC Worker Thread 3)' failed to extend beyond 25391 elements.
    [memdbg ][Mon Mar 17 18:17:48 2008][15174] Too many pointers, compaction aborted
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Final marking phase promoted 126119 objects (4396K)
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Removing 18 permanent work packets from pool, now 339 packets
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] total concurrent mark time: 4378.189 ms
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] ending marking phase
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] current error: -162.086164
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Requesting to run parallel sweep since Alloc Queue is non-empty.
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Will use parallel sweep instead of concurrent sweep due to request.
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] starting parallel sweeping phase
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] total sweep time: 18.297 ms
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] ending sweeping phase
    [nursery][Mon Mar 17 18:17:49 2008][15174] Setting mmNurseryMarker[0] to 0xf75c4e8
    [nursery][Mon Mar 17 18:17:49 2008][15174] Setting mmNurseryMarker[1] to 0x14dd6788
    [nursery][Mon Mar 17 18:17:49 2008][15174] Nursery size increased from 0kb to 327680kb. Parts: 1307
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Alloc Queue size after GC: 0, tlas: 0, oldest: 0
    [compact][Mon Mar 17 18:17:49 2008][15174] Compaction pause: 0.000 (target 127.074), update ref pause: 0.000 (target 6.259)
    [compact][Mon Mar 17 18:17:49 2008][15174] Updated 55716 refs: 0 inside compaction area, and 55716 outside (limit: 54054).
    [compact][Mon Mar 17 18:17:49 2008][15174] Compaction ended at index 31, object end address 0x69f8058.
    [compact][Mon Mar 17 18:17:49 2008][15174] Summary: 183;24;31;512;0;1;0.000;127.074;0.000;6.259;55716;54054
    [compact][Mon Mar 17 18:17:49 2008][15174] Exceptional compaction size requested by allocator, not adjusting compact ratio.
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] gc-trigger is 12.339 %
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Nursery size after OC: 335544320
    [gcpause][Mon Mar 17 18:17:49 2008][15174] old collection phase 4-0 pause time: 301.755262 ms, (start time: 1939.257 s)
    [gcpause][Mon Mar 17 18:17:49 2008][15174] (pause includes compaction: 0.000 ms (no compaction), update ref: 0.000 ms)
    [memory ][Mon Mar 17 18:17:49 2008][15174] 1935.151-1939.559: GC 226936K->166214K (1048576K), sum of pauses 435.345 ms
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Page faults before GC: 4527, page faults after GC: 4529, pages in heap: 131072
    [finaliz][Mon Mar 17 18:17:49 2008][15174] (OC) Pending finalizers 0->24
    [memdbg ][Mon Mar 17 18:17:49 2008][15174] Restarting of javathreads took 0.928 ms
    [gcpause][Mon Mar 17 18:17:49 2008][15174] Thread "Dispatcher-Thread-19" id=79 idx=0x138 tid=78 was in object alloc 11134.471 ms from 1928.426 s
    [memory ][Mon Mar 17 18:17:49 2008][15174] Throwing OutOfMemory: allocLargeObjectOrArray - Object size: 7057136, Num elements: 3528560Why was the thread "Dispatcher-Thread-19" not able to allocate 7057136 bytes within 11134.471 ms?
    Cheers,
    Torsten

    Is it expected by your application to create StringBuilders with more than 3 million characters in them?Yes. Our application (which runs happily for years now on Sun's JVM) is producing and consuming large data streams (mostly XML). So a StringBuilder producing a string of 3 million characters ist very common. Some XML data streams consumed from database clobs have up to 10 million characters. As I said, with Sun's JVM we have no problems with fragmentation or aborted compactions with a similar sized heap (-server -Xms1g -Xmx1g -Xmn256m -XX:MaxPermSize=256m -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=95 -XX:MaxTenuringThreshold=31 -XX:+UseParNewGC -XX:ParallelGCThreads=4 -XX:-BindGCTaskThreadsToCPUs -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC).
    Cheers,
    Torsten

  • Concurrent mode failure- low memory -- OutOfMemory Exception

    WebLogic Web-tier instance frequently causing low memory. Where GC logs shows
    Total time for which application threads were stopped: 0.7241688 seconds
    46903.931: [GC [1 CMS-initial-mark: 375909K(524288K)] 441597K(983040K), 0.1330196 secs]
    Total time for which application threads were stopped: 0.2807213 seconds
    46904.064: [CMS-concurrent-mark-start]
    46907.915: [CMS-concurrent-mark: 3.850/3.850 secs]
    46907.915: [CMS-concurrent-preclean-start]
    46907.952: [CMS-concurrent-preclean: 0.035/0.037 secs]
    46907.952: [CMS-concurrent-abortable-preclean-start]
    46938.527: [GC {Heap before gc invocations=218:
    par new generation total 458752K, used 458752K [0xaf800000, 0xcf800000, 0xcf800000)
    eden space 393216K, 100% used [0xaf800000, 0xc7800000, 0xc7800000)
    from space 65536K, 100% used [0xc7800000, 0xcb800000, 0xcb800000)
    to space 65536K, 0% used [0xcb800000, 0xcb800000, 0xcf800000)
    concurrent mark-sweep generation total 524288K, used 375909K [0xcf800000, 0xef800000, 0xef800000)
    concurrent-mark-sweep perm gen total 131072K, used 97681K [0xef800000, 0xf7800000, 0xf7800000)
    46938.528: [ParNew (promotion failed): 458752K->458752K(458752K), 2.2293867 secs]46940.757: [CMS46941.097: [CMS-concurrent-abortable-preclean: 8.221/33.145 secs]
    (concurrent mode failure): 500649K->524287K(524288K), 9.2616939 secs] 834661K->599632K(983040K)Heap after gc invocations=219:
    par new generation total 458752K, used 75345K [0xaf800000, 0xcf800000, 0xcf800000)
    eden space 393216K, 19% used [0xaf800000, 0xb4194420, 0xc7800000)
    from space 65536K, 0% used [0xc7800000, 0xc7800000, 0xcb800000)
    to space 65536K, 0% used [0xcb800000, 0xcb800000, 0xcf800000)
    concurrent mark-sweep generation total 524288K, used 524287K [0xcf800000, 0xef800000, 0xef800000)
    concurrent-mark-sweep perm gen total 131072K, used 96428K [0xef800000, 0xf7800000, 0xf7800000)
    , 11.4919889 secs]
    For your reference JVM arguments for these instances are as follows:
    -Xms1024m -Xmx1024m
    -XX:NewSize=512m
    -XX:MaxNewSize=512m
    -XX:SurvivorRatio=6
    -Xnoclassgc -XX:PermSize=128m
    -XX:MaxPermSize=128m
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:+DisableExplicitGC
    -XX:+UseConcMarkSweepGC
    -verbose:gc -XX:+PrintGCTimeStamps
    -XX:+PrintGCDetails
    -XX:+PrintGCApplicationStoppedTime
    -XX:+PrintHeapAtGC
    -Xloggc:gc.log
    -XX:MaxPermSize=128m
    JDK: Sun JDK 1.5
    Operating Environment: Solaris 5.10
    WebLogic version 9.2 MP3
    Thread dump shows:
    "[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x009679c0 nid=0x26 waiting for monitor entry [0xa3e7e000..0xa3e7f970]
    at weblogic.utils.io.Chunk.<init>(Chunk.java:293)
    at weblogic.utils.io.Chunk.getChunk(Chunk.java:141)
    at weblogic.utils.io.ChunkedOutputStream.advance(ChunkedOutputStream.java:52)
    at weblogic.utils.io.ChunkedOutputStream.write(ChunkedOutputStream.java:65)
    at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1685)
    We had applied patches as suggested by Oracle WebLogic. But, still issue persists :(
    Any help on this issue appriciated.

    1) Yes, a System.gc() will be reported by cms as a concurrent mode failure.
    2) See if the release you're using has the flag -XX:+ExplicitGCInvokesConcurrent.
    This will tell cms to start a concurrent collection when a System.gc() is involved.
    It is in jdk6. It may still not do what you want if the the concurrent collections
    are happening too frequently.
    RMI schedules the full GC's because GC's are local to a JVM and
    the collector on one JVM (call it JVM A) does not know about state of the heap
    on another JVM (call it JVM B).
    3) I would have expected setting the gcInterval to
    a larger number would have had more of an affect. What was the value?
    Post a fragment of a gc log (-XX:+PrintGCDetails) that shows the concurrent
    mode failure. Maybe there's something else going on.

  • Weblogic GC log shows 32 bit heap??

    Hello Gurus,
    Following is the environment:
    OS: Windows 2008 R2 x64 bit
    We have installed Oracle's EPM suite on the server which is again 64 bit. We are currently having an OOM issue for Financial Reporting server and to understand the Garbage collection we enabled the Garbage collection logging.
    ISSUE: When we saw the GC log we found the following line in it --
    [INFO ][memory ] Running with 32 bit heap and compressed references.
    [INFO ][memory ] GC mode: Garbage collection optimized for short pausetimes, strategy: Generational Concurrent Mark & Sweep.I am confused with the first two lines in the log as shown above. What does it mean? Why is it telling that we are running 32 bit heap as we have 64 bit EPM installed in our environment.
    Any help will be highly appreciated.
    Thanks,
    hyperionEPM

    Do a java -version to get the WL bit
    and Which JVM are you using, is it Sun or Oracle
    or
    java -d64 -showversion - > will give
    Running a 64-bit JVM is not supported on this platform
    Edited by: Phani potumarti on May 29, 2013 4:01 AM
    Edited by: Phani potumarti on May 29, 2013 4:03 AM

  • Help interpreting verbose GC output

    Below is the output of a verbose GC using concurrent market sweep and parallel new GC's (XX:+UseParNewGC -XX:+UseConcMarkSweepGC)
    java version "1.4.2_07"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_07-b05)
    Java HotSpot(TM) Client VM (build 1.4.2_07-b05, mixed mode)
    I'd like to be able to understand this output better.
    What are the steps involved with concurrent market sweeps as outlined below? And how do I best read this?
    During a GC what state is the application in? My understanding is that is it effectively paused and not serving requests. Below suggests that the application has paused for 3+ seconds.
    290903.690: [GC [1 CMS-initial-mark: 714288K(1048576K)] 721114K(1298816K), 0.0211860 secs]
    290903.712: [CMS-concurrent-mark-start]
    290905.235: [CMS-concurrent-mark: 1.523/1.523 secs]
    290905.236: [CMS-concurrent-preclean-start]
    290905.236: [CMS-concurrent-preclean: 0.000/0.000 secs]
    290905.296: [GC290905.296: [Rescan (parallel) , 0.0478730 secs]
    290905.344: [weak refs processing, 0.0102490 secs] [1 CMS-remark: 714288K(1048576K)] 732512K(1298816K), 0.0586340 secs]
    290905.355: [CMS-concurrent-sweep-start]
    290906.948: [CMS-concurrent-sweep: 1.593/1.593 secs]
    290906.948: [CMS-concurrent-reset-start]
    290906.978: [CMS-concurrent-reset: 0.031/0.031 secs]
    The total time spent doing a GC appears to be
    initial mark           0.02
    concurrent mark      1.523
    preclean           0.00
    rescan           0.047
    weak refs           0.01
    remark           0.05
    sweep                1.59
    reset                0.031
    total gc time          3.271sec
    Is this a correct interpretation?
    Many thanks.

    Your are using the low pause garbage collector or Concurrent Mark Sweep (CMS). Some of those times are done without pauses. See http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/#11.2.1. Just two of those phases are stop the world pauses. According to the above article it is the:
    initial mark 0.02
    remark 0.05
    Total stop the world pause 0.07. The rest of them are down concurrenlty with the application still running.

Maybe you are looking for

  • Problem with shell commands and scripts from an Applescript Application

    Hi- I am fairly new to OSX software development. I am trying to build an application that creates a reverse SSH tunnel to my computer and starts OSXvnc. The idea is that my Mom or anyone else who needs help can connect to me without having to tinker

  • Creating report using catalog manager

    Hi All, Currently we have a option in catalog manager to create reports on catalog metadata like report names, path, etc. I can create a report by going into catalog manager -> options -> create report, Is there a way to automate this process other t

  • 5th gen ipod counter stuck at 0:00

    Already tried the 5 r's with no success. which is to say ipod restored and all music seems to be loaded, but will not play with counter stuck on 0:00. Any ideas?

  • German - English Dictionary to translate words in iBooks

    Anyone know of a way to install a German - English dictionary that can be used to define words within iBooks?  This functionality is in Kindle, would be useful to have it for my German iBooks as well.

  • Adding Tracks and being Organized Question?

    Hey All, Sometimes I download tracks to my desktop or move them over from a zip onto my desktop. Then I go to put them into a my iTunes library and the tracks are just added into my library on a one by one basis. What I'd really like is to be able to