Tomcat/Websphere JVM Crashes due to JCO / sapjcorfc.dll

We have developed a web application on Tomcat/Websphere that uses JCO to connect to SAP/R3 and request data.
When requesting more than aproximately 60.000 records the JVM crashes with:
Unhandled exception
Type=GPF vmState=0xffffffff
Target=20030915 (Windows XP 5.1 build 2600 Service Pack 2 x86)
ExceptionCode=0xc0000005 ExceptionAddress=0x0a4da787 ContextFlags=0x0001003f
Handler1=0x111016c0 Handler2=0x10019730
<b>Module=C:WINDOWSsystem32sapjcorfc.dll</b>
When monitoring the JCO with -Djco.trace_level=10 the last entry inside the log is
Servlet.Engine.Transports : 0 [11:35:27:234]: [JNI-LAYER] <b>RFC.reallocRecordBuffers</b>("/ACZENOS/S_BUFFILE"(00088EBC),0562DA60) enter [SUCCESS]
Finalizer thread [11:35:28:046]: [JAV_LAYER] JCO.Client.finalize() on 0
For me it looks like a memory problem on the JNI layer... but my JVM have sufficient memory(1GB) and there is left another GB free Mem.
Any ideas what to do would be helpful.

Kitten,
RAM 1GB is not mean that you have free space. Just do garbage collecting before processing. What I mean is break your records recording in such a way that, at a time you read only 50% of crashing capacity

Similar Messages

  • JVM Crashes due to memory shortage ...

    I have a Java application which runs as a server on my Windows 2000 Server machine having 2GB RAM and virtual memory specified as 2-4GB. I start the application with the options -Xms786m -Xms-786m.
    I have around 1500 clients that connect to this server and I open 2 dedicated server sockets for each client. The clients keep sending reuests over these connections for which they get the response over these connections. I connect to an Oracle database using the oracle 'thin' driver for this.
    Till recently everything was running fine until I made some very small changes such as addition of some 50-100 users and a few more request-response sets due to which I have run into trouble. My server keeps crashing as soon as it gets around 1600 clients and the memory used by the process reaches around 2GB(seen through Perfmon). Sometimes I get the error :
    Exception in thread �main� java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space
    and sometimes the jvm terminates without any error/message.
    I also profiled the server but there dont seem to be any problems with the objects on the heap. I know that this is because I am running short of native heap memory. How can I identify what is causing this and is there any way to increase this limit ?

    Did you mean you use -Xms786m -Xmx786m?
    If so, did you try increasing that to -Xms1024 -Xmx1024?
    I thought I read somewhere that there is a ultimate limit of 2G for JRE on 32-bit Windows.
    I would say that ultimately, if you find there is a JRE memory limitation that prevents a single JRE from using all available system memory, that you could always split your application into several separate JRE's that communicate with each other using the best interprocess service available. Split your app into an upper tier to handle all the client setup/teardown and a lower tier to handle other stuff. Or maybe some other logic split that works better for your application/design.

  • JVM crashing due to JNI call

    Hi All
    My application uses JNI to interact with other classes developed in C++. When I run the application ,which is comiled in java1.4, against java 1.5 ,it is crashing for any operation related to JNI. When I run the application with -Xint JVM option it worked fine. What is this Xint all about ..
    Thanks in advance
    Deepa K

    Most likely, you are holding on to a reference back into Java's managed memory that you should not be. The JIT is moving things around to optimize them (and 1.5 is probably doing other optimizations that 1.4 wasn't).
    Even if your app runs under 1.4, or with the JIT disabled under 1.5, it is not implemented properly. Are you properly releasing all array references on the JNI side? Are you managing threads properly?
    What exactly are you doing in the JNI code? Have you tried attaching a debugger to the native code to see exactly what is crashing? Worst case, but a bunch of print statements in there and run the Java app in a console so you can see the output...
    - K
    Hi Folks:
    You are right . Using -Xint will not crash the JVM in
    Java 1.5. I too had the same problems.
    Well by using -Xint we are switching off the Just in
    Time compilation in HotSpot VM . So everything will
    be interpreted rather then being compiled to native
    code at run time. But I really dont know whats
    happening in Java 1.5 because my program is running
    fine in with Java 1.4.02 without using -Xint option.
    -Kiran

  • JCo: "sapjcorfc.dll already loaded in another classloader"

    Hi,
    using Eclipse 3.2 I created some Java classes, which are using com.sap.mw.jco.*.
    Now I create two JSP projects, which using my Java classes.
    The JSPs are hosted by a Tomcat 5.5.
    The reason of the error above is written in SAP note 542175.
    Can somebody explain me please, which settings inside Eclipse 3.2 I have to make, that I can use my JSP applications at the same time.
    What's to do on the Tomcat server?
    In thread 15622 was written: "Add the library reference to the J2EE module ...", but how to do that?
    Any help would be appreciated!
    Regards,
    Peter

    Inside Eclipse open project properties, go to "Java Build Path" and then I pressed "Add variable", there I picked the SAPJCo.jar.
    On tomcat installation directory I added the SAPJCo.jar inside file directory
    "C:\Program Files\Apache Software Foundation\Tomcat 5.5\shared\lib"
    I also extended the environment PATH variable with SAP JCo installation directory "C:\Program Files\SAP\sapjco-ntintel-2.1.7"
    Hope this helps!
    Peter

  • JVM crashes on Tomcat Start up :  An unexpected error has been detected by

    My application is running fine on tomcat server from last 2 months, but when I did a restart JVM crashed.
    It happened twice, after that its restarted normally.
    ERROR Log
    # An unexpected error has been detected by HotSpot Virtual Machine:
    # Internal Error (53484152454432554E54494D450E43505001A3), pid=31446, tid=234257328
    # Java VM: Java HotSpot(TM) Server VM (1.5.0_07-b03 mixed mode)
    # An error report file with more information is saved as hs_err_pid31446.log
    # If you would like to submit a bug report, please visit:
    # http://java.sun.com/webapps/bugreport/crash.jsp
    contents of hs_err_pid31446.log
    # An unexpected error has been detected by HotSpot Virtual Machine:
    # Internal Error (53484152454432554E54494D450E43505001A3), pid=31446, tid=234257328
    # Java VM: Java HotSpot(TM) Server VM (1.5.0_07-b03 mixed mode)
    --------------- T H R E A D ---------------
    Current thread (0x0817bf48): JavaThread "main" [_thread_in_Java, id=31465]
    Stack: [0x0dee7000,0x0df68000), sp=0x0df62c8c, free space=495k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V [libjvm.so+0x500c93]
    V [libjvm.so+0x1c600c]
    V [libjvm.so+0x48f97f]
    V [libjvm.so+0x429a16]
    V [libjvm.so+0x426c24]
    --------------- P R O C E S S ---------------
    Java Threads: ( => current thread )
    0x077fcdc8 JavaThread "Connection Consumer for dest Subscription[subId=-  0x08b2df28 JavaThread "IdleRemover" daemon [_thread_blocked, id=31525]
    0x08b61070 JavaThread "TimeoutFactory" daemon [_thread_blocked, id=31524]
    0x078b0608 JavaThread "JCA PoolFiller" [_thread_blocked, id=31523]
    0x08b2d998 JavaThread "Timer-2" daemon [_thread_blocked, id=31522]
    0x0964f4f8 JavaThread "HSQLDB Timer @1f9b804" daemon [_thread_blocked, id=31521]
    0x062fcbe0 JavaThread "WorkManager(3)-1" daemon [_thread_blocked, id=31519]
    0x06eb7948 JavaThread "JBossMQ Cache Reference Softner" daemon [_thread_blocked, id=31515]
    0x081547b0 JavaThread "CompilerThread1" daemon [_thread_in_native, id=31462]
    0x08153708 JavaThread "CompilerThread0" daemon [_thread_in_native, id=31461]
    0x08152590 JavaThread "AdapterThread" daemon [_thread_blocked, id=31460]
    0x08151778 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=31459]
    0x08147500 JavaThread "Finalizer" daemon [_thread_blocked, id=31458]
    0x08146fb8 JavaThread "Reference Handler" daemon [_thread_blocked, id=31457]
    Other Threads:
    0x08144ae0 VMThread [id=31456]
    0x08157168 WatcherThread [id=31464]
    VM state:not at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread: None
    VM Arguments:
    jvm_args: -Xms1024m -Xmx3100m -XX:MaxPermSize=512m -verbose:gc -XX:+DisableExplicitGC -DAppDir= -Djboss.server.name=RatesServices-live-server4 -Dprogram.name
    =run.sh -Djava.endorsed.dirs=/ecomm/Jboss/RatesServices-live/lib/endorsed
    java_command: org.jboss.Main -c rbs
    Launcher Type: SUN_STANDARD
    Environment Variables:
    PATH=/usr/bin:/bin
    LD_LIBRARY_PATH=/opt/app/ecomm/java/jdk1.5.0_07/jre/lib/i386/server:/opt/app/ecomm/java/jdk1.5.0_07/jre/lib/i386:/opt/app/ecomm/java/jdk1.5.0_07/jre/../lib/i
    386:/opt/tibco/rv/lib:/ecomm/caf/DSlibs:/ecomm/caf/gcc/3.4.1/p4/lib:/ecomm/caf/boost/lib:/ecomm/caf/xerces/lib:/ecomm/caf/python/lib:/ecomm/caf/caf_4.8.0.17/
    build/linux_gcc_release/bin:/ecomm/caf/caf_4.8.0.17/build/linux_gcc_release/bin/lib/python2.2/config:/ecomm/caf/caf_4.8.0.17/build/linux_gcc_release/bin/lib:
    SHELL=/bin/sh
    Signal Handlers:
    SIGSEGV: [libjvm.so+0x5016a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGBUS: [libjvm.so+0x5016a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGFPE: [libjvm.so+0x426c00], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGPIPE: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
    SIGILL: [libjvm.so+0x426c00], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
    SIGUSR2: [libjvm.so+0x429050], sa_mask[0]=0x00000000, sa_flags=0x10000004
    SIGHUP: [libjvm.so+0x428a80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
    SIGQUIT: [libjvm.so+0x428a80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGTERM: [libjvm.so+0x428a80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    --------------- S Y S T E M ---------------
    OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
    uname:Linux 2.6.9-42.0.3.ELsmp #1 SMP Mon Sep 25 17:24:31 EDT 2006 x86_64
    libc:glibc 2.3.4 NPTL 2.3.4
    rlimit: STACK 10240k, CORE 0k, NPROC 131071, NOFILE 1024, AS infinity
    load average:2.20 1.03 0.62
    CPU:total 8 family 47, cmov, cx8, fxsr, mmx, sse, sse2, ht
    Memory: 4k page, physical 3968k(432k free), swap 2047k(2047k free)
    vm_info: Java HotSpot(TM) Server VM (1.5.0_07-b03) for linux-x86, built on May 3 2006 00:32:58 by java_re with gcc 3.2.1-7a (J2SE release)

    keep only the earlier versionKeep only the later version
    your problem will be resolvedYou don't know that. It's likely but not certain.

  • JVM Crashes Sporadically with ORA-24550 while Running Tomcat Web App

    We have experienced a problem that has been troubling us for some time now. Our website goes down sporadically as a result of a JVM crash. The Tomcat catalina.out log reads as follows:
    # A fatal error has been detected by the Java Runtime Environment:
    # SIGSEGV (0xb) at pc=0x00002aaac5938c2b, pid=2214, tid=47766235233056
    # JRE version: 7.0_07-b10
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.3-b01 mixed mode linux-amd64 compressed oops)
    # Problematic frame:
    # C [libclntsh.so.11.1+0x15f7c2b] nnfltimexp+0x1f
    # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    # An error report file with more information is saved as:
    # /usr/local/tomcat/tomcat-gbco2/logs/hs_err_pid2214.log
    # If you would like to submit a bug report, please visit:
    # http://bugreport.sun.com/bugreport/crash.jsp
    ORA-24550: signal received: [si_signo=6] [si_errno=0] [si_code=-6] [si_int=367714944] [si_ptr=0xffff810115eae280] [si_addr=0x69000008a6]
    kpedbg_dmp_stack()+360<-kpeDbgCrash()+192<-kpeDbgSignalHandler()+119<-skgesig_sigactionHandler()+218<-__sighandler()<-gsignal()+53<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0<-00002B71717BC1F0
    I am no expert in reading crash logs, but I can provide it if it would be helpful.
    We are running our app on
    Intel (R) Xeon (R) CPU X5550 @ 2.67GHz with 4 cpu cores.
    CentOS Release 5.7 (Linux xena.pcbi.upenn.edu 2.6.18-274.3.1.el5 #1 SMP Tue Sep 6 20:13:52 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux)
    Total server RAM: 24GB
    Tomcat 6.0.26
    Java 7.0_07-b10 (Java VM: Java HotSpot(TM) 64-Bit Server VM)
    Oracle Enterprise database server (11.2.0.3) - (separate server)
    OCI 11.2.0.3 (64-bit client)
    Our sqlnet.ora file contains the following:
    NAMES.DIRECTORY_PATH= (LDAP, TNSNAMES, EZCONNECT)
    DIAG_ADR_ENABLED=OFF
    DIAG_SIGHANDLER_ENABLED=FALSE
    DIAG_DDE_ENABLED=FALSE
    We read about issues with OCI 11.2.0.1 in the Oracle support website and we upgraded. But the issue remains. We are at a loss as to where to look/what to do next. Any suggestions would indeed be most welcome.
    Many thanks -
    Cris

    Chris,
    Another developer is reporting a very similar problem:
    http://www.jfree.org/phpBB2/viewtopic.php?t=10452
    Hopefully we can reduce this to a fairly simple and reproducible test case and then get it fixed.
    Regards,
    Dave Gilbert
    JFreeChart Project Leader

  • JVM crash stopping the Tomcat server

    Hello. I am currently having issues on a Linux RHEL AS 2.1 kernel 2.4.9 server, running Tomcat 5.0.28 and j2sdk1.4.2_06. We keep experiencing JVM crashes, and are not sure what the cause is. Not even sure where to begin to look for the cause. The only clue I can find is Hot Spot Virtual Machine error:11. Error 11 could be multiple things. Can anyone suggest where to start on this issue. Java is definately not my area of expertise so feel free to talk to me as if I was two. All OS hardware seems to be good. There are no indications of bad memory or any other hardware issues.

    Inadvertent duplicate post, reply to the other one.

  • Jrockit crashes due to outofmemory and illegal memory acces

    Hello,
    We have been using jrmc-3.1.2-1.6.0 and lately we are seeing JVM crashes after every couple days. Note: We have been seeing these crashes recently , for last 1 year we did not see such crashes.
    Following are different issues that we have seen over last few days:
    1. Crash due to illegal memory access
    2. Crash due to out of memory error
    3. Sever becoming unresponsive/Idle
    For illegal memory access and out of memory crashes, there was jRockit dump file created and Dump file seems to be pointing at Jrockit libjvm.so module for the exception/crash.
    Any help would be appreciated!
    *** Dump for Illegal Memory Access
    Error Message: Illegal memory access. [54]
    Signal info : si_signo=11, si_code=1 si_addr=0x10
    Version : BEA JRockit(R) R27.6.5-32_o-121899-1.6.0_14-20091001-2113-linux-ia32
    CPU : Intel Core i7 (HT) SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Core Intel64
    Number CPUs : 4
    Tot Phys Mem : 12762509312 (12171 MB)
    OS version : Red Hat Enterprise Linux Server release 5.6 (Tikanga)
    Linux version 2.6.18-164.15.1.el5PAE ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Mon Mar 1 11:14:09 EST 2010 (i686)
    Thread System: NPTL
    Java locking : Lazy unlocking enabled (class banning) (transfer banning)
    State : JVM is running
    Command Line : -Djava.util.logging.config.file=/usr/local/springsource/tcServer-6.0/zplus/conf/logging.properties -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Xmx2048m -Xms2048m -Djava.rmi.server.hostname=300714-web8.echovox.com -XgcPrio:pausetime -DTERRACOTTA_URL=338449-web10.echovox.com:9510,338450-web11.echovox.com:9510 -Dapp_version= -Xmanagement -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dehcache.monitor.enabled=True -Djava.endorsed.dirs=/usr/local/springsource/tcServer-6.0/tomcat-6.0.20.A/endorsed -Dcatalina.base=/usr/local/springsource/tcServer-6.0/zplus -Dcatalina.home=/usr/local/springsource/tcServer-6.0/tomcat-6.0.20.A -Djava.io.tmpdir=/usr/local/springsource/tcServer-6.0/zplus/temp -Dsun.java.launcher=SUN_STANDARD org.apache.catalina.startup.Bootstrap start
    java.home : /usr/local/jrmc-3.1.2-1.6.0/jre
    j.class.path : :/usr/local/springsource/tcServer-6.0/tomcat-6.0.20.A/bin/bootstrap.jar
    j.lib.path : /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/jrockit:/usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386:/usr/local/jrmc-3.1.2-1.6.0/jre/../lib/i386
    JAVA_HOME : <not set>
    JAVAOPTIONS: <not set>
    LD_LIBRARY_PATH: /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/jrockit:/usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386:/usr/local/jrmc-3.1.2-1.6.0/jre/../lib/i386
    LD_PRELOAD : <not set>
    LD_ASSUME_KERNEL: <not set>
    StackOverFlow: 0 StackOverFlowErrors have occured
    OutOfMemory : 0 OutOfMemoryErrors have occured
    C Heap : Good; no memory allocations have failed
    GC Strategy : Mode: pausetime. Currently using strategy: genconcon
    GC Status : OC currently running, in phase: marking. This is OC#2905.
    : YC is not running. Last finished YC was YC#100045.
    OC History : Strategy genconcon was used for OC#2776.
    : Strategy genconpar was used for OC#2777 to OC#2781.
    : Strategy genconcon was used for OC#2782 to OC#2783.
    : Strategy genconpar was used for OC#2784 to OC#2790.
    : Strategy genconcon was used for OC#2791 to OC#2905.
    YC History : Ran 1 YCs before OC#2901.
    : Ran 0 YCs before OC#2902.
    : Ran 1 YCs before OC#2903.
    : Ran 0 YCs before OC#2904.
    : Ran 2 YCs before OC#2905.
    YC Promotion : Last YC successfully promoted all objects
    Heap : 0x8b00000 - 0x88b00000 (Size: 2048 MB)
    Compaction : 0x49b00000 - 0x51b00000 (Current compaction type: internal)
    NurseryList : 0xaf67e18 - 0x8226c988
    KeepArea : 0x817031e8 - 0x8226c988
    NurseryMarker: [ 0x80d031f0,  0x817031e8 ]
    CompRefs : References are 32-bit.
    Registers (from ThreadContext: 0x96b75420 / OS context: 0x96b7551c):
    eax = 00000000 ecx = ffffffcc edx = 8b80eb78 ebx = 8b80eb78
    esp = 96b75814 ebp = 96b75960 esi = 96b75a48 edi = 00000000
    es = 0000007b cs = 00000073 ss = 0000007b ds = 0000007b
    fs = 00000000 gs = 00000033
    eip = b7d3a397 eflags = 00210296
    Loaded modules:
    (* denotes the module causing the exception)
    08048000-08058233 /usr/local/jrmc-3.1.2-1.6.0/jre/bin/java
    b7f24000-b7f2462b /usr/local/jrmc-3.1.2-1.6.0/jre/bin/java
    00754000-00768f17 /lib/libpthread.so.0
    006b2000-006d8a23 /lib/libm.so.6
    006ab000-006ad0fb /lib/libdl.so.2
    00550000-006a2723 /lib/libc.so.6
    00531000-0054b4f7 /lib/ld-linux.so.2
    b7c46000-b7e9fea7 */usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/jrockit/libjvm.so
    007a3000-007a9ebf /lib/librt.so.1
    b722f000-b723839b /lib/libnss_files.so.2
    b7118000-b71229bb /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libverify.so
    b70f3000-b7115f57 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libjava.so
    007ae000-007c27a7 /lib/libnsl.so.1
    b723e000-b7243ef0 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/native_threads/libhpi.so
    b59f0000-b59fe3e4 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libzip.so
    b5805000-b580a666 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libmanagement.so
    b5224000-b5236a18 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libnet.so
    b723c000-b723c6ad /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/librmi.so
    b2d42000-b2d45c2f /lib/libnss_dns.so.2
    b2d2e000-b2d3d74b /lib/libresolv.so.2
    b2d4a000-b2d503a4 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libnio.so
    b1f9d000-b1fc5857 /usr/local/springsource/tcServer-6.0/zplus/temp/tmpSigarJars841342613926113968186721535772/libsigar-x86-linux-1.6.4.so
    Stack:
    (* marks the word pointed to by the stack pointer)
    96b75814: b7cbce41* 00000000 006772e2 b04dbab8 006a4ff4 00000030
    96b7582c: 00000002 96b758a0 00000000 00000000 00010000 0061eea8
    96b75844: b7e48510 00010000 00000000 0061eeb9 b7d8f0f8 20000000
    96b7585c: 00010000 00000000 00004022 ffffffff 00000000 96b75890
    Code:
    (* marks the word pointed to by the instruction pointer)
    b7d3a364: c0a108ec e8b7f0fe ffffffa0 f0fed4a1 ff96e8b7 b8c9ffff
    b7d3a37c: 00000001 900debc3 90909090 90909090 90909090 8be58955
    b7d3a394: 8b5d0845* d2851050 0fc0950f b60fc0b6 768dc3c0 27bc8d00
    b7d3a3ac: 00000000 53e58955 0134ec81 9d8d0000 fffffee8 04245c89
    "RMI TCP Connection(idle)" id=440712 idx=0x7c4 tid=825 lastJavaFrame=0x96b75ecc
    Stack 0: start=0x96b54000, end=0x96b78000, guards=0x96b59000 (ok), forbidden=0x96b57000
    Thread Stack Trace:
    at jniExceptionCheck+7()@0xb7d3a397
    at cmgrGenerateCode+260()@0xb7cbe024
    at generate_code2+937()@0xb7da6c99
    at generate_code+97()@0xb7da6f11
    at get_runnable_codeinfo2+275()@0xb7da74c3
    at call_java+317()@0xb7d42f0d
    at jniInvoke+110()@0xb7d451be
    -- Java stack --
    at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
        at sun/reflect/NativeConstructorAccessorImpl.newInstance0(Ljava/lang/reflect/Constructor;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
        at sun/reflect/NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun/reflect/DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)[optimized]
    at java/lang/reflect/Constructor.newInstance(Constructor.java:513)[optimized]
    at java/lang/Class.newInstance0(Class.java:355)[inlined]
    at java/lang/Class.newInstance(Class.java:308)[optimized]
    at sun/reflect/MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:233)[inlined]
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:241)[inlined]
    at sun/reflect/MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)[optimized]
    at sun/reflect/MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
    at java/lang/reflect/Method.invoke(Method.java:597)[inlined]
    at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)[inlined]
    at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)[optimized]
    at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)[inlined]
    at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)[inlined]
    at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:326)[optimized]
    at sun/rmi/server/UnicastRef.marshalValue(UnicastRef.java:274)
    at sun/rmi/server/UnicastServerRef.dispatch(UnicastServerRef.java:315)
    at sun/rmi/transport/Transport$1.run(Transport.java:159)
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:255)[optimized]
    at sun/rmi/transport/Transport.serviceCall(Transport.java:155)
    at sun/rmi/transport/tcp/TCPTransport.handleMessages(TCPTransport.java:535)
    at sun/rmi/transport/tcp/TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at sun/rmi/transport/tcp/TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[inlined]
    at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)[optimized]
    at java/lang/Thread.run(Thread.java:619)[optimized]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    -- end of trace
    Extended, platform specific info:
    libc release: 2.5-stable
    Elf headers:
    libc ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00565fe0 PHOFF: 00000034 SHOFF: 0019ccbc EF: 0x0 HS: 52 PS: 32 PHN; 10 SS: 40 SHN: 75 STIDX: 74
    libpthread ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 00758850 PHOFF: 00000034 SHOFF: 00021474 EF: 0x0 HS: 52 PS: 32 PHN; 9 SS: 40 SHN: 40 STIDX: 39
    libjvm ehdrs: EI: 7f454c46010101000000000000000000 ET: 3 EM: 3 V: 1 ENTRY: 0004c3a0 PHOFF: 00000034 SHOFF: 012c9a58 EF: 0x0 HS: 52 PS: 32 PHN; 4 SS: 40 SHN: 29 STIDX: 26
    * If you see this dump, please go to *
    * http://edocs.bea.com/jrockit/go2troubleshooting.html *
    * for troubleshooting information. *
    ===== END DUMP ===============================================================
    ****** Dump for OutOfMemory error
    Error Message: Out of memory [68]
    Signal info : si_signo=11, si_code=1 si_addr=(nil)
    Fatal Error : Reference Iteration refIterInit src/jvm/code/runtime/refiter.c:167
    Version : BEA JRockit(R) R27.6.5-32_o-121899-1.6.0_14-20091001-2113-linux-ia32
    CPU : Intel Core i7 (HT) SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Core Intel64
    Number CPUs : 4
    Tot Phys Mem : 12762509312 (12171 MB)
    OS version : Red Hat Enterprise Linux Server release 5.6 (Tikanga)
    Linux version 2.6.18-164.15.1.el5PAE ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Mon Mar 1 11:14:09 EST 2010 (i686)
    Thread System: NPTL
    Java locking : Lazy unlocking enabled (class banning) (transfer banning)
    State : JVM is running
    Command Line : -Djava.util.logging.config.file=/usr/local/springsource/tcServer-6.0/zplus/conf/logging.properties -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Xmx2048m -Xms2048m -Djava.rmi.server.hostname=300714-web8.echovox.com -XgcPrio:pausetime -DTERRACOTTA_URL=338449-web10.echovox.com:9510,338450-web11.echovox.com:9510 -Dapp_version= -Xmanagement -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dehcache.monitor.enabled=True -Djava.endorsed.dirs=/usr/local/springsource/tcServer-6.0/tomcat-6.0.20.A/endorsed -Dcatalina.base=/usr/local/springsource/tcServer-6.0/zplus -Dcatalina.home=/usr/local/springsource/tcServer-6.0/tomcat-6.0.20.A -Djava.io.tmpdir=/usr/local/springsource/tcServer-6.0/zplus/temp -Dsun.java.launcher=SUN_STANDARD org.apache.catalina.startup.Bootstrap start
    java.home : /usr/local/jrmc-3.1.2-1.6.0/jre
    j.class.path : :/usr/local/springsource/tcServer-6.0/tomcat-6.0.20.A/bin/bootstrap.jar
    j.lib.path : /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/jrockit:/usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386:/usr/local/jrmc-3.1.2-1.6.0/jre/../lib/i386
    JAVA_HOME : <not set>
    JAVAOPTIONS: <not set>
    LD_LIBRARY_PATH: /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/jrockit:/usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386:/usr/local/jrmc-3.1.2-1.6.0/jre/../lib/i386
    LD_PRELOAD : <not set>
    LD_ASSUME_KERNEL: <not set>
    StackOverFlow: 0 StackOverFlowErrors have occured
    OutOfMemory : 0 OutOfMemoryErrors have occured
    C Heap : 1 memory allocations have failed
    : First failure was a mmMalloc of 20 bytes
    : Last failure was a mmMalloc of 20 bytes
    GC Strategy : Mode: pausetime. Currently using strategy: genconpar
    GC Status : OC is not running. Last finished OC was OC#1945.
    : YC is not running. Last finished YC was YC#90824.
    OC History : Strategy genconpar was used for OC#1745 to OC#1755.
    : Strategy genparpar was used for OC#1756 to OC#1757.
    : Strategy genconpar was used for OC#1758 to OC#1938.
    : Strategy genparpar was used for OC#1939 to OC#1940.
    : Strategy genconpar was used for OC#1941 to OC#1945.
    YC History : Ran 1 YCs before OC#1941.
    : Ran 6 YCs before OC#1942.
    : Ran 4 YCs before OC#1943.
    : Ran 6 YCs before OC#1944.
    : Ran 4 YCs before OC#1945.
    : Ran 5 YCs since last OC.
    YC Promotion : Last YC successfully promoted all objects
    Heap : 0x8100000 - 0x88100000 (Size: 2048 MB)
    Compaction : 0x64100000 - 0x68100020 (Current compaction type: internal)
    NurseryList : 0x8214640 - 0x84438f20
    KeepArea : (no keeparea in use)
    NurseryMarker: [ 0x827bfd10,  0x83996a00 ]
    CompRefs : References are 32-bit.
    Registers (from ThreadContext: 0x2a96c20 / OS context: 0x2a96d1c):
    eax = 00001267 ecx = 00000000 edx = 00000042 ebx = b7eb0de4
    esp = 02a97010 ebp = 02a97028 esi = 00000044 edi = 02a97078
    es = 0000007b cs = 00000073 ss = 0000007b ds = 0000007b
    fs = 00000000 gs = 00000033
    eip = b7cfca45 eflags = 00010206
    Loaded modules:
    (* denotes the module causing the exception)
    08048000-08058233 /usr/local/jrmc-3.1.2-1.6.0/jre/bin/java
    b7f44000-b7f4462b /usr/local/jrmc-3.1.2-1.6.0/jre/bin/java
    00754000-00768f17 /lib/libpthread.so.0
    006b2000-006d8a23 /lib/libm.so.6
    006ab000-006ad0fb /lib/libdl.so.2
    00550000-006a2723 /lib/libc.so.6
    00531000-0054b4f7 /lib/ld-linux.so.2
    b7c66000-b7ebfea7 */usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/jrockit/libjvm.so
    007a3000-007a9ebf /lib/librt.so.1
    b724f000-b725839b /lib/libnss_files.so.2
    b7138000-b71429bb /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libverify.so
    b7113000-b7135f57 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libjava.so
    007ae000-007c27a7 /lib/libnsl.so.1
    b725e000-b7263ef0 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/native_threads/libhpi.so
    b5972000-b59803e4 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libzip.so
    b5803000-b5808666 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libmanagement.so
    b51a8000-b51baa18 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libnet.so
    b725c000-b725c6ad /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/librmi.so
    b2dae000-b2db1c2f /lib/libnss_dns.so.2
    b2d9a000-b2da974b /lib/libresolv.so.2
    b47ae000-b47b43a4 /usr/local/jrmc-3.1.2-1.6.0/jre/lib/i386/libnio.so
    b1ec7000-b1eef857 /usr/local/springsource/tcServer-6.0/zplus/temp/tmpSigarJars551489478996731294326116727652638760/libsigar-x86-linux-1.6.4.so
    Stack:
    (* marks the word pointed to by the stack pointer)
    02a97010: b7ec5040* 00000200 b7eb0de4 02a97078 02a97078 02a97034
    02a97028: 02a97048 b7e78894 00000044 b7eb0de4 02a97078 00000001
    02a97040: 02a970d0 02a97110 02a97068 b7e788bf 00000044 b7eb0de4
    02a97058: 02a97078 b666692a 00000001 02a970d0 02a97088 b7e2147a
    Code:
    (* marks the word pointed to by the instruction pointer)
    b7cfca14: 4c892074 458b0824 2404c710 b7ec5040 0c244489 000200b8
    b7cfca2c: 24448900 b5dae804 01b80017 a3000000 b7ec5024 001267b8
    b7cfca44: 0000a300* 04c70000 00003f24 bdcae800 768d0017 27bc8d00
    b7cfca5c: 00000000 e589fc55 53c03157 b9e87d8d 00000004 00c0ec81
    "tomcat-http--119" id=24059 idx=0x60c tid=8771 lastJavaFrame=0xfffffffc
    Stack 0: start=0x2a74000, end=0x2a98000, guards=0x2a79000 (ok), forbidden=0x2a77000
    Thread Stack Trace:
    at dumpForceDump+117()@0xb7cfca45
    at vmFatalErrorMsgV+84()@0xb7e78894
    at vmFatalErrorMsg+31()@0xb7e788bf
    at fatalError+42()@0xb7e2147a
    at refIterInit+111()@0xb7e215bf
    at trProcessLocksForThread+41()@0xb7e300f9
    at get_all_locks+106()@0xb7d476ca
    at javaLockConvertLazyToThin+99()@0xb7d477b3
    at javaLockUnmatchedLock+802()@0xb7d488f2
    at jniMonitorEnter+48()@0xb7d66cf0
    at vmtiDetachFromThreadObject+85()@0xb7d9ec85
    at tsiThreadStub+147()@0xb7d9f0f3
    at ptiThreadStub+18()@0xb7e0e1d2
    at start_thread+226()@0x759832
    at __clone+94()@0x62245e
    -- Java stack --

    From the command-line (-Dehcache.monitor.enabled=True) you are using some form of caching.
    The out-of-memory occured as the JVM was unable the allocate an object: C Heap : 1 memory allocations have failed
    Could you check how the live data set is going (or the memory leak detector)
    Some concerns with regard to tune a JVM that runs a cache can be found here: http://middlewaremagic.com/weblogic/?p=7083
    Note that the example given discusses Coherence, but can be adopted for another caching mechanism as well.

  • JVM Crash: ConcMarkSweepGC related?

    We're experiencing the following JVM crash approximately 2-5 times per day on a well traffic application server running Tomcat 5.5 under Java 1.5 (05) on Linux with a 2.4.21-37 kernel. I'm posting here because we kind of have no route to take: I can't seem to work around the crash in any way. There's a potential that UseParallelGC might fix this, but I'm unable to run the app reliably with a ParallelGC, as we have a very large cache and the Full GC's get unbelievably long (2GB heap).
    How can we find out what's going on? Is there a way to obtain a debug build so that:
    V [libjvm.so+0x4c2c53]
    V [libjvm.so+0x1bea7c]
    We're also seeing this in the standard output:
    An exception occurred on the SessionReaperDaemon[null, 2005-11-01 20:02:14.81] thread while processing (task="com.tangosol.coherence.servlet.SessionHelper$SessionReaperDaemon$IncrementalReapTask@1ba4ac8"):
    java.lang.NullPointerException
         at com.tangosol.coherence.servlet.SessionHelper$SessionReaperDaemon$IncrementalReapTask.run(SessionHelper.java:3072)
         at com.tangosol.util.TaskDaemon.run(TaskDaemon.java:69)
         at com.tangosol.coherence.servlet.SessionHelper$SessionReaperDaemon.run(SessionHelper.java:2709)
         at com.tangosol.util.Daemon$1.run(Daemon.java:63)
    Prior to the crash, but this is an impossible place for an exception:
    3070: if (iter.hasNext())
    3071: {
    3072: String sId = (String) iter.next();3073: ...
    Any help or next steps would be greatly appreciated.
    An attached dump:
    # An unexpected error has been detected by HotSpot Virtual Machine:
    # Internal Error (4E4D4554484F440E4350500521), pid=1705, tid=907377584
    # Java VM: Java HotSpot(TM) Server VM (1.5.0_05-b05 mixed mode)
    --------------- T H R E A D ---------------
    Current thread (0x36bc7068): JavaThread "SessionReaperDaemon[null, 2005-11-01 22:38:23.388]" daemon [_thread_in_Java, id=1739]
    Stack: [0x360d7000,0x36158000), sp=0x36156eec, free space=511k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V [libjvm.so+0x4c2c53]
    V [libjvm.so+0x1bea7c]
    V [libjvm.so+0x3e2bee]
    V [libjvm.so+0x4564cb]
    V [libjvm.so+0x3f8956]
    V [libjvm.so+0x3f5c64]
    --------------- P R O C E S S ---------------
    Java Threads: ( => current thread )
    0x0894eba0 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=4871]
    0x0864ea88 JavaThread "RMI TCP Connection(341)-192.168.100.62" daemon [_thread_in_native, id=4802]
    0x083e6568 JavaThread "http-80-Processor300" daemon [_thread_in_native, id=4708]
    0x08d4e058 JavaThread "http-80-Processor299" daemon [_thread_in_native, id=4707]
    0x08bc7c20 JavaThread "http-80-Processor298" daemon [_thread_blocked, id=4706]
    0x0845fd78 JavaThread "http-80-Processor297" daemon [_thread_in_native, id=4705]
    0x08e39330 JavaThread "http-80-Processor296" daemon [_thread_in_native, id=4704]
    0x08bceb30 JavaThread "http-80-Processor295" daemon [_thread_blocked, id=4703]
    0x08bb9de8 JavaThread "http-80-Processor294" daemon [_thread_in_native, id=4702]
    0x08b692d0 JavaThread "http-80-Processor293" daemon [_thread_in_native, id=4701]
    0x084fa8e0 JavaThread "http-80-Processor292" daemon [_thread_in_native, id=4700]
    0x08446908 JavaThread "http-80-Processor291" daemon [_thread_blocked, id=4699]
    0x08e4c4c0 JavaThread "http-80-Processor290" daemon [_thread_in_native, id=4698]
    0x08625e60 JavaThread "http-80-Processor289" daemon [_thread_blocked, id=4697]
    0x0843d5b8 JavaThread "http-80-Processor288" daemon [_thread_in_native, id=4696]
    0x08ca13b8 JavaThread "http-80-Processor287" daemon [_thread_in_native, id=4695]
    0x08395e00 JavaThread "http-80-Processor286" daemon [_thread_in_native, id=4694]
    0x084c8f28 JavaThread "http-80-Processor285" daemon [_thread_in_native, id=4693]
    0x083f6490 JavaThread "http-80-Processor284" daemon [_thread_in_native, id=4692]
    0x08e69368 JavaThread "http-80-Processor281" daemon [_thread_blocked, id=4689]
    0x08df8218 JavaThread "http-80-Processor280" daemon [_thread_blocked, id=4688]
    0x0861ba28 JavaThread "http-80-Processor279" daemon [_thread_blocked, id=4687]
    0x083e22d0 JavaThread "http-80-Processor278" daemon [_thread_blocked, id=4686]
    0x08e44e80 JavaThread "http-80-Processor277" daemon [_thread_blocked, id=4685]
    0x08625478 JavaThread "http-80-Processor276" daemon [_thread_blocked, id=4684]
    0x08547890 JavaThread "http-80-Processor275" daemon [_thread_in_native, id=4632]
    0x0846c598 JavaThread "http-80-Processor274" daemon [_thread_in_native, id=4631]
    0x08beadb8 JavaThread "http-80-Processor270" daemon [_thread_in_native, id=4627]
    0x0865b3a0 JavaThread "http-80-Processor269" daemon [_thread_in_native, id=4626]
    0x092899e0 JavaThread "http-80-Processor268" daemon [_thread_in_native, id=4625]
    0x09277f58 JavaThread "http-80-Processor267" daemon [_thread_blocked, id=4624]
    0x086790b0 JavaThread "http-80-Processor265" daemon [_thread_in_native, id=4622]
    0x0844d920 JavaThread "http-80-Processor263" daemon [_thread_blocked, id=4620]
    0x0846f478 JavaThread "http-80-Processor262" daemon [_thread_in_native, id=4619]
    0x08c5f470 JavaThread "http-80-Processor261" daemon [_thread_in_native, id=4618]
    0x088cd8d0 JavaThread "http-80-Processor260" daemon [_thread_in_native, id=4617]
    0x08d5af30 JavaThread "http-80-Processor259" daemon [_thread_blocked, id=4616]
    0x0884bc18 JavaThread "http-80-Processor258" daemon [_thread_blocked, id=4615]
    0x08e618a0 JavaThread "http-80-Processor257" daemon [_thread_in_native, id=4614]
    0x08be3de0 JavaThread "http-80-Processor256" daemon [_thread_in_native, id=4613]
    0x085de680 JavaThread "http-80-Processor255" daemon [_thread_blocked, id=4612]
    0x083b8420 JavaThread "http-80-Processor254" daemon [_thread_blocked, id=4611]
    0x08e533e0 JavaThread "http-80-Processor253" daemon [_thread_in_native, id=4610]
    0x083c5af8 JavaThread "http-80-Processor252" daemon [_thread_in_native, id=4609]
    0x084701c0 JavaThread "http-80-Processor251" daemon [_thread_blocked, id=4608]
    0x3639cff0 JavaThread "http-80-Processor250" daemon [_thread_in_native, id=4402]
    0x2f41cd88 JavaThread "http-80-Processor249" daemon [_thread_in_native, id=4401]
    0x36bf7a48 JavaThread "http-80-Processor247" daemon [_thread_blocked, id=4399]
    0x35160748 JavaThread "http-80-Processor246" daemon [_thread_in_native, id=4398]
    0x36b42ed0 JavaThread "http-80-Processor245" daemon [_thread_blocked, id=4397]
    0x359b3ec0 JavaThread "http-80-Processor244" daemon [_thread_in_native, id=4396]
    0x3633a1f8 JavaThread "http-80-Processor240" daemon [_thread_blocked, id=4392]
    0x35a9ef28 JavaThread "http-80-Processor239" daemon [_thread_in_native, id=4391]
    0x35a52840 JavaThread "http-80-Processor238" daemon [_thread_blocked, id=4390]
    0x35a8e9e8 JavaThread "http-80-Processor236" daemon [_thread_blocked, id=4388]
    0x35a8de38 JavaThread "http-80-Processor235" daemon [_thread_in_native, id=4387]
    0x304703b8 JavaThread "http-80-Processor232" daemon [_thread_in_native, id=4384]
    0x36b32498 JavaThread "http-80-Processor231" daemon [_thread_blocked, id=4383]
    0x2e149a40 JavaThread "http-80-Processor230" daemon [_thread_blocked, id=4382]
    0x304731f8 JavaThread "http-80-Processor229" daemon [_thread_blocked, id=4381]
    0x2e186cd0 JavaThread "http-80-Processor228" daemon [_thread_in_native, id=4380]
    0x35a53c68 JavaThread "http-80-Processor227" daemon [_thread_blocked, id=4379]
    0x083a85d0 JavaThread "RMI TCP Connection(315)-192.168.100.62" daemon [_thread_blocked, id=4343]
    0x08cdd748 JavaThread "JMX server connection timeout 1017" daemon [_thread_blocked, id=3885]
    0x08df9fc0 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=3884]
    0x083c7f68 JavaThread "http-80-Processor225" daemon [_thread_blocked, id=3816]
    0x083c85b8 JavaThread "http-80-Processor223" daemon [_thread_blocked, id=3814]
    0x08df3be8 JavaThread "http-80-Processor222" daemon [_thread_in_native, id=3813]
    0x087a49b8 JavaThread "http-80-Processor221" daemon [_thread_in_native, id=3812]
    0x084c9db8 JavaThread "http-80-Processor220" daemon [_thread_in_native, id=3811]
    0x086203e0 JavaThread "http-80-Processor218" daemon [_thread_in_native, id=3809]
    0x088e2090 JavaThread "http-80-Processor216" daemon [_thread_in_native, id=3807]
    0x088bd2d0 JavaThread "http-80-Processor214" daemon [_thread_in_native, id=3805]
    0x08863490 JavaThread "http-80-Processor211" daemon [_thread_in_native, id=3802]
    0x08e9f4b8 JavaThread "http-80-Processor207" daemon [_thread_in_native, id=3798]
    0x08c9f748 JavaThread "http-80-Processor206" daemon [_thread_in_native, id=3797]
    0x0861fa00 JavaThread "http-80-Processor205" daemon [_thread_in_native, id=3796]
    0x08408ec8 JavaThread "http-80-Processor204" daemon [_thread_blocked, id=3795]
    0x0869bf28 JavaThread "http-80-Processor203" daemon [_thread_blocked, id=3794]
    0x08ba2648 JavaThread "http-80-Processor201" daemon [_thread_blocked, id=3792]
    0x08e70980 JavaThread "http-80-Processor200" daemon [_thread_blocked, id=3550]
    0x08e707e8 JavaThread "http-80-Processor199" daemon [_thread_blocked, id=3549]
    0x088e83b0 JavaThread "http-80-Processor196" daemon [_thread_blocked, id=3546]
    0x08634758 JavaThread "http-80-Processor195" daemon [_thread_blocked, id=3545]
    0x08198ea8 JavaThread "http-80-Processor194" daemon [_thread_blocked, id=3544]
    0x0831d5a8 JavaThread "http-80-Processor191" daemon [_thread_in_native, id=3541]
    0x088c6788 JavaThread "http-80-Processor189" daemon [_thread_in_native, id=3539]
    0x0835d868 JavaThread "http-80-Processor187" daemon [_thread_blocked, id=3537]
    0x08be2788 JavaThread "http-80-Processor186" daemon [_thread_in_native, id=3536]
    0x08d5b638 JavaThread "http-80-Processor182" daemon [_thread_blocked, id=3532]
    0x088c6488 JavaThread "http-80-Processor180" daemon [_thread_in_native, id=3530]
    0x08d83d70 JavaThread "http-80-Processor173" daemon [_thread_blocked, id=3158]
    0x088f3ca8 JavaThread "http-80-Processor171" daemon [_thread_blocked, id=3156]
    0x08862858 JavaThread "http-80-Processor168" daemon [_thread_blocked, id=3153]
    0x08629e38 JavaThread "http-80-Processor162" daemon [_thread_blocked, id=3147]
    0x086b9dd0 JavaThread "http-80-Processor159" daemon [_thread_in_native, id=3144]
    0x086e1c88 JavaThread "http-80-Processor153" daemon [_thread_in_native, id=3138]
    0x082fae10 JavaThread "http-80-Processor151" daemon [_thread_in_native, id=3136]
    0x084c7c40 JavaThread "http-80-Processor149" daemon [_thread_blocked, id=2489]
    0x087a5568 JavaThread "http-80-Processor147" daemon [_thread_in_native, id=2487]
    0x08c5f648 JavaThread "http-80-Processor146" daemon [_thread_blocked, id=2486]
    0x08c64090 JavaThread "http-80-Processor135" daemon [_thread_blocked, id=2475]
    0x08d60c68 JavaThread "http-80-Processor133" daemon [_thread_in_native, id=2473]
    0x08cdc058 JavaThread "http-80-Processor129" daemon [_thread_blocked, id=2469]
    0x0854a978 JavaThread "http-80-Processor128" daemon [_thread_blocked, id=2468]
    0x08d940c0 JavaThread "http-80-Processor126" daemon [_thread_in_native, id=2466]
    0x087c4998 JavaThread "http-80-Processor114" daemon [_thread_blocked, id=2186]
    0x0895ec90 JavaThread "http-80-Processor113" daemon [_thread_in_native, id=2185]
    0x0895e028 JavaThread "http-80-Processor112" daemon [_thread_blocked, id=2184]
    0x08e9bbb8 JavaThread "http-80-Processor82" daemon [_thread_in_native, id=2147]
    0x08634cf8 JavaThread "http-80-Processor80" daemon [_thread_in_native, id=2145]
    0x085d93c0 JavaThread "http-80-Processor68" daemon [_thread_blocked, id=2129]
    0x085d8658 JavaThread "http-80-Processor67" daemon [_thread_blocked, id=2128]
    0x085d7310 JavaThread "http-80-Processor66" daemon [_thread_in_native, id=2127]
    0x084c57e0 JavaThread "http-80-Processor60" daemon [_thread_blocked, id=2121]
    0x084c4a78 JavaThread "http-80-Processor59" daemon [_thread_blocked, id=2120]
    0x08588db8 JavaThread "http-80-Processor56" daemon [_thread_blocked, id=2117]
    0x086a2130 JavaThread "http-80-Processor54" daemon [_thread_in_native, id=2115]
    0x0878cfa8 JavaThread "http-80-Processor50" daemon [_thread_in_native, id=2108]
    0x087b9148 JavaThread "http-80-Processor39" daemon [_thread_blocked, id=2097]
    0x082f38c8 JavaThread "http-80-Processor29" daemon [_thread_blocked, id=2087]
    0x087a6a30 JavaThread "http-80-Processor28" daemon [_thread_in_native, id=2086]
    0x36b33540 JavaThread "http-80-Monitor" [_thread_blocked, id=1769]
    0x351178e0 JavaThread "http-80-Processor4" daemon [_thread_blocked, id=1747]
    0x35a72210 JavaThread "http-80-Processor1" daemon [_thread_blocked, id=1744]
    0x363ed3b0 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=1743]
    0x35a60238 JavaThread "Thread-4" [_thread_blocked, id=1742]
    =>0x36bc7068 JavaThread "SessionReaperDaemon[null, 2005-11-01 22:38:23.388]" daemon [_thread_in_Java, id=1739]
    0x363633c8 JavaThread "DistributedCache:DistributedSessions:EventDispatcher" daemon [_thread_blocked, id=1738]
    0x363771a0 JavaThread "DistributedCache:DistributedSessions" daemon [_thread_blocked, id=1737]
    0x36ba85f0 JavaThread "ReplicatedCache:ReplicatedSessionsMisc" daemon [_thread_blocked, id=1736]
    0x36bbb118 JavaThread "TcpRingListener" daemon [_thread_in_native, id=1735]
    0x36373478 JavaThread "Cluster" daemon [_thread_blocked, id=1733]
    0x36351268 JavaThread "PacketPublisher" daemon [_thread_blocked, id=1732]
    0x36350d60 JavaThread "PacketReceiver" daemon [_thread_blocked, id=1731]
    0x36ba8858 JavaThread "PacketListenerN" daemon [_thread_in_native, id=1730]
    0x36bb11e8 JavaThread "PacketListener1" daemon [_thread_in_native, id=1729]
    0x36b9f530 JavaThread "Logger@9508926 3.0/315" daemon [_thread_blocked, id=1728]
    0x081b0910 JavaThread "RMI TCP Accept-9000" daemon [_thread_in_native, id=1726]
    0x081b5c90 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=1725]
    0x081b57c0 JavaThread "Timer-0" daemon [_thread_blocked, id=1724]
    0x08157b60 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1722]
    0x081566e8 JavaThread "CompilerThread1" daemon [_thread_blocked, id=1721]
    0x08155698 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1720]
    0x08154578 JavaThread "AdapterThread" daemon [_thread_blocked, id=1719]
    0x08153618 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1718]
    0x081528e8 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=1717]
    0x08148480 JavaThread "Finalizer" daemon [_thread_blocked, id=1716]
    0x08148008 JavaThread "Reference Handler" daemon [_thread_blocked, id=1715]
    0x0805c598 JavaThread "main" [_thread_in_native, id=1705]
    Other Threads:
    0x08145bc8 VMThread [id=1714]
    0x082237b0 WatcherThread [id=1727]
    VM state:not at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread: None
    Heap
    par new generation total 523840K, used 289536K [0x3d1c0000, 0x5d1c0000, 0x5d1c0000)
    eden space 523392K, 55% used [0x3d1c0000, 0x4ec800f0, 0x5d0e0000)
    from space 448K, 0% used [0x5d150000, 0x5d150000, 0x5d1c0000)
    to space 448K, 0% used [0x5d0e0000, 0x5d0e0000, 0x5d150000)
    concurrent mark-sweep generation total 1318912K, used 521347K [0x5d1c0000, 0xad9c0000, 0xad9c0000)
    concurrent-mark-sweep perm gen total 30720K, used 30499K [0xad9c0000, 0xaf7c0000, 0xb19c0000)
    Dynamic libraries:
    08048000-08056000 r-xp 00000000 08:02 180163 /usr/java/jdk1.5.0_05/bin/java
    08056000-08058000 rwxp 0000e000 08:02 180163 /usr/java/jdk1.5.0_05/bin/java
    08058000-0aea7000 rwxp 00000000 00:00 0
    2af40000-2b067000 rwxp 00001000 00:00 0
    2b067000-2b06a000 ---p 00000000 00:00 0
    2b06a000-2b0e8000 rwxp 00003000 00:00 0
    2b0e8000-2b0eb000 ---p 00081000 00:00 0
    2b0eb000-2b169000 rwxp 00084000 00:00 0
    2b169000-2b16c000 ---p 00102000 00:00 0
    2b16c000-2b1ea000 rwxp 00105000 00:00 0
    2b1ea000-2b1ed000 ---p 00081000 00:00 0
    2b1ed000-2b26b000 rwxp 00084000 00:00 0
    2b26b000-2b26e000 ---p 00081000 00:00 0
    2b26e000-2b2ec000 rwxp 00084000 00:00 0
    2b2ec000-2b367000 r-xs 00000000 08:02 1784003 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xsdlib.jar
    2b367000-2b382000 r-xs 00000000 08:02 1784002 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xmlrpc-1.2-b1.jar
    2b382000-2b39d000 r-xs 00000000 08:02 1784001 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xml-apis.jar
    2b39d000-2b476000 r-xs 00000000 08:02 1784000 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xercesImpl.jar
    2b476000-2b554000 r-xs 00000000 08:02 1783999 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/xalan.jar
    2b554000-2b557000 ---p 00000000 00:00 0
    2b557000-2b5d5000 rwxp 00003000 00:00 0
    2b5d5000-2b5d8000 ---p 00081000 00:00 0
    2b5d8000-2b656000 rwxp 00084000 00:00 0
    2b656000-2b659000 ---p 00081000 00:00 0
    2b659000-2b6d7000 rwxp 00084000 00:00 0
    2b6d7000-2b6da000 ---p 00102000 00:00 0
    2b6da000-2b758000 rwxp 00105000 00:00 0
    2b758000-2b75b000 ---p 00183000 00:00 0
    2b75b000-2b7d9000 rwxp 00186000 00:00 0
    2b7d9000-2b7dc000 rwxp 00000000 00:00 0
    2b7dc000-2b85a000 rwxp 00207000 00:00 0
    2b85a000-2b85d000 ---p 00285000 00:00 0
    2b85d000-2b8db000 rwxp 00288000 00:00 0
    2b8db000-2b8de000 ---p 00102000 00:00 0
    2b8de000-2b95c000 rwxp 00105000 00:00 0
    2b95c000-2b95f000 ---p 00183000 00:00 0
    2b95f000-2ba5d000 rwxp 00186000 00:00 0
    2ba5d000-2ba60000 ---p 00182000 00:00 0
    2ba60000-2bade000 rwxp 00185000 00:00 0
    2bade000-2bae1000 ---p 00203000 00:00 0
    2bae1000-2bb5f000 rwxp 00206000 00:00 0
    2bb5f000-2bb62000 ---p 00284000 00:00 0
    2bb62000-2bbe0000 rwxp 00287000 00:00 0
    2bbe0000-2bbe3000 ---p 00305000 00:00 0
    2bbe3000-2bc61000 rwxp 00308000 00:00 0
    2bc61000-2bc64000 ---p 00204000 00:00 0
    2bc64000-2bce2000 rwxp 00207000 00:00 0
    2bce2000-2bce5000 ---p 00285000 00:00 0
    2bce5000-2bd63000 rwxp 00288000 00:00 0
    2bd63000-2bd66000 rwxp 00000000 00:00 0
    2bd66000-2bde4000 rwxp 00186000 00:00 0
    2bde4000-2bde7000 ---p 00204000 00:00 0
    2bde7000-2be65000 rwxp 00207000 00:00 0
    2be65000-2be68000 rwxp 00000000 00:00 0
    2be68000-2bee6000 rwxp 00084000 00:00 0
    2bee6000-2bee9000 ---p 00102000 00:00 0
    2bee9000-2bf67000 rwxp 00105000 00:00 0
    2bf67000-2bf6a000 ---p 00183000 00:00 0
    2bf6a000-2bfe8000 rwxp 00186000 00:00 0
    2bfe8000-2bfeb000 ---p 00204000 00:00 0
    2bfeb000-2c069000 rwxp 00207000 00:00 0
    2c069000-2c06c000 ---p 00081000 00:00 0
    2c06c000-2c0ea000 rwxp 00084000 00:00 0
    2c0ea000-2c0ed000 ---p 00102000 00:00 0
    2c0ed000-2c16b000 rwxp 00105000 00:00 0
    2c16b000-2c16e000 ---p 00183000 00:00 0
    2c16e000-2c1ec000 rwxp 00186000 00:00 0
    2c1ec000-2c1ef000 rwxp 00000000 00:00 0
    2c1ef000-2c26d000 rwxp 00207000 00:00 0
    2c26d000-2c270000 ---p 00285000 00:00 0
    2c270000-2c2ee000 rwxp 00288000 00:00 0
    2c2ee000-2c2f1000 ---p 00306000 00:00 0
    2c2f1000-2c36f000 rwxp 00309000 00:00 0
    2c36f000-2c372000 ---p 00387000 00:00 0
    2c372000-2c3f0000 rwxp 0038a000 00:00 0
    2c3f0000-2c3f3000 ---p 00204000 00:00 0
    2c3f3000-2c471000 rwxp 00207000 00:00 0
    2c471000-2c474000 ---p 00285000 00:00 0
    2c474000-2c4f2000 rwxp 00288000 00:00 0
    2c4f2000-2c4f5000 ---p 00102000 00:00 0
    2c4f5000-2c573000 rwxp 00105000 00:00 0
    2c573000-2c576000 ---p 00183000 00:00 0
    2c576000-2c5f4000 rwxp 00186000 00:00 0
    2c5f4000-2c5f7000 ---p 00204000 00:00 0
    2c5f7000-2c675000 rwxp 00207000 00:00 0
    2c675000-2c678000 ---p 00285000 00:00 0
    2c678000-2c6f6000 rwxp 00288000 00:00 0
    2c6f6000-2c6f9000 ---p 00306000 00:00 0
    2c6f9000-2c777000 rwxp 00309000 00:00 0
    2c777000-2c77a000 ---p 00387000 00:00 0
    2c77a000-2c7f8000 rwxp 0038a000 00:00 0
    2c7f8000-2c7fb000 ---p 00081000 00:00 0
    2c7fb000-2c879000 rwxp 00084000 00:00 0
    2c879000-2c87c000 ---p 00102000 00:00 0
    2c87c000-2c8fa000 rwxp 00105000 00:00 0
    2c8fa000-2c8fd000 ---p 00183000 00:00 0
    2c8fd000-2c97b000 rwxp 00186000 00:00 0
    2c97b000-2c97e000 rwxp 00000000 00:00 0
    2c97e000-2c9fc000 rwxp 00207000 00:00 0
    2c9fc000-2c9ff000 ---p 00285000 00:00 0
    2c9ff000-2ca7d000 rwxp 00288000 00:00 0
    2ca7d000-2ca80000 ---p 00285000 00:00 0
    2ca80000-2cafe000 rwxp 00288000 00:00 0
    2cafe000-2cb01000 rwxp 00000000 00:00 0
    2cb01000-2cb7f000 rwxp 00309000 00:00 0
    2cb7f000-2cb82000 ---p 00387000 00:00 0
    2cb82000-2cc58000 rwxp 0038a000 00:00 0
    2cc58000-2cd00000 ---p 00029000 00:00 0
    2cd7f000-2cd82000 ---p 00000000 00:00 0
    2cd82000-2cefe000 rwxp 00003000 00:00 0
    2cefe000-2cf00000 ---p 00029000 00:00 0
    2cf00000-2d0fe000 rwxp 0002b000 00:00 0
    2d0fe000-2d100000 ---p 00038000 00:00 0
    2d17b000-2d17e000 ---p 00000000 00:00 0
    2d17e000-2d1fc000 rwxp 00003000 00:00 0
    2d1fc000-2d1ff000 ---p 00081000 00:00 0
    2d1ff000-2d27d000 rwxp 00084000 00:00 0
    2d27d000-2d280000 rwxp 00000000 00:00 0
    2d280000-2d2fe000 rwxp 00105000 00:00 0
    2d2fe000-2d301000 ---p 00183000 00:00 0
    2d301000-2d37f000 rwxp 00186000 00:00 0
    2d37f000-2d382000 ---p 00204000 00:00 0
    2d382000-2d600000 rwxp 00207000 00:00 0
    2d67f000-2d682000 ---p 00000000 00:00 0
    2d682000-2e1ff000 rwxp 00003000 00:00 0
    2e1ff000-2e200000 ---p 00029000 00:00 0
    2e200000-2e25d000 rwxp 0002a000 00:00 0
    2e25d000-2e300000 ---p 00059000 00:00 0
    2e300000-2e700000 rwxp 00000000 00:00 0
    2e77f000-2e782000 ---p 00000000 00:00 0
    2e782000-30500000 rwxp 00003000 00:00 0
    3057f000-30582000 rwxp 00000000 00:00 0
    30582000-30700000 rwxp 00003000 00:00 0
    3077d000-30780000 ---p 00000000 00:00 0
    30780000-307fe000 rwxp 00003000 00:00 0
    307fe000-30801000 ---p 00081000 00:00 0
    30801000-3087f000 rwxp 00084000 00:00 0
    3087f000-30882000 ---p 00102000 00:00 0
    30882000-30a00000 rwxp 00105000 00:00 0
    30a7f000-30a82000 ---p 00000000 00:00 0
    30a82000-30ffc000 rwxp 00003000 00:00 0
    30ffc000-31000000 ---p 00029000 00:00 0
    31000000-311fc000 rwxp 00095000 00:00 0
    311fc000-31200000 ---p 00029000 00:00 0
    31200000-313fc000 rwxp 0002d000 00:00 0
    313fc000-31400000 ---p 000fd000 00:00 0
    31416000-31434000 r-xs 00000000 08:02 1783998 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/wsdl4j.jar
    31434000-31442000 r-xs 00000000 08:02 1783997 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/wrapper.jar
    31442000-3147f000 r-xs 00000000 08:02 1783995 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/webInstaller.jar
    3147f000-31482000 ---p 00000000 00:00 0
    31482000-315ff000 rwxp 00003000 00:00 0
    315ff000-31600000 ---p 00000000 00:00 0
    31625000-3167f000 r-xs 00000000 08:02 1783994 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/velocity-1.4-rc1.jar
    3167f000-31682000 ---p 00000000 00:00 0
    31682000-31dfb000 rwxp 00003000 00:00 0
    31dfb000-31e00000 ---p 00000000 00:00 0
    31e14000-31e17000 ---p 00000000 00:00 0
    31e17000-31e95000 rwxp 00003000 00:00 0
    31e95000-31e98000 ---p 00081000 00:00 0
    31e98000-31f16000 rwxp 00084000 00:00 0
    31f16000-31f19000 ---p 00102000 00:00 0
    31f19000-31f97000 rwxp 00105000 00:00 0
    31f97000-31f9a000 ---p 00183000 00:00 0
    31f9a000-32018000 rwxp 00186000 00:00 0
    32018000-3201b000 ---p 00183000 00:00 0
    3201b000-32099000 rwxp 00186000 00:00 0
    32099000-3209c000 ---p 00204000 00:00 0
    3209c000-3211a000 rwxp 00207000 00:00 0
    3211a000-3211d000 ---p 00285000 00:00 0
    3211d000-3219b000 rwxp 00288000 00:00 0
    3219b000-3219e000 ---p 00102000 00:00 0
    3219e000-3221c000 rwxp 00105000 00:00 0
    3221c000-3221f000 ---p 00183000 00:00 0
    3221f000-3229d000 rwxp 00186000 00:00 0
    3229d000-322a0000 ---p 00204000 00:00 0
    322a0000-3231e000 rwxp 00207000 00:00 0
    3231e000-32321000 ---p 00285000 00:00 0
    32321000-3239f000 rwxp 00288000 00:00 0
    3239f000-323a2000 ---p 00306000 00:00 0
    323a2000-32420000 rwxp 00309000 00:00 0
    32420000-32423000 ---p 00387000 00:00 0
    32423000-324a1000 rwxp 0038a000 00:00 0
    324a1000-324a4000 ---p 00408000 00:00 0
    324a4000-32522000 rwxp 0040b000 00:00 0
    32522000-32525000 ---p 00285000 00:00 0
    32525000-325a3000 rwxp 00288000 00:00 0
    325a3000-325a6000 ---p 00102000 00:00 0
    325a6000-32624000 rwxp 00105000 00:00 0
    32624000-32627000 ---p 00183000 00:00 0
    32627000-326a5000 rwxp 00186000 00:00 0
    326a5000-326a8000 ---p 00204000 00:00 0
    326a8000-32726000 rwxp 00207000 00:00 0
    32726000-32729000 ---p 00285000 00:00 0
    32729000-327a7000 rwxp 00288000 00:00 0
    327a7000-327aa000 ---p 00102000 00:00 0
    327aa000-32828000 rwxp 00105000 00:00 0
    32828000-3282b000 rwxp 00000000 00:00 0
    3282b000-328a9000 rwxp 00186000 00:00 0
    328a9000-328ac000 ---p 00204000 00:00 0
    328ac000-3292a000 rwxp 00207000 00:00 0
    3292a000-3292d000 ---p 00285000 00:00 0
    3292d000-329ab000 rwxp 00288000 00:00 0
    329ab000-329ae000 rwxp 00000000 00:00 0
    329ae000-32a2c000 rwxp 00309000 00:00 0
    32a2c000-32a2f000 ---p 00183000 00:00 0
    32a2f000-32aad000 rwxp 00186000 00:00 0
    32aad000-32ab0000 rwxp 00000000 00:00 0
    32ab0000-32b2e000 rwxp 00207000 00:00 0
    32b2e000-32b31000 ---p 00000000 00:00 0
    32b31000-32baf000 rwxp 00003000 00:00 0
    32baf000-32bb2000 ---p 00081000 00:00 0
    32bb2000-32c30000 rwxp 00084000 00:00 0
    32c30000-32c33000 ---p 00102000 00:00 0
    32c33000-32cb1000 rwxp 00105000 00:00 0
    32cb1000-32cb4000 rwxp 00000000 00:00 0
    32cb4000-32d32000 rwxp 00186000 00:00 0
    32d32000-32d35000 rwxp 00000000 00:00 0
    32d35000-32db3000 rwxp 00084000 00:00 0
    32db3000-32db6000 ---p 00102000 00:00 0
    32db6000-32e34000 rwxp 00105000 00:00 0
    32e34000-32e37000 rwxp 00000000 00:00 0
    32e37000-32eb5000 rwxp 00186000 00:00 0
    32eb5000-32eb8000 ---p 00204000 00:00 0
    32eb8000-32f36000 rwxp 00207000 00:00 0
    32f36000-32f39000 rwxp 00000000 00:00 0
    32f39000-32fb7000 rwxp 00288000 00:00 0
    32fb7000-32fba000 ---p 00306000 00:00 0
    32fba000-33038000 rwxp 00309000 00:00 0
    33038000-3303b000 rwxp 00000000 00:00 0
    3303b000-330b9000 rwxp 00186000 00:00 0
    330b9000-330bc000 ---p 00204000 00:00 0
    330bc000-3313a000 rwxp 00207000 00:00 0
    3313a000-3313d000 ---p 00285000 00:00 0
    3313d000-331bb000 rwxp 00288000 00:00 0
    331bb000-331be000 ---p 00102000 00:00 0
    331be000-3323c000 rwxp 00105000 00:00 0
    3323c000-3323f000 ---p 00183000 00:00 0
    3323f000-332bd000 rwxp 00186000 00:00 0
    332bd000-332c0000 ---p 00204000 00:00 0
    332c0000-3333e000 rwxp 00207000 00:00 0
    3333e000-33341000 ---p 00285000 00:00 0
    33341000-333bf000 rwxp 00288000 00:00 0
    333bf000-333c2000 ---p 00306000 00:00 0
    333c2000-33440000 rwxp 00309000 00:00 0
    33440000-33443000 ---p 00183000 00:00 0
    33443000-334c1000 rwxp 00186000 00:00 0
    334e2000-33531000 r-xs 00000000 08:02 1783993 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/torque-3.0.2.jar
    33531000-33537000 r-xs 00000000 08:02 1783992 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/textile4j-1.20.jar
    33537000-336e9000 r-xs 00000000 08:02 1783991 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/tangosol.jar
    336e9000-33710000 r-xs 00000000 08:02 1783990 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/ss_css2.jar
    33710000-33716000 r-xs 00000000 08:02 1783989 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/smtp.jar
    33716000-3371b000 r-xs 00000000 08:02 1783987 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/saaj.jar
    3371b000-33720000 r-xs 00000000 08:02 1783986 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/relaxngDatatype.jar
    33720000-33726000 r-xs 00000000 08:02 1783985 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/pop3.jar
    33726000-3373e000 r-xs 00000000 08:02 1783984 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/p6spy.jar
    3373e000-3375b000 r-xs 00000000 08:02 1783983 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/oscache-2.1.1.jar
    3375b000-337ff000 r-xs 00000000 08:02 1783982 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/optional.jar
    337ff000-3396a000 r-xs 00000000 08:02 1783981 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/msv.jar
    3396a000-33979000 r-xs 00000000 08:02 1783980 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/msutil.jar
    33979000-3398a000 r-xs 00000000 08:02 1783979 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/mssqlserver.jar
    3398a000-339d1000 r-xs 00000000 08:02 1783978 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/msbase.jar
    339d1000-339e8000 r-xs 00000000 08:02 1783977 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/metadata-extractor-2.2.2.jar
    339e8000-33a15000 r-xs 00000000 08:02 1783976 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/mailapi.jar
    33a15000-33a6c000 r-xs 00000000 08:02 1783975 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/log4j-1.2.8.jar
    33a6c000-33a6e000 r-xs 00000000 08:02 1783974 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jdbc2_0-stdext.jar
    33a6e000-33ab6000 r-xs 00000000 08:02 1783973 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jcs.jar
    33ab6000-33ac2000 r-xs 00000000 08:02 1783972 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jazzy-core.jar
    33ac2000-33acb000 r-xs 00000000 08:02 1783971 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jaxrpc.jar
    33acb000-33b06000 r-xs 00000000 08:02 1783970 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jaxen-1.1-beta-4.jar
    33b06000-33b0e000 r-xs 00000000 08:02 1783969 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jakarta-regexp-1.2.jar
    33b0e000-33b1e000 r-xs 00000000 08:02 1783968 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jakarta-oro-2.0.6.jar
    33b1e000-33c9f000 r-xs 00000000 08:02 1783967 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jai_core.jar
    33c9f000-33cd3000 r-xs 00000000 08:02 1783966 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/jai_codec.jar
    33cd3000-33d03000 r-xs 00000000 08:02 1783965 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/isorelax.jar
    33d03000-33d25000 r-xs 00000000 08:02 1783964 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/imap.jar
    33d25000-33d54000 r-xs 00000000 08:02 1783963 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/hessian-3.0.13.jar
    33d54000-33d58000 r-xs 00000000 08:02 1783962 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/fast-md5-2.5.jar
    33d58000-33db8000 r-xs 00000000 08:02 1783961 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/dom4j-1.5-rc1.jar
    33db8000-33df5000 r-xs 00000000 08:02 1783960 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/dnsjava-1.6.6.jar
    33df5000-33ef6000 r-xs 00000000 08:02 1783958 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/db-ojb-1.0.1.jar
    33ef6000-33f0f000 r-xs 00000000 08:02 1783957 /usr/local/apache-tomcat-5.5.12/webapps/x/WEB-INF/lib/crossdb.jar
    33f0f000-33f1d000

    An aside: When I switch to ParallelGC, afterrunning
    for about 5-6 hours, GC times start doubling
    (approximately) in time, eventually moving upwardsof
    2 minutes per full GC.That sounds to me like you're swapping during full
    GCs. I wouldn't assume that 2 GB of physical RAM is
    sufficient to run an app with an 1800 MB heap. Have
    you done any heap profiling to determine the real
    working set of your application? When the GCs get
    this pathologically long, is there disk activity on
    the machine?Hmm.. So what should we size our heap to on a server with 2GB? The server is essentially only running the Java process. What would you consider "safe"?
    While heap profiling with jconsole, the application only utilizes 1.1GB of the heap at peaks. We want to expand that eventually as we size our caches larger.
    I don't believe there's disk activity with long GCs, but I can't confirm that 100% right now. The server is reading files from another machine, so the disks are more or less untouched while the app is running. There's heavy caching. The only reason disks would be hititng if your guess regarding the swapping turns out correct. After you suggest a safer way to calc a max heap size for our machine, I'll try again with ParallelGC and a smaller heap.
    Some points:
    - I tried increasing perm generation size. It'snot
    even close to filled (according to gc log).(As you mentioned, "old generation" instead of "perm
    generation".) What about when the full collection
    times are taking 2 minutes in duration? Is there any
    disk activity indicating swapping?Again, not positive on the disk situation -- should get an answer above to figure this out.
    - I have not tried the -client VM, will this break
    performance on a production application?The client VM does not generate as high quality
    machine code as the server VM but on the x86
    architecture many benchmark scores for the two
    compilers are comparable. I would recommend you try
    that option if only to narrow down the scope of the
    crash you're seeing.
    - I saw a bug related to a ConcMarkSweepGC crashon
    4-cpu machines opened on Oct 17th. I've testedthis
    on 4-cpu (2 proc with hyperthreading) and 2-cpu
    (hyperthreading off) -- both exhibit this issue.What bug ID is this?http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6337544
    - Are there any known issues with this collectorand
    Linux 2.4.21-37.ELsmp? Should we be using
    LD_ASSUME_KERNEL=2.2.5 or anything? For java to
    recognize this variable, do we just need itspecified
    anywhere in the environment at JVM start? How dowe
    test if it worked?No issues as far as I know, but I am not a Linux
    expert. I think in order to toggle which thread
    library is used you just need that variable exported
    in the environment when the JVM starts up. You can
    probably use pmap to see which thread library got
    loaded to verify the one you expect is present in the
    loaded objects.Trying various combinations of LD_ASSUME_KERNEL to switch threading versions has no real effect on stability there. We're seeing crashes repeatedly still.
    I'm going to attempt to deploy some machines with a new kernel as well and see if we can reproduce on those. I'll try ParallelGC again as soon as I get your recommendations on the configuration.

  • JVM crashes with Out of SWAP space

    Hello every body,
    We have a problem in a production environnement. The JVM crashes every time with the log file below.
    The application which is running don't occupate all memory and we don't understand why an outofmemory exception is raised.
    It's not probably due to an allocation inside the application because the exception should be an outofmemory exception : Java Heap space.
    Please, somebody can help me to understant what append or to give me some advices?
    Thank you a lot for help.
    # An unexpected error has been detected by Java Runtime Environment:
    # java.lang.OutOfMemoryError: requested 41943040 bytes for GrET in C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
    #  Internal Error (allocation.inline.hpp:42), pid=2700, tid=2112
    #  Error: GrET in C:\BUILD_AREA\jdk6_07\hotspot\src\share\vm\utilities\growableArray.cpp
    # Java VM: Java HotSpot(TM) Server VM (10.0-b23 mixed mode windows-x86)
    # If you would like to submit a bug report, please visit:
    #   http://java.sun.com/webapps/bugreport/crash.jsp
    ---------------  T H R E A D  ---------------
    Current thread (0x65b97400):  VMThread [stack: 0x65bd0000,0x65c20000] [id=2112]
    Stack: [0x65bd0000,0x65c20000]
    [error occurred during error reporting (printing stack bounds), id 0x80000001]
    VM_Operation (0x66dffa14): ParallelGCSystemGC, mode: safepoint, requested by thread 0x65fb9400
    ---------------  P R O C E S S  ---------------
    Java Threads: ( => current thread )
      0x65f3cc00 JavaThread "pool-3-thread-1" [_thread_blocked, id=2780, stack(0x67300000,0x67350000)]
      0x66547800 JavaThread "RMI RenewClean-[192.16.252.55:3916]" daemon [_thread_blocked, id=3076, stack(0x672b0000,0x67300000)]
      0x66547400 JavaThread "RMI RenewClean-[192.16.252.53:2046]" daemon [_thread_blocked, id=2292, stack(0x67260000,0x672b0000)]
      0x6650d800 JavaThread "RMI RenewClean-[192.16.254.46:3126]" daemon [_thread_blocked, id=3012, stack(0x67210000,0x67260000)]
      0x663e8400 JavaThread "RMI RenewClean-[192.16.254.45:3026]" daemon [_thread_blocked, id=2596, stack(0x671c0000,0x67210000)]
      0x65f99800 JavaThread "RMI RenewClean-[192.16.252.54:1623]" daemon [_thread_blocked, id=3128, stack(0x67170000,0x671c0000)]
      0x65f99400 JavaThread "RMI RenewClean-[192.16.254.55:1992]" daemon [_thread_blocked, id=2720, stack(0x67120000,0x67170000)]
      0x66480800 JavaThread "RMI RenewClean-[192.16.254.54:4954]" daemon [_thread_blocked, id=2920, stack(0x670d0000,0x67120000)]
      0x66480000 JavaThread "RMI RenewClean-[192.16.254.53:3093]" daemon [_thread_blocked, id=2068, stack(0x67080000,0x670d0000)]
      0x6653f800 JavaThread "RMI RenewClean-[192.16.254.52:2543]" daemon [_thread_blocked, id=2156, stack(0x67030000,0x67080000)]
      0x66411400 JavaThread "RMI RenewClean-[192.16.254.51:3893]" daemon [_thread_blocked, id=1688, stack(0x66fe0000,0x67030000)]
      0x66468800 JavaThread "RMI RenewClean-[192.16.254.50:3371]" daemon [_thread_blocked, id=2688, stack(0x66f90000,0x66fe0000)]
      0x663f8400 JavaThread "RMI RenewClean-[192.16.254.49:4374]" daemon [_thread_blocked, id=2656, stack(0x66f40000,0x66f90000)]
      0x65f27800 JavaThread "RMI RenewClean-[192.16.254.48:3311]" daemon [_thread_blocked, id=748, stack(0x66ef0000,0x66f40000)]
      0x65f27400 JavaThread "RMI RenewClean-[192.16.254.47:2571]" daemon [_thread_blocked, id=1048, stack(0x66ea0000,0x66ef0000)]
      0x65f9e400 JavaThread "RMI RenewClean-[192.16.238.145:1673]" daemon [_thread_blocked, id=2332, stack(0x66e50000,0x66ea0000)]
      0x6602e800 JavaThread "RMI RenewClean-[192.16.238.144:2332]" daemon [_thread_blocked, id=3100, stack(0x66e00000,0x66e50000)]
      0x65fb9400 JavaThread "GC Daemon" daemon [_thread_blocked, id=1608, stack(0x66db0000,0x66e00000)]
      0x6657a400 JavaThread "RMI RenewClean-[192.16.238.143:3902]" daemon [_thread_blocked, id=2668, stack(0x66d60000,0x66db0000)]
      0x65fa9c00 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=2624, stack(0x66d10000,0x66d60000)]
      0x65ff4c00 JavaThread "pool-4-thread-1" [_thread_blocked, id=2864, stack(0x66390000,0x663e0000)]
      0x65fbf400 JavaThread "pool-2-thread-1" [_thread_blocked, id=2696, stack(0x66340000,0x66390000)]
      0x65f6d400 JavaThread "Thread-0" [_thread_blocked, id=3016, stack(0x662f0000,0x66340000)]
      0x65ebf000 JavaThread "WrapperSimpleAppMain" [_thread_blocked, id=696, stack(0x662a0000,0x662f0000)]
      0x00265c00 JavaThread "DestroyJavaVM" [_thread_blocked, id=564, stack(0x00680000,0x006d0000)]
      0x65fecc00 JavaThread "Wrapper-Connection" daemon [_thread_in_native, id=1224, stack(0x66170000,0x661c0000)]
      0x65ec9c00 JavaThread "Wrapper-Control-Event-Monitor" daemon [_thread_blocked, id=2896, stack(0x660d0000,0x66120000)]
      0x65bbf800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1284, stack(0x65e00000,0x65e50000)]
      0x65bbd000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=3040, stack(0x65db0000,0x65e00000)]
      0x65bb8000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1472, stack(0x65d60000,0x65db0000)]
      0x65bb6c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=2000, stack(0x65d10000,0x65d60000)]
      0x65bb6000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2732, stack(0x65cc0000,0x65d10000)]
      0x65b9ec00 JavaThread "Finalizer" daemon [_thread_blocked, id=2008, stack(0x65c70000,0x65cc0000)]
      0x65b9a800 JavaThread "Reference Handler" daemon [_thread_blocked, id=1304, stack(0x65c20000,0x65c70000)]
    Other Threads:
    =>0x65b97400 VMThread [stack: 0x65bd0000,0x65c20000] [id=2112]
      0x65bc1000 WatcherThread [stack: 0x65e50000,0x65ea0000] [id=192]
    VM state:at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
    [0x00265408/0x00000310] Threads_lock - owner thread: 0x65b97400
    [0x002655a8/0x000002d0] Heap_lock - owner thread: 0x65fb9400
    Heap
    PSYoungGen      total 76928K, used 344K [0x5ad20000, 0x60d40000, 0x653c0000)
      eden space 76544K, 0% used [0x5ad20000,0x5ad20000,0x5f7e0000)
      from space 384K, 89% used [0x5f7e0000,0x5f836070,0x5f840000)
      to   space 10944K, 0% used [0x60290000,0x60290000,0x60d40000)
    PSOldGen        total 1341888K, used 529414K [0x077c0000, 0x59630000, 0x5ad20000)
      object space 1341888K, 39% used [0x077c0000,0x27cc1b38,0x59630000)
    PSPermGen       total 16384K, used 10204K [0x037c0000, 0x047c0000, 0x077c0000)
      object space 16384K, 62% used [0x037c0000,0x041b73a0,0x047c0000)
    Dynamic libraries:
    0x00400000 - 0x00423000      D:\Java\jdk1.6.0_07\bin\java.exe
    0x78460000 - 0x784e2000      C:\WINNT\system32\ntdll.dll
    0x78ed0000 - 0x78f35000      C:\WINNT\system32\ADVAPI32.dll
    0x77e70000 - 0x77f2f000      C:\WINNT\system32\KERNEL32.dll
    0x770c0000 - 0x7712f000      C:\WINNT\system32\RPCRT4.dll
    0x78fb0000 - 0x78fbf000      C:\WINNT\system32\Secur32.dll
    0x7c340000 - 0x7c396000      D:\Java\jdk1.6.0_07\jre\bin\msvcr71.dll
    0x6dac0000 - 0x6de7a000      D:\Java\jdk1.6.0_07\jre\bin\server\jvm.dll
    0x77e00000 - 0x77e5f000      C:\WINNT\system32\USER32.dll
    0x77f40000 - 0x77f7c000      C:\WINNT\system32\GDI32.dll
    0x77540000 - 0x77571000      C:\WINNT\system32\WINMM.dll
    0x6d320000 - 0x6d328000      D:\Java\jdk1.6.0_07\jre\bin\hpi.dll
    0x68ea0000 - 0x68eab000      C:\WINNT\system32\PSAPI.DLL
    0x6d820000 - 0x6d82c000      D:\Java\jdk1.6.0_07\jre\bin\verify.dll
    0x6d3c0000 - 0x6d3df000      D:\Java\jdk1.6.0_07\jre\bin\java.dll
    0x6d860000 - 0x6d86f000      D:\Java\jdk1.6.0_07\jre\bin\zip.dll
    0x660a0000 - 0x660b7000      D:\Prism8.2\lib\wrapper.dll
    0x6d620000 - 0x6d633000      D:\Java\jdk1.6.0_07\jre\bin\net.dll
    0x74fb0000 - 0x74fc4000      C:\WINNT\system32\WS2_32.dll
    0x78000000 - 0x78045000      C:\WINNT\system32\MSVCRT.DLL
    0x74fa0000 - 0x74fa8000      C:\WINNT\system32\WS2HELP.DLL
    0x74f50000 - 0x74f6e000      C:\WINNT\system32\msafd.dll
    0x74f90000 - 0x74f97000      C:\WINNT\System32\wshtcpip.dll
    0x77830000 - 0x7783c000      C:\WINNT\System32\rnr20.dll
    0x77970000 - 0x77994000      C:\WINNT\system32\DNSAPI.DLL
    0x74fd0000 - 0x74fd9000      C:\WINNT\system32\WSOCK32.dll
    0x77310000 - 0x77323000      C:\WINNT\system32\iphlpapi.dll
    0x774f0000 - 0x774f5000      C:\WINNT\system32\ICMP.dll
    0x772f0000 - 0x77307000      C:\WINNT\system32\MPRAPI.dll
    0x750d0000 - 0x750e0000      C:\WINNT\system32\SAMLIB.DLL
    0x7ccc0000 - 0x7cd10000      C:\WINNT\system32\NETAPI32.DLL
    0x77be0000 - 0x77bf1000      C:\WINNT\system32\NTDSAPI.dll
    0x77940000 - 0x7796c000      C:\WINNT\system32\WLDAP32.DLL
    0x75140000 - 0x75146000      C:\WINNT\system32\NETRAP.dll
    0x7cd20000 - 0x7ce0f000      C:\WINNT\system32\OLE32.DLL
    0x779a0000 - 0x77a3c000      C:\WINNT\system32\OLEAUT32.DLL
    0x77380000 - 0x773b0000      C:\WINNT\system32\ACTIVEDS.DLL
    0x77350000 - 0x77373000      C:\WINNT\system32\ADSLDPC.DLL
    0x77820000 - 0x7782e000      C:\WINNT\system32\RTUTILS.DLL
    0x783c0000 - 0x78451000      C:\WINNT\system32\SETUPAPI.DLL
    0x78d20000 - 0x78d85000      C:\WINNT\system32\USERENV.DLL
    0x774b0000 - 0x774e4000      C:\WINNT\system32\RASAPI32.dll
    0x77490000 - 0x774a1000      C:\WINNT\system32\rasman.dll
    0x77500000 - 0x77522000      C:\WINNT\system32\TAPI32.dll
    0x77b40000 - 0x77bc4000      C:\WINNT\system32\COMCTL32.DLL
    0x77c70000 - 0x77cb9000      C:\WINNT\system32\SHLWAPI.DLL
    0x77330000 - 0x77349000      C:\WINNT\system32\DHCPCSVC.DLL
    0x777d0000 - 0x777d8000      C:\WINNT\System32\winrnr.dll
    0x777e0000 - 0x777e5000      C:\WINNT\system32\rasadhlp.dll
    0x7ca00000 - 0x7ca23000      C:\WINNT\system32\rsaenh.dll
    0x79570000 - 0x795fd000      C:\WINNT\system32\CRYPT32.dll
    0x77400000 - 0x77411000      C:\WINNT\system32\MSASN1.dll
    0x7ce30000 - 0x7d07d000      C:\WINNT\system32\shell32.dll
    0x6d7c0000 - 0x6d7c6000      D:\Java\jdk1.6.0_07\jre\bin\rmi.dll
    VM Arguments:
    jvm_args: -Djava.rmi.server.codebase=file:.\Prism.jar -Djava.security.policy=.\policy -Xms64m -Xmx1500m -Djava.library.path=.\lib;.\config -Dwrapper.key=ouZmthkGY4rb0RsE -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=1388 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1
    java_command: org.tanukisoftware.wrapper.WrapperSimpleApp com.socgen.prism.simulateur.client.PrismSimulatorClientOld
    Launcher Type: SUN_STANDARD
    Environment Variables:
    JAVA_HOME=D:\Java\jdk1.6.0_07
    PATH=D:\Java\jdk1.6.0_07;D:\Java\jdk1.6.0_07\bin;C:\Program Files\nsr\bin;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\MSDE\BINN;ITAPaging;D:\PROGRA~1\patrol\Patrol3\bin;D:\Program Files\patrol\Patrol3\bin;D:\Program Files\patrol\Patrol3\Pamo;C:\Program Files\Resource Kit\;C:\Program Files\Support Tools\;C:\Program Files\Dell\SysMgt\oma\bin;C:\Program Files\Dell\SysMgt\RAC3
    OS=Windows_NT
    PROCESSOR_IDENTIFIER=x86 Family 15 Model 2 Stepping 7, GenuineIntel
    ---------------  S Y S T E M  ---------------
    OS: Windows 2000 Build 2195 Service Pack 4
    CPU:total 4 (1 cores per cpu, 2 threads per core) family 15 model 2 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, ht
    Memory: 4k page, physical 2097151k(2059808k free), swap 4194303k(4194303k free)
    vm_info: Java HotSpot(TM) Server VM (10.0-b23) for windows-x86 JRE (1.6.0_07-b06), built on Jun 10 2008 01:09:43 by "java_re" with MS VC++ 7.1
    time: Sat Aug 09 17:44:47 2008
    elapsed time: 79016 seconds

    Answers are in this thread: http://forum.java.sun.com/thread.jspa?threadID=5139476
    Be sure to read the whole thing.

  • JVM Crash When Using JNI and COM

    I'm trying to call a DLL compiled from VB6 source code that I do not have access to. The VB6 code simply retrieves data from a DB2 database using ADO and my client code grabs that data and marshals it to my Java code. I'm attempting to achieve this using JNI and COM (without a third-party bridge). It works 75% of the time, but the other 25% of the time, the JVM crashes with the usual Hotspot crash log containing an access violation exception. However, I don't know what in my C++ code (VC++ 8) could be causing this except for passing a "wild" pointer to the code lying underneath the COM object interface. If that is the case, I don't know how I am doing that.
    The Java code that is calling my native method is running on Tomcat 5.5.25 and just to be safe, I am not allowing multiple threads to concurrently call the method in my JNI DLL (though I realize that this will kill performance). Once I can get past this problem, I'll do the COM interfacing on a worker thread in my native code so I don't screw up CoInitialize and CoUninitialize calls in the case the same thread is concurrently executing multiple calls to my native code.
    I've noticed that in most cases, the JVM crashes during my call to the pClsAccount->OpenConnection method. However, my DLL isn't what is listed on the top of the call stack, which is why I suspect the passing of a wild pointer, though I'm just taking a guess at that. Does anyone have an idea as to what's going on ?
    JNIEXPORT jobject JNICALL Java_CustomerInfo_nGetCustomerAccountInfo(JNIEnv *env, jobject customerInfo, jstring accountNumber, jstring iniFileName)
    jboolean isCopy;
    // Account info class and instance to be instantiated
    jclass accountInfoCls = NULL;
    jobject accountInfoObj = NULL;
    // The constructor ID of the accountInfoCls
    jmethodID ctorID = NULL;
    // Pointer to the interface for the ClsAccount COM object
    _clsAccount *pClsAccount = NULL;
    HRESULT hr;
    BSTR bstrIniFileName(L"");
    try
    const char *nativeAccountNumber = NULL;
    if (accountNumber != NULL)
    nativeAccountNumber = env->GetStringUTFChars(accountNumber, &isCopy);
    else
    jclass newExcCls;
    env->ExceptionDescribe();
    env->ExceptionClear();
    newExcCls = env->FindClass("java/lang/IllegalArgumentException");
    env->ThrowNew(newExcCls, "accountNumber passed in was null !");
    return NULL;
    // Initialization
    variantt varConnectionSucceeded = variantt(false);
    variantt varGetAccountInfoSucceeded = variantt(false);
    variantt varAccountNumber = variantt(nativeAccountNumber);
    bstrt bstrLastPaymentDate = bstrt();
    bstrt bstrLastErrorMessage = bstrt();
    bstrt bstrLastErrorNumber = bstrt();
    jlong jTotalDue = NULL;
    jlong jEstablishedDueDay = NULL;
    jlong jLastPaymentAmount = NULL;
    jstring jLastPaymentDate = NULL;
    jstring jLastErrorMessage = NULL;
    jstring jLastErrorNumber = NULL;
    jthrowable jException = NULL;
    const char *chLastPaymentDate = NULL;
    const char *chLastErrorMessage = NULL;
    const char *chLastErrorNumber = NULL;
    long long totalDue;
    long long lastPaymentAmount;
    long establishedDueDateDay;
    //Convert string from Java string to C string to VB string
    const char *nativeIniFileName = NULL;
    if (iniFileName != NULL)
    nativeIniFileName = env->GetStringUTFChars(iniFileName, &isCopy);
    else
    jclass newExcCls;
    env->ExceptionDescribe();
    env->ExceptionClear();
    newExcCls = env->FindClass("java/lang/IllegalArgumentException");
    env->ThrowNew(newExcCls, "iniFileName passed in was null");
    return NULL;
    bstrIniFileName = comutil::ConvertStringToBSTR(nativeIniFileName);
    CoInitialize(NULL);
    // Create an instance of the COClass with the interface over it
    hr = CoCreateInstance(__uuidof(clsAccount), NULL, CLSCTX_INPROC_SERVER, __uuidof(_clsAccount), (void **)&pClsAccount);
    if (hr == S_OK)
    varConnectionSucceeded.boolVal = pClsAccount->OpenConnection(&bstrIniFileName);
    &#12288;
    if (varConnectionSucceeded.boolVal == -1)
    varGetAccountInfoSucceeded.boolVal = pClsAccount->GetAccountPaymentInformation(&(varAccountNumber.GetVARIANT()));
    env->ReleaseStringUTFChars(accountNumber, nativeAccountNumber);
    // Extract all available account information from the ClsAccount object
    if (varGetAccountInfoSucceeded.boolVal == -1)
    totalDue = pClsAccount->TotalDue.int64;
    establishedDueDateDay = pClsAccount->EstablishedDueDateDay;
    lastPaymentAmount = pClsAccount->LastPaymentAmount.int64;
    bstrLastPaymentDate = pClsAccount->LastPaymentDate;
    chLastPaymentDate = comutil::ConvertBSTRToString(bstrLastPaymentDate.GetBSTR());
    jTotalDue = (jlong)totalDue;
    jEstablishedDueDay = (jlong)establishedDueDateDay;
    jLastPaymentAmount = (jlong)lastPaymentAmount;
    jLastPaymentDate = env->NewStringUTF(chLastPaymentDate);
    delete[] chLastPaymentDate;
    pClsAccount->CloseConnection();
    // Populate error fields if any errors occur
    bstrLastErrorMessage = pClsAccount->LastErrMessage;
    chLastErrorMessage = comutil::ConvertBSTRToString(bstrLastErrorMessage.GetBSTR());
    bstrLastErrorNumber = pClsAccount->LastErrNumber;
    chLastErrorNumber = comutil::ConvertBSTRToString(bstrLastErrorNumber.GetBSTR());
    jLastErrorMessage = env->NewStringUTF(chLastErrorMessage);
    jLastErrorNumber = env->NewStringUTF(chLastErrorNumber);
    delete[] chLastErrorMessage;
    delete[] chLastErrorNumber;
    const char* clsName = "com/nuance/merchantsmutual/businessentities/CustomerAccountInfo";
    // Find the Java class and the ID of its constructor
    accountInfoCls = env->FindClass(clsName);
    ctorID = env->GetMethodID(accountInfoCls, "<init>", "(JJJLjava/lang/String;Ljava/lang/String;Ljava/lang/String;)V");
    jException = env->ExceptionOccurred();
    if (jException != NULL)
    env->ExceptionDescribe();
    env->ExceptionClear();
    //Release all resources associated with the ClsAccount instance
    pClsAccount->Release();
    //Instantiate the class with the given parameters
    accountInfoObj = env->NewObject(accountInfoCls, ctorID, jTotalDue, jEstablishedDueDay, jLastPaymentAmount, jLastPaymentDate, jLastErrorMessage, jLastErrorNumber);
    jException = env->ExceptionOccurred();
    if (jException != NULL)
    env->ExceptionDescribe();
    env->ExceptionClear();
    else if (hr == REGDB_E_CLASSNOTREG)
    cout << "COM class not registered" << endl;
    else if ( hr == CLASS_E_NOAGGREGATION)
    cout << "COM class can't be aggregated" << endl;
    else if (hr == E_NOINTERFACE)
    cout << "No interface for COM class clsAccount" << endl;
    else if (hr == E_POINTER)
    cout << "*ppv pointer was NULL !" << endl;
    else
    cout << "Error occurred while creating COM object. HR is [" << hr << "]" << endl;
    // Free the BSTR because a new one was returned with a call to comutil::ConvertStringToBSTR
    SysFreeString(bstrIniFileName);
    // Release the string when it's no longer needed. MUST call if string won't be used
    // anymore or else a memory leak will occur
    env->ReleaseStringUTFChars(iniFileName, nativeIniFileName);
    CoUninitialize();
    &#12288;
    catch (_com_error &e)
    cout << "Encountered an exception in GetCustomerAccountInfo: Error was " << e.ErrorMessage();
    pClsAccount->Release();
    catch (...)
    pClsAccount->Release();
    return accountInfoObj;
    Edited by: Cthulhu76 on Jan 5, 2010 9:18 AM

    0x49202400 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=5340, stack(0x49bf0000,0x49c40000)]
    0x48a7e800 JavaThread "Thread-1" [_thread_in_native, id=5976, stack(0x48f00000,0x48f50000)]
    0x48a0dc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3072, stack(0x48c60000,0x48cb0000)]
    0x48a09000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=4988, stack(0x48c10000,0x48c60000)]
    0x48a07c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=3124, stack(0x48bc0000,0x48c10000)]
    0x48a07000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2572, stack(0x48b70000,0x48bc0000)]
    0x489f5c00 JavaThread "Finalizer" daemon [_thread_blocked, id=5752, stack(0x48b20000,0x48b70000)]
    0x489f4c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=2596, stack(0x48ad0000,0x48b20000)]
    0x003c6000 JavaThread "main" [_thread_in_native, id=4252, stack(0x00820000,0x00870000)]
    Other Threads:
    0x489f0400 VMThread [stack: 0x48a80000,0x48ad0000] [id=5624]
    0x48a18800 WatcherThread [stack: 0x48cb0000,0x48d00000] [id=1192]
    VM state:not at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread: None
    Heap
    def new generation total 36288K, used 12762K [0x02940000, 0x050a0000, 0x07800000)
    eden space 32256K, 34% used [0x02940000, 0x0343af58, 0x048c0000)
    from space 4032K, 37% used [0x04cb0000, 0x04e2ba28, 0x050a0000)
    to space 4032K, 0% used [0x048c0000, 0x048c0000, 0x04cb0000)
    tenured generation total 483968K, used 7518K [0x07800000, 0x250a0000, 0x42940000)
    the space 483968K, 1% used [0x07800000, 0x07f57958, 0x07f57a00, 0x250a0000)
    compacting perm gen total 14080K, used 14016K [0x42940000, 0x43700000, 0x46940000)
    the space 14080K, 99% used [0x42940000, 0x436f0320, 0x436f0400, 0x43700000)
    No shared spaces configured.
    Dynamic libraries:
    0x00400000 - 0x0040f000      C:\Program Files\Apache Software Foundation\Tomcat 5.5\bin\tomcat5.exe
    0x7c800000 - 0x7c8c0000      C:\WINDOWS\system32\ntdll.dll
    0x77e40000 - 0x77f42000      C:\WINDOWS\system32\kernel32.dll
    0x77380000 - 0x77411000      C:\WINDOWS\system32\USER32.dll
    0x77c00000 - 0x77c48000      C:\WINDOWS\system32\GDI32.dll
    0x77f50000 - 0x77feb000      C:\WINDOWS\system32\ADVAPI32.dll
    0x77c50000 - 0x77cef000      C:\WINDOWS\system32\RPCRT4.dll
    0x76f50000 - 0x76f63000      C:\WINDOWS\system32\Secur32.dll
    0x77ba0000 - 0x77bfa000      C:\WINDOWS\system32\MSVCRT.dll
    0x7c8d0000 - 0x7d0cf000      C:\WINDOWS\system32\SHELL32.dll
    0x77da0000 - 0x77df2000      C:\WINDOWS\system32\SHLWAPI.dll
    0x76290000 - 0x762ad000      C:\WINDOWS\system32\IMM32.DLL
    0x77420000 - 0x77523000      C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x-ww_D8713E55\comctl32.dll
    0x6d7c0000 - 0x6da10000      C:\Program Files\Java\jre1.6.0_07\bin\client\jvm.dll
    0x76aa0000 - 0x76acd000      C:\WINDOWS\system32\WINMM.dll
    0x7c340000 - 0x7c396000      C:\WINDOWS\system32\MSVCR71.dll
    0x6d270000 - 0x6d278000      C:\Program Files\Java\jre1.6.0_07\bin\hpi.dll
    0x76b70000 - 0x76b7b000      C:\WINDOWS\system32\PSAPI.DLL
    0x6d770000 - 0x6d77c000      C:\Program Files\Java\jre1.6.0_07\bin\verify.dll
    0x6d310000 - 0x6d32f000      C:\Program Files\Java\jre1.6.0_07\bin\java.dll
    0x6d7b0000 - 0x6d7bf000      C:\Program Files\Java\jre1.6.0_07\bin\zip.dll
    0x6d570000 - 0x6d583000      C:\Program Files\Java\jre1.6.0_07\bin\net.dll
    0x71c00000 - 0x71c17000      C:\WINDOWS\system32\WS2_32.dll
    0x71bf0000 - 0x71bf8000      C:\WINDOWS\system32\WS2HELP.dll
    0x71b20000 - 0x71b61000      C:\WINDOWS\system32\mswsock.dll
    0x5f270000 - 0x5f2ca000      C:\WINDOWS\system32\hnetcfg.dll
    0x71ae0000 - 0x71ae8000      C:\WINDOWS\System32\wshtcpip.dll
    0x76ed0000 - 0x76efa000      C:\WINDOWS\system32\DNSAPI.dll
    0x76f70000 - 0x76f77000      C:\WINDOWS\System32\winrnr.dll
    0x76f10000 - 0x76f3e000      C:\WINDOWS\system32\WLDAP32.dll
    0x76f80000 - 0x76f85000      C:\WINDOWS\system32\rasadhlp.dll
    0x4a6a0000 - 0x4a6ac000      C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\MMI\WEB-INF\lib\CustomerInfoProxy.dll
    0x77670000 - 0x777a9000      C:\WINDOWS\system32\ole32.dll
    0x77d00000 - 0x77d8b000      C:\WINDOWS\system32\OLEAUT32.dll
    0x7c420000 - 0x7c4a7000      C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.4053_x-ww_E6967989\MSVCP80.dll
    0x78130000 - 0x781cb000      C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.4053_x-ww_E6967989\MSVCR80.dll
    0x777b0000 - 0x77833000      C:\WINDOWS\system32\CLBCatQ.DLL
    0x77010000 - 0x770d6000      C:\WINDOWS\system32\COMRes.dll
    0x77b90000 - 0x77b98000      C:\WINDOWS\system32\VERSION.dll
    0x75da0000 - 0x75e5d000      C:\WINDOWS\system32\SXS.DLL
    0x75e60000 - 0x75e87000      C:\WINDOWS\system32\apphelp.dll
    0x4dc30000 - 0x4dc5e000      C:\WINDOWS\system32\msctfime.ime
    0x4b0d0000 - 0x4b395000      C:\WINDOWS\system32\xpsp2res.dll
    0x71bb0000 - 0x71bb9000      C:\WINDOWS\system32\WSOCK32.dll
    0x4bbe0000 - 0x4bbea000      C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\MMI\WEB-INF\lib\ClearTranProxy.dll
    0x745e0000 - 0x7489e000      C:\WINDOWS\system32\msi.dll
    0x71c40000 - 0x71c97000      C:\WINDOWS\system32\NETAPI32.dll
    0x4bc50000 - 0x4bc6c000      C:\WINDOWS\system32\DBNETLIB.DLL
    0x71f60000 - 0x71f64000      C:\WINDOWS\system32\security.dll
    0x76c90000 - 0x76cb7000      C:\WINDOWS\system32\msv1_0.dll
    0x76cf0000 - 0x76d0a000      C:\WINDOWS\system32\iphlpapi.dll
    0x761b0000 - 0x76243000      C:\WINDOWS\system32\crypt32.dll
    0x76190000 - 0x761a2000      C:\WINDOWS\system32\MSASN1.dll
    0x4bcf0000 - 0x4bcff000      C:\Program Files\Common Files\System\Ole DB\SQLOLEDB.RLL
    0x4a8a0000 - 0x4a8aa000      C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\MMI\WEB-INF\lib\MIGI.DLL
    0x73570000 - 0x736c2000      C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\MMI\WEB-INF\lib\MSVBVM60.DLL
    0x4a950000 - 0x4a9e2000      C:\Program Files\Common Files\System\ado\msado15.dll
    0x74a50000 - 0x74a6a000      C:\WINDOWS\system32\MSDART.DLL
    0x4c850000 - 0x4c8c9000      C:\Program Files\Common Files\System\Ole DB\oledb32.dll
    0x4dbb0000 - 0x4dbc1000      C:\Program Files\Common Files\System\Ole DB\OLEDB32R.DLL
    VM Arguments:
    jvm_args: -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 5.5 -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 5.5 -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat 5.5\common\endorsed -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat 5.5\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Program Files\Apache Software Foundation\Tomcat 5.5\conf\logging.properties -Djava.library.path=C:\Program Files\Apache Software Foundation\Tomcat 5.5\webapps\MMI\WEB-INF\lib vfprintf -Xms512m -Xmx1024m
    java_command: <unknown>
    Launcher Type: generic
    Environment Variables:
    JAVA_HOME=C:\Program Files\Java\jdk1.6.0_07
    [error occurred during error reporting (printing environment variables), id 0xc0000005]
    --------------- S Y S T E M ---------------
    OS: Windows Server 2003 family Build 3790 Service Pack 2
    CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 7 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
    Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)
    vm_info: Java HotSpot(TM) Client VM (10.0-b23) for windows-x86 JRE (1.6.0_07-b06), built on Jun 10 2008 01:14:11 by "java_re" with MS VC++ 7.1
    time: Mon Dec 28 15:24:00 2009
    elapsed time: 600 seconds

  • JVM Crash with latest FB3 SA on WXP-sp2

    I've been experiencing JVM crash intermittently through-out
    the beta program and now also with the released version.
    I don't see an outcry from the users, so I'm guessing very
    few others are experiencing this ? and it is due to something weird
    in my own PC environment ( e.g. like several other JVMs with
    different versions that are running Apache, Mysql and the like and
    somehow cause some cross-talk ?). However if there are others out
    there with similar situation please report ..

    if you have a reproducible case. Please submit a bug at
    http://bugs.adobe.com/flex.
    also attach the .log file (located in the workspace folder).
    Thanks,
    Gaurav

  • JVM crashes with illegal Memory Access ..

    [JRockit] ERROR: The JVM has crashed. Writing crash information to C:\bea10.3\wlserver_10.3\samples\domains\wl_server\jrockit.3712.dump.
    ===== BEGIN DUMP =============================================================
    JRockit dump produced after 0 days, 00:13:27 on Mon Dec 07 20:06:05 2009
    * If you see this dump, please go to *
    * http://edocs.bea.com/jrockit/go2troubleshooting.html *
    * for troubleshooting information. *
    Additional information is available in:
    C:\bea10.3\wlserver_10.3\samples\domains\wl_server\jrockit.3712.dump
    C:\bea10.3\wlserver_10.3\samples\domains\wl_server\jrockit.3712.mdmp
    Error Message: Illegal memory access. [54]
    Exception Rec: EXCEPTION_ACCESS_VIOLATION (c0000005) at 0x00466F21 - memory at 0x00000000 could not be read.
    Minidump : Wrote mdmp. Size is 472MB
    SafeDllMode : -1
    Version : BEA JRockit(R) R27.6.0-50_o-100423-1.6.0_05-20080626-2105-windows-ia32
    GC Strategy : Mode: throughput. Currently using strategy: genparpar
    GC Status : OC is not running. Last finished OC was OC#2.
    : YC is not running. Last finished YC was YC#24.
    OC History : Strategy genparpar was used for OC#1 to OC#2.
    YC History : Ran 3 YCs before OC#1.
    : Ran 3 YCs before OC#2.
    : Ran 18 YCs since last OC.
    YC Promotion : Last YC successfully promoted all objects
    Heap : 0x00C00000 - 0x10C00000 (Size: 256 MB)
    Compaction : 0x00C00000 - 0x01C00000 (Current compaction type: internal)
    NurseryList : 0x00E57CF0 - 0x08132978
    KeepArea : 0x04693C88 - 0x06320918
    NurseryMarker: [ 0x04693C88,  0x06320918 ]
    CompRefs : References are 32-bit.
    CPU : Intel Core 2 SSE SSE2 SSE3 SSSE3 SSE4.1 EM64T
    Number CPUs : 2
    Tot Phys Mem : 2096984064 (1999 MB)
    OS version : Microsoft Windows XP version 5.1 Service Pack 3 (Build 2600) (32-bit)
    Thread System: Windows Threads
    Java locking : Lazy unlocking enabled (class banning) (transfer banning)
    State : JVM is running
    Command Line : -Xverify:all -Xms256m -Xmx512m -Xverify:none -da -Dplatform.home=C:\bea10.3\WLSERV~1.3 -Dwls.home=C:\bea10.3\WLSERV~1.3\server -Dweblogic.home=C:\bea10.3\WLSERV~1.3\server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=C:\bea10.3\patch_wlw1030\profiles\default\sysext_manifest_classpath;C:\bea10.3\patch_wls1030\profiles\default\sysext_manifest_classpath;C:\bea10.3\patch_cie660\profiles\default\sysext_manifest_classpath -Dweblogic.Name=examplesServer -Djava.security.policy=C:\bea10.3\WLSERV~1.3\server\lib\weblogic.policy -Dsun.java.launcher=SUN_STANDARD weblogic.Server
    java.home : C:\bea10.3\JROCKI~1\jre
    j.class.path : C:\bea10.3\wlserver_10.3\samples\server\examples\build\serverclasses;C:\bea10.3\patch_wlw1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\bea10.3\patch_wls1030\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\bea10.3\patch_cie660\profiles\default\sys_manifest_classpath\weblogic_patch.jar;C:\bea10.3\JROCKI~1\lib\tools.jar;C:\bea10.3\WLSERV~1.3\server\lib\weblogic_sp.jar;C:\bea10.3\WLSERV~1.3\server\lib\weblogic.jar;C:\bea10.3\modules\features\weblogic.server.modules_10.3.0.0.jar;C:\bea10.3\WLSERV~1.3\server\lib\webservices.jar;C:\bea10.3\modules\ORGAPA~1.5/lib/ant-all.jar;C:\bea10.3\modules\NETSFA~1.0_1/lib/ant-contrib.jar;;C:\bea10.3\WLSERV~1.3\common\eval\pointbase\lib\pbembedded57.jar;C:\bea10.3\WLSERV~1.3\common\eval\pointbase\lib\pbclient57.jar;C:\bea10.3\WLSERV~1.3\server\lib\xqrl.jar;;C:\bea10.3\wlserver_10.3\server\lib\wlspy.jar;;
    j.lib.path : C:\bea10.3\JROCKI~1\bin;.;C:\WINDOWS\system32;C:\WINDOWS;C:\bea10.3\patch_wlw1030\profiles\default\native;C:\bea10.3\patch_wls1030\profiles\default\native;C:\bea10.3\patch_cie660\profiles\default\native;C:\bea10.3\WLSERV~1.3\server\native\win\32;C:\bea10.3\WLSERV~1.3\server\bin;C:\bea10.3\modules\ORGAPA~1.5\bin;C:\bea10.3\JROCKI~1\jre\bin;C:\bea10.3\JROCKI~1\bin;C:/Program Files/Java/jre1.6.0_02/bin/client;C:/Program Files/Java/jre1.6.0_02/bin;C:\Program Files\IBM\WebSphere MQ\Java\lib;C:\oracle\product\10.2.0\bin;C:\Program Files\Documentum\Shared;c:\windows\system32;c:\windows;c:\windows\system32\wbem;c:\program files\ntru cryptosystems\ntru tcg software stack\bin\;c:\program files\wave systems corp\gemalto\access client\v5\;c:\program files\intel\dmix;c:\program files\intel\wifi\bin\;c:\program files\common files\roxio shared\dllshared\;c:\program files\common files\roxio shared\9.0\dllshared\;c:\program files\utimaco\safeguard easy\;C:\Program Files\Windows Imaging\;c:\program files\java\jdk1.6.0_05\bin;;C:\bea10.3\wlserver_10.3\server\lib\wlspy.jar;C:\Program Files\IBM\WebSphere MQ\bin;C:\Program Files\IBM\WebSphere MQ\tools\c\samples\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Subversion\bin;C:\bea10.3\WLSERV~1.3\server\native\win\32\oci920_8
    JAVA_HOME : C:\bea10.3\JROCKI~1
    JAVAOPTIONS: <not set>
    PATH : C:\bea10.3\JROCKI~1\jre\bin;C:\bea10.3\patch_wlw1030\profiles\default\native;C:\bea10.3\patch_wls1030\profiles\default\native;C:\bea10.3\patch_cie660\profiles\default\native;C:\bea10.3\WLSERV~1.3\server\native\win\32;C:\bea10.3\WLSERV~1.3\server\bin;C:\bea10.3\modules\ORGAPA~1.5\bin;C:\bea10.3\JROCKI~1\jre\bin;C:\bea10.3\JROCKI~1\bin;C:/Program Files/Java/jre1.6.0_02/bin/client;C:/Program Files/Java/jre1.6.0_02/bin;C:\Program Files\IBM\WebSphere MQ\Java\lib;C:\oracle\product\10.2.0\bin;C:\Program Files\Documentum\Shared;c:\windows\system32;c:\windows;c:\windows\system32\wbem;c:\program files\ntru cryptosystems\ntru tcg software stack\bin\;c:\program files\wave systems corp\gemalto\access client\v5\;c:\program files\intel\dmix;c:\program files\intel\wifi\bin\;c:\program files\common files\roxio shared\dllshared\;c:\program files\common files\roxio shared\9.0\dllshared\;c:\program files\utimaco\safeguard easy\;C:\Program Files\Windows Imaging\;c:\program files\java\jdk1.6.0_05\bin;;C:\bea10.3\wlserver_10.3\server\lib\wlspy.jar;C:\Program Files\IBM\WebSphere MQ\bin;C:\Program Files\IBM\WebSphere MQ\tools\c\samples\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Subversion\bin;C:\bea10.3\WLSERV~1.3\server\native\win\32\oci920_8
    C Heap : Good; no memory allocations have failed
    StackOverFlow: 0 StackOverFlowErrors have occured
    OutOfMemory : 0 OutOfMemoryErrors have occured
    Registers (from ThreadContext: 0x319BD1FC / OS context: 0x319BD5F8):
    eax = 2b1cdf58 ecx = 31b647a8 edx = 00000000 ebx = 0000002d
    esp = 319bd8c4 ebp = 31b66400 esi = 00000000 edi = 00000001
    es = 00000023 cs = 0000001b ss = 00000023 ds = 00000023
    fs = 0000003b gs = 00000000
    eip = 00466f21 eflags = 00010202
    Stack:
    (* marks the word pointed to by the stack pointer)
    319bd8c4: 319bda40* 28ac5bdc 2d61e1c4 31b636f8 0000002c 31b66590
    319bd8dc: 00000000 00000001 00000001 00000001 31b66588 00000000
    319bd8f4: 0000002c 00000000 00000005 31b647a8 2d61e1c4 2d630700
    319bd90c: 00001df5 00462f77 28ac5bdc 31b636f8 319bdec8 319bdf04
    Code:
    (* marks the word pointed to by the instruction pointer)
    00466ef0: 08c8448d 28244489 89f8048d 8b142444 431c2444 c085f633
    00466f08: 00f9860f ff8b0000 54244c8b 8b34518b 4c8bb204 518b3c24
    00466f20: f20c8b2c* 1824548b 8bca0c8d 89142454 8b38244c 8928244c
    00466f38: 5489f11c b70f04f1 00014e88 b8b70f00 0000014a 44c7f92b
    Loaded modules:
    (* denotes the module causing the exception)
    00400000-00410fff C:\bea10.3\JROCKI~1\bin\java.exe
    7c900000-7c9b1fff C:\WINDOWS\system32\ntdll.dll
    7c800000-7c8f5fff C:\WINDOWS\system32\kernel32.dll
    77dd0000-77e6afff C:\WINDOWS\system32\ADVAPI32.dll
    77e70000-77f01fff C:\WINDOWS\system32\RPCRT4.dll
    77fe0000-77ff0fff C:\WINDOWS\system32\Secur32.dll
    77c10000-77c67fff C:\WINDOWS\system32\MSVCRT.dll
    7c340000-7c395fff C:\bea10.3\JROCKI~1\jre\bin\msvcr71.dll
    00420000-006c3fff *C:\bea10.3\JROCKI~1\jre\bin\jrockit\jvm.dll
    76b40000-76b6cfff C:\WINDOWS\system32\WINMM.dll
    77f10000-77f58fff C:\WINDOWS\system32\GDI32.dll
    7e410000-7e4a0fff C:\WINDOWS\system32\USER32.dll
    71ab0000-71ac6fff C:\WINDOWS\system32\WS2_32.dll
    71aa0000-71aa7fff C:\WINDOWS\system32\WS2HELP.dll
    76390000-763acfff C:\WINDOWS\system32\IMM32.DLL
    6d820000-6d82bfff C:\bea10.3\JROCKI~1\jre\bin\verify.dll
    6d3c0000-6d3defff C:\bea10.3\JROCKI~1\jre\bin\java.dll
    6d320000-6d327fff C:\bea10.3\JROCKI~1\jre\bin\hpi.dll
    6d860000-6d86efff C:\bea10.3\jrockit_160_05\jre\bin\zip.dll
    6d570000-6d578fff C:\bea10.3\jrockit_160_05\jre\bin\management.dll
    6d620000-6d632fff C:\bea10.3\jrockit_160_05\jre\bin\net.dll
    237a0000-237cffff C:\Program Files\F-Secure\FSPS\program\FSLSP.DLL
    237d0000-237e6fff c:\program files\f-secure\scanner-interface\fsgkiapi.dll
    71a50000-71a8efff C:\WINDOWS\system32\mswsock.dll
    662b0000-66307fff C:\WINDOWS\system32\hnetcfg.dll
    71a90000-71a97fff C:\WINDOWS\System32\wshtcpip.dll
    23ca0000-23cc6fff C:\WINDOWS\system32\DNSAPI.dll
    23cd0000-23cd7fff C:\WINDOWS\System32\winrnr.dll
    23ce0000-23d0bfff C:\WINDOWS\system32\WLDAP32.dll
    23d60000-23d65fff C:\WINDOWS\system32\rasadhlp.dll
    6d640000-6d648fff C:\bea10.3\jrockit_160_05\jre\bin\nio.dll
    68000000-68035fff C:\WINDOWS\system32\rsaenh.dll
    769c0000-76a73fff C:\WINDOWS\system32\USERENV.dll
    5b860000-5b8b4fff C:\WINDOWS\system32\netapi32.dll
    25c20000-25c2afff C:\bea10.3\jrockit_160_05\jre\bin\jmapi.dll
    25e30000-25e3dfff C:\bea10.3\wlserver_10.3\server\native\win\32\wlfileio2.dll
    26120000-26138fff C:\WINDOWS\system32\iphlpapi.dll
    26150000-26167fff C:\WINDOWS\system32\MPRAPI.dll
    77cc0000-77cf1fff C:\WINDOWS\system32\ACTIVEDS.dll
    26170000-26194fff C:\WINDOWS\system32\adsldpc.dll
    76b20000-76b30fff C:\WINDOWS\system32\ATL.DLL
    774e0000-7761cfff C:\WINDOWS\system32\ole32.dll
    261a0000-2622afff C:\WINDOWS\system32\OLEAUT32.dll
    26230000-2623dfff C:\WINDOWS\system32\rtutils.dll
    71bf0000-71c02fff C:\WINDOWS\system32\SAMLIB.dll
    77920000-77a12fff C:\WINDOWS\system32\SETUPAPI.dll
    26280000-26284fff C:\bea10.3\wlserver_10.3\server\native\win\32\wlntio.dll
    6d800000-6d807fff C:\bea10.3\jrockit_160_05\jre\bin\sunmscapi.dll
    77a80000-77b14fff C:\WINDOWS\system32\CRYPT32.dll
    77b20000-77b31fff C:\WINDOWS\system32\MSASN1.dll
    76bf0000-76bfafff C:\WINDOWS\system32\psapi.dll
    6d7c0000-6d7c5fff C:\bea10.3\jrockit_160_05\jre\bin\rmi.dll
    31c20000-31d32fff C:\bea10.3\JROCKI~1\jre\bin\dbghelp.dll
    "[ACTIVE] ExecuteThread: '18' fo" id=54 idx=0xc8 tid=5456 lastJavaFrame=0x319BDEF4
    Stack 0: start=0x31980000, end=0x319C0000, guards=0x31983000 (ok), forbidden=0x31981000
    Thread Stack Trace:
    at dtCreateDTable+1025(dispatchtables.c:298+7)@0x00466F21
    at defineClass+167(defineclass.c:110+7)@0x00462F77
    at lib_define_class+34(library.c:516+16)@0x005164C2
    at libDefineClass+39(library.c:576+38)@0x00516707
    at libDefineClass2+58(library.c:597+34)@0x0051685A
    at JVM_DefineClassWithSource+43(jvmclass.c:59+43)@0x004E609B
    at Javajava_lang_ClassLoader_defineClass1@32+235()@0x6D3C15DA
    -- Java stack --
    at java/lang/ClassLoader.defineClass1(Ljava/lang/String;[BIILjava/security/ProtectionDomain;Ljava/lang/String;)Ljava/lang/Class;(Native Method)
        at java/lang/ClassLoader.defineClass(ClassLoader.java:620)
        at java/security/SecureClassLoader.defineClass(SecureClassLoader.java:124)
        at weblogic/utils/classloaders/GenericClassLoader.defineClass(GenericClassLoader.java:335)
        at weblogic/utils/classloaders/GenericClassLoader.findLocalClass(GenericClassLoader.java:288)
        ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E7FCE78[recursive]
    at weblogic/utils/classloaders/GenericClassLoader.findClass(GenericClassLoader.java:256)
    at java/lang/ClassLoader.loadClass(ClassLoader.java:306)
    ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E7FCE78[recursive]
    at java/lang/ClassLoader.loadClass(ClassLoader.java:251)
    at weblogic/utils/classloaders/GenericClassLoader.loadClass(GenericClassLoader.java:176)
    at jrockit/vm/Classes.loadClassInternal(Classes.java:75)
    ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E7FCE78[recursive]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    at java/lang/ClassLoader.defineClass1(Ljava/lang/String;[BIILjava/security/ProtectionDomain;Ljava/lang/String;)Ljava/lang/Class;(Native Method)
        at java/lang/ClassLoader.defineClass(ClassLoader.java:620)
        at java/security/SecureClassLoader.defineClass(SecureClassLoader.java:124)
        at weblogic/utils/classloaders/GenericClassLoader.defineClass(GenericClassLoader.java:335)
        at weblogic/utils/classloaders/GenericClassLoader.findLocalClass(GenericClassLoader.java:288)
        ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E7FCE78[recursive]
    at weblogic/utils/classloaders/GenericClassLoader.findClass(GenericClassLoader.java:256)
    at java/lang/ClassLoader.loadClass(ClassLoader.java:306)
    ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E7FCE78[recursive]
    at java/lang/ClassLoader.loadClass(ClassLoader.java:251)
    at weblogic/utils/classloaders/GenericClassLoader.loadClass(GenericClassLoader.java:176)
    at jrockit/vm/Classes.loadClassInternal(Classes.java:75)
    ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E7FCE78[biased lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    at jrockit/vm/Classes.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;(Native Method)
    at java/lang/Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;(Native Method)
    at java/lang/Class.forName(Class.java:247)
    at sun/reflect/generics/factory/CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:95)
    at sun/reflect/generics/visitor/Reifier.visitClassTypeSignature(Reifier.java:107)
    at sun/reflect/generics/tree/ClassTypeSignature.accept(ClassTypeSignature.java:31)
    at sun/reflect/annotation/AnnotationParser.parseSig(AnnotationParser.java:370)
    at sun/reflect/annotation/AnnotationParser.parseClassValue(AnnotationParser.java:351)
    at sun/reflect/annotation/AnnotationParser.parseMemberValue(AnnotationParser.java:280)
    at sun/reflect/annotation/AnnotationParser.parseAnnotation(AnnotationParser.java:222)
    at sun/reflect/annotation/AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
    at sun/reflect/annotation/AnnotationParser.parseAnnotations(AnnotationParser.java:52)
    at java/lang/reflect/Field.declaredAnnotations(Field.java:1016)
    ^-- Holding lock: java/lang/reflect/Field@0x032BFE90[biased lock]
    at java/lang/reflect/Field.getDeclaredAnnotations(Field.java:1009)
    at java/lang/reflect/AccessibleObject.getAnnotations(AccessibleObject.java:175)
    at org/apache/openjpa/lib/util/J2DoPriv5Helper$1.run(J2DoPriv5Helper.java:51)
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:233)
    at jrockit/vm/AccessController.doPrivileged(AccessController.java:241)
    at org/apache/openjpa/persistence/PersistenceMetaDataDefaults.annotated(PersistenceMetaDataDefaults.java:293)
    at org/apache/openjpa/persistence/PersistenceMetaDataDefaults.getAccessType(PersistenceMetaDataDefaults.java:262)
    at org/apache/openjpa/persistence/PersistenceMetaDataDefaults.getAccessType(PersistenceMetaDataDefaults.java:269)
    at org/apache/openjpa/persistence/PersistenceMetaDataDefaults.getAccessType(PersistenceMetaDataDefaults.java:250)
    at org/apache/openjpa/meta/AbstractMetaDataDefaults.populate(AbstractMetaDataDefaults.java:155)
    at org/apache/openjpa/persistence/PersistenceMetaDataDefaults.populate(PersistenceMetaDataDefaults.java:227)
    at org/apache/openjpa/meta/MetaDataRepository.addMetaData(MetaDataRepository.java:794)
    at org/apache/openjpa/meta/MetaDataRepository.addMetaData(MetaDataRepository.java:780)
    at org/apache/openjpa/persistence/AnnotationPersistenceMetaDataParser.getMetaData(AnnotationPersistenceMetaDataParser.java:657)
    at org/apache/openjpa/persistence/AnnotationPersistenceMetaDataParser.parseClassAnnotations(AnnotationPersistenceMetaDataParser.java:480)
    at org/apache/openjpa/persistence/AnnotationPersistenceMetaDataParser.parse(AnnotationPersistenceMetaDataParser.java:352)
    at org/apache/openjpa/persistence/PersistenceMetaDataFactory.load(PersistenceMetaDataFactory.java:229)
    at org/apache/openjpa/meta/MetaDataRepository.getMetaDataInternal(MetaDataRepository.java:474)
    at org/apache/openjpa/meta/MetaDataRepository.getMetaData(MetaDataRepository.java:294)
    ^-- Holding lock: org/apache/openjpa/meta/MetaDataRepository@0x0E7FCDF0[biased lock]
    at org/apache/openjpa/enhance/PCEnhancer.<init>(PCEnhancer.java:248)
    at org/apache/openjpa/enhance/PCEnhancer.<init>(PCEnhancer.java:219)
    at org/apache/openjpa/enhance/PCClassFileTransformer.transform0(PCClassFileTransformer.java:139)
    at org/apache/openjpa/enhance/PCClassFileTransformer.transform(PCClassFileTransformer.java:120)
    at org/apache/openjpa/persistence/PersistenceProviderImpl$ClassTransformerImpl.transform(PersistenceProviderImpl.java:210)
    at weblogic/deployment/PersistenceUnitInfoImpl$ClassPreProcessorImpl.preProcess(PersistenceUnitInfoImpl.java:497)
    at weblogic/utils/classloaders/ClassPreProcessor$ClassPreProcessorSupport.preProcess(ClassPreProcessor.java:95)
    at weblogic/utils/classloaders/GenericClassLoader.doPreProcess(GenericClassLoader.java:348)
    at weblogic/utils/classloaders/GenericClassLoader.defineClass(GenericClassLoader.java:329)
    at weblogic/utils/classloaders/GenericClassLoader.findLocalClass(GenericClassLoader.java:288)
    ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E093168[recursive]
    at weblogic/utils/classloaders/GenericClassLoader.findClass(GenericClassLoader.java:256)
    at java/lang/ClassLoader.loadClass(ClassLoader.java:306)
    ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E093168[recursive]
    at java/lang/ClassLoader.loadClass(ClassLoader.java:251)
    at weblogic/utils/classloaders/GenericClassLoader.loadClass(GenericClassLoader.java:176)
    at jrockit/vm/Classes.loadClassInternal(Classes.java:75)
    ^-- Holding lock: weblogic/utils/classloaders/GenericClassLoader@0x0E093168[biased lock]
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    at jrockit/vm/Classes.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;(Native Method)
    at java/lang/Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;(Native Method)
    at java/lang/Class.forName(Class.java:247)
    at sun/reflect/generics/factory/CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:95)
    at sun/reflect/generics/visitor/Reifier.visitClassTypeSignature(Reifier.java:107)
    at sun/reflect/generics/tree/ClassTypeSignature.accept(ClassTypeSignature.java:31)
    at sun/reflect/generics/visitor/Reifier.reifyTypeArguments(Reifier.java:50)
    at sun/reflect/generics/visitor/Reifier.visitClassTypeSignature(Reifier.java:120)
    at sun/reflect/generics/tree/ClassTypeSignature.accept(ClassTypeSignature.java:31)
    at sun/reflect/generics/repository/MethodRepository.getReturnType(MethodRepository.java:50)
    at java/lang/reflect/Method.getGenericReturnType(Method.java:236)
    at java/lang/reflect/Method.toGenericString(Method.java:482)
    at weblogic/ejb/container/ejbc/EjbCodeGenerator.setMethod(EjbCodeGenerator.java:3511)
    at weblogic/ejb/container/ejbc/EjbCodeGenerator.remote_interface_methods(EjbCodeGenerator.java:2698)
    at weblogic/ejb/container/ejbc/EjbCodeGenerator.remote_business_interface_methods(EjbCodeGenerator.java:2685)
    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
    at jrockit/vm/Reflect.invokeMethod(Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
        at sun/reflect/NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Native Method)
        at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java/lang/reflect/Method.invoke(Method.java:597)
        at weblogic/utils/compiler/CodeGenerator.processAt(CodeGenerator.java:688)
        at weblogic/utils/compiler/CodeGenerator.parse(CodeGenerator.java:582)
        at weblogic/utils/compiler/CodeGenerator.parse(CodeGenerator.java:539)
        at weblogic/utils/compiler/CodeGenerator.generateCode(CodeGenerator.java:341)
        at weblogic/utils/compiler/CodeGenerator.generate(CodeGenerator.java:242)
        at weblogic/utils/compiler/CodeGenerator.generate(CodeGenerator.java:197)
        at weblogic/ejb/container/ejbc/EJBCompiler.generateSourcesFromBeanInfo(EJBCompiler.java:1019)
        at weblogic/ejb/container/ejbc/EJBCompiler.doCompile(EJBCompiler.java:299)
        at weblogic/ejb/container/ejbc/EJBCompiler.compileEJB(EJBCompiler.java:552)
        at weblogic/ejb/container/ejbc/EJBCompiler.compileEJB(EJBCompiler.java:519)
        at weblogic/ejb/container/deployer/EJBDeployer.runEJBC(EJBDeployer.java:393)
        at weblogic/ejb/container/deployer/EJBDeployer.compileJar(EJBDeployer.java:715)
        at weblogic/ejb/container/deployer/EJBDeployer.compileIfNecessary(EJBDeployer.java:618)
        at weblogic/ejb/container/deployer/EJBDeployer.prepare(EJBDeployer.java:1154)
        at weblogic/ejb/container/deployer/EJBModule.prepare(EJBModule.java:425)
    Stopping PointBase server...
    PointBase server stopped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                &n

    Looks like a JVM crash during classloading. Have no idea what might cause it. I suggest you try upgrading to a more recent JRockit version (google for "download jrockit" and get the latest JRMC version you can find, it includes JRockit R27.6.5). If that doesn't help, open a ticket with Oracle Support and provide the text dump and the minidump for analysis.
    Henrik

  • Variety of periodic JVM crashes (no JNI)

    Hello,
    I have a J2SE application that runs on a Linux box that is getting a number of various JVM crashes. The program has no JNI and does not use any libraries, it's all just Java. The call stacks are always different, and I don't see any consistency in what is happening in the application when the JVM crashes. It does however always seem to be a SIGSEGV. Sometimes they happen after 2 minutes of uptime, sometimes after days.
    I tried upgrading the JVM from 6.0_27-b07 to 7.0_06-b24 but it has made no difference. I uploaded all the hs_err logs here: http://www.tacticstudios.com/downloads/logs.rar if you would like to take a look.
    Due to the inconsistent call stacks my only assumption is that memory is getting corrupted in some way, and the crash is occurring later on.  Maybe it is a hardware problem? I'm not certain what would cause this.
    Any assistance you can provide in how to debug this, or any ideas would be extremely appreciated. I have pasted the most recent error log below.
    Jesse
    # A fatal error has been detected by the Java Runtime Environment:
    #  SIGSEGV (0xb) at pc=0x00007fbcc8a16bd4, pid=3126, tid=140448514426624
    # JRE version: 7.0_06-b24
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.2-b09 mixed mode linux-amd64 compressed oops)
    # Problematic frame:
    # V  [libjvm.so+0x7b5bd4]  PhaseChaitin::Split(unsigned int)+0x8e4
    # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    # If you would like to submit a bug report, please visit:
    #   http://bugreport.sun.com/bugreport/crash.jsp
    ---------------  T H R E A D  ---------------
    Current thread (0x00007fbcc40e7800):  JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=3145, stack(0x00007fbcb7bfc000,0x00007fbcb7cfd000)]
    siginfo:si_signo=SIGSEGV: si_errno=0, si_code=128 (), si_addr=0x0000000000000000
    Registers:
    RAX=0x00007fbc89897540, RBX=0x00007fbc88e499f0, RCX=0x00007fbc898974b0, RDX=0x0000000000000000
    RSP=0x00007fbcb7cf8390, RBP=0x00007fbcb7cf8820, RSI=0x0000000000000004, RDI=0x00007fbc88e499f0
    R8 =0x0000000000000181, R9 =0x0000000000000004, R10=0x0000000000000001, R11=0x00007fbc88e69390
    R12=0x0000000000000000, R13=0x0000000000000001, R14=0x00007fbc88e499f0, R15=0xf100000000000000
    RIP=0x00007fbcc8a16bd4, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000000
      TRAPNO=0x000000000000000d
    Top of Stack: (sp=0x00007fbcb7cf8390)
    0x00007fbcb7cf8390:   00007fbcb7cf83d0 00007fbcc897d002
    0x00007fbcb7cf83a0:   0000000000000000 0000000000000000
    0x00007fbcb7cf83b0:   000000000000002a 00007fbcb7cf8a40
    0x00007fbcb7cf83c0:   0101010101010101 0101010101010101
    0x00007fbcb7cf83d0:   0000000000000000 0000000000000000
    0x00007fbcb7cf83e0:   00007fbcb7cf8690 00007fbcb7cf8650
    0x00007fbcb7cf83f0:   00007fbcb7cf8670 00007fbcb7cf8750
    0x00007fbcb7cf8400:   0000000000000007 00007fbc88f97d90
    0x00007fbcb7cf8410:   0000000000000000 00007fbcb7cf8a40
    0x00007fbcb7cf8420:   00007fbc8908f210 00007fbcc848c156
    0x00007fbcb7cf8430:   00007fbcb7cf8490 0000000000000004
    0x00007fbcb7cf8440:   00007fbc88712fc8 00007fbc8908f188
    0x00007fbcb7cf8450:   0000000000000008 0000000000002940
    0x00007fbcb7cf8460:   00007fbcb7cf9f90 00007fbc883f9cc0
    0x00007fbcb7cf8470:   00007fbc883fa590 0000000000000000
    0x00007fbcb7cf8480:   0000000000000090 00007fbcb7cf8a40
    0x00007fbcb7cf8490:   00007fbc8909b580 00007fbc8908f188
    0x00007fbcb7cf84a0:   00007fbcb7cfa058 00007fbcc846c6ce
    0x00007fbcb7cf84b0:   0000000000000003 0000011ac8ac9296
    0x00007fbcb7cf84c0:   00007fbc883fb850 00007fbc88b58600
    0x00007fbcb7cf84d0:   00007fbc88b58640 00007fbc883fc120
    0x00007fbcb7cf84e0:   01007fbc8908f260 00007fbc898974b0
    0x00007fbcb7cf84f0:   00007fbc88b51000 00007fbc88348b90
    0x00007fbcb7cf8500:   00007fbc883f4990 0000007888107a90
    0x00007fbcb7cf8510:   00007fbcb7cf8500 00007fbcc848c03f
    0x00007fbcb7cf8520:   00007fbcb7cf89e0 00000000000000f0
    0x00007fbcb7cf8530:   0000000000000009 0000000000000013
    0x00007fbcb7cf8540:   00007fbcc8f37280 00007fbcc8763c94
    0x00007fbcb7cf8550:   0000000000000008 00007fbcb7cfade0
    0x00007fbcb7cf8560:   00007fbcc8f37280 00007fbcc8763ac0
    0x00007fbcb7cf8570:   00007fbc88071000 0000000000000048
    0x00007fbcb7cf8580:   0000000000000002 0000000000070d60
    Instructions: (pc=0x00007fbcc8a16bd4)
    0x00007fbcc8a16bb4:   84 f6 0f 84 dc 1b 00 00 48 8b 8d c8 fc ff ff 45
    0x00007fbcc8a16bc4:   31 f6 8b 71 18 85 f6 74 07 48 8b 41 20 4c 8b 30
    0x00007fbcc8a16bd4:   49 8b 07 4c 89 ff ff 50 28 48 8b b5 c8 fc ff ff
    0x00007fbcc8a16be4:   48 89 85 78 fc ff ff 31 c0 8b 56 18 85 d2 74 07
    Register to memory mapping:
    RAX=0x00007fbc89897540 is an unknown value
    RBX=0x00007fbc88e499f0 is an unknown value
    RCX=0x00007fbc898974b0 is an unknown value
    RDX=0x0000000000000000 is an unknown value
    RSP=0x00007fbcb7cf8390 is pointing into the stack for thread: 0x00007fbcc40e7800
    RBP=0x00007fbcb7cf8820 is pointing into the stack for thread: 0x00007fbcc40e7800
    RSI=0x0000000000000004 is an unknown value
    RDI=0x00007fbc88e499f0 is an unknown value
    R8 =0x0000000000000181 is an unknown value
    R9 =0x0000000000000004 is an unknown value
    R10=0x0000000000000001 is an unknown value
    R11=0x00007fbc88e69390 is an unknown value
    R12=0x0000000000000000 is an unknown value
    R13=0x0000000000000001 is an unknown value
    R14=0x00007fbc88e499f0 is an unknown value
    R15=0xf100000000000000 is an unknown value
    Stack: [0x00007fbcb7bfc000,0x00007fbcb7cfd000],  sp=0x00007fbcb7cf8390,  free space=1008k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V  [libjvm.so+0x7b5bd4]  PhaseChaitin::Split(unsigned int)+0x8e4
    V  [libjvm.so+0x347d4e]  PhaseChaitin::Register_Allocate()+0x48e
    V  [libjvm.so+0x3ba1dd]  Compile::Code_Gen()+0x3ad
    V  [libjvm.so+0x3bcde5]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool)+0xe15
    V  [libjvm.so+0x32eaf2]  C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x142
    V  [libjvm.so+0x3c1d03]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x323
    V  [libjvm.so+0x3c2e6d]  CompileBroker::compiler_thread_loop()+0x43d
    V  [libjvm.so+0x864ec8]  JavaThread::thread_main_inner()+0xc8
    V  [libjvm.so+0x865018]  JavaThread::run()+0x138
    V  [libjvm.so+0x746a00]  java_start(Thread*)+0x100
    Current CompileTask:
    C2:5652858  298             Main.GameObject::aiMoveUnit (1510 bytes)
    ---------------  P R O C E S S  ---------------
    Java Threads: ( => current thread )
      0x00007fbc68018000 JavaThread "Thread-107" [_thread_in_native, id=4598, stack(0x00007fbcb53d9000,0x00007fbcb54da000)]
      0x00007fbc68017000 JavaThread "Thread-108" [_thread_blocked, id=4597, stack(0x00007fbcb52d8000,0x00007fbcb53d9000)]
      0x00007fbc68012000 JavaThread "Thread-100" [_thread_in_native, id=4442, stack(0x00007fbcb59df000,0x00007fbcb5ae0000)]
      0x00007fbc68011800 JavaThread "Thread-101" [_thread_blocked, id=4441, stack(0x00007fbcb58de000,0x00007fbcb59df000)]
      0x00007fbc68016000 JavaThread "Thread-93" [_thread_in_native, id=4405, stack(0x00007fbcb57dd000,0x00007fbcb58de000)]
      0x00007fbc68015000 JavaThread "Thread-94" [_thread_blocked, id=4404, stack(0x00007fbcb56dc000,0x00007fbcb57dd000)]
      0x00007fbc68014000 JavaThread "Thread-85" [_thread_in_native, id=4322, stack(0x00007fbcb54da000,0x00007fbcb55db000)]
      0x00007fbc68013000 JavaThread "Thread-86" [_thread_blocked, id=4321, stack(0x00007fbcb55db000,0x00007fbcb56dc000)]
      0x00007fbc6801d000 JavaThread "Thread-79" [_thread_in_native, id=4308, stack(0x00007fbcb68f2000,0x00007fbcb69f3000)]
      0x00007fbc68001000 JavaThread "Thread-80" [_thread_blocked, id=4307, stack(0x00007fbcb69f3000,0x00007fbcb6af4000)]
      0x00007fbc68010800 JavaThread "Thread-41" [_thread_in_native, id=3373, stack(0x00007fbcb5be1000,0x00007fbcb5ce2000)]
      0x00007fbc6800f800 JavaThread "Thread-42" [_thread_blocked, id=3372, stack(0x00007fbcb5ae0000,0x00007fbcb5be1000)]
      0x00007fbc6800e800 JavaThread "Thread-36" [_thread_in_native, id=3343, stack(0x00007fbcb5de3000,0x00007fbcb5ee4000)]
      0x00007fbc6800d800 JavaThread "Thread-37" [_thread_blocked, id=3342, stack(0x00007fbcb5ce2000,0x00007fbcb5de3000)]
      0x00007fbc6800c800 JavaThread "Thread-28" [_thread_in_native, id=3325, stack(0x00007fbcb5ee4000,0x00007fbcb5fe5000)]
      0x00007fbc6800b800 JavaThread "Thread-29" [_thread_blocked, id=3324, stack(0x00007fbcb5fe5000,0x00007fbcb60e6000)]
      0x00007fbc6800a000 JavaThread "Thread-19" [_thread_in_native, id=3214, stack(0x00007fbcb60e6000,0x00007fbcb61e7000)]
      0x00007fbc68008800 JavaThread "Thread-20" [_thread_blocked, id=3213, stack(0x00007fbcb61e7000,0x00007fbcb62e8000)]
      0x00007fbc68007800 JavaThread "Thread-17" [_thread_in_native, id=3211, stack(0x00007fbcb62e8000,0x00007fbcb63e9000)]
      0x00007fbc68006800 JavaThread "Thread-18" [_thread_blocked, id=3210, stack(0x00007fbcb63e9000,0x00007fbcb64ea000)]
      0x00007fbc68003000 JavaThread "Thread-11" [_thread_in_native, id=3190, stack(0x00007fbcb67f1000,0x00007fbcb68f2000)]
      0x00007fbc68002000 JavaThread "Thread-12" [_thread_blocked, id=3189, stack(0x00007fbcb64ea000,0x00007fbcb65eb000)]
      0x00007fbc74005800 JavaThread "Thread-5" [_thread_in_native, id=3162, stack(0x00007fbcb6af4000,0x00007fbcb6bf5000)]
      0x00007fbc74004000 JavaThread "Thread-1" [_thread_blocked, id=3161, stack(0x00007fbcb6bf5000,0x00007fbcb6cf6000)]
      0x00007fbc7800e800 JavaThread "Thread-3" [_thread_in_native, id=3160, stack(0x00007fbcb6cf6000,0x00007fbcb6df7000)]
      0x00007fbc7800d000 JavaThread "Thread-4" [_thread_blocked, id=3159, stack(0x00007fbcb6df7000,0x00007fbcb6ef8000)]
      0x00007fbcc4007800 JavaThread "DestroyJavaVM" [_thread_blocked, id=3135, stack(0x00007fbcc8160000,0x00007fbcc8261000)]
      0x00007fbcc4625800 JavaThread "Thread-2" [_thread_in_native, id=3149, stack(0x00007fbcb6ef8000,0x00007fbcb6ff9000)]
      0x00007fbcc4619800 JavaThread "Thread-0" [_thread_blocked, id=3148, stack(0x00007fbcb6ff9000,0x00007fbcb70fa000)]
      0x00007fbcc40ea000 JavaThread "Service Thread" daemon [_thread_blocked, id=3146, stack(0x00007fbcb7afb000,0x00007fbcb7bfc000)]
    =>0x00007fbcc40e7800 JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=3145, stack(0x00007fbcb7bfc000,0x00007fbcb7cfd000)]
      0x00007fbcc40e4800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3144, stack(0x00007fbcb7cfd000,0x00007fbcb7dfe000)]
      0x00007fbcc40e2800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3143, stack(0x00007fbcb7dfe000,0x00007fbcb7eff000)]
      0x00007fbcc4099000 JavaThread "Finalizer" daemon [_thread_blocked, id=3142, stack(0x00007fbcb7eff000,0x00007fbcb8000000)]
      0x00007fbcc4096800 JavaThread "Reference Handler" daemon [_thread_blocked, id=3141, stack(0x00007fbcbc0ef000,0x00007fbcbc1f0000)]
    Other Threads:
      0x00007fbcc408f800 VMThread [stack: 0x00007fbcbc1f0000,0x00007fbcbc2f1000] [id=3140]
      0x00007fbcc40f4800 WatcherThread [stack: 0x00007fbcb79fa000,0x00007fbcb7afb000] [id=3147]
    VM state:not at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread: None
    Heap
    PSYoungGen      total 108032K, used 64372K [0x00000000eaab0000, 0x00000000f53b0000, 0x0000000100000000)
      eden space 100224K, 63% used [0x00000000eaab0000,0x00000000ee8dd1b8,0x00000000f0c90000)
      from space 7808K, 9% used [0x00000000f0c90000,0x00000000f0d40000,0x00000000f1430000)
      to   space 7552K, 0% used [0x00000000f4c50000,0x00000000f4c50000,0x00000000f53b0000)
    ParOldGen       total 29952K, used 15594K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 52% used [0x00000000c0000000,0x00000000c0f3a848,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6931K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 32% used [0x00000000bae00000,0x00000000bb4c4c10,0x00000000bc2c0000)
    Card table byte_map: [0x00007fbcc0574000,0x00007fbcc079e000] byte_map_base: 0x00007fbcbff9d000
    Polling page: 0x00007fbcc91ab000
    Code Cache  [0x00007fbcc079e000, 0x00007fbcc0a0e000, 0x00007fbcc379e000)
    total_blobs=549 nmethods=296 adapters=205 free_code_cache=47745Kb largest_free_block=48842112
    Compilation events (10 events):
    Event: 5283.138 Thread 0x00007fbcc40e4800 nmethod 293 0x00007fbcc08e5950 code [0x00007fbcc08e5a80, 0x00007fbcc08e5af8]
    Event: 5447.824 Thread 0x00007fbcc40e7800  294             Main.GameObject::aiGetMovementPath (1009 bytes)
    Event: 5447.912 Thread 0x00007fbcc40e7800 nmethod 294 0x00007fbcc0901610 code [0x00007fbcc0901a00, 0x00007fbcc0903d70]
    Event: 5481.887 Thread 0x00007fbcc40e4800  295             Main.GameObject::aiGetTargetInRange (212 bytes)
    Event: 5481.896 Thread 0x00007fbcc40e4800 nmethod 295 0x00007fbcc08e5f90 code [0x00007fbcc08e61c0, 0x00007fbcc08e68f0]
    Event: 5536.215 Thread 0x00007fbcc40e7800  296             Main.GameObject::getBestPath (333 bytes)
    Event: 5536.245 Thread 0x00007fbcc40e7800 nmethod 296 0x00007fbcc0906950 code [0x00007fbcc0906c20, 0x00007fbcc0908190]
    Event: 5553.050 Thread 0x00007fbcc40e4800  297             IsoInfo.Main.Unit::canMove (58 bytes)
    Event: 5553.051 Thread 0x00007fbcc40e4800 nmethod 297 0x00007fbcc08e77d0 code [0x00007fbcc08e7920, 0x00007fbcc08e7a78]
    Event: 5652.525 Thread 0x00007fbcc40e7800  298             Main.GameObject::aiMoveUnit (1510 bytes)
    GC Heap History (10 events):
    Event: 907.123 GC heap before
    {Heap before GC invocations=8 (full 1):
    PSYoungGen      total 39808K, used 32448K [0x00000000eaab0000, 0x00000000eda10000, 0x0000000100000000)
      eden space 31936K, 100% used [0x00000000eaab0000,0x00000000ec9e0000,0x00000000ec9e0000)
      from space 7872K, 6% used [0x00000000ed1e0000,0x00000000ed260000,0x00000000ed990000)
      to   space 8192K, 0% used [0x00000000ec9e0000,0x00000000ec9e0000,0x00000000ed1e0000)
    ParOldGen       total 29952K, used 15297K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0ef0798,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6721K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 31% used [0x00000000bae00000,0x00000000bb4904f8,0x00000000bc2c0000)
    Event: 907.125 GC heap after
    Heap after GC invocations=8 (full 1):
    PSYoungGen      total 40128K, used 768K [0x00000000eaab0000, 0x00000000ef8d0000, 0x0000000100000000)
      eden space 31936K, 0% used [0x00000000eaab0000,0x00000000eaab0000,0x00000000ec9e0000)
      from space 8192K, 9% used [0x00000000ec9e0000,0x00000000ecaa0000,0x00000000ed1e0000)
      to   space 8256K, 0% used [0x00000000ef0c0000,0x00000000ef0c0000,0x00000000ef8d0000)
    ParOldGen       total 29952K, used 15297K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0ef0798,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6721K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 31% used [0x00000000bae00000,0x00000000bb4904f8,0x00000000bc2c0000)
    Event: 1532.671 GC heap before
    {Heap before GC invocations=9 (full 1):
    PSYoungGen      total 40128K, used 32704K [0x00000000eaab0000, 0x00000000ef8d0000, 0x0000000100000000)
      eden space 31936K, 100% used [0x00000000eaab0000,0x00000000ec9e0000,0x00000000ec9e0000)
      from space 8192K, 9% used [0x00000000ec9e0000,0x00000000ecaa0000,0x00000000ed1e0000)
      to   space 8256K, 0% used [0x00000000ef0c0000,0x00000000ef0c0000,0x00000000ef8d0000)
    ParOldGen       total 29952K, used 15297K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0ef0798,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6764K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 31% used [0x00000000bae00000,0x00000000bb49b0d0,0x00000000bc2c0000)
    Event: 1532.674 GC heap after
    Heap after GC invocations=9 (full 1):
    PSYoungGen      total 71232K, used 640K [0x00000000eaab0000, 0x00000000ef870000, 0x0000000100000000)
      eden space 63360K, 0% used [0x00000000eaab0000,0x00000000eaab0000,0x00000000ee890000)
      from space 7872K, 8% used [0x00000000ef0c0000,0x00000000ef160000,0x00000000ef870000)
      to   space 8128K, 0% used [0x00000000ee890000,0x00000000ee890000,0x00000000ef080000)
    ParOldGen       total 29952K, used 15297K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0ef0798,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6764K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 31% used [0x00000000bae00000,0x00000000bb49b0d0,0x00000000bc2c0000)
    Event: 2415.833 GC heap before
    {Heap before GC invocations=10 (full 1):
    PSYoungGen      total 71232K, used 64000K [0x00000000eaab0000, 0x00000000ef870000, 0x0000000100000000)
      eden space 63360K, 100% used [0x00000000eaab0000,0x00000000ee890000,0x00000000ee890000)
      from space 7872K, 8% used [0x00000000ef0c0000,0x00000000ef160000,0x00000000ef870000)
      to   space 8128K, 0% used [0x00000000ee890000,0x00000000ee890000,0x00000000ef080000)
    ParOldGen       total 29952K, used 15297K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0ef0798,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6811K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 32% used [0x00000000bae00000,0x00000000bb4a6d00,0x00000000bc2c0000)
    Event: 2415.834 GC heap after
    Heap after GC invocations=10 (full 1):
    PSYoungGen      total 71488K, used 960K [0x00000000eaab0000, 0x00000000f1c60000, 0x0000000100000000)
      eden space 63360K, 0% used [0x00000000eaab0000,0x00000000eaab0000,0x00000000ee890000)
      from space 8128K, 11% used [0x00000000ee890000,0x00000000ee980000,0x00000000ef080000)
      to   space 7936K, 0% used [0x00000000f14a0000,0x00000000f14a0000,0x00000000f1c60000)
    ParOldGen       total 29952K, used 15393K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0f087e8,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6811K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 32% used [0x00000000bae00000,0x00000000bb4a6d00,0x00000000bc2c0000)
    Event: 3564.860 GC heap before
    {Heap before GC invocations=11 (full 1):
    PSYoungGen      total 71488K, used 64320K [0x00000000eaab0000, 0x00000000f1c60000, 0x0000000100000000)
      eden space 63360K, 100% used [0x00000000eaab0000,0x00000000ee890000,0x00000000ee890000)
      from space 8128K, 11% used [0x00000000ee890000,0x00000000ee980000,0x00000000ef080000)
      to   space 7936K, 0% used [0x00000000f14a0000,0x00000000f14a0000,0x00000000f1c60000)
    ParOldGen       total 29952K, used 15393K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0f087e8,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6848K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 32% used [0x00000000bae00000,0x00000000bb4b0338,0x00000000bc2c0000)
    Event: 3564.862 GC heap after
    Heap after GC invocations=11 (full 1):
    PSYoungGen      total 107584K, used 576K [0x00000000eaab0000, 0x00000000f1bd0000, 0x0000000100000000)
      eden space 100224K, 0% used [0x00000000eaab0000,0x00000000eaab0000,0x00000000f0c90000)
      from space 7360K, 7% used [0x00000000f14a0000,0x00000000f1530000,0x00000000f1bd0000)
      to   space 7808K, 0% used [0x00000000f0c90000,0x00000000f0c90000,0x00000000f1430000)
    ParOldGen       total 29952K, used 15562K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0f32848,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6848K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 32% used [0x00000000bae00000,0x00000000bb4b0338,0x00000000bc2c0000)
    Event: 4661.819 GC heap before
    {Heap before GC invocations=12 (full 1):
    PSYoungGen      total 107584K, used 100800K [0x00000000eaab0000, 0x00000000f1bd0000, 0x0000000100000000)
      eden space 100224K, 100% used [0x00000000eaab0000,0x00000000f0c90000,0x00000000f0c90000)
      from space 7360K, 7% used [0x00000000f14a0000,0x00000000f1530000,0x00000000f1bd0000)
      to   space 7808K, 0% used [0x00000000f0c90000,0x00000000f0c90000,0x00000000f1430000)
    ParOldGen       total 29952K, used 15562K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 51% used [0x00000000c0000000,0x00000000c0f32848,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6868K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 32% used [0x00000000bae00000,0x00000000bb4b53c8,0x00000000bc2c0000)
    Event: 4661.820 GC heap after
    Heap after GC invocations=12 (full 1):
    PSYoungGen      total 108032K, used 704K [0x00000000eaab0000, 0x00000000f53b0000, 0x0000000100000000)
      eden space 100224K, 0% used [0x00000000eaab0000,0x00000000eaab0000,0x00000000f0c90000)
      from space 7808K, 9% used [0x00000000f0c90000,0x00000000f0d40000,0x00000000f1430000)
      to   space 7552K, 0% used [0x00000000f4c50000,0x00000000f4c50000,0x00000000f53b0000)
    ParOldGen       total 29952K, used 15594K [0x00000000c0000000, 0x00000000c1d40000, 0x00000000eaab0000)
      object space 29952K, 52% used [0x00000000c0000000,0x00000000c0f3a848,0x00000000c1d40000)
    PSPermGen       total 21248K, used 6868K [0x00000000bae00000, 0x00000000bc2c0000, 0x00000000c0000000)
      object space 21248K, 32% used [0x00000000bae00000,0x00000000bb4b53c8,0x00000000bc2c0000)
    Deoptimization events (10 events):
    Event: 1030.601 Thread 0x00007fbcc4619800 Uncommon trap -34 fr.pc 0x00007fbcc0863418
    Event: 1175.846 Thread 0x00007fbcc4619800 Uncommon trap -34 fr.pc 0x00007fbcc0863418
    Event: 1182.839 Thread 0x00007fbcc4619800 Uncommon trap -34 fr.pc 0x00007fbcc0863418
    Event: 1297.158 Thread 0x00007fbcc4619800 Uncommon trap -122 fr.pc 0x00007fbcc0851b3c
    Event: 1665.442 Thread 0x00007fbcc4619800 Uncommon trap -83 fr.pc 0x00007fbcc08a20fc
    Event: 1665.442 Thread 0x00007fbcc4619800 Uncommon trap -83 fr.pc 0x00007fbcc0881eb4
    Event: 1682.396 Thread 0x00007fbcc4619800 Uncommon trap -83 fr.pc 0x00007fbcc086908c
    Event: 1800.266 Thread 0x00007fbcc4619800 Uncommon trap -83 fr.pc 0x00007fbcc085da7c
    Event: 2579.579 Thread 0x00007fbcc4619800 Uncommon trap -12 fr.pc 0x00007fbcc08b2584
    Event: 4892.024 Thread 0x00007fbcc4619800 Uncommon trap -12 fr.pc 0x00007fbcc08b7598
    Internal exceptions (10 events):
    Event: 135.214 Thread 0x00007fbcc4619800 Threw 0x00000000eaf84e08 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jvm.cpp:1166
    Event: 135.215 Thread 0x00007fbcc4619800 Threw 0x00000000eafed3c8 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jvm.cpp:1166
    Event: 263.679 Thread 0x00007fbcc4619800 Threw 0x00000000eb855ba8 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jvm.cpp:1166
    Event: 769.972 Thread 0x00007fbcc4619800 Threw 0x00000000ebe42428 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jvm.cpp:1166
    Event: 769.973 Thread 0x00007fbcc4619800 Threw 0x00000000ebe46b98 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jvm.cpp:1166
    Event: 1297.157 Thread 0x00007fbcc4619800 Threw 0x00000000ebdad4e0 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jvm.cpp:1166
    Event: 1665.441 Thread 0x00007fbcc4619800 Threw 0x00000000eb8d6848 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jvm.cpp:1166
    Event: 2400.274 Thread 0x00007fbc68014000 Threw 0x00000000eabbec48 at /HUDSON/workspace/jdk7u6-2-build-linux-amd64-product/jdk7u6/hotspot/src/share/vm/prims/jni.cpp:742
    Event: 2579.579 Thread 0x00007fbcc4619800 Implicit null exception at 0x00007fbcc08b11b6 to 0x00007fbcc08b2571
    Event: 4892.024 Thread 0x00007fbcc4619800 Implicit null exception at 0x00007fbcc08b621a to 0x00007fbcc08b7585
    Events (10 events):
    Event: 4948.484 Thread 0x00007fbc74265000 Thread added: 0x00007fbc74265000
    Event: 4948.484 Thread 0x00007fbc7426b800 Thread added: 0x00007fbc7426b800
    Event: 4948.616 Executing VM operation: RevokeBias
    Event: 4948.616 Executing VM operation: RevokeBias done
    Event: 4948.616 Thread 0x00007fbc7426b800 Thread exited: 0x00007fbc7426b800
    Event: 4948.622 Executing VM operation: RevokeBias
    Event: 4948.622 Executing VM operation: RevokeBias done
    Event: 4948.622 Thread 0x00007fbc74265000 Thread exited: 0x00007fbc74265000
    Event: 5481.896 Thread 0x00007fbcc40e4800 flushing nmethod 0x00007fbcc0851690
    Event: 5536.245 Thread 0x00007fbcc40e7800 flushing nmethod 0x00007fbcc0862510
    Dynamic libraries:
    00400000-00401000 r-xp 00000000 08:02 658256                             /usr/java/jre1.7.0_06/bin/java
    00600000-00601000 rw-p 00000000 08:02 658256                             /usr/java/jre1.7.0_06/bin/java
    01ae6000-01b07000 rw-p 00000000 00:00 0                                  [heap]
    bae00000-bc2c0000 rw-p 00000000 00:00 0
    bc2c0000-c0000000 rw-p 00000000 00:00 0
    c0000000-c1d40000 rw-p 00000000 00:00 0
    c1d40000-eaab0000 rw-p 00000000 00:00 0
    eaab0000-f53b0000 rw-p 00000000 00:00 0
    f53b0000-100000000 rw-p 00000000 00:00 0
    3647c00000-3647c20000 r-xp 00000000 08:06 343                            /lib64/ld-2.12.so
    3647e1f000-3647e20000 r--p 0001f000 08:06 343                            /lib64/ld-2.12.so
    3647e20000-3647e21000 rw-p 00020000 08:06 343                            /lib64/ld-2.12.so
    3647e21000-3647e22000 rw-p 00000000 00:00 0
    3648000000-364818a000 r-xp 00000000 08:06 344                            /lib64/libc-2.12.so
    364818a000-3648389000 ---p 0018a000 08:06 344                            /lib64/libc-2.12.so
    3648389000-364838d000 r--p 00189000 08:06 344                            /lib64/libc-2.12.so
    364838d000-364838e000 rw-p 0018d000 08:06 344                            /lib64/libc-2.12.so
    364838e000-3648393000 rw-p 00000000 00:00 0
    3648400000-3648483000 r-xp 00000000 08:06 348                            /lib64/libm-2.12.so
    3648483000-3648682000 ---p 00083000 08:06 348                            /lib64/libm-2.12.so
    3648682000-3648683000 r--p 00082000 08:06 348                            /lib64/libm-2.12.so
    3648683000-3648684000 rw-p 00083000 08:06 348                            /lib64/libm-2.12.so
    3648800000-3648802000 r-xp 00000000 08:06 351                            /lib64/libdl-2.12.so
    3648802000-3648a02000 ---p 00002000 08:06 351                            /lib64/libdl-2.12.so
    3648a02000-3648a03000 r--p 00002000 08:06 351                            /lib64/libdl-2.12.so
    3648a03000-3648a04000 rw-p 00003000 08:06 351                            /lib64/libdl-2.12.so
    3648c00000-3648c17000 r-xp 00000000 08:06 345                            /lib64/libpthread-2.12.so
    3648c17000-3648e17000 ---p 00017000 08:06 345                            /lib64/libpthread-2.12.so
    3648e17000-3648e18000 r--p 00017000 08:06 345                            /lib64/libpthread-2.12.so
    3648e18000-3648e19000 rw-p 00018000 08:06 345                            /lib64/libpthread-2.12.so
    3648e19000-3648e1d000 rw-p 00000000 00:00 0
    3649400000-3649407000 r-xp 00000000 08:06 352                            /lib64/librt-2.12.so
    3649407000-3649606000 ---p 00007000 08:06 352                            /lib64/librt-2.12.so
    3649606000-3649607000 r--p 00006000 08:06 352                            /lib64/librt-2.12.so
    3649607000-3649608000 rw-p 00007000 08:06 352                            /lib64/librt-2.12.so
    364a000000-364a016000 r-xp 00000000 08:06 386                            /lib64/libresolv-2.12.so
    364a016000-364a216000 ---p 00016000 08:06 386                            /lib64/libresolv-2.12.so
    364a216000-364a217000 r--p 00016000 08:06 386                            /lib64/libresolv-2.12.so
    364a217000-364a218000 rw-p 00017000 08:06 386                            /lib64/libresolv-2.12.so
    364a218000-364a21a000 rw-p 00000000 00:00 0
    7fbc34000000-7fbc34021000 rw-p 00000000 00:00 0
    7fbc34021000-7fbc38000000 ---p 00000000 00:00 0
    7fbc38000000-7fbc38021000 rw-p 00000000 00:00 0
    7fbc38021000-7fbc3c000000 ---p 00000000 00:00 0
    7fbc3c000000-7fbc3c021000 rw-p 00000000 00:00 0
    7fbc3c021000-7fbc40000000 ---p 00000000 00:00 0
    7fbc40000000-7fbc40021000 rw-p 00000000 00:00 0
    7fbc40021000-7fbc44000000 ---p 00000000 00:00 0
    7fbc44000000-7fbc44021000 rw-p 00000000 00:00 0
    7fbc44021000-7fbc48000000 ---p 00000000 00:00 0
    7fbc48000000-7fbc48021000 rw-p 00000000 00:00 0
    7fbc48021000-7fbc4c000000 ---p 00000000 00:00 0
    7fbc4c000000-7fbc4c021000 rw-p 00000000 00:00 0
    7fbc4c021000-7fbc50000000 ---p 00000000 00:00 0
    7fbc50000000-7fbc50021000 rw-p 00000000 00:00 0
    7fbc50021000-7fbc54000000 ---p 00000000 00:00 0
    7fbc54000000-7fbc54040000 rw-p 00000000 00:00 0
    7fbc54040000-7fbc58000000 ---p 00000000 00:00 0
    7fbc58000000-7fbc58021000 rw-p 00000000 00:00 0
    7fbc58021000-7fbc5c000000 ---p 00000000 00:00 0
    7fbc5c000000-7fbc5c021000 rw-p 00000000 00:00 0
    7fbc5c021000-7fbc60000000 ---p 00000000 00:00 0
    7fbc60000000-7fbc60021000 rw-p 00000000 00:00 0
    7fbc60021000-7fbc64000000 ---p 00000000 00:00 0
    7fbc64000000-7fbc64021000 rw-p 00000000 00:00 0
    7fbc64021000-7fbc68000000 ---p 00000000 00:00 0
    7fbc68000000-7fbc68024000 rw-p 00000000 00:00 0
    7fbc68024000-7fbc6c000000 ---p 00000000 00:00 0
    7fbc6c000000-7fbc6c021000 rw-p 00000000 00:00 0
    7fbc6c021000-7fbc70000000 ---p 00000000 00:00 0
    7fbc70000000-7fbc70021000 rw-p 00000000 00:00 0
    7fbc70021000-7fbc74000000 ---p 00000000 00:00 0
    7fbc74000000-7fbc7428b000 rw-p 00000000 00:00 0
    7fbc7428b000-7fbc78000000 ---p 00000000 00:00 0
    7fbc78000000-7fbc78021000 rw-p 00000000 00:00 0
    7fbc78021000-7fbc7c000000 ---p 00000000 00:00 0
    7fbc7c000000-7fbc7c021000 rw-p 00000000 00:00 0
    7fbc7c021000-7fbc80000000 ---p 00000000 00:00 0
    7fbc80000000-7fbc80021000 rw-p 00000000 00:00 0
    7fbc80021000-7fbc84000000 ---p 00000000 00:00 0
    7fbc84000000-7fbc84a1b000 rw-p 00000000 00:00 0
    7fbc84a1b000-7fbc88000000 ---p 00000000 00:00 0
    7fbc88000000-7fbc8a3cf000 rw-p 00000000 00:00 0
    7fbc8a3cf000-7fbc8c000000 ---p 00000000 00:00 0
    7fbc8c000000-7fbc8c021000 rw-p 00000000 00:00 0
    7fbc8c021000-7fbc90000000 ---p 00000000 00:00 0
    7fbc92170000-7fbc98000000 r--p 00000000 08:02 796195                     /usr/lib/locale/locale-archive
    7fbc98000000-7fbc98021000 rw-p 00000000 00:00 0
    7fbc98021000-7fbc9c000000 ---p 00000000 00:00 0
    7fbc9c000000-7fbc9c021000 rw-p 00000000 00:00 0
    7fbc9c021000-7fbca0000000 ---p 00000000 00:00 0
    7fbca0000000-7fbca0021000 rw-p 00000000 00:00 0
    7fbca0021000-7fbca4000000 ---p 00000000 00:00 0
    7fbca4000000-7fbca4021000 rw-p 00000000 00:00 0
    7fbca4021000-7fbca8000000 ---p 00000000 00:00 0
    7fbca8000000-7fbca8021000 rw-p 00000000 00:00 0
    7fbca8021000-7fbcac000000 ---p 00000000 00:00 0
    7fbcb0000000-7fbcb0021000 rw-p 00000000 00:00 0
    7fbcb0021000-7fbcb4000000 ---p 00000000 00:00 0
    7fbcb50d6000-7fbcb50d9000 ---p 00000000 00:00 0
    7fbcb50d9000-7fbcb51d7000 rw-p 00000000 00:00 0
    7fbcb51d7000-7fbcb51da000 ---p 00000000 00:00 0
    7fbcb51da000-7fbcb52d8000 rw-p 00000000 00:00 0
    7fbcb52d8000-7fbcb52db000 ---p 00000000 00:00 0
    7fbcb52db000-7fbcb53d9000 rw-p 00000000 00:00 0
    7fbcb53d9000-7fbcb53dc000 ---p 00000000 00:00 0
    7fbcb53dc000-7fbcb54da000 rw-p 00000000 00:00 0
    7fbcb54da000-7fbcb54dd000 ---p 00000000 00:00 0
    7fbcb54dd000-7fbcb55db000 rw-p 00000000 00:00 0
    7fbcb55db000-7fbcb55de000 ---p 00000000 00:00 0
    7fbcb55de000-7fbcb56dc000 rw-p 00000000 00:00 0
    7fbcb56dc000-7fbcb56df000 ---p 00000000 00:00 0
    7fbcb56df000-7fbcb57dd000 rw-p 00000000 00:00 0
    7fbcb57dd000-7fbcb57e0000 ---p 00000000 00:00 0
    7fbcb57e0000-7fbcb58de000 rw-p 00000000 00:00 0
    7fbcb58de000-7fbcb58e1000 ---p 00000000 00:00 0
    7fbcb58e1000-7fbcb59df000 rw-p 00000000 00:00 0
    7fbcb59df000-7fbcb59e2000 ---p 00000000 00:00 0
    7fbcb59e2000-7fbcb5ae0000 rw-p 00000000 00:00 0
    7fbcb5ae0000-7fbcb5ae3000 ---p 00000000 00:00 0
    7fbcb5ae3000-7fbcb5be1000 rw-p 00000000 00:00 0
    7fbcb5be1000-7fbcb5be4000 ---p 00000000 00:00 0
    7fbcb5be4000-7fbcb5ce2000 rw-p 00000000 00:00 0
    7fbcb5ce2000-7fbcb5ce5000 ---p 00000000 00:00 0
    7fbcb5ce5000-7fbcb5de3000 rw-p 00000000 00:00 0
    7fbcb5de3000-7fbcb5de6000 ---p 00000000 00:00 0
    7fbcb5de6000-7fbcb5ee4000 rw-p 00000000 00:00 0
    7fbcb5ee4000-7fbcb5ee7000 ---p 00000000 00:00 0
    7fbcb5ee7000-7fbcb5fe5000 rw-p 00000000 00:00 0
    7fbcb5fe5000-7fbcb5fe8000 ---p 00000000 00:00 0
    7fbcb5fe8000-7fbcb60e6000 rw-p 00000000 00:00 0
    7fbcb60e6000-7fbcb60e9000 ---p 00000000 00:00 0
    7fbcb60e9000-7fbcb61e7000 rw-p 00000000 00:00 0
    7fbcb61e7000-7fbcb61ea000 ---p 00000000 00:00 0
    7fbcb61ea000-7fbcb62e8000 rw-p 00000000 00:00 0
    7fbcb62e8000-7fbcb62eb000 ---p 00000000 00:00 0
    7fbcb62eb000-7fbcb63e9000 rw-p 00000000 00:00 0
    7fbcb63e9000-7fbcb63ec000 ---p 00000000 00:00 0
    7fbcb63ec000-7fbcb64ea000 rw-p 00000000 00:00 0
    7fbcb64ea000-7fbcb64ed000 ---p 00000000 00:00 0
    7fbcb64ed000-7fbcb65eb000 rw-p 00000000 00:00 0
    7fbcb65eb000-7fbcb65f0000 r-xp 00000000 08:06 8064                       /lib64/libnss_dns-2.12.so
    7fbcb65f0000-7fbcb67ef000 ---p 00005000 08:06 8064                       /lib64/libnss_dns-2.12.so
    7fbcb67ef000-7fbcb67f0000 r--p 00004000 08:06 8064                       /lib64/libnss_dns-2.12.so
    7fbcb67f0000-7fbcb67f1000 rw-p 00005000 08:06 8064                       /lib64/libnss_dns-2.12.so
    7fbcb67f1000-7fbcb67f4000 ---p 00000000 00:00 0
    7fbcb67f4000-7fbcb68f2000 rw-p 00000000 00:00 0
    7fbcb68f2000-7fbcb68f5000 ---p 00000000 00:00 0
    7fbcb68f5000-7fbcb69f3000 rw-p 00000000 00:00 0
    7fbcb69f3000-7fbcb69f6000 ---p 00000000 00:00 0
    7fbcb69f6000-7fbcb6af4000 rw-p 00000000 00:00 0
    7fbcb6af4000-7fbcb6af7000 ---p 00000000 00:00 0
    7fbcb6af7000-7fbcb6bf5000 rw-p 00000000 00:00 0
    7fbcb6bf5000-7fbcb6bf8000 ---p 00000000 00:00 0
    7fbcb6bf8000-7fbcb6cf6000 rw-p 00000000 00:00 0
    7fbcb6cf6000-7fbcb6cf9000 ---p 00000000 00:00 0
    7fbcb6cf9000-7fbcb6df7000 rw-p 00000000 00:00 0
    7fbcb6df7000-7fbcb6dfa000 ---p 00000000 00:00 0
    7fbcb6dfa000-7fbcb6ef8000 rw-p 00000000 00:00 0
    7fbcb6ef8000-7fbcb6efb000 ---p 00000000 00:00 0
    7fbcb6efb000-7fbcb6ff9000 rw-p 00000000 00:00 0
    7fbcb6ff9000-7fbcb6ffc000 ---p 00000000 00:00 0
    7fbcb6ffc000-7fbcb70fa000 rw-p 00000000 00:00 0
    7fbcb70fa000-7fbcb710f000 r-xp 00000000 08:02 658313                     /usr/java/jre1.7.0_06/lib/amd64/libnet.so
    7fbcb710f000-7fbcb730f000 ---p 00015000 08:02 658313                     /usr/java/jre1.7.0_06/lib/amd64/libnet.so
    7fbcb730f000-7fbcb7310000 rw-p 00015000 08:02 658313                     /usr/java/jre1.7.0_06/lib/amd64/libnet.so
    7fbcb7310000-7fbcb7320000 r-xp 00000000 08:02 658314                     /usr/java/jre1.7.0_06/lib/amd64/libnio.so
    7fbcb7320000-7fbcb7520000 ---p 00010000 08:02 658314                     /usr/java/jre1.7.0_06/lib/amd64/libnio.so
    7fbcb7520000-7fbcb7521000 rw-p 00010000 08:02 658314                     /usr/java/jre1.7.0_06/lib/amd64/libnio.so
    7fbcb7521000-7fbcb7527000 r-xp 00000000 08:02 658276                     /usr/java/jre1.7.0_06/lib/amd64/headless/libmawt.so
    7fbcb7527000-7fbcb7727000 ---p 00006000 08:02 658276                     /usr/java/jre1.7.0_06/lib/amd64/headless/libmawt.so
    7fbcb7727000-7fbcb7728000 rw-p 00006000 08:02 658276                     /usr/java/jre1.7.0_06/lib/amd64/headless/libmawt.so
    7fbcb7728000-7fbcb77ca000 r-xp 00000000 08:02 658281                     /usr/java/jre1.7.0_06/lib/amd64/libawt.so
    7fbcb77ca000-7fbcb79ca000 ---p 000a2000 08:02 658281                     /usr/java/jre1.7.0_06/lib/amd64/libawt.so
    7fbcb79ca000-7fbcb79d6000 rw-p 000a2000 08:02 658281                     /usr/java/jre1.7.0_06/lib/amd64/libawt.so
    7fbcb79d6000-7fbcb79fa000 rw-p 00000000 00:00 0
    7fbcb79fa000-7fbcb79fb000 ---p 00000000 00:00 0
    7fbcb79fb000-7fbcb7afb000 rw-p 00000000 00:00 0
    7fbcb7afb000-7fbcb7afe000 ---p 00000000 00:00 0
    7fbcb7afe000-7fbcb7bfc000 rw-p 00000000 00:00 0
    7fbcb7bfc000-7fbcb7bff000 ---p 00000000 00:00 0
    7fbcb7bff000-7fbcb7cfd000 rw-p 00000000 00:00 0
    7fbcb7cfd000-7fbcb7d00000 ---p 00000000 00:00 0
    7fbcb7d00000-7fbcb7dfe000 rw-p 00000000 00:00 0
    7fbcb7dfe000-7fbcb7e01000 ---p 00000000 00:00 0
    7fbcb7e01000-7fbcb7eff000 rw-p 00000000 00:00 0
    7fbcb7eff000-7fbcb7f02000 ---p 00000000 00:00 0
    7fbcb7f02000-7fbcb8000000 rw-p 00000000 00:00 0
    7fbcb8000000-7fbcb8021000 rw-p 00000000 00:00 0
    7fbcb8021000-7fbcbc000000 ---p 00000000 00:00 0
    7fbcbc0ef000-7fbcbc0f2000 ---p 00000000 00:00 0
    7fbcbc0f2000-7fbcbc1f0000 rw-p 00000000 00:00 0
    7fbcbc1f0000-7fbcbc1f1000 ---p 00000000 00:00 0
    7fbcbc1f1000-7fbcbd31a000 rw-p 00000000 00:00 0
    7fbcbd31a000-7fbcbd4e6000 r--s 01665000 08:02 658895                     /usr/java/jre1.7.0_06/lib/rt.jar
    7fbcbd4e6000-7fbcc001a000 rw-p 00000000 00:00 0
    7fbcc001a000-7fbcc001b000 ---p 00000000 00:00 0
    7fbcc001b000-7fbcc011b000 rw-p 00000000 00:00 0
    7fbcc011b000-7fbcc011c000 ---p 00000000 00:00 0
    7fbcc011c000-7fbcc021c000 rw-p 00000000 00:00 0
    7fbcc021c000-7fbcc021d000 ---p 00000000 00:00 0
    7fbcc021d000-7fbcc031d000 rw-p 00000000 00:00 0
    7fbcc031d000-7fbcc031e000 ---p 00000000 00:00 0
    7fbcc031e000-7fbcc042d000 rw-p 00000000 00:00 0
    7fbcc042d000-7fbcc0574000 rw-p 00000000 00:00 0
    7fbcc0574000-7fbcc057f000 rw-p 00000000 00:00 0
    7fbcc057f000-7fbcc059d000 rw-p 00000000 00:00 0
    7fbcc059d000-7fbcc05ac000 rw-p 00000000 00:00 0
    7fbcc05ac000-7fbcc06f2000 rw-p 00000000 00:00 0
    7fbcc06f2000-7fbcc0747000 rw-p 00000000 00:00 0
    7fbcc0747000-7fbcc079d000 rw-p 00000000 00:00 0
    7fbcc079d000-7fbcc079e000 rw-p 00000000 00:00 0
    7fbcc079e000-7fbcc0a0e000 rwxp 00000000 00:00 0
    7fbcc0a0e000-7fbcc379e000 rw-p 00000000 00:00 0
    7fbcc379e000-7fbcc37b8000 r-xp 00000000 08:02 658325                     /usr/java/jre1.7.0_06/lib/amd64/libzip.so
    7fbcc37b8000-7fbcc39b8000 ---p 0001a000 08:02 658325                     /usr/java/jre1.7.0_06/lib/amd64/libzip.so
    7fbcc39b8000-7fbcc39b9000 rw-p 0001a000 08:02 658325                     /usr/java/jre1.7.0_06/lib/amd64/libzip.so
    7fbcc39b9000-7fbcc39c5000 r-xp 00000000 08:06 8068                       /lib64/libnss_files-2.12.so
    7fbcc39c5000-7fbcc3bc5000 ---p 0000c000 08:06 8068                       /lib64/libnss_files-2.12.so
    7fbcc3bc5000-7fbcc3bc6000 r--p 0000c000 08:06 8068                       /lib64/libnss_files-2.12.so
    7fbcc3bc6000-7fbcc3bc7000 rw-p 0000d000 08:06 8068                       /lib64/libnss_files-2.12.so
    7fbcc3bc7000-7fbcc3bf0000 r-xp 00000000 08:02 658295                     /usr/java/jre1.7.0_06/lib/amd64/libjava.so
    7fbcc3bf0000-7fbcc3df0000 ---p 00029000 08:02 658295                     /usr/java/jre1.7.0_06/lib/amd64/libjava.so
    7fbcc3df0000-7fbcc3df2000 rw-p 00029000 08:02 658295                     /usr/java/jre1.7.0_06/lib/amd64/libjava.so
    7fbcc3df2000-7fbcc3dff000 r-xp 00000000 08:02 658324                     /usr/java/jre1.7.0_06/lib/amd64/libverify.so
    7fbcc3dff000-7fbcc3ffe000 ---p 0000d000 08:02 658324                     /usr/java/jre1.7.0_06/lib/amd64/libverify.so
    7fbcc3ffe000-7fbcc4000000 rw-p 0000c000 08:02 658324                     /usr/java/jre1.7.0_06/lib/amd64/libverify.so
    7fbcc4000000-7fbcc462c000 rw-p 00000000 00:00 0
    7fbcc462c000-7fbcc8000000 ---p 00000000 00:00 0
    7fbcc803d000-7fbcc804f000 r--s 000d8000 08:08 21758031                   /home/jesse/isoserver/IsoServer.jar
    7fbcc804f000-7fbcc8082000 rw-p 00000000 00:00 0
    7fbcc8082000-7fbcc80a0000 rw-p 00000000 00:00 0
    7fbcc80a0000-7fbcc80aa000 rw-p 00000000 00:00 0
    7fbcc80aa000-7fbcc8160000 rw-p 00000000 00:00 0
    7fbcc8160000-7fbcc8163000 ---p 00000000 00:00 0
    7fbcc8163000-7fbcc8261000 rw-p 00000000 00:00 0
    7fbcc8261000-7fbcc8ca8000 r-xp 00000000 08:02 658329                     /usr/java/jre1.7.0_06/lib/amd64/server/libjvm.so
    7fbcc8ca8000-7fbcc8ea7000 ---p 00a47000 08:02 658329                     /usr/java/jre1.7.0_06/lib/amd64/server/libjvm.so
    7fbcc8ea7000-7fbcc8f4a000 rw-p 00a46000 08:02 658329                     /usr/java/jre1.7.0_06/lib/amd64/server/libjvm.so
    7fbcc8f4a000-7fbcc8f89000 rw-p 00000000 00:00 0
    7fbcc8f89000-7fbcc8f9f000 r-xp 00000000 08:02 658278                     /usr/java/jre1.7.0_06/lib/amd64/jli/libjli.so
    7fbcc8f9f000-7fbcc919e000 ---p 00016000 08:02 658278                     /usr/java/jre1.7.0_06/lib/amd64/jli/libjli.so
    7fbcc919e000-7fbcc919f000 rw-p 00015000 08:02 658278                     /usr/java/jre1.7.0_06/lib/amd64/jli/libjli.so
    7fbcc919f000-7fbcc91a0000 rw-p 00000000 00:00 0
    7fbcc91a1000-7fbcc91a2000 rw-p 00000000 00:00 0
    7fbcc91a2000-7fbcc91aa000 rw-s 00000000 08:07 292                        /tmp/hsperfdata_root/3126
    7fbcc91aa000-7fbcc91ab000 rw-p 00000000 00:00 0
    7fbcc91ab000-7fbcc91ac000 r--p 00000000 00:00 0
    7fbcc91ac000-7fbcc91ad000 rw-p 00000000 00:00 0
    7fffad99a000-7fffad9af000 rw-p 00000000 00:00 0                          [stack]
    7fffad9ff000-7fffada00000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
    VM Arguments:
    jvm_args: -Xmx1024m
    java_command: Main.IsoServer nogui
    Launcher Type: SUN_STANDARD
    Environment Variables:
    JAVA_HOME=/usr/local/jdk
    CLASSPATH=.:/usr/local/jdk/lib/classes.zip:/usr/local/jdk/lib/mysql-connector-java-5.1.6.jar
    PATH=/usr/local/jdk/bin:/usr/lib64/qt-3.3/bin:/usr/lib/courier-imap/sbin:/usr/lib/courier-imap/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/root/bin
    SHELL=/bin/bash
    Signal Handlers:
    SIGSEGV: [libjvm.so+0x8a5a80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGBUS: [libjvm.so+0x8a5a80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGFPE: [libjvm.so+0x741b60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGPIPE: [libjvm.so+0x741b60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGXFSZ: [libjvm.so+0x741b60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGILL: [libjvm.so+0x741b60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
    SIGUSR2: [libjvm.so+0x7414b0], sa_mask[0]=0x00000000, sa_flags=0x10000004
    SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
    SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
    SIGTERM: [libjvm.so+0x743840], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    SIGQUIT: [libjvm.so+0x743840], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
    ---------------  S Y S T E M  ---------------
    OS:CentOS release 6.4 (Final)
    uname:Linux 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST 2012 x86_64
    libc:glibc 2.12 NPTL 2.12
    rlimit: STACK 10240k, CORE 0k, NPROC 15830, NOFILE 1024, AS infinity
    load average:0.08 0.02 0.01
    /proc/meminfo:
    MemTotal:        2045524 kB
    MemFree:           93212 kB
    Buffers:          215936 kB
    Cached:           957604 kB
    SwapCached:         7300 kB
    Active:           977036 kB
    Inactive:         786260 kB
    Active(anon):     338308 kB
    Inactive(anon):   253632 kB
    Active(file):     638728 kB
    Inactive(file):   532628 kB
    Unevictable:           0 kB
    Mlocked:               0 kB
    SwapTotal:       4095992 kB
    SwapFree:        4084172 kB
    Dirty:                20 kB
    Writeback:             0 kB
    AnonPages:        586236 kB
    Mapped:            31028 kB
    Shmem:              2128 kB
    Slab:             133656 kB
    SReclaimable:     102016 kB
    SUnreclaim:        31640 kB
    KernelStack:        2056 kB
    PageTables:        14972 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:     5118752 kB
    Committed_AS:    2418356 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:      275704 kB
    VmallocChunk:   34359355248 kB
    HardwareCorrupted:     0 kB
    AnonHugePages:    233472 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    DirectMap4k:        7744 kB
    DirectMap2M:     2080768 kB
    CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 30 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, tsc, tscinvbit
    /proc/cpuinfo:
    processor    : 0
    vendor_id    : GenuineIntel
    cpu family    : 6
    model        : 30
    model name    : Intel(R) Core(TM) i5 CPU         760  @ 2.80GHz
    stepping    : 5
    cpu MHz        : 2800.022
    cache size    : 8192 KB
    physical id    : 0
    siblings    : 4
    core id        : 0
    cpu cores    : 4
    apicid        : 0
    initial apicid    : 0
    fpu        : yes
    fpu_exception    : yes
    cpuid level    : 11
    wp        : yes
    flags        : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
    bogomips    : 5600.04
    clflush size    : 64
    cache_alignment    : 64
    address sizes    : 36 bits physical, 48 bits virtual
    power management:
    processor    : 1
    vendor_id    : GenuineIntel
    cpu family    : 6
    model        : 30
    model name    : Intel(R) Core(TM) i5 CPU         760  @ 2.80GHz
    stepping    : 5
    cpu MHz        : 2800.022
    cache size    : 8192 KB
    physical id    : 0
    siblings    : 4
    core id        : 1
    cpu cores    : 4
    apicid        : 2
    initial apicid    : 2
    fpu        : yes
    fpu_exception    : yes
    cpuid level    : 11
    wp        : yes
    flags        : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
    bogomips    : 5599.14
    clflush size    : 64
    cache_alignment    : 64
    address sizes    : 36 bits physical, 48 bits virtual
    power management:
    processor    : 2
    vendor_id    : GenuineIntel
    cpu family    : 6
    model        : 30
    model name    : Intel(R) Core(TM) i5 CPU         760  @ 2.80GHz
    stepping    : 5
    cpu MHz        : 2800.022
    cache size    : 8192 KB
    physical id    : 0
    siblings    : 4
    core id        : 2
    cpu cores    : 4
    apicid        : 4
    initial apicid    : 4
    fpu        : yes
    fpu_exception    : yes
    cpuid level    : 11
    wp        : yes
    flags        : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
    bogomips    : 5599.14
    clflush size    : 64
    cache_alignment    : 64
    address sizes    : 36 bits physical, 48 bits virtual
    power management:
    processor    : 3
    vendor_id    : GenuineIntel
    cpu family    : 6
    model        : 30
    model name    : Intel(R) Core(TM) i5 CPU         760  @ 2.80GHz
    stepping    : 5
    cpu MHz        : 2800.022
    cache size    : 8192 KB
    physical id    : 0
    siblings    : 4
    core id        : 3
    cpu cores    : 4
    apicid        : 6
    initial apicid    : 6
    fpu        : yes
    fpu_exception    : yes
    cpuid level    : 11
    wp        : yes
    flags        : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm ida dts tpr_shadow vnmi flexpriority ept vpid
    bogomips    : 5599.14
    clflush size    : 64
    cache_alignment    : 64
    address sizes    : 36 bits physical, 48 bits virtual
    power management:
    Memory: 4k page, physical 2045524k(93212k free), swap 4095992k(4084172k free)
    vm_info: Java HotSpot(TM) 64-Bit Server VM (23.2-b09) for linux-amd64 JRE (1.7.0_06-b24), built on Aug  9 2012 19:49:48 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
    time: Tue Jun 18 13:37:03 2013
    elapsed time: 5652 seconds

    Hi!
    I've got the jvm crashing when it calls native methods
    via the JNI.
    The same native method run without a glitch in "pure"
    C. So there is no bug in the native method, I suppose.
    What is more interesting
    is that the same java program with the same native
    interface runs fine using gij (from gnu) too!
    $ java -version
    java version "1.4.2"
    Java(TM) 2 Runtime Environment, Standard Edition
    (build 1.4.2-b28)
    Java HotSpot(TM) Client VM (build 1.4.2-b28, mixed
    mode)
    The native methods are in C. Compiled using gcc.
    $ gcc -v
    Reading specs from
    /usr/lib/gcc-lib/i386-redhat-linux/3.2/specs
    Configured with: ../configure --prefix=/usr
    mandir=/usr/share/man infodir=/usr/share/info
    enable-shared enable-threads=posix
    disable-checking host=i386-redhat-linux
    with-system-zlib enable-__cxa_atexit
    Thread model: posix
    gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)
    The machine is P4 on Red hat linux 8.0
    $ uname -a
    Linux vanee.mlaproxy.iitk.ac.in 2.4.18-14 #1 Wed Sep 4
    13:35:50 EDT 2002 i686 i686 i386 GNU/Linux
    The error message is
    An unexpected exception has been detected in native
    code outside the VM.
    Unexpected Signal : 11 occurred at PC=0x4CDBEDA3
    Function=Java_Medico_Producer_vanee+0x2B3
    Library=/home/asingh/Java/medico/develop/Source/libnati
    elib.so
    Current Java thread:
    at Medico.Producer.vanee(Native Method)
    at Medico.Producer.run(Producer.java:18)
    If you are interested, I can also post the dyanamic
    libraries.
    Any help is appreciated. Please cc me a copy at
    [email protected]
    Thanks,
    Abhishek.Signal 11 is a segmentation fault. If this runs fine in pure C then you may be passing it a null value on the Java side. I am unfamilar with gij, so I don't know what to say about that

  • Investigation on JVM crash without core dump (by JIT compiler)

    Hi, All
    I posted "JVM crash without core dump due to CompilerThread1" couple months ago.
    http://forum.java.sun.com/thread.jspa?threadID=5253434
    I would like show what we found, and ask couple questions
    1) The reason of JVM crash is "CompilerThread0/1 wanted to allocate more native memory and eventually exceeded the limit of 32-bit Linux (on Redhat, the limit of VIRT is 3G)
    2) After we lower the heap size (to reduce process size) and added process size monitor (track process size every minute) and JIT compilation log (-XX:+PrintCompilation),
    We found sometimes JVM process jumped more than 800M when compiling one method. with the following log
    Total time for which application threads were stopped: 0.1997400 seconds
    5828 xxx.xxx.SomeClass::someMethod (1507 bytes)
    5828 COMPILE SKIPPED: out of nodes during split (not retryable)
    549002.449: [GC [PSYoungGen: 517897K->28390K(551296K)] 942855K->453348K(1229952K), 0.0726350 secs]
    if we saw "COMPILE SKIPPED: out of nodes during split (not retryable)", the process size of JVM (VIRT/RES) always jumped 800M~1000M, and sometimes the memory get reverted in 30mins to couple hours, and sometimes it lasted forever. (so before we have larger footprint, this jump will kill JVM, right now, if the jump lasts forever, any more allocation on top of it also can kill jvm (much rare))
    3) This is definitely a bug of JVM, because
    it only happened on server mode, not client mode
    it is random, that method can be compiled success on other JVM or next restart (we have 20 JVMs)
    the method is not that complicate, like 100 lines, (bigger method gets compiled success)
    this only happen on PROD environment, we can't reproduce it locally or QA (the method is always compiled success)
    It mostly like under some condition (maybe node space is not enough), to compile that method will trigger JVM trying to allocate much more native memory.
    4) We are going to disable this method by ".hotspot_compiler" to fix it, right now we are using lower footprint, JVM dies rarely.
    +To understand more about this, I have some questions about JIT compiling+
    I saw same method are compiled more than once from JIT log, like the method caused our problem, it only happens on 2nd time compilation.
    Is it that JIT compiler will recompile the method with deeper optimization level some time, and more optimization it uses, the more memory it requires?
    (like gcc has -O2 -O3)
    Thanks!
    Neo

    Today, one JVM crashed again, with
    {Heap before gc invocations=4188:
    PSYoungGen      total 540864K, used 487958K [0x8ba50000, 0xb1250000, 0xb1250000)
      eden space 467328K, 100% used [0x8ba50000,0xa82b0000,0xa82b0000)
      from space 73536K, 28% used [0xa82b0000,0xa96d5850,0xaca80000)
      to   space 71680K, 0% used [0xacc50000,0xacc50000,0xb1250000)
    ParOldGen       total 1024000K, used 493477K [0x4d250000, 0x8ba50000, 0x8ba50000)
      object space 1024000K, 48% used [0x4d250000,0x6b439610,0x8ba50000)
    PSPermGen       total 101760K, used 101240K [0x2d250000, 0x335b0000, 0x4d250000)
      object space 101760K, 99% used [0x2d250000,0x3352e1f8,0x335b0000)
    405290.711: [GC [PSYoungGen: 487958K->17397K(544192K)] 981435K->518248K(1568192K), 0.0769350 secs]
    Heap after gc invocations=4188:
    PSYoungGen total 544192K, used 17397K [0x8ba50000, 0xb1250000, 0xb1250000)
    eden space 472512K, 0% used [0x8ba50000,0x8ba50000,0xa87c0000)
    from space 71680K, 24% used [0xacc50000,0xadd4d480,0xb1250000)
    to space 70208K, 0% used [0xa87c0000,0xa87c0000,0xacc50000)
    ParOldGen total 1024000K, used 500851K [0x4d250000, 0x8ba50000, 0x8ba50000)
    object space 1024000K, 48% used [0x4d250000,0x6bb6cf70,0x8ba50000)
    PSPermGen total 101760K, used 101240K [0x2d250000, 0x335b0000, 0x4d250000)
    object space 101760K, 99% used [0x2d250000,0x3352e1f8,0x335b0000)
    Total time for which application threads were stopped: 0.0779620 seconds
    Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 4522768 bytes for Chunk::new. Out of swap space?

Maybe you are looking for