Rapid memory growth in jvm
Hi,
We have a production java based server that experienced large memory growth recently and we are looking for help in diagnosing what might be the cause. The memory footprint of this java server application is limited to 120M using the jvm -Xmx switch. The server process size held consistently at around 180M until an event happened that caused the process size to increase rapidly to aprox 3G within a 4 minute period. We used the java jstat command to check the application heap usage within the jvm and it was within the specified limits.
We have collected a variety of diagnostic information (pmap, pstack, pfiles, etc - available on request) as well as output from dtrace using a script that is used to count the number of memory allocations. The dtrace script and tail end of this message.
Any help in interpreting the dtrace output in the hopes of pinpointing a cause for the large memory allocations would be greatly appreciated.
Steve
java version information:
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode)
dtrace script:
#!/usr/sbin/dtrace -d -F
pid$target::malloc:entry
@totalbytes[ ustack() ] = sum( arg0 );
pid$target::valloc:entry
@totalbytes[ ustack() ] = sum( arg0 );
pldd information:
8348: /export/home/sonusComm/jre/bin/java -Dname=agent -Xmx120M -DAGENT_HOME
/lib/libumem.so.1
/lib/libthread.so.1
/lib/libdl.so.1
/lib/libc.so.1
/platform/sun4u-us3/lib/libc_psr.so.1
/export/home/sonusComm/jre/lib/sparc/server/libjvm.so
/lib/libsocket.so.1
/usr/lib/libsched.so.1
/usr/lib/libCrun.so.1
/lib/libm.so.1
/lib/libnsl.so.1
/lib/libm.so.2
/lib/libscf.so.1
/lib/libdoor.so.1
/lib/libuutil.so.1
/lib/libmd5.so.1
/platform/sun4u/lib/libmd5_psr.so.1
/lib/libmp.so.2
/export/home/sonusComm/jre/lib/sparc/native_threads/libhpi.so
/export/home/sonusComm/jre/lib/sparc/libverify.so
/export/home/sonusComm/jre/lib/sparc/libjava.so
/export/home/sonusComm/jre/lib/sparc/libzip.so
/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3
/export/home/sonusComm/jre/lib/sparc/libnet.so
/export/home/sonusComm/jre/lib/sparc/libmanagement.so
/export/home/sonusComm/jre/lib/sparc/libj2pkcs11.so
/usr/lib/libpkcs11.so.1
/usr/lib/libcryptoutil.so.1
/usr/lib/security/pkcs11_softtoken.so.1
/export/home/sonusComm/jre/lib/sparc/librmi.so
dtrace output snippet (the numeric value at the end of each stack is the number of bytes allocated)
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2a8
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x720
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
9269948
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cHMatcherFxform6MpnENode_i_2_+0xac
libjvm.so`__1cHMatcherFmatch6M_v_+0x644
libjvm.so`__1cHCompileICode_Gen6M_v_+0xd4
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10008720
libumem.so.1`malloc
libc.so.1`_real_gettext_u+0x98
libc.so.1`dgettext+0x98
libhpi.so`0xff0623d0
libjava.so`JNU_ThrowIOExceptionWithLastError+0x28
libjava.so`Java_java_lang_UNIXProcess_forkAndExec+0x714
0xf8f4f250
0xf90dd264
0xf90e6cdc
0xf9101be0
0xf8ed8078
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10313472
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1180
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10371104
libumem.so.1`malloc
libnet.so`Java_java_net_SocketInputStream_socketRead0+0xcc
0xf8c668ec
0xf8f01088
0xf8c76344
0xf8c058b8
0xf8c05764
0xf8c595b4
0xf8e7e594
0xf8c96238
0xf8c05c2c
0xf8c05d3c
0xf8c05d3c
0xf8c05d3c
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
10469376
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cICHeapObj2n6FI_pv_+0x1c
libjvm.so`__1cKJavaThreadKinitialize6M_v_+0xc0
libjvm.so`__1cKJavaThread2t5B6MpFp0pnGThread__vI_v_+0x60
libjvm.so`JVM_StartThread+0x1c0
0xf8c0c280
0xf8c0c224
0xf8f40c7c
0xf9065edc
0xf913a744
0xf9132638
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
10552560
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cICHeapObj2n6FI_pv_+0x1c
libjvm.so`__1cFMutex2t5B6Mipkci_v_+0x20
libjvm.so`__1cHMonitor2t5B6Mipkci_v_+0x10
libjvm.so`__1cGThread2t5B6M_v_+0x118
libjvm.so`__1cKJavaThread2t5B6MpFp0pnGThread__vI_v_+0x34
libjvm.so`JVM_StartThread+0x1c0
0xf8c0c280
0xf8c0c224
0xf8f40c7c
0xf9065edc
0xf913a744
0xf9132638
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
11607816
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
12653032
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cJOopMapSetGall_do6FpknFframe_pnICodeBlob_pknLRegisterMap_pnKOopClosure_pFppnHoopDesc_9E_v9B9B_v_+0x5a0
libjvm.so`__1cJOopMapSetHoops_do6FpknFframe_pnICodeBlob_pknLRegisterMap_pnKOopClosure__v_+0x44
libjvm.so`__1cFframeRoops_code_blob_do6MpnKOopClosure_pknLRegisterMap__v_+0x28
libjvm.so`__1cKJavaThreadHoops_do6MpnKOopClosure__v_+0x19c
libjvm.so`__1cPThreadRootsTaskFdo_it6MpnNGCTaskManager_I_v_+0x50
libjvm.so`__1cMGCTaskThreadDrun6M_v_+0x1e0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
18146824
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x47c
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
20415552
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`jni_GetByteArrayElements+0x144
libjava.so`Java_java_lang_UNIXProcess_forkAndExec+0x58
0xf8f4f250
0xf90dd264
0xf90e6cdc
0xf9101be0
0xf8ed8078
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
38030987
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cICHeapObj2n6FI_pv_+0x1c
libjvm.so`JVM_StartThread+0x1a0
0xf8c0c280
0xf8c0c224
0xf8f40c7c
0xf9065edc
0xf913a744
0xf9132638
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
59094336
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cHThreadsZcreate_thread_roots_tasks6FpnLGCTaskQdDueue__v_+0x4c
libjvm.so`__1cKPSScavengeQinvoke_no_policy6Fpi_i_+0x7d0
libjvm.so`__1cKPSScavengeGinvoke6Fpi_v_+0x5c
libjvm.so`__1cUParallelScavengeHeapTfailed_mem_allocate6MpiIii_pnIHeapWord__+0xbc
libjvm.so`__1cbDVM_ParallelGCFailedAllocationEdoit6M_v_+0xa4
libjvm.so`__1cMVM_OperationIevaluate6M_v_+0x80
libjvm.so`__1cIVMThreadDrun6M_v_+0x6e0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
60336552
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeIout_grow6MI_v_+0xac
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0xacc
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
100790212
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeFclone6kM_p0_+0xb0
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0x8c8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
101445332
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0x244
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cMPhaseIterGVNJtransform6MpnENode__2_+0x98
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0xa3c
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
268435464
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cINodeHashEgrow6M_v_+0x74
libjvm.so`__1cINodeHashQhash_find_insert6MpnENode__2_+0x1ac
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x2f8
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
536936472
libumem.so.1`malloc
libjava.so`JNU_GetStringPlatformChars+0xd8
libjava.so`Java_java_io_UnixFileSystem_getBooleanAttributes0+0x80
0xf8c0c280
0xf8c0c224
0xf8c058b8
0xf8c36cf8
0xf8f83678
0xf8fd515c
0xf8c96238
0xf8c05764
0xf8c05764
0xf8c05764
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
-8588966992
This isn't a heap space issue with the java application though, which is what these tools you mentioned would be used to diagnose. The heap space usage of our java app is remaining constant and below the specified limit of 120M but the process continues to grow as we speak.
I can only conclude that this is most likely a jvm memory leak. I should have probably made this clearer in the subject heading.
Anyone know how to file a bug against the solaris jvm ?
Similar Messages
-
How to diagnose rapid memory growth in Solaris 10 java server application
Hi,
We have a production java based server that experienced large memory growth recently and we are looking for help in diagnosing what might be the cause. The memory footprint of this java server application is limited to 120M using the jvm -Xmx switch. The server process size held consistently at around 180M until an event happened that caused the process size to rapidly increase to aprox 3G. We used the java jstat command to check the application heap usage within the jvm was within the specified limits.
This application can be performs java Runtime exec's to run simple shell scripts from a variety of java threads in the server app.
We have attempted to search for a cause by analysing the output of the pmap, pfiles, and pstack commands but have been unsuccessfull at this. We have also attempted to run the libumem.so library with the application in the hopes of using mdb to find out where the heap usage is. Unfortunately when we followed the instruction on using limumem the application stopped working due an ELF error.
The following are what we used for libumem:
LD_PRELOAD=/usr/lib/libumem.so
UMEM_DEBUG=default
UMEM_LOGGING=transaction
export LD_PRELOAD UMEM_DEBUG UMEM_LOGGING
<run the application>
The error we got is:
ld.so.1: nohup: fatal: /usr/lib/libumem.so: wrong ELF class: ELFCLASS32
We are looking for advice on how to proceed in finding the cause for this growth. In addition to the diag info below we have captured the core file for the application from gcore. The following is the output from pfiles, pmap, pstack, and the showrev command.
Any help/suggestions appreciated.
Steve
java version information:
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode)
pmap:
24584: /export/home/sonusComm/jre/bin/java -Dname=agent -Xmx120M -DAGENT_HOME
Address Kbytes RSS Anon Locked Mode Mapped File
00010000 64 8 - - r-x-- java
0002E000 16 8 8 - rwx-- java
00032000 3896 2648 1400 - rwx-- [ heap ]
00400000 3579904 717400 591704 - rwx-- [ heap ]
E50F6000 32 32 24 - rw--R [ anon ]
E50FE000 8 8 8 - rw--R [ stack tid=189562 ]
E52F6000 32 32 16 - rw--R [ anon ]
E52FE000 8 8 8 - rw--R [ stack tid=180717 ]
E5CF6000 32 32 16 - rw--R [ anon ]
E5CFE000 8 8 8 - rw--R [ stack tid=170864 ]
E6DF6000 40 40 40 - rw--R [ stack tid=220763 ]
E6EF6000 40 40 40 - rw--R [ anon ]
E6FF6000 8 8 - - rw--R [ anon ]
E6FF8000 32 32 32 - rw--R [ stack tid=220225 ]
E71F6000 40 40 40 - rw--R [ anon ]
E72F6000 40 40 40 - rw--R [ anon ]
E73F8000 24 24 - - rw--R [ anon ]
E73FE000 8 8 8 - rw--R [ stack tid=193659 ]
E74F6000 40 40 40 - rw--R [ stack tid=220762 ]
E75F6000 40 40 40 - rw--R [ stack tid=215672 ]
E76F6000 8 8 - - rw--R [ anon ]
E76F8000 32 32 32 - rw--R [ stack tid=211403 ]
E77F8000 8 8 - - rw--R [ anon ]
E77FA000 24 24 24 - rw--R [ stack tid=206975 ]
E78F6000 40 40 40 - rw--R [ anon ]
E79F8000 32 32 32 - rw--R [ stack tid=220760 ]
E7AF6000 40 40 40 - rw--R [ anon ]
E7CF6000 40 40 40 - rw--R [ stack tid=220764 ]
E7EF6000 32 32 8 - rw--R [ anon ]
E7EFE000 8 8 8 - rw--R [ stack tid=185268 ]
E7FF6000 16 16 16 - rw--R [ anon ]
E7FFC000 16 16 16 - rw--R [ stack tid=140567 ]
E80F6000 8 8 8 - rw--R [ anon ]
E80FE000 8 8 8 - rw--R [ stack tid=145776 ]
E81FC000 16 16 16 - rw--R [ stack tid=124964 ]
E82F6000 40 40 40 - rw--R [ anon ]
E83F8000 32 32 32 - rw--R [ stack tid=1512 ]
E84F6000 8 8 8 - rw--R [ anon ]
E84FA000 8 8 8 - rw--R [ anon ]
E84FE000 8 8 8 - rw--R [ stack tid=119761 ]
E85F6000 32 32 8 - rw--R [ anon ]
E85FE000 8 8 8 - rw--R [ stack tid=202234 ]
E86F6000 16 16 8 - rw--R [ anon ]
E86FA000 24 24 24 - rw--R [ stack tid=197719 ]
E87F8000 32 32 32 - rw--R [ stack tid=1518 ]
E88F8000 8 8 8 - rw--R [ anon ]
E88FC000 16 16 16 - rw--R [ stack tid=173 ]
E89FA000 24 24 24 - rw--R [ stack tid=161359 ]
E8AF8000 32 32 32 - rw--R [ stack tid=34505 ]
E8BFE000 8 8 8 - rw--R [ stack tid=135384 ]
E8CF8000 8 8 8 - rw--R [ anon ]
E8CFC000 16 16 16 - rw--R [ stack tid=172 ]
E8DF6000 8 8 8 - rw--R [ anon ]
E8DFC000 16 16 16 - rw--R [ stack tid=150966 ]
E8EFA000 8 8 8 - rw--R [ anon ]
E8EFE000 8 8 8 - rw--R [ stack tid=130168 ]
E8FFA000 8 8 8 - rw--R [ anon ]
E8FFE000 8 8 8 - rw--R [ stack tid=166557 ]
E90FE000 8 8 8 - rw--R [ stack tid=44270 ]
E91F8000 32 32 32 - rw--R [ stack tid=368 ]
E92FA000 8 8 - - rw--R [ anon ]
E92FE000 8 8 8 - rw--R [ stack tid=156163 ]
E93F8000 32 32 32 - rw--R [ stack tid=82 ]
E94F8000 8 8 8 - rw--R [ anon ]
E94FC000 16 16 16 - rw--R [ stack tid=72 ]
E95F8000 16 16 8 - rw--R [ anon ]
E95FC000 16 16 16 - rw--R [ stack tid=175828 ]
E96FE000 8 8 8 - rw--R [ stack tid=67 ]
E97F8000 32 32 32 - rw--R [ stack tid=62 ]
E98FE000 8 8 8 - rw--R [ stack tid=65 ]
E99FE000 8 8 8 - rw--R [ stack tid=51517 ]
E9AF8000 32 32 32 - rw--R [ stack tid=59 ]
E9BF8000 32 32 32 - rw--R [ stack tid=56 ]
E9CF8000 8 8 8 - rw--R [ anon ]
E9CFC000 16 16 16 - rw--R [ stack tid=57 ]
E9DF8000 16 16 8 - rw--R [ anon ]
E9DFC000 16 16 16 - rw--R [ stack tid=58 ]
E9EF8000 32 32 32 - rw--R [ stack tid=52 ]
E9FF8000 32 32 32 - rw--R [ stack tid=51 ]
EA0F8000 32 32 32 - rw--R [ stack tid=50 ]
EA1F8000 32 32 32 - rw--R [ stack tid=49 ]
EA2F8000 32 32 32 - rw--R [ stack tid=48 ]
EA3F8000 32 32 32 - rw--R [ stack tid=46 ]
EA4F8000 32 32 32 - rw--R [ stack tid=45 ]
EA5F8000 8 8 8 - rw--R [ anon ]
EA5FC000 8 8 - - rw--R [ anon ]
EA5FE000 8 8 8 - rw--R [ stack tid=44 ]
EA6FE000 8 8 8 - rw--R [ stack tid=43 ]
EA7FE000 8 8 8 - rw--R [ stack tid=42 ]
EA8F6000 40 40 40 - rw--R [ stack tid=41 ]
EA9F8000 32 32 32 - rw--R [ stack tid=38 ]
EAAF8000 32 32 32 - rw--R [ stack tid=40 ]
EABFE000 8 8 8 - rw--R [ stack tid=39 ]
EACFE000 8 8 8 - rw--R [ stack tid=31 ]
EADFE000 8 8 8 - rw--R [ stack tid=30 ]
EAEF8000 32 32 32 - rw--R [ stack tid=33 ]
EAFF8000 32 32 32 - rw--R [ stack tid=28 ]
EB0F8000 32 32 32 - rw--R [ stack tid=27 ]
EB1F8000 8 8 8 - rw--R [ anon ]
EB1FE000 8 8 8 - rw--R [ stack tid=26 ]
EB2F8000 8 8 - - rw--R [ anon ]
EB2FA000 24 24 24 - rw--R [ stack tid=25 ]
EB3F8000 8 8 8 - rw--R [ anon ]
EB3FE000 8 8 8 - rw--R [ stack tid=24 ]
EB4F8000 8 8 8 - rw--R [ anon ]
EB4FC000 16 16 16 - rw--R [ stack tid=21 ]
EB580000 816 - - - r--s- dev:32,24 ino:716444
EB680000 760 - - - r--s- dev:32,24 ino:716443
EB780000 1160 - - - r--s- dev:32,24 ino:716441
EB900000 816 - - - r--s- dev:32,24 ino:728313
EBA00000 760 - - - r--s- dev:32,24 ino:728312
EBB00000 1160 - - - r--s- dev:32,24 ino:728310
EBC80000 1120 - - - r--s- dev:32,24 ino:728308
EBE7E000 8 8 8 - rw--R [ stack tid=20 ]
EBF78000 32 32 32 - rw--R [ stack tid=19 ]
EC07E000 8 8 8 - rw--R [ stack tid=18 ]
EC17E000 8 8 8 - rw--R [ stack tid=17 ]
EC27E000 8 8 8 - rw--R [ stack tid=16 ]
EC378000 32 32 32 - rw--R [ stack tid=15 ]
EC400000 6880 - - - r--s- dev:32,24 ino:716451
ECB78000 32 32 32 - rw--R [ stack tid=14 ]
ECC00000 808 - - - r--s- dev:32,24 ino:716464
ECD00000 592 - - - r--s- dev:32,24 ino:716465
ECE78000 16 16 8 - rw--R [ anon ]
ECE7C000 16 16 16 - rw--R [ stack tid=83 ]
ECF00000 784 - - - r--s- dev:32,24 ino:716758
ED07E000 8 8 8 - rw--R [ stack tid=12 ]
ED27C000 16 16 16 - rw--R [ stack tid=10 ]
ED37E000 8 8 8 - rw--R [ stack tid=9 ]
ED3D0000 128 64 64 - rwx-- [ anon ]
ED400000 16384 15424 15232 - rwx-- [ anon ]
F1400000 1408 1408 1408 - rwx-- [ anon ]
F1560000 3264 3264 3136 - rwx-- [ anon ]
F1890000 5568 5568 5504 - rwx-- [ anon ]
F1E00000 7424 768 704 - rwx-- [ anon ]
F6400000 4096 4096 3968 - rwx-- [ anon ]
F6800000 3072 3072 3072 - rwx-- [ anon ]
F6B00000 1408 1408 1408 - rwx-- [ anon ]
F6C60000 448 448 448 - rwx-- [ anon ]
F6CD0000 1344 1344 1344 - rwx-- [ anon ]
F6E20000 512 512 512 - rwx-- [ anon ]
F6EA0000 832 832 832 - rwx-- [ anon ]
F6F70000 128 128 128 - rwx-- [ anon ]
F6F90000 1344 1344 1344 - rwx-- [ anon ]
F70E0000 448 448 448 - rwx-- [ anon ]
F7150000 384 384 384 - rwx-- [ anon ]
F71B0000 128 128 128 - rwx-- [ anon ]
F71D0000 192 192 192 - rwx-- [ anon ]
F7200000 128 128 128 - rwx-- [ anon ]
F7220000 128 128 128 - rwx-- [ anon ]
F7240000 128 128 128 - rwx-- [ anon ]
F7260000 192 192 192 - rwx-- [ anon ]
F7290000 128 128 128 - rwx-- [ anon ]
F72B0000 64 64 64 - rwx-- [ anon ]
F72C0000 384 384 384 - rwx-- [ anon ]
F7320000 448 448 448 - rwx-- [ anon ]
F7390000 896 896 896 - rwx-- [ anon ]
F7470000 768 768 768 - rwx-- [ anon ]
F7530000 704 704 704 - rwx-- [ anon ]
F75E0000 576 576 576 - rwx-- [ anon ]
F7670000 576 576 576 - rwx-- [ anon ]
F7700000 448 448 448 - rwx-- [ anon ]
F7770000 384 384 384 - rwx-- [ anon ]
F77D0000 128 128 128 - rwx-- [ anon ]
F77F0000 192 192 192 - rwx-- [ anon ]
F7820000 320 320 320 - rwx-- [ anon ]
F7870000 128 128 128 - rwx-- [ anon ]
F7890000 128 128 128 - rwx-- [ anon ]
F78B0000 448 448 448 - rwx-- [ anon ]
F7920000 896 896 896 - rwx-- [ anon ]
F7A00000 768 768 768 - rwx-- [ anon ]
F7AC0000 704 704 704 - rwx-- [ anon ]
F7B70000 832 832 640 - rwx-- [ anon ]
F7C40000 704 704 384 - rwx-- [ anon ]
F7CF0000 448 448 384 - rwx-- [ anon ]
F7D60000 704 704 384 - rwx-- [ anon ]
F7E10000 384 384 192 - rwx-- [ anon ]
F7E70000 384 384 192 - rwx-- [ anon ]
F7ED0000 320 320 256 - rwx-- [ anon ]
F7F20000 256 256 192 - rwx-- [ anon ]
F7F60000 256 256 192 - rwx-- [ anon ]
F7FA0000 128 128 128 - rwx-- [ anon ]
F7FC0000 192 192 128 - rwx-- [ anon ]
F7FF0000 192 192 128 - rwx-- [ anon ]
F8020000 256 256 128 - rwx-- [ anon ]
F8060000 128 128 128 - rwx-- [ anon ]
F8080000 128 128 64 - rwx-- [ anon ]
F80A0000 192 192 192 - rwx-- [ anon ]
F80D0000 384 384 192 - rwx-- [ anon ]
F8130000 64 64 64 - rwx-- [ anon ]
F8140000 64 64 - - rwx-- [ anon ]
F8150000 128 128 128 - rwx-- [ anon ]
F8170000 128 128 128 - rwx-- [ anon ]
F8190000 128 128 128 - rwx-- [ anon ]
F81B0000 128 128 64 - rwx-- [ anon ]
F81D0000 1600 1600 768 - rwx-- [ anon ]
F8360000 1152 1152 384 - rwx-- [ anon ]
F8480000 960 472 - - rwx-- [ anon ]
F8570000 832 80 - - rwx-- [ anon ]
F8640000 128 8 - - rwx-- [ anon ]
F8660000 832 16 - - rwx-- [ anon ]
F8730000 576 - - - rwx-- [ anon ]
F87C0000 448 - - - rwx-- [ anon ]
F8830000 256 - - - rwx-- [ anon ]
F8870000 320 - - - rwx-- [ anon ]
F88C0000 256 - - - rwx-- [ anon ]
F8900000 256 - - - rwx-- [ anon ]
F8940000 256 - - - rwx-- [ anon ]
F8980000 192 - - - rwx-- [ anon ]
F89B0000 384 - - - rwx-- [ anon ]
F8A10000 320 - - - rwx-- [ anon ]
F8A60000 192 - - - rwx-- [ anon ]
F8A90000 256 - - - rwx-- [ anon ]
F8AD0000 256 - - - rwx-- [ anon ]
F8B10000 192 128 64 - rwx-- [ anon ]
F8B40000 128 128 128 - rwx-- [ anon ]
F8B60000 64 64 64 - rwx-- [ anon ]
F8B70000 192 192 128 - rwx-- [ anon ]
F8BA0000 128 128 64 - rwx-- [ anon ]
F8BC0000 128 - - - rwx-- [ anon ]
F8C10000 64 - - - rwx-- [ anon ]
F8C30000 8 8 - - r-x-- librmi.so
F8C40000 8 8 8 - rwx-- librmi.so
F8C60000 64 - - - r--s- dev:32,24 ino:728318
F8C80000 376 144 - - r-x-- pkcs11_softtoken.so.1
F8CEE000 16 - - - rwx-- pkcs11_softtoken.so.1
F8D7E000 8 8 8 - rw--R [ stack tid=8 ]
F8D90000 16 - - - r--s- dev:32,24 ino:728317
F8DA0000 232 - - - r--s- dev:32,24 ino:728316
F8DE0000 88 - - - r--s- dev:32,24 ino:716445
F8E90000 16 - - - r--s- dev:32,24 ino:728315
F8EA0000 136 - - - r--s- dev:32,24 ino:728309
F8ED0000 136 - - - r--s- dev:32,24 ino:728307
F8F78000 32 32 32 - rw--R [ stack tid=6 ]
F8F90000 88 - - - r--s- dev:32,24 ino:728314
F8FB0000 8 8 8 - rwx-- [ anon ]
F8FC0000 88 88 - - r-x-- libpkcs11.so.1
F8FE6000 24 16 8 - rwx-- libpkcs11.so.1
F8FEC000 8 - - - rwx-- libpkcs11.so.1
F9000000 4096 4096 4096 - rwx-- [ anon ]
F9400000 4096 4096 4096 - rwx-- [ anon ]
FB010000 24 24 - - r-x-- libcryptoutil.so.1
FB026000 8 8 - - rwx-- libcryptoutil.so.1
FB030000 56 8 - - r-x-- libj2pkcs11.so
FB04C000 8 - - - rwx-- libj2pkcs11.so
FB060000 352 - - - r--s- dev:32,24 ino:716547
FB0C0000 184 - - - r--s- dev:32,24 ino:716545
FB178000 8 8 8 - rw--R [ anon ]
FB17E000 8 8 8 - rw--R [ stack tid=5 ]
FB190000 8 - - - r---- [ anon ]
FB1A0000 160 - - - r--s- dev:32,24 ino:716544
FB1D0000 72 64 - - r-x-- libnet.so
FB1F0000 8 8 8 - rwx-- libnet.so
FB27E000 8 8 8 - rw--R [ stack tid=4 ]
FB290000 24 8 - - r-x-- libmanagement.so
FB2A4000 8 8 - - rwx-- libmanagement.so
FB2B0000 256 - - - r--s- dev:32,24 ino:716466
FB37E000 8 8 8 - rw--R [ stack tid=3 ]
FB390000 32 - - - r--s- dev:32,24 ino:716442
FB3A0000 120 - - - r--s- dev:32,24 ino:716467
FB3D0000 176 - - - r--s- dev:32,24 ino:716455
FB400000 8520 - - - r--s- dev:32,24 ino:717379
FBC60000 32 - - - r--s- dev:32,24 ino:728311
FBC70000 40 - - - r--s- dev:32,24 ino:716546
FBC80000 304 - - - r--s- dev:32,24 ino:716456
FBCD0000 176 - - - r--s- dev:32,24 ino:716759
FBD7E000 8 8 8 - rw--R [ stack tid=2 ]
FBD90000 32 - - - r--s- dev:32,24 ino:716463
FBDA0000 8 - - - r--s- dev:32,24 ino:716757
FBDB0000 160 - - - r--s- dev:32,24 ino:716756
FBDE0000 64 64 64 - rw--- [ anon ]
FBE00000 64 40 40 - rw--- [ anon ]
FBE20000 32 32 32 - rwx-- [ anon ]
FBE50000 8 8 8 - rwx-- [ anon ]
FBE52000 8 8 8 - rwx-- [ anon ]
FBE54000 8 8 8 - rwx-- [ anon ]
FBE56000 16 16 16 - rwx-- [ anon ]
FBE80000 24 24 24 - rwx-- [ anon ]
FBE86000 8 8 - - rwx-- [ anon ]
FBE88000 8 8 8 - rwx-- [ anon ]
FBE8A000 8 8 8 - rwx-- [ anon ]
FBE8C000 8 8 8 - rwx-- [ anon ]
FBE8E000 8 8 8 - rwx-- [ anon ]
FBE90000 8 8 8 - rwx-- [ anon ]
FBE92000 8 8 8 - rwx-- [ anon ]
FBE94000 8 8 8 - rwx-- [ anon ]
FBE96000 8 8 8 - rwx-- [ anon ]
FBE98000 8 8 8 - rwx-- [ anon ]
FBE9A000 8 8 8 - rwx-- [ anon ]
FBE9C000 8 8 - - rwx-- [ anon ]
FBF10000 32 32 32 - rwx-- [ anon ]
FBF30000 8 8 8 - rwx-- [ anon ]
FBF32000 8 8 8 - rwx-- [ anon ]
FBF34000 8 8 8 - rwx-- [ anon ]
FBF36000 16 16 16 - rwx-- [ anon ]
FBF58000 8 8 8 - rwx-- [ anon ]
FBF5A000 8 8 8 - rwx-- [ anon ]
FBF5C000 8 8 8 - rwx-- [ anon ]
FBF5E000 8 8 8 - rwx-- [ anon ]
FBF60000 8 8 8 - rwx-- [ anon ]
FBF62000 8 8 8 - rwx-- [ anon ]
FBF64000 8 8 8 - rwx-- [ anon ]
FBF66000 8 8 8 - rwx-- [ anon ]
FBF68000 8 8 - - rwx-- [ anon ]
FBF6A000 8 8 8 - rwx-- [ anon ]
FBF6C000 8 - - - rwx-- [ anon ]
FBF80000 480 - - - r--s- dev:32,24 ino:716808
FC000000 35016 - - - r--s- dev:32,24 ino:716764
FE240000 64 64 64 - rwx-- [ anon ]
FE260000 88 - - - r--s- dev:32,24 ino:716807
FE280000 1184 - - - r--s- dev:32,24 ino:716460
FE3B0000 8 - - - r--s- dev:32,24 ino:716468
FE3C0000 200 - - - r--s- dev:32,24 ino:716461
FE400000 1744 - - - r--s- dev:32,24 ino:716459
FE5C0000 64 - - - r--s- dev:32,24 ino:716458
FE5E0000 8 8 8 - rwx-- [ anon ]
FE5F0000 64 24 - - r-x-- libzip.so
FE600000 8 8 8 - rwx-- libzip.so
FE610000 144 96 - - r-x-- libjava.so
FE644000 8 8 8 - rwx-- libjava.so
FE650000 56 16 - - r-x-- libverify.so
FE66E000 8 8 8 - rwx-- libverify.so
FE680000 40 - - - r--s- dev:32,24 ino:716397
FE692000 8 8 - - rwxs- [ anon ]
FE6A0000 32 16 - - rw-s- dev:295,2 ino:5858787
FE6B0000 32 8 - - r-x-- libhpi.so
FE6C8000 8 8 8 - rwx-- libhpi.so
FE6CA000 8 - - - rwx-- libhpi.so
FE6E0000 16 16 - - r-x-- libmp.so.2
FE6F4000 8 8 - - rwx-- libmp.so.2
FE700000 680 160 - - r-x-- libm.so.2
FE7B8000 32 32 - - rwx-- libm.so.2
FE7D0000 8 8 8 - rwx-- [ anon ]
FE7E0000 8 8 - - r-x-- libmd5_psr.so.1
FE7F2000 8 8 - - rwx-- libmd5_psr.so.1
FE800000 7984 4696 - - r-x-- libjvm.so
FEFDC000 408 392 320 - rwx-- libjvm.so
FF042000 56 48 48 - rwx-- libjvm.so
FF060000 8 8 8 - rwx-- [ anon ]
FF070000 8 8 - - r-x-- libmd5.so.1
FF082000 8 8 - - rwx-- libmd5.so.1
FF090000 32 24 - - r-x-- libuutil.so.1
FF0A8000 8 8 - - rwx-- libuutil.so.1
FF0B0000 8 8 - - r-x-- libdoor.so.1
FF0C2000 8 8 - - rwx-- libdoor.so.1
FF0D0000 96 88 - - r-x-- libscf.so.1
FF0F8000 8 8 - - rwx-- libscf.so.1
FF100000 584 584 - - r-x-- libnsl.so.1
FF1A2000 40 40 8 - rwx-- libnsl.so.1
FF1AC000 24 - - - rwx-- libnsl.so.1
FF1C0000 8 8 8 - rwx-- [ anon ]
FF1D0000 16 16 - - r-x-- libm.so.1
FF1E2000 8 8 - - rwx-- libm.so.1
FF1F0000 48 48 - - r-x-- libCrun.so.1
FF20A000 8 8 8 - rwx-- libCrun.so.1
FF20C000 16 - - - rwx-- libCrun.so.1
FF220000 8 8 - - r---- [ anon ]
FF230000 48 48 - - r-x-- libsocket.so.1
FF24C000 8 8 8 - rwx-- libsocket.so.1
FF260000 8 8 - - r-x-- libsched.so.1
FF270000 8 8 - - r-x-- libc_psr.so.1
FF280000 864 864 - - r-x-- libc.so.1
FF368000 32 32 24 - rwx-- libc.so.1
FF370000 8 8 8 - rwx-- libc.so.1
FF380000 8 8 8 - rwx-- [ anon ]
FF390000 24 24 24 - rwx-- [ anon ]
FF3A0000 8 8 - - r-x-- libdl.so.1
FF3B0000 184 184 - - r-x-- ld.so.1
FF3EE000 8 8 8 - rwx-- ld.so.1
FF3F0000 8 8 8 - rwx-- ld.so.1
FF3F8000 16 16 - - r-x-- libthread.so.1
FFB80000 24 - - - ----- [ anon ]
FFBF4000 48 8 8 - rw--- [ stack ]
pfiles:
24584: /export/home/sonusComm/jre/bin/java -Dname=agent -Xmx120M -DAGENT_HOME
Current rlimit: 65536 file descriptors
0: S_IFCHR mode:0666 dev:291,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDONLY|O_LARGEFILE
/devices/pseudo/mm@0:null
1: S_IFREG mode:0644 dev:32,24 ino:716359 uid:50004 gid:6003 size:36526979
O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE
/export/home/sonusComm/logs/console.log
2: S_IFREG mode:0644 dev:32,24 ino:716359 uid:50004 gid:6003 size:36526979
O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE
/export/home/sonusComm/logs/console.log
3: S_IFCHR mode:0666 dev:291,0 ino:6815772 uid:0 gid:3 rdev:13,12
O_RDWR FD_CLOEXEC
/devices/pseudo/mm@0:zero
4: S_IFDOOR mode:0444 dev:300,0 ino:58 uid:0 gid:0 size:0
O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[104]
/var/run/name_service_door
5: S_IFCHR mode:0644 dev:291,0 ino:99614724 uid:0 gid:3 rdev:190,0
O_RDONLY|O_LARGEFILE
/devices/pseudo/random@0:random
6: S_IFCHR mode:0644 dev:291,0 ino:99614726 uid:0 gid:3 rdev:190,1
O_RDONLY|O_LARGEFILE
/devices/pseudo/random@0:urandom
7: S_IFREG mode:0644 dev:32,24 ino:716361 uid:50004 gid:6003 size:323022
O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE
/export/home/sonusComm/server/communicator/log/server.log
8: S_IFIFO mode:0000 dev:299,0 ino:47868548 uid:50004 gid:6003 size:0
O_RDWR
9: S_IFSOCK mode:0666 dev:297,0 ino:4151 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 6098
10: S_IFSOCK mode:0666 dev:297,0 ino:4155 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 42405
12: S_IFSOCK mode:0666 dev:297,0 ino:19198 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 6099
14: S_IFIFO mode:0000 dev:299,0 ino:47865419 uid:50004 gid:6003 size:0
O_RDWR
15: S_IFCHR mode:0644 dev:291,0 ino:99614724 uid:0 gid:3 rdev:190,0
O_RDONLY|O_LARGEFILE
/devices/pseudo/random@0:random
16: S_IFSOCK mode:0666 dev:297,0 ino:48720 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 127.0.0.1 port: 2371
17: S_IFIFO mode:0000 dev:299,0 ino:47865419 uid:50004 gid:6003 size:0
O_RDWR
18: S_IFSOCK mode:0666 dev:297,0 ino:37454 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 9991
19: S_IFREG mode:0644 dev:32,24 ino:717406 uid:50004 gid:6003 size:23436
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/export/home/sonusComm/logs/application.monitor.audit
20: S_IFSOCK mode:0666 dev:297,0 ino:3896 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAM
SO_BROADCAST,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.0.224.0)
sockname: AF_INET 0.0.0.0 port: 161
21: S_IFSOCK mode:0666 dev:297,0 ino:3897 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 7788
22: S_IFSOCK mode:0666 dev:297,0 ino:22116 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAM
SO_BROADCAST,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.0.224.0)
sockname: AF_INET 0.0.0.0 port: 9993
23: S_IFSOCK mode:0666 dev:297,0 ino:64893 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAM
SO_BROADCAST,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.0.224.0)
sockname: AF_INET 0.0.0.0 port: 5200
24: S_IFSOCK mode:0666 dev:297,0 ino:26901 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 2020
26: S_IFREG mode:0644 dev:32,24 ino:716364 uid:50004 gid:6003 size:658924
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/export/home/sonusComm/logs/EventMgr.audit
27: S_IFSOCK mode:0666 dev:297,0 ino:4149 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 127.0.0.1 port: 7788
peername: AF_INET 127.0.0.1 port: 42407
29: S_IFREG mode:0644 dev:32,24 ino:716365 uid:50004 gid:6003 size:49140
O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
/export/home/sonusComm/logs/EventMgrLog.audit
30: S_IFSOCK mode:0666 dev:297,0 ino:4150 uid:0 gid:0 size:0
O_RDWR
SOCK_STREAM
SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_NEXTHOP(0.0.192.0)
sockname: AF_INET 0.0.0.0 port: 19991
31: S_IFSOCK mode:0666 dev:297,0 ino:3714 uid:0 gid:0 size:0
O_RDWR
SOCK_DGRAMI was finally able to reproduce this problem where I observed that within a 5 minute time span the jvm process grew from around 180M (VS) / 100M (RSS) to 2G (VS) / 1G (RSS). Here are some samplings at the end of my dtrace run showing the largest allocations that were made. We suspect this might be a JVM bug (since jstat reports our heap is well below the 120M limit we specified). Do you have any suggestions on how we might resolve this issue ?
Thanks,
Steve
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x31c
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
2352344
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0x90
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x5d0
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
2807672
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0xd4
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cKNode_ArrayEgrow6MI_v_+0x7c
libjvm.so`__1cMPhaseChaitinbApost_allocate_copy_removal6M_v_+0x544
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1488
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
2817016
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0xd4
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cKNode_ArrayEgrow6MI_v_+0x7c
libjvm.so`__1cMPhaseChaitinbApost_allocate_copy_removal6M_v_+0x518
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1488
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3013552
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2a8
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x100c
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3341112
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x5d0
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3402976
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0x90
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x8b8
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
3952576
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2a8
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x720
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
4782376
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x8b8
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
4881328
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cHMatcherFxform6MpnENode_i_2_+0xac
libjvm.so`__1cHMatcherFmatch6M_v_+0x644
libjvm.so`__1cHCompileICode_Gen6M_v_+0xd4
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5090648
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x200
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5252816
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0x90
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1180
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5820336
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
5822560
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cMPhaseChaitinFSplit6MI_I_+0x2c0
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x720
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
6747736
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cIPhaseIFGEinit6MI_v_+0xdc
libjvm.so`__1cMPhaseChaitinRRegister_Allocate6M_v_+0x1180
libjvm.so`__1cHCompileICode_Gen6M_v_+0x2b0
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xc08
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
7129056
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0x214
libjvm.so`__1cOPhaseIdealLoopKDominators6M_v_+0xb4
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0x7fc
libjvm.so`__1cHCompileIOptimize6M_v_+0x47c
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
16349616
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0x244
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cENodeIout_grow6MI_v_+0x150
libjvm.so`__1cENodeFclone6kM_p0_+0x1f0
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0x8c8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
134217736
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFChunk2n6FII_pv_+0x244
libjvm.so`__1cFArenaIArealloc6MpvII_1_+0xc4
libjvm.so`__1cKNode_ArrayEgrow6MI_v_+0x7c
libjvm.so`__1cMPhaseIterGVNWadd_users_to_worklist06MpnENode__v_+0xa4
libjvm.so`__1cMPhaseIterGVNVadd_users_to_worklist6MpnENode__v_+0x8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x3c
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
134217736
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeIout_grow6MI_v_+0xac
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0xacc
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
165122996
libumem.so.1`malloc
libjvm.so`__1cCosGmalloc6FI_pv_+0x28
libjvm.so`__1cFArenaEgrow6MI_pv_+0xd4
libjvm.so`__1cENodeFclone6kM_p0_+0xb0
libjvm.so`__1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__+0x8c8
libjvm.so`__1cMPhaseIterGVNNtransform_old6MpnENode__2_+0x28
libjvm.so`__1cMPhaseIterGVNIoptimize6M_v_+0xac
libjvm.so`__1cOPhaseIdealLoop2t5B6MrnMPhaseIterGVN_pk0i_v_+0xc34
libjvm.so`__1cHCompileIOptimize6M_v_+0x174
libjvm.so`__1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_+0xbdc
libjvm.so`__1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_+0xb0
libjvm.so`__1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_+0x4cc
libjvm.so`__1cNCompileBrokerUcompiler_thread_loop6F_v_+0x44c
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
165122996
libumem.so.1`malloc
libjava.so`JNU_GetStringPlatformChars+0xd8
libnet.so`Java_java_net_Inet4AddressImpl_lookupAllHostAddr+0x54
0xf8c0c280
0xf8c0c224
0xf8c05d3c
0xf8c05d3c
0xf8c05874
0xf8c05874
0xf8c05764
0xf8c05c2c
0xf8c00218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`__1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_+0x188
libjvm.so`__1cMthread_entry6FpnKJavaThread_pnGThread__v_+0x134
libjvm.so`__1cKJavaThreadDrun6M_v_+0x2b0
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
-4294966636 -
How to modify the memory quota of jvm???
hello
how can i modify the memory size of jvm within the weblogic platform?The following link http://e-docs.bea.com/wls/docs81/perform/JVMTuning.html#1109778 gives all the information required for setting JVM heap size. This is a nice link for tuning up your JVM settings too.
Hope this helps.
-Vishwas -
We have a multi-master Sun DS setup , wherein we are observing a huge memory growth on only one of them. Memory growth started after third party plug-ins were installed on them, the issue should have been seen on both as plug-in were installed on both of them. We have esclated it to plug-in OEM.
Have you faced any such issue in the past or do u know any tools preferably command to pull out these extra memory consumption?
FYI, the plug-in being used is CP metadirectory for Sun Directory Syncronisations with other databases.
TIAHi Ludovic,
Obviously, The first though was to hit 3rd party vendor for this cause, but they have a point " Plug-ins should have cause memory growth on both of them", whereas this is not the case. On the otherhand, We have the same plug-ins installed on our Dev & Stg environment, we dont see any similar issue there"
Is there any way to list out memory consumptions specifically by connections, lib etc?
TIA,
Nawaz -
Hi
How do I allocate Contiguous memory space to JVM 1.3 in Windows and AIX?
Regards
SudhindraHi
You are right. But the issues here is performance
enhancement. I feel that when we are talking about
huge volumes of data and transaction performance will
be enhanced if the memory allocated is contigous.
As far as I know all modern platforms use virtual addressing. The overhead of that applies regardless of layout of physical memory. So I am not sure how contiguous blocks would help.
So can you please tell me how to do this? Where can i
get more information about this?
Like I said maybe it is specific to your platform. If so you need to look to the docs about your platform. And the only way you are going to get any advantage from java objects from that is if you write your own jvm. -
Agent resident memory utilization and Agent Virtual Memory Growth is HIGH
I have been receiving many alerts from EM on agents resident memory and virtual memory growth is high. Every time the agent is getting restarted by watchdog.
I am wondering where exactly the problem is ...!
anilWhat platform and what Agent version are you using?
Regards
Rob
For more Tips and Tricks on OEM GC see: http://oemgc.wordpress.com -
How much memory is allocated to JVM when we run a java program, and does it keep increasing as and when more memory is required or it remains fixed?
Is it possible to change the memory allocated to JVM,
say make it 64 MB, or reduce it to 16 MB?
java -Xmx<memory size here/> ... -
Rapid database growth - how to check?
We have been experiencing rapid database growth in the past few days.
How do we check what is causing this?
We are on ECC 5.0 / Win / SQL Server
Thanks
PrasadHi,
How do we check what is causing this?
What is your business ? what interfaces do you have into your system
Have you done a recent take on or influx of orders....
What do you call rapid DB growth..
Mark -
Linux command/shell script to find Memory consumed by JVM in EBS,SOA
Hi All,
Could anyone please let me know the following:
a)Linux command/shell script to find Memory consumed by JVM for every 1 minute frequency interval in EBS R12, SOA suite
b)Percentage of the Heap memory usage by the JVM for every 1 minute frequency interval in EBS R12, SOA suite
Thanks for your time!
Regards,user10088255 wrote:
Hi All,
Could anyone please let me know the following:
a)Linux command/shell script to find Memory consumed by JVM for every 1 minute frequency interval in EBS R12, SOA suite
b)Percentage of the Heap memory usage by the JVM for every 1 minute frequency interval in EBS R12, SOA suite
Thanks for your time!
Regards,Please see these docs.
JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 [ID 362851.1]
Generate JVM heap dump in E-Business suite 11i and R12 [ID 462550.1]
11i: How To Use Jconsole to Monitor JVM with Oracle E-Business Suite 11i [ID 415455.1]
Configuring various JVM tuning parameters for Oracle E-Business suite 11i and R12 [ID 567551.1]
Script to find Apache, Java, Jinitiator, Forms version and JVM details for Oracle E-Business Suite 11i [ID 466890.1]
JVM Memory Monitoring, Tuning, Garbage Collection, Out of Memory, and Heap Dump Analysis For SOA Suite Integration 11g [ID 1358719.1]
How to Perform Memory and Thread Analysis on SOA [ID 1450293.1]
Thanks,
Hussein -
"Not enough disk space" when there seems to be (+ rapid memory consumption)
[Note: I originally posted this in the iDVD section, but due to an answer I got there to another question and further conversations with my daughter, figured out this is probably an iMovie question?]
I'm a newbie to iDVD and iMovie and Forums (my last Mac was a Performa 6360--still using it for the family budget). I got my daughter a MacBook as a graduation gift. She is using it to combine movie inputs from a wedding/reception that come from digital camreas and a couple of video cameras. We went to Flip4Mac to get the necessary software/features to import .wmv files (only way we could download one of the video cameras was by going from PC to Mac--can't remember if it was a Sony or Canon).
Issue: When she tries to import vidoes into iMovie HD 6.0.3 which have already been transferred to her computer, she gets: "The file could not be imported: There is not enough disk space to import Macintosh HD/Users/julie/pictures/iPhoto Library/Originals/2007/Roll8/CIMG1442.AVI" [She says this is about a 4 min. clip.]
Basically the question is how to get more room on her disk to import and to save? She seems to only be able to Save--she can't Save Project as an option because it says she doesn't have enough disk space (she's already deleted the Trash). Help says you need to have more than 400mb of free space or it gets difficult but the indicator shows 28.8GB free space left...so there seems to be some kind of disconnect?? (Or maybe we're talking apples and oranges?)
[Even more disturbing, just in the last couple of days with only a little time spent on the computer, the 28.8 GB has dropped to 2.3GB--obviously there is something she is doing that she is not aware of or shouldn't be doing that is consuming memory at a rapid pace.] On top of that, she also gets the same error message when she tries to burn the project onto a disk using iDVD???
I went through the first 5 pages of the forum and couldn't see a similar problem, but then I may have not known what to look for?
Any help would be greatly appreciated...she's trying to give the DVD as a Wedding gift--somewhat belated.Hello Karstn Schluter,
This is fm's daughter who is havin ghte difficulites. at the end you tell me to try to answer 1-8 so i will try.
1- how can you tell if the avi is containing divx? or not?? do I need to just keep this in mind when importing these files
2- I have not brought in an external drive. I did try making my laptop computer a harddrive so I could import my imovie poject to my eMAC at one point, but it was not allowing me to do that either. Because I had to have it in DVD form or something
3- No I did not segment the internal drive....sorry How do you do that??? Right now I am trying to find out what is taking up all the memory and Hard Disk Drive space....i do not know how to do that either....Do you know how???
4-Yes I know flip4mac is for wmv....I guess what that question was emplying was that I also have wmv files that I am trying to import to imovieHD and that is not working either...I can get my avi to get into imovieHD, but not my wmv. files
5-THank you for the tip with the trash then another question to this one how do you delet projects that are taking up memory that are no longer in use?
6-Thank you for the tip...so then ignore the not enough disk space.....how can you alter the disk space so that you can save the project under save project as and not just save project?
7- Sorry my dad was miss informed I cannot import the wmv. files to imovie and I cannot import imovie to iDVD I only have transferred one and then all of the other files are not seen as movies so I can not import them?? Does that make since
8- When we were useing the Sony Camera originally we could not import the video to the MAC because it did not have the correct firewire cable and it would not been read onto the MAC computer. So, we have to buy the flip4mac equipment so that I would be able ot download the inoformation to the computer by a wmv.file dvd so we are having difficulites trying to import that file.
I hope these answers clear up the question a little bit more and sorry that your answers brought out more questions. Thank you very much for helping me. I was using a different program in my college class so changing to imovie and idvd has been a huge struggle for me. Thank you once again. -
Memory leak in jvm? totalMemory differs real memory usage
I working on a server application under linux with java 1.4.1_02.
The problem is that the OS reports after approx. 2 days a memory usage of about 200 MB while the JVM's totalMemory method says, that only about 20 MB are allocated and more than 10 MB of these 20 MB are free. Calling System.gc() doesn't help.
Is there a memory leak in the JVM itself ?
Is there a bugfix or workaround for preventing out of memory ?hi, here is some of the messages from our tomcat log. Please let us know if you see something here.
thanks
wh
14:04 Started Tomcat0.000: [GC 23352K->3056K(259264K), 0.0425550 secs]
6.396: [GC 26416K->3837K(259264K), 0.0428260 secs]
67.830: [GC 27197K->7675K(259264K), 0.0526750 secs]
14:11 Started OpenSTA load test (Tomcat 103M/103M SIZE/RSIZE)417.805: [Full GC 23900K->10468K(259264K), 0.1470690 secs]
464.368: [Full GC 29592K->12380K(259264K), 0.1984410 secs]
474.295: [GC 35740K->16076K(259264K), 0.0362230 secs]
482.470: [Full GC 32672K->16120K(259264K), 0.1574280 secs]
497.299: [Full GC 32266K->15347K(259264K), 0.1993480 secs]
513.132: [GC 38707K->15644K(259264K), 0.0067610 secs]
513.552: [Full GC 21105K->15833K(259264K), 0.1484350 secs]
524.393: [Full GC 31980K->15975K(259264K), 0.1507760 secs]
535.314: [Full GC 36832K->16164K(259264K), 0.1561530 secs]
544.704: [GC 39524K->16360K(259264K), 0.0035200 secs]
544.724: [Full GC 17439K->15435K(259264K), 0.1654420 secs]
550.837: [GC 38790K->15669K(259264K), 0.0031210 secs]
550.961: [Full GC 20198K->15612K(259264K), 0.1458520 secs]
561.816: [Full GC 36712K->15768K(259264K), 0.1535080 secs]
567.997: [Full GC 38928K->15944K(259264K), 0.1586890 secs]
572.313: [Full GC 29129K->15618K(259264K), 0.1764170 secs]
581.769: [Full GC 36749K->15768K(259264K), 0.1538980 secs]
588.514: [GC 39123K->16114K(259264K), 0.0042700 secs]
588.573: [Full GC 20576K->15878K(259264K), 0.1476950 secs]
592.833: [Full GC 34359K->16027K(259264K), 0.1563230 secs]
594.394: [GC 39387K->16189K(259264K), 0.0030660 secs]
596.453: [Full GC 30677K->15678K(259264K), 0.1698180 secs]
603.878: [GC 39038K->15839K(259264K), 0.0043830 secs]
610.088: [Full GC 32709K->15843K(259264K), 0.1554470 secs]
613.281: [Full GC 25096K->15932K(259264K), 0.1500100 secs]
618.982: [Full GC 35727K->16060K(259264K), 0.1558180 secs]
625.228: [Full GC 39090K->15677K(259264K), 0.1778470 secs]
638.611: [GC 39037K->15885K(259264K), 0.0034330 secs]
639.430: [Full GC 22116K->15866K(259264K), 0.1508080 secs]
646.904: [Full GC 26171K->16062K(259264K), 0.1532020 secs]
652.457: [Full GC 25748K->16156K(259264K), 0.1512670 secs]
658.609: [GC 39516K->16367K(259264K), 0.0032440 secs]
660.209: [Full GC 21359K->15677K(259264K), 0.1657590 secs]
664.911: [Full GC 34660K->15809K(259264K), 0.1562110 secs]
670.953: [Full GC 33108K->15937K(259264K), 0.1571950 secs]
673.849: [Full GC 29438K->16016K(259264K), 0.1531250 secs]
680.074: [GC 39376K->16523K(259264K), 0.0060770 secs]
680.279: [Full GC 26022K->15968K(259264K), 0.1805950 secs]
686.274: [Full GC 39141K->16092K(259264K), 0.1616190 secs]
688.844: [Full GC 36586K->16180K(259264K), 0.1567480 secs]
693.700: [Full GC 35574K->16344K(259264K), 0.1574960 secs]
700.598: [GC 39704K->16637K(259264K), 0.0035300 secs]
700.624: [Full GC 18174K->15872K(259264K), 0.1651170 secs]
703.509: [Full GC 34099K->16082K(259264K), 0.1573090 secs]
707.642: [GC 39442K->16171K(259264K), 0.0025780 secs]
709.815: [Full GC 28634K->16206K(259264K), 0.1534170 secs]
716.272: [Full GC 36047K->16381K(259264K), 0.1587900 secs]
723.495: [Full GC 39059K->16219K(259264K), 0.1683900 secs]
728.392: [GC 39577K->16618K(259264K), 0.0040680 secs]
728.415: [Full GC 18013K->16400K(259264K), 0.1536650 secs]
731.039: [Full GC 34082K->16531K(259264K), 0.1571980 secs]
737.472: [Full GC 32417K->16713K(259264K), 0.1577520 secs]
742.717: [Full GC 37469K->16111K(259264K), 0.1606780 secs]
744.274: [GC 39471K->16290K(259264K), 0.0033230 secs]
744.613: [Full GC 21102K->16265K(259264K), 0.1536350 secs]
748.395: [GC 39621K->16436K(259264K), 0.0031140 secs]
748.677: [Full GC 25242K->16359K(259264K), 0.1576130 secs]
750.854: [Full GC 31336K->16451K(259264K), 0.1585730 secs]
756.051: [Full GC 36466K->16119K(259264K), 0.1612460 secs]
759.115: [GC 39479K->16346K(259264K), 0.0028810 secs]
760.093: [Full GC 21902K->16228K(259264K), 0.1520320 secs]
763.760: [GC 39582K->16700K(259264K), 0.0059930 secs]
764.034: [Full GC 22526K->16387K(259264K), 0.1544760 secs]
768.679: [Full GC 32133K->16544K(259264K), 0.1591930 secs]
771.644: [Full GC 31230K->16325K(259264K), 0.1612140 secs]
776.770: [Full GC 38785K->16445K(259264K), 0.1620900 secs]
780.396: [Full GC 37938K->16896K(259264K), 0.1673300 secs]
782.776: [Full GC 30184K->17037K(259264K), 0.1590650 secs]
786.573: [GC 40397K->17228K(259264K), 0.0034880 secs]
786.731: [Full GC 19211K->16266K(259264K), 0.1549430 secs]
789.000: [GC 39619K->16361K(259264K), 0.0026020 secs]
789.269: [Full GC 19125K->16399K(259264K), 0.1513090 secs]
792.386: [Full GC 36010K->16508K(259264K), 0.1637400 secs]
795.102: [GC 39868K->16833K(259264K), 0.0040440 secs]
795.207: [Full GC 21389K->16739K(259264K), 0.1533440 secs]
799.324: [Full GC 23240K->16280K(259264K), 0.1577710 secs]
802.196: [GC 39639K->16518K(259264K), 0.0033770 secs]
802.399: [Full GC 22872K->16698K(259264K), 0.1572730 secs]
807.161: [GC 40058K->16776K(259264K), 0.0027580 secs]
809.900: [Full GC 36185K->16876K(259264K), 0.1607120 secs]
811.791: [Full GC 30282K->16978K(259264K), 0.1593370 secs]
813.563: [GC 40337K->17376K(259264K), 0.0064020 secs]
813.804: [Full GC 29057K->16982K(259264K), 0.1816130 secs]
815.297: [GC 40336K->17338K(259264K), 0.0039630 secs]
815.310: [Full GC 17806K->17082K(259264K), 0.1522910 secs]
819.516: [GC 40438K->17223K(259264K), 0.0028420 secs]
821.285: [Full GC 26923K->17272K(259264K), 0.1588350 secs]
823.559: [GC 40632K->17448K(259264K), 0.0031100 secs]
824.265: [Full GC 21492K->17462K(259264K), 0.1550870 secs]
825.397: [GC 40822K->17655K(259264K), 0.0031120 secs]
825.486: [Full GC 18871K->16337K(259264K), 0.1562730 secs]
829.469: [Full GC 38369K->16447K(259264K), 0.1650720 secs]
831.816: [GC 39807K->16601K(259264K), 0.0026710 secs]
831.827: [Full GC 17119K->16512K(259264K), 0.1522890 secs]
835.329: [GC 39872K->17080K(259264K), 0.0053700 secs]
835.465: [Full GC 23458K->16733K(259264K), 0.1581070 secs]
837.668: [GC 40093K->16935K(259264K), 0.0031290 secs]
838.251: [Full GC 20039K->16495K(259264K), 0.1564500 secs]
841.767: [Full GC 25895K->16613K(259264K), 0.1577470 secs]
844.476: [Full GC 35311K->16695K(259264K), 0.1628390 secs]
848.103: [GC 40044K->17069K(259264K), 0.0032710 secs]
848.628: [Full GC 22870K->16916K(259264K), 0.1580780 secs]
854.887: [Full GC 31905K->16323K(259264K), 0.1653390 secs]
857.228: [Full GC 36492K->16530K(259264K), 0.1634600 secs]
858.956: [GC 39890K->16815K(259264K), 0.0036330 secs]
862.023: [Full GC 21092K->16631K(259264K), 0.1564650 secs]
863.249: [Full GC 26164K->16781K(259264K), 0.1562600 secs]
865.711: [GC 40141K->17043K(259264K), 0.0031530 secs]
865.729: [Full GC 18180K->16457K(259264K), 0.1577410 secs]
869.627: [GC 39817K->16729K(259264K), 0.0033990 secs]
870.098: [Full GC 24966K->16659K(259264K), 0.1562460 secs]
872.350: [Full GC 31145K->16718K(259264K), 0.1604970 secs]
876.105: [Full GC 39279K->16941K(259264K), 0.1668260 secs]
880.207: [GC 40299K->17205K(259264K), 0.0034920 secs]
880.251: [Full GC 20652K->16517K(259264K), 0.1571090 secs]
883.114: [Full GC 37902K->16737K(259264K), 0.1646110 secs]
884.628: [Full GC 24661K->16849K(259264K), 0.1560830 secs]
888.022: [GC 40203K->17459K(259264K), 0.0051950 secs]
889.142: [Full GC 22380K->17051K(259264K), 0.1583020 secs]
891.554: [GC 40411K->17208K(259264K), 0.0027960 secs]
891.582: [Full GC 18063K->16781K(259264K), 0.1585470 secs]
892.405: [Full GC 29120K->17053K(259264K), 0.1622140 secs]
894.017: [GC 40406K->17464K(259264K), 0.0045690 secs]
895.447: [Full GC 31179K->17223K(259264K), 0.1628010 secs]
897.067: [Full GC 38729K->17329K(259264K), 0.1647360 secs]
900.396: [GC 40689K->17720K(259264K), 0.0044380 secs]
900.441: [Full GC 20648K->16903K(259264K), 0.1598090 secs]
902.678: [Full GC 35946K->17128K(259264K), 0.1663350 secs]
905.853: [GC 40488K->17532K(259264K), 0.0064380 secs]
905.891: [Full GC 19672K->17499K(259264K), 0.1621960 secs]
907.750: [GC 40847K->17670K(259264K), 0.0030520 secs]
908.460: [Full GC 20024K->17620K(259264K), 0.1565920 secs]
910.624: [GC 40980K->17765K(259264K), 0.0030090 secs]
910.671: [Full GC 18684K->16921K(259264K), 0.1605090 secs]
912.509: [Full GC 29066K->17017K(259264K), 0.1614450 secs]
913.757: [GC 40377K->17483K(259264K), 0.0047620 secs]
914.763: [Full GC 25902K->17141K(259264K), 0.1608170 secs]
917.689: [GC 40501K->17647K(259264K), 0.0049530 secs]
918.806: [Full GC 28871K->17294K(259264K), 0.1601960 secs]
921.558: [GC 40654K->17457K(259264K), 0.0029970 secs]
922.333: [Full GC 30464K->16925K(259264K), 0.1662820 secs]
923.373: [GC 40285K->17159K(259264K), 0.0042550 secs]
923.555: [Full GC 24496K->17082K(259264K), 0.1599500 secs]
927.044: [Full GC 37046K->17226K(259264K), 0.1683760 secs]
929.061: [GC 40586K->17346K(259264K), 0.0026490 secs]
929.162: [Full GC 21815K->17310K(259264K), 0.1590250 secs]
931.978: [GC 40670K->17413K(259264K), 0.0024780 secs]
932.513: [Full GC 23342K->16981K(259264K), 0.1626270 secs]
935.043: [GC 40341K->17151K(259264K), 0.0028210 secs]
935.520: [Full GC 29674K->17213K(259264K), 0.1660230 secs]
937.492: [GC 40573K->17558K(259264K), 0.0039620 secs]
937.670: [Full GC 23593K->17320K(259264K), 0.1597560 secs]
938.607: [GC 40680K->17648K(259264K), 0.0045610 secs]
939.678: [Full GC 33877K->17515K(259264K), 0.1648540 secs]
942.806: [GC 40869K->18021K(259264K), 0.0051340 secs]
944.254: [Full GC 35743K->17147K(259264K), 0.1719980 secs]
945.155: [Full GC 36250K->17530K(259264K), 0.1705550 secs]
946.222: [Full GC 34120K->17623K(259264K), 0.1649620 secs]
949.645: [Full GC 30572K->17807K(259264K), 0.1649170 secs]
952.442: [GC 41163K->17964K(259264K), 0.0030280 secs]
952.566: [Full GC 20289K->16895K(259264K), 0.1610340 secs]
954.853: [Full GC 28279K->17090K(259264K), 0.1638610 secs]
957.884: [GC 40445K->17584K(259264K), 0.0047550 secs]
958.667: [Full GC 32363K->17427K(259264K), 0.1661990 secs]
959.881: [Full GC 33152K->17499K(259264K), 0.1664660 secs]
964.265: [Full GC 37131K->17078K(259264K), 0.1695560 secs]
967.544: [Full GC 39971K->17286K(259264K), 0.1698470 secs]
972.467: [GC 40645K->17716K(259264K), 0.0058330 secs]
972.573: [Full GC 19899K->17688K(259264K), 0.1596110 secs]
973.323: [GC 41048K->18078K(259264K), 0.0039180 secs]
973.767: [Full GC 30629K->18056K(259264K), 0.1642740 secs]
975.978: [Full GC 32502K->16772K(259264K), 0.1692030 secs]
977.636: [GC 40132K->17237K(259264K), 0.0055500 secs]
978.935: [Full GC 25669K->16985K(259264K), 0.1628640 secs]
980.311: [Full GC 36664K->17060K(259264K), 0.1661300 secs]
984.254: [GC 40420K->17283K(259264K), 0.0035880 secs]
984.947: [Full GC 28397K->17339K(259264K), 0.1645970 secs]
988.111: [Full GC 30208K->16821K(259264K), 0.1669820 secs]
991.456: [Full GC 24124K->16941K(259264K), 0.1607770 secs]
994.575: [GC 40301K->17145K(259264K), 0.0035460 secs]
995.050: [Full GC 26065K->17242K(259264K), 0.1658660 secs]
996.625: [Full GC 37922K->17434K(259264K), 0.1697800 secs]
999.246: [Full GC 31259K->17049K(259264K), 0.1676220 secs]
1001.492: [GC 40408K->17779K(259264K), 0.0078300 secs]
1001.859: [Full GC 34893K->17307K(259264K), 0.1669200 secs]
1004.393: [GC 40666K->17525K(259264K), 0.0030640 secs]
1004.425: [Full GC 20060K->17436K(259264K), 0.1593370 secs]
1007.645: [GC 40794K->17669K(259264K), 0.0030540 secs]
1008.124: [Full GC 30847K->17795K(259264K), 0.1688060 secs]
1009.680: [Full GC 27073K->17588K(259264K), 0.1798900 secs]
1011.538: [Full GC 30414K->17677K(259264K), 0.1661910 secs]
1016.696: [GC 41037K->17878K(259264K), 0.0043140 secs]
1017.675: [GC 41238K->18305K(259264K), 0.0076230 secs]
1017.697: [Full GC 18965K->17980K(259264K), 0.1633270 secs]
1018.701: [Full GC 40089K->18034K(259264K), 0.1699260 secs]
1022.387: [Full GC 40085K->17215K(259264K), 0.1718460 secs]
1024.329: [GC 40575K->17390K(259264K), 0.0033200 secs]
1025.327: [Full GC 32127K->17405K(259264K), 0.1646980 secs]
1027.948: [GC 40765K->18028K(259264K), 0.0072570 secs]
1028.257: [Full GC 33726K->17576K(259264K), 0.1646440 secs]
1029.759: [Full GC 35673K->17688K(259264K), 0.1722710 secs]
1032.229: [Full GC 29552K->17224K(259264K), 0.1670930 secs]
1036.764: [GC 40584K->17488K(259264K), 0.0044870 secs]
1037.789: [Full GC 23532K->17375K(259264K), 0.1604060 secs]
1040.193: [Full GC 40682K->17543K(259264K), 0.1722250 secs]
1041.362: [GC 40902K->17948K(259264K), 0.0044240 secs]
1042.386: [Full GC 29926K->17651K(259264K), 0.1651240 secs]
1046.173: [GC 41011K->17786K(259264K), 0.0030940 secs]
1046.186: [Full GC 18169K->17442K(259264K), 0.1657440 secs]
1048.882: [GC 40802K->17670K(259264K), 0.0032420 secs]
1049.728: [Full GC 26921K->17623K(259264K), 0.1666840 secs]
1050.667: [Full GC 27281K->17714K(259264K), 0.1655080 secs]
1053.205: [Full GC 39795K->17784K(259264K), 0.1699730 secs]
1055.242: [Full GC 37892K->17524K(259264K), 0.1736060 secs]
1058.278: [GC 40877K->17783K(259264K), 0.0032240 secs]
1058.934: [Full GC 24636K->17670K(259264K), 0.1629560 secs]
1061.572: [Full GC 32462K->17738K(259264K), 0.1658050 secs]
1064.860: [GC 41090K->18049K(259264K), 0.0031520 secs]
1064.873: [Full GC 18402K->17913K(259264K), 0.1598100 secs]
1068.969: [GC 41272K->18129K(259264K), 0.0030650 secs]
1069.415: [Full GC 25714K->17421K(259264K), 0.1689390 secs]
1070.981: [Full GC 40072K->17529K(259264K), 0.1719780 secs]
1072.694: [Full GC 40298K->17684K(259264K), 0.1755570 secs]
1074.349: [Full GC 29692K->17760K(259264K), 0.1646700 secs]
1079.137: [GC 41120K->18001K(259264K), 0.0032570 secs]
1079.901: [Full GC 29267K->17912K(259264K), 0.1952200 secs]
1081.705: [GC 41272K->18459K(259264K), 0.0059700 secs]
1082.812: [Full GC 28652K->18115K(259264K), 0.1686030 secs]
1084.535: [GC 41475K->18286K(259264K), 0.0029690 secs]
1085.759: [Full GC 22825K->18304K(259264K), 0.1620300 secs]
1088.874: [GC 41664K->18557K(259264K), 0.0032180 secs]
1090.304: [Full GC 23507K->18428K(259264K), 0.1613190 secs]
1092.560: [GC 41787K->18731K(259264K), 0.0037290 secs]
1092.568: [Full GC 18918K->17280K(259264K), 0.1863890 secs]
1094.698: [Full GC 40048K->17487K(259264K), 0.1717440 secs]
1097.608: [GC 40847K->17639K(259264K), 0.0029460 secs]
1097.770: [Full GC 21453K->17546K(259264K), 0.1593820 secs]
1101.222: [Full GC 34584K->17699K(259264K), 0.1714350 secs]
1103.842: [Full GC 29518K->17398K(259264K), 0.1673060 secs]
1105.429: [Full GC 36543K->17480K(259264K), 0.1687430 secs]
1107.633: [GC 40839K->17863K(259264K), 0.0037960 secs]
1109.418: [Full GC 38489K->17844K(259264K), 0.1714760 secs]
1110.644: [GC 41201K->18176K(259264K), 0.0037780 secs]
1110.708: [Full GC 22032K->18031K(259264K), 0.1629450 secs]
1112.769: [Full GC 33242K->17641K(259264K), 0.1746310 secs]
1115.871: [GC 41001K->17913K(259264K), 0.0042400 secs]
1115.936: [Full GC 18849K->17868K(259264K), 0.1649840 secs]
1116.903: [GC 41228K->18158K(259264K), 0.0038590 secs]
1117.142: [Full GC 23635K->18147K(259264K), 0.1661090 secs]
1119.657: [Full GC 33873K->18237K(259264K), 0.1687550 secs]
1122.138: [GC 41597K->18843K(259264K), 0.0057360 secs]
1122.460: [Full GC 30191K->17352K(259264K), 0.1718040 secs]
1124.176: [GC 40711K->17534K(259264K), 0.0032200 secs]
1124.188: [Full GC 18088K->17408K(259264K), 0.1585710 secs]
1126.471: [GC 40768K->17610K(259264K), 0.0030090 secs]
1126.621: [Full GC 23207K->17585K(259264K), 0.1639700 secs]
1130.601: [GC 40945K->18032K(259264K), 0.0057690 secs]
1130.987: [Full GC 33251K->17715K(259264K), 0.1698070 secs]
1132.400: [GC 41075K->18200K(259264K), 0.0048990 secs]
1133.852: [Full GC 22412K->17396K(259264K), 0.1678410 secs]
1134.988: [GC 40756K->17811K(259264K), 0.0047240 secs]
1135.108: [Full GC 22610K->17538K(259264K), 0.1632820 secs]
1138.140: [Full GC 35484K->17677K(259264K), 0.1697280 secs]
1140.949: [GC 41037K->18028K(259264K), 0.0039610 secs]
1142.579: [Full GC 26182K->17886K(259264K), 0.1663820 secs]
1145.332: [Full GC 38186K->17413K(259264K), 0.1775910 secs]
1147.998: [Full GC 31086K->17542K(259264K), 0.1694310 secs]
1150.183: [GC 40902K->17680K(259264K), 0.0028430 secs]
1151.988: [Full GC 23320K->17714K(259264K), 0.1662670 secs]
1153.556: [GC 41074K->18019K(259264K), 0.0033240 secs]
1153.694: [Full GC 22297K->17819K(259264K), 0.1636210 secs]
1156.910: [Full GC 32491K->17555K(259264K), 0.1708090 secs]
1159.295: [GC 40915K->17925K(259264K), 0.0041940 secs]
1159.584: [Full GC 27356K->17749K(259264K), 0.1684860 secs]
1162.179: [Full GC 40840K->17842K(259264K), 0.1709320 secs]
1165.462: [Full GC 40836K->17998K(259264K), 0.1745900 secs]
1169.213: [Full GC 38228K->17387K(259264K), 0.1773150 secs]
1173.839: [Full GC 38730K->17553K(259264K), 0.1721990 secs]
1176.118: [Full GC 28517K->17724K(259264K), 0.1694920 secs]
1178.427: [GC 41084K->18236K(259264K), 0.0056840 secs]
1178.476: [Full GC 21916K->17907K(259264K), 0.1636330 secs]
1180.035: [Full GC 28804K->17629K(259264K), 0.1706580 secs]
1183.111: [GC 40989K->17821K(259264K), 0.0039230 secs]
1183.470: [Full GC 27647K->17770K(259264K), 0.1642930 secs]
1186.098: [Full GC 34782K->18020K(259264K), 0.1711540 secs]
1189.374: [GC 41380K->18148K(259264K), 0.0030680 secs]
1189.773: [Full GC 19108K->18108K(259264K), 0.1620400 secs]
1193.545: [GC 41468K->18264K(259264K), 0.0030810 secs]
1195.049: [Full GC 33109K->17520K(259264K), 0.1722670 secs]
1197.419: [Full GC 34615K->17775K(259264K), 0.1743520 secs]
1198.468: [Full GC 27126K->17819K(259264K), 0.1658660 secs]
1201.775: [GC 41179K->18064K(259264K), 0.0032670 secs]
1202.434: [Full GC 34303K->18285K(259264K), 0.1741190 secs]
1204.504: [GC 41645K->18591K(259264K), 0.0040820 secs]
1205.473: [Full GC 28617K->17816K(259264K), 0.1725460 secs]
1206.615: [Full GC 23644K->17914K(259264K), 0.1654130 secs]
1208.603: [GC 41274K->18294K(259264K), 0.0047020 secs]
1208.659: [Full GC 21617K->18046K(259264K), 0.1671710 secs]
1210.184: [Full GC 38308K->18107K(259264K), 0.1723240 secs]
1212.135: [Full GC 26647K->17955K(259264K), 0.1731150 secs]
1216.077: [GC 41314K->18170K(259264K), 0.0041080 secs]
1216.475: [Full GC 28069K->18058K(259264K), 0.1722190 secs]
1218.160: [GC 41418K->18256K(259264K), 0.0029660 secs]
1218.196: [Full GC 19422K->18244K(259264K), 0.1601460 secs]
1220.002: [Full GC 26298K->18354K(259264K), 0.1672780 secs]
1222.871: [Full GC 41007K->17486K(259264K), 0.1767970 secs]
1224.257: [Full GC 38459K->17581K(259264K), 0.1708260 secs]
1228.508: [Full GC 31027K->17678K(259264K), 0.1934270 secs]
1230.842: [GC 41038K->17846K(259264K), 0.0027980 secs]
1231.091: [Full GC 23773K->17876K(259264K), 0.1648090 secs]
1233.377: [GC 41235K->18030K(259264K), 0.0043450 secs]
1234.627: [Full GC 27018K->17623K(259264K), 0.1681140 secs]
1238.083: [GC 40983K->18145K(259264K), 0.0053580 secs]
1238.236: [Full GC 26915K->17828K(259264K), 0.1658790 secs]
1240.558: [Full GC 32151K->17956K(259264K), 0.1687010 secs]
1243.685: [GC 41310K->18474K(259264K), 0.0058920 secs]
1246.029: [Full GC 41795K->18533K(259264K), 0.1784250 secs]
1249.400: [GC 41893K->18739K(259264K), 0.0035550 secs]
1249.741: [Full GC 22503K->17989K(259264K), 0.1702740 secs]
1252.402: [GC 41347K->18176K(259264K), 0.0039630 secs]
1252.516: [Full GC 20445K->18258K(259264K), 0.1678890 secs]
1254.473: [Full GC 37435K->18439K(259264K), 0.1891770 secs]
1257.912: [GC 41797K->18658K(259264K), 0.0034350 secs]
1258.627: [Full GC 28099K->18792K(259264K), 0.1701140 secs]
1259.558: [GC 42151K->19185K(259264K), 0.0040460 secs]
1259.685: [Full GC 24326K->17830K(259264K), 0.1797960 secs]
1261.675: [Full GC 35858K->17895K(259264K), 0.1747070 secs]
1265.409: [GC 41253K->18235K(259264K), 0.0045390 secs]
1265.480: [Full GC 21386K->18167K(259264K), 0.1647190 secs]
1266.987: [GC 41527K->18518K(259264K), 0.0042120 secs]
1267.097: [Full GC 23121K->18225K(259264K), 0.1638840 secs]
1269.256: [GC 41579K->18556K(259264K), 0.0034600 secs]
1269.907: [Full GC 25021K->17589K(259264K), 0.1682260 secs]
1272.214: [Full GC 29448K->17641K(259264K), 0.1673120 secs]
1274.376: [Full GC 32613K->17885K(259264K), 0.1710000 secs]
1278.034: [GC 41245K->18309K(259264K), 0.0044890 secs]
1278.139: [Full GC 20559K->18359K(259264K), 0.1699630 secs]
1281.824: [GC 41719K->18836K(259264K), 0.0042590 secs]
1281.869: [Full GC 22173K->17938K(259264K), 0.1710780 secs]
1282.883: [Full GC 38905K->18085K(259264K), 0.1729570 secs]
1285.470: [Full GC 29364K->18221K(259264K), 0.1712110 secs]
1287.186: [GC 41579K->18392K(259264K), 0.0037710 secs]
1287.648: [Full GC 34833K->18379K(259264K), 0.1722590 secs]
1293.044: [Full GC 36896K->17895K(259264K), 0.1790080 secs]
1295.344: [Full GC 39122K->17997K(259264K), 0.1739800 secs]
1298.695: [GC 41356K->18180K(259264K), 0.0035400 secs]
1299.139: [Full GC 27101K->18151K(259264K), 0.1662500 secs]
1301.236: [Full GC 29897K->18243K(259264K), 0.1670530 secs]
1304.222: [GC 41599K->18690K(259264K), 0.0051650 secs]
1304.631: [Full GC 26785K->17570K(259264K), 0.1685990 secs]
1307.040: [Full GC 35984K->17736K(259264K), 0.1714940 secs]
1308.610: [Full GC 33194K->17867K(259264K), 0.1732130 secs]
1310.245: [GC 41227K->18348K(259264K), 0.0046460 secs]
1310.270: [Full GC 20079K->17997K(259264K), 0.1622390 secs]
1316.119: [Full GC 31797K->17701K(259264K), 0.1767490 secs]
1317.765: [GC 41061K->17911K(259264K), 0.0034320 secs]
1318.459: [Full GC 36597K->18147K(259264K), 0.1772120 secs]
1319.623: [Full GC 32165K->18210K(259264K), 0.1702940 secs]
1323.339: [GC 41570K->18394K(259264K), 0.0035930 secs]
1323.362: [Full GC 20003K->18302K(259264K), 0.1621850 secs]
1328.455: [GC 41662K->18528K(259264K), 0.0031090 secs]
1329.080: [Full GC 26141K->17601K(259264K), 0.1714570 secs]
1330.502: [GC 40960K->18190K(259264K), 0.0055370 secs]
1331.248: [Full GC 33207K->17743K(259264K), 0.1716600 secs]
1332.504: [Full GC 33325K->17919K(259264K), 0.1707440 secs]
1334.322: [Full GC 25962K->17964K(259264K), 0.1647600 secs]
1339.043: [GC 41324K->18445K(259264K), 0.0053560 secs]
1339.431: [Full GC 30802K->18036K(259264K), 0.1760700 secs]
1340.706: [GC 41396K->18435K(259264K), 0.0049790 secs]
1340.788: [Full GC 19653K->18362K(259264K), 0.1679480 secs]
1341.402: [Full GC 37930K->18541K(259264K), 0.1789520 secs]
1343.397: [Full GC 38599K->18598K(259264K), 0.1745700 secs]
1345.654: [Full GC 38131K->17863K(259264K), 0.1768640 secs]
1349.079: [Full GC 40810K->18034K(259264K), 0.1751080 secs]
1351.329: [GC 41394K->18181K(259264K), 0.0031310 secs]
1351.630: [Full GC 22737K->18131K(259264K), 0.1667750 secs]
1353.752: [GC 41491K->18415K(259264K), 0.0039180 secs]
1353.771: [Full GC 19663K->18329K(259264K), 0.1658580 secs]
1356.021: [Full GC 33020K->17962K(259264K), 0.1772410 secs]
1362.446: [GC 41322K->18319K(259264K), 0.0041650 secs]
1362.707: [Full GC 28140K->18280K(259264K), 0.1717800 secs]
1364.104: [GC 41640K->18440K(259264K), 0.0035700 secs]
1364.219: [Full GC 19931K->18444K(259264K), 0.1673940 secs]
1367.044: [Full GC 41175K->18638K(259264K), 0.1774620 secs]
1369.008: [GC 41998K->18904K(259264K), 0.0033770 secs]
1370.808: [Full GC 39500K->17736K(259264K), 0.1779480 secs]
1372.395: [Full GC 38579K->17798K(259264K), 0.1732010 secs]
1374.965: [GC 41158K->17977K(259264K), 0.0032530 secs]
1375.530: [Full GC 32640K->18465K(259264K), 0.1768610 secs]
1376.498: [Full GC 37663K->18530K(259264K), 0.1755930 secs]
1379.187: [Full GC 34360K->17772K(259264K), 0.1767890 secs]
1381.635: [GC 41132K->17916K(259264K), 0.0030470 secs]
1382.740: [Full GC 33087K->17979K(259264K), 0.1725710 secs]
1384.342: [Full GC 40704K->18127K(259264K), 0.1782130 secs]
1386.079: [Full GC 37351K->18238K(259264K), 0.1749410 secs]
1388.360: [Full GC 37386K->17696K(259264K), 0.1768570 secs]
1390.025: [Full GC 29469K->17756K(259264K), 0.1694570 secs]
1393.734: [GC 41112K->18057K(259264K), 0.0038490 secs]
1394.247: [Full GC 26711K->17971K(259264K), 0.1697510 secs]
1395.559: [GC 41331K->18421K(259264K), 0.0046160 secs]
1395.724: [Full GC 22935K->18369K(259264K), 0.1683440 secs]
1399.015: [GC 41729K->18557K(259264K), 0.0031830 secs]
1400.459: [Full GC 24074K->17934K(259264K), 0.1763530 secs]
1403.708: [GC 41285K->18399K(259264K), 0.0047150 secs]
1403.906: [Full GC 26160K->18149K(259264K), 0.1707510 secs]
1406.124: [GC 41509K->18387K(259264K), 0.0035380 secs]
1406.848: [Full GC 29898K->18662K(259264K), 0.1756120 secs]
1407.882: [Full GC 36353K->18762K(259264K), 0.1732850 secs]
1409.476: [Full GC 31392K->18111K(259264K), 0.1766410 secs]
1413.341: [GC 41471K->18259K(259264K), 0.0029240 secs]
1414.016: [Full GC 23700K->18347K(259264K), 0.1689310 secs]
1416.973: [GC 41707K->18621K(259264K), 0.0032220 secs]
1417.033: [Full GC 21370K->18493K(259264K), 0.1729750 secs]
1418.844: [Full GC 41535K->18652K(259264K), 0.1788500 secs]
1421.566: [Full GC 37085K->17599K(259264K), 0.1771590 secs]
1426.447: [Full GC 39674K->17772K(259264K), 0.1779320 secs]
1427.655: [Full GC 34052K->17910K(259264K), 0.1752830 secs]
1428.736: [Full GC 31329K->18038K(259264K), 0.1709880 secs]
1431.035: [GC 41398K->18234K(259264K), 0.0033650 secs]
1431.545: [Full GC 32607K->17774K(259264K), 0.1744760 secs]
1435.344: [Full GC 35392K->17901K(259264K), 0.1731200 secs]
1440.290: [Full GC 39635K->18103K(259264K), 0.1759120 secs]
1443.216: [GC 41463K->18459K(259264K), 0.0039860 secs]
1443.627: [Full GC 28455K->18606K(259264K), 0.1717400 secs]
1446.677: [Full GC 31845K->17807K(259264K), 0.1757170 secs]
1448.217: [Full GC 36029K->17929K(259264K), 0.1723040 secs]
1451.897: [Full GC 40425K->18068K(259264K), 0.1773810 secs]
1453.303: [GC 41428K->18399K(259264K), 0.0043310 secs]
1454.383: [GC 41747K->18891K(259264K), 0.0066130 secs]
1454.603: [Full GC 26649K->18470K(259264K), 0.1726960 secs]
1455.862: [GC 41830K->19071K(259264K), 0.0057860 secs]
1455.919: [Full GC 23860K->18080K(259264K), 0.1716680 secs]
1459.396: [GC 41440K->18551K(259264K), 0.0048670 secs]
1459.560: [Full GC 21678K->18433K(259264K), 0.1685140 secs]
1460.305: [Full GC 28004K->18540K(259264K), 0.1708560 secs]
1463.360: [GC 41898K->19157K(259264K), 0.0053660 secs]
1463.385: [Full GC 19875K->18963K(259264K), 0.1695990 secs]
1465.299: [GC 42322K->19230K(259264K), 0.0037680 secs]
1466.243: [Full GC 30800K->18146K(259264K), 0.2014600 secs]
1468.739: [GC 41506K->18435K(259264K), 0.0042590 secs]
1469.240: [Full GC 23481K->18360K(259264K), 0.1682010 secs]
1470.849: [Full GC 37140K->18509K(259264K), 0.1753980 secs]
1473.283: [Full GC 28643K->18566K(259264K), 0.1716950 secs]
1475.513: [GC 41926K->18655K(259264K), 0.0024490 secs]
1476.303: [Full GC 28013K->17893K(259264K), 0.1730760 secs]
1479.100: [GC 41253K->18106K(259264K), 0.0032820 secs]
1479.114: [Full GC 19042K->18011K(259264K), 0.1668360 secs]
1480.705: [GC 41371K->18216K(259264K), 0.0033410 secs]
1480.820: [Full GC 21912K->18126K(259264K), 0.1680140 secs]
1483.212: [GC 41483K->18657K(259264K), 0.0068210 secs]
1483.612: [Full GC 32191K->18531K(259264K), 0.1748200 secs]
1484.237: [Full GC 27101K->17945K(259264K), 0.1720220 secs]
1487.355: [GC 41304K->18052K(259264K), 0.0028180 secs]
1487.453: [Full GC 18350K->18034K(259264K), 0.1671100 secs]
1491.399: [Full GC 30457K->18216K(259264K), 0.1750190 secs]
1493.152: [GC 41576K->18373K(259264K), 0.0030600 secs]
1495.040: [Full GC 35834K->18553K(259264K), 0.1844910 secs]
1496.486: [Full GC 40056K->18404K(259264K), 0.1839830 secs]
1497.604: [Full GC 40636K->18576K(259264K), 0.1802690 secs]
1499.708: [Full GC 32806K->18699K(259264K), 0.1755440 secs]
1502.358: [GC 42056K->19017K(259264K), 0.0035950 secs]
1503.448: [Full GC 39416K->19186K(259264K), 0.1810510 secs]
1504.321: [GC 42546K->19502K(259264K), 0.0041760 secs]
1504.523: [Full GC 26135K->18075K(259264K), 0.1737100 secs]
1506.421: [Full GC 34630K->18190K(259264K), 0.1749040 secs]
1509.521: [Full GC 34218K->18280K(259264K), 0.1738500 secs]
1512.404: [Full GC 40123K->18419K(259264K), 0.1785160 secs]
1514.910: [GC 41779K->18978K(259264K), 0.0046400 secs]
1515.057: [Full GC 21217K->18346K(259264K), 0.1729520 secs]
1516.312: [Full GC 32009K->18445K(259264K), 0.1742600 secs]
1519.535: [GC 41800K->18681K(259264K), 0.0033170 secs]
1521.938: [Full GC 39950K->18817K(259264K), 0.1806540 secs]
1522.918: [Full GC 39996K->18896K(259264K), 0.1770260 secs]
1525.240: [Full GC 27208K->18087K(259264K), 0.1766390 secs]
1527.746: [GC 41446K->18283K(259264K), 0.0030650 secs]
1529.193: [Full GC 31590K->18230K(259264K), 0.1739110 secs]
1531.467: [GC 41590K->18408K(259264K), 0.0031390 secs]
1531.621: [Full GC 20536K->18343K(259264K), 0.1684640 secs]
1534.052: [Full GC 39614K->18673K(259264K), 0.1790600 secs]
1536.308: [Full GC 41947K->18277K(259264K), 0.1831870 secs]
1540.110: [GC 41634K->18676K(259264K), 0.0037740 secs]
1540.655: [Full GC 22982K->18554K(259264K), 0.1718110 secs]
1543.403: [Full GC 40455K->18762K(259264K), 0.1808010 secs]
1544.456: [GC 42122K->19049K(259264K), 0.0056720 secs]
1544.623: [Full GC 24100K->19040K(259264K), 0.1734450 secs]
1547.598: [GC 42399K->19313K(259264K), 0.0031750 secs]
1547.782: [Full GC 21803K->17896K(259264K), 0.1733430 secs]
1550.856: [Full GC 38776K->18049K(259264K), 0.1790000 secs]
1554.027: [Full GC 39925K->18209K(259264K), 0.1794380 secs]
1554.868: [Full GC 36385K->18297K(259264K), 0.1765220 secs]
1557.076: [GC 41657K->18481K(259264K), 0.0033910 secs]
1557.170: [Full GC 23085K->17844K(259264K), 0.1733090 secs]
1558.265: [Full GC 21638K->17878K(259264K), 0.1694620 secs]
1563.060: [GC 41237K->18097K(259264K), 0.0033500 secs]
1564.856: [Full GC 32663K->18154K(259264K), 0.1754150 secs]
1566.025: [GC 41510K->18340K(259264K), 0.0031060 secs]
1566.042: [Full GC 19331K->18242K(259264K), 0.1685920 secs]
1568.406: [GC 41602K->18499K(259264K), 0.0034100 secs]
1568.794: [Full GC 29429K->18048K(259264K), 0.1775530 secs]
1571.351: [Full GC 41276K->18129K(259264K), 0.1805000 secs]
1573.344: [Full GC 38881K->18195K(259264K), 0.1760390 secs]
1575.800: [Full GC 26060K->18243K(259264K), 0.1733030 secs]
1577.789: [Full GC 30691K->18180K(259264K), 0.1765060 secs]
1580.336: [GC 41540K->18408K(259264K), 0.0033330 secs]
1582.005: [Full GC 32348K->18433K(259264K), 0.1778330 secs]
1583.438: [GC 41793K->18751K(259264K), 0.0039520 secs]
1583.913: [Full GC 25360K->18656K(259264K), 0.1724900 secs]
1587.323: [Full GC 35278K->18824K(259264K), 0.1789120 secs]
1590.612: [GC 42183K->19194K(259264K), 0.0040080 secs]
1591.305: [Full GC 32714K->18075K(259264K), 0.1783630 secs]
1591.920: [Full GC 32296K->18141K(259264K), 0.1737070 secs]
1593.748: [ -
Hi,
I am getting a jvm Memory Leak when i try to load and parse XMLs. The version of JDK is 1.4.1_05. Has anyone faced this problem?
Any pointers to this will be very helpful.
Thanks
RajdeepThere is at least one bug associated with xml (although not because of it) which usually makes an appearance in long running applications.
http://forum.java.sun.com/thread.jsp?forum=31&thread=351925
If this is an internal application you could conceivably replace the StringBuffer class yourself to fix this. -
Hi,
I'm using the following set up
OS: Redhat linux 7.1
Kernel: 2.4.2-1
glibc-2.1.x for i386
other set up: ulimit -s 2048, JDK 1.3.1_01
app: Jboss app server
I'm running my application in Jboss 2.2.1 which is a I/O intensive. This is a distributed system using visigenic corba 4.5.
The problem:
If I run my application the application memory remains stable over a period of time but my JVM memory (system) increases linearly. i got the app memory using : Runtime.getRuntime().totalMemory()
and JVM memory using : top
Is the JVM leaking ??? I'm not getting OutOfMemeory error anywhere but my system freezes when it reaches a point where it cannot allocate more memory to java process
Reg
VedFreezes? You mean that it stops running at 8 at night and when you come in at 8 the next morning it is still doing nothing? Or that it stops for several minutes?
Memory problems are not usually the cause of this. Some possibilites:
-Blocked, deadlocked, threads.
-Blocked io.
-Endless spins (code runs but does nothing.)
-JNI code. -
Easy to produce - Memory Leak in JVM Using thread
I was just debugging the problem with our server and got this error (not from the original code, I reproduced it with small code)
at java.lang.Thread.run(Unknown Source)
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at SomeObject.resursiveThreadGeneratorMethod(ThreadRecursionLeak.java:12
at ClientThread.run(ThreadRecursionLeak.java:27)
it is just a simple program generating thread recursively with a lock on an object
import java.lang.Runnable;
import java.lang.Thread;
// <<Objec/Monitor Class>>
class SomeObject{
public synchronized void resursiveThreadGeneratorMethod(int threadNumber, int k){
if(k<=0)
return;
else{
System.out.println("Thread: "+threadNumber + "call number: "+k);
Thread thread=new Thread(new ClientThread(this));
thread.start();
resursiveThreadGeneratorMethod(threadNumber,k-1);
// <<Thread Class>>
class ClientThread implements Runnable{
SomeObject someObject=null;
public static int threadNumber;
public ClientThread(SomeObject someObject){
this.someObject=someObject;
threadNumber++;
public void run(){
someObject.resursiveThreadGeneratorMethod(this.threadNumber,10);
// <<Driver Class>>
public class ThreadRecursionLeak{
public static void main(String [] args){
SomeObject someObject=new SomeObject();
Thread thread=new Thread(new ClientThread(someObject));
thread.start();
is there any way I can monitor JVM memory status (programatically), kill those threads leading to memory leak and send notification to the system admin.
We run JVM 1.4 on the server and need to address this issue as we do not have control over the code that can be submitted to the server, we just do some checks and then let it run on the servers.
Our main requirement is to keep JVM alive and kill all those code and report them.
Message was edited by:
rajanikant_malviyaYou can monitor the JVM with the
[url=http://java.sun.com/j2se/1.4.2/docs/guide/jvmpi/i
ndex.html]JVMPI if you're using the Sun JVM.This is native code that basically gets
notified anytime the JVM does something
interesting.
However, you're going to have a very difficult time
catching code that does what the code you show does.
I'm not sure I understand your environment - you
allow people to submit code and you run it? Are you
running it in the same JVM as your server? That
seems like a huge issue. A simple System.exit()
code will take down your server.
Let us know some more details.
We are actually having a portal through witch users can submit their code, witch are basically utility codes for different operations teams.
we then provide a way to schedule them.
At the background, we are having 6 (Win) servers where at each server we are having 20 to 25 users on different ports (managed by our application) running there own JVMs. When a schedule is met we just invoke a servlet (on any free port) and pass code id (stored in DB). And this servlet is responsible for compilation and exicution of the code (We just use Runtime to fork this new process).
I don't know, but is there any way through witch I can monitor this process and restrict to generate threads recursively??? -
I am losing memory rapidly. I have no idea why? I've lost about 5oo gigs in just a few weeks.. I am a photographer, but all my files are stored on exteral drives..
You have a memory leak in an application. Go to User>Applications>Activity Monitor and see what's hogging the memory.
But are you sure it's not early onset Alzheimer's?
Maybe you are looking for
-
I recently purchased the Sony HD camera (model #Z1U) and shot some footage in HD. I was able to capture it on my Sony HDV deck(model #M15U) and need to use it in FCP. What should my sequence settings be to view the video in HD and I have a monitor bu
-
1.1: New features, but does it work better? (XP)
Sorry if this has been covered, but I've out the last week. I've installed 1.1 on XP, but it doesn't seem any faster? Am I missing something? Is it more stable? thoughts? Thanks! Reid
-
I am so angry I can't stop shaking. Today iTunes blew over $100 of my 4G broadband by downloading content on my mac laptop that I didn't start. Several series that I watch on the apple tv just brought down gigs of data on the laptop. This has happene
-
Cannot connect to internet with iMac connected to airport via Ethernet
Hi all, Firstly please forgive if this problem has been posted before but unfortunately i haven't yet been able to find a solution to my problem. I cannot seem to get my wireless network to work since updating my Airport to v5.7. I have reverted back
-
Urgent in format of excelsheet
HI HOW TO Convert the data in the internal table into the SPECIFIED('T 'STRCTURE) format and performing the mapping of the data as GL-GL etc and following the rule for SPECIFIED('T 'STRCTURE) format. The rules are as follow : a) Left justif