Debug a hung Java Process

hi Folks,
we have this script that connects to a server and retrieve bunch of data and parses and indexes it. One of the ways to improve the speed of this indexing, we used the ExecutorService class, for our thread needs.
Seems like there is a hang every now and then ( intemittent ). I did google and found a deluge of articles on how to debug a Hung process, can some please suggest from their real time experience on the most efficient way of debugging this process hang ?.
Thanks a million
_ddt                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

Well, if it's still hung, then kill it. Put a bunch of println's in it and run it from a console session, when you get an idea of where it is, then get your debugger out and set a breakpoint around where you're having the problem and then single step looking for abnormalities until you find it or it hangs again. Repeat the part with the debugger until you find the problem.

Similar Messages

  • Signal ILL trying to debug java process with dbx

    Hello,
    I'm trying to debug a java process to find an error in JNI code. I get this message when dbx stops in a C breakpoint:
    t@18 (l@19) signal ILL (illegal trap) in (unknown) t@18 (l@19) signal ILL (illegal trap) in (unknown) at 0xf848ed28
    0xf848ed28:     tne     0x12This signal is not launched when running application without debugger, and after launching this exception and exiting dbx, the java process generates a core.
    I've tried to find out unsuccessfully whether my dbx version is compatible with my Java version, but it seems this could be the problem.
    This is my current configuration:
    java �version:
    java version "1.5.0_06"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
    Java HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode)
    opt/SUNWspro/bin/version:
    Machine hardware:   sun4u
    OS version:         5.8
    Processor type:     sparc
    Hardware:           SUNW,Sun-Blade-2500
    The following components are installed on your system:
    Forte C++ 6 update 2
            Sun WorkShop 6 update 2 Compilers C    
            Sun WorkShop 6 update 2 Compilers C++
            Sun WorkShop 6 update 2 Tools.h++ 7.1
            Sun WorkShop 6 update 2 Standard 64-bit Class Library for C++
            Sun WorkShop 6 update 2 Memory Monitor for C++
            Sun WorkShop 6 update 2 Visual
            Sun WorkShop 6 update 2 Development Environment
            Sun WorkShop 6 update 2 TeamWare
            Sun WorkShop 6 update 2 FileMerge
            Sun WorkShop 6 update 2 DBX Debugging Tools
            Sun WorkShop 6 update 2 Performance Analyzer
            Sun WorkShop 6 update 2 LockLint
            XEmacs 20.4 Source Distribution
            Sun WorkShop 6 update 2 Documentation Set
    Sun WorkShop 6 update 2 Building SoftwareAnd this is how I launch java application:
    java -Djava.compiler=NONE -Xdebug -Xnoagent -Xrundbx_agent -jar jarname.jar -mainOptionsDid I miss something important? Does anyone know if it is possible debugging with this configuration?
    Best Regards,
    Alberto

    - Native code has been compiled with -g option.
    - It fails in my code, but only when debugging.
    - Stack trace (executing "where" in dbx):
    (dbx) where
    current thread: t@18
      [1] 0xf848ef68(0xf01537d8, 0xf443b7b0, 0x0, 0xf84181b0, 0x21e9, 0xef581050), at 0xf848ef68
      [2] 0xf8458354(0xf15faea8, 0xf01537d8, 0xf0151a88, 0xf8419368, 0x21e8, 0x0), at 0xf8458354
      [3] 0xf848c9a0(0xf15faea8, 0xf01537d8, 0xf0151a58, 0xf0151a38, 0xf0153830, 0x0), at 0xf848c9a0
      [4] 0xf84577c8(0xf15faea8, 0xb6, 0x0, 0xf84181b0, 0xf0158f80, 0xef581118), at 0xf84577c8
      [5] 0xf8405764(0xf15faea8, 0xb6, 0x0, 0xf8419320, 0xf0158f80, 0xef5811c8), at 0xf8405764
      [6] 0xf8405764(0xf15faea8, 0xb6, 0x0, 0xf84195a0, 0xf0158f80, 0xef581250), at 0xf8405764
      [7] 0xf8405764(0xf0151a38, 0xb8, 0x0, 0xf8419320, 0xf0125258, 0xef5812d8), at 0xf8405764
      [8] 0xf8405874(0xf15fb6c8, 0xb8, 0x0, 0xf8419720, 0xf0125258, 0xef581360), at 0xf8405874
      [9] 0xf8405764(0xf15fae98, 0xb6, 0x0, 0xf8419770, 0xf00f0088, 0xef5813f0), at 0xf8405764
      [10] 0xf8405874(0xf15fae98, 0xb6, 0x0, 0xf8419320, 0x1465, 0xef581478), at 0xf8405874
      [11] 0xf8405874(0xf16315b0, 0xb6, 0x0, 0xf8419320, 0xf00f0088, 0xef581500), at 0xf8405874
      [12] 0xf8405764(0xf16315b0, 0xf45744c0, 0x0, 0xf8419320, 0xf45744c0, 0xef581588), at 0xf8405764
      [13] 0xf8405c2c(0xf16315d0, 0xb7, 0x0, 0xf84198a0, 0xf00f0088, 0xef581618), at 0xf8405c2c
      [14] 0xf8405764(0xf009c018, 0xf402c8b8, 0x0, 0xf84195a0, 0x8800, 0xef5816a8), at 0xf8405764
      [15] 0xf8405c2c(0x2000, 0x2, 0x0, 0xf84198f0, 0xef50e0f8, 0xef581720), at 0xf8405c2c
      [16] 0xf8400218(0xef581808, 0xef581a08, 0xa, 0xf402dae8, 0xf840ba20, 0xef581920), at 0xf8400218
      [17] JavaCalls::call_helper(0x1, 0x65d188, 0xef581918, 0xef581818, 0x4, 0xef581a08), at 0xfdd9a808
      [18] JavaCalls::call_virtual(0x7a9c, 0x65d188, 0x3570f8, 0x357104, 0x357100, 0xf00bc770), at 0xfdec319c
      [19] thread_entry(0xf402fc58, 0x65d188, 0x3574e4, 0xfe424640, 0xfe424710, 0xfe424130), at 0xfdee2700
      [20] JavaThread::run(0x65d188, 0x18, 0xfe40c790, 0x6c00, 0x7dd4, 0x7c00), at 0xfdede29c
      [21] _start(0x65d188, 0x3b62, 0xfe3be000, 0x0, 0x4f24, 0x4c00), at 0xfe264240 - Stack trace (obtained from core generated after ILL signal):
    -----------------  lwp# 19 / thread# 18  --------------------
    ff3791f0 __sigprocmask (ff37b7f4, 0, 0, ef581d98, ff38c000, 0) + 8
    ff36dd0c _sigon   (ef581d98, ff3938a8, 6, ef58083c, ef581d98, ff39321c) + d0
    ff370d4c _thrp_kill (0, 12, 6, ff38c000, 12, ff2c0458) + f8
    ff24bce0 raise    (6, 0, 0, ffffffff, ff2c03c4, 8400) + 40
    ff235984 abort    (ff2bc008, fe3be000, fe26a8e0, fe4133c4, 0, 331c8) + 100
    fe264724 __1cCosFabort6Fi_v_ (1, 0, fe3f2614, fe3be000, 73b4, 7000) + 58
    fe2f08f0 __1cHVMErrorOreport_and_die6M_v_ (0, fe419ad8, fe41339c, 1, fe268a14, fe41339c) + c84
    fde76878 JVM_handle_solaris_signal (4, ef580fd8, ef580d20, 7800, 11, 65d188) + aac
    ff37b118 __sighndlr (4, ef580fd8, ef580d20, fde75d94, ef581e40, ef581e30) + c
    ff37811c sigacthandler (4, ef581d98, 0, 0, 0, ff38c000) + 708
    --- called from signal handler with signal 4 (SIGILL) ---
    f848ef68 ???????? (f15faea8, f01537d8, f0151a88, f8419368, 21e8, 0)
    f848c9a0 ???????? (f15faea8, f01537d8, f0151a58, f0151a38, f0153830, 0)
    f84577c8 ???????? (f15faea8, b6, 0, f84181b0, f0158f80, ef581118)
    f8405764 ???????? (f15faea8, b6, 0, f8419320, f0158f80, ef5811c8)
    f8405764 ???????? (f15faea8, b6, 0, f84195a0, f0158f80, ef581250)
    f8405764 ???????? (f0151a38, b8, 0, f8419320, f0125258, ef5812d8)
    f8405874 ???????? (f15fb6c8, b8, 0, f8419720, f0125258, ef581360)
    f8405764 ???????? (f15fae98, b6, 0, f8419770, f00f0088, ef5813f0)
    f8405874 ???????? (f15fae98, b6, 0, f8419320, 1465, ef581478)
    f8405874 ???????? (f16315b0, b6, 0, f8419320, f00f0088, ef581500)
    f8405764 ???????? (f16315b0, f45744c0, 0, f8419320, f45744c0, ef581588)
    f8405c2c ???????? (f16315d0, b7, 0, f84198a0, f00f0088, ef581618)
    f8405764 ???????? (f009c018, f402c8b8, 0, f84195a0, 8800, ef5816a8)
    f8405c2c ???????? (2000, 2, 0, f84198f0, ef50e0f8, ef581720)
    f8400218 ???????? (ef581808, ef581a08, a, f402dae8, f840ba20, ef581920)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 65d188, ef581918, ef581818, 4, ef581a08) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 65d188, 3570f8, 357104, 357100, f00bc770) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 65d188, 3574e4, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (65d188, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (65d188, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (65d188, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 20 / thread# 19  --------------------
    ff29f474 _lwp_sema_wait (ff3934c8, ff392e40, 0, 1, 0, 0) + c
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 21  --------------------------------
    ff379200 private___lwp_cond_wait (fa5a3d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa5a3d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 22  --------------------------------
    ff379200 private___lwp_cond_wait (fa4d3d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa4d3d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 23  --------------------------------
    ff379200 private___lwp_cond_wait (fa4c3d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa4c3d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 24  --------------------------------
    ff379200 private___lwp_cond_wait (fa493d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa493d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 25  --------------------------------
    ff379200 private___lwp_cond_wait (fa483d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa483d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 26  --------------------------------
    ff379200 private___lwp_cond_wait (fa473d98, ff38cd6c, ff38c000, 3, ff38c000, 1) + 8
    ff36a358 _lwp_start (fa473d98, 0, 4000, fb28162c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 1 / thread# 1  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, 18e, 37020, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (1c08, fe406e44, 37020, fe412d84, 38680, fe3be000) + 260
    f840e22c ???????? (b9, ffbee73c, ffbee6b8, ffffff80, 0, 0)
    f840e1d0 ???????? (0, b8, 0, 8, f402fc58, ffbee6d0)
    f8405764 ???????? (f15996c8, fe4163d4, 0, f8419770, 8800, ffbee768)
    f8405764 ???????? (f01778f8, fe4163d4, 0, f8419370, 8800, ffbee820)
    f8405764 ???????? (2000, 2, 0, f8419320, ffb7c110, ffbee8b8)
    f8400218 ???????? (ffbee9a0, ffbeeb00, a, f41c4038, f840ba20, ffbeea88)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 37020, ffbeea80, ffbee9b0, 4, ffbeeb00) + 5a0
    fdedea14 jni_CallStaticVoidMethod (fe412d84, 381ec, 36750, 37020, 36748, 37cb8) + 500
    000121fc main     (38648, 1e7fa2, fe40a630, 1d8, fded413c, 1) + 115c
    00011088 _start   (0, 0, 0, 0, 0, 0) + 108
    -----------------  lwp# 2 / thread# 2  --------------------
    ff29edc4 _signotifywait (ff38c000, 0, 0, 0, 0, 0) + 8
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 3  --------------------------------
    ff379200 private___lwp_cond_wait (ff325d98, ff38cd6c, ff38c000, 0, 0, 4) + 8
    ff29c9c0 _door_return (ff325cd8, ff36a358, 0, 0, 0, 0) + 68
    -----------------  lwp# 4 / thread# 4  --------------------
    ff29f428 ___lwp_cond_wait (35d68, 35d50, fdb81978, 0, 0, fdb81970) + 8
    ff296a7c _lwp_cond_timedwait (2958790, 44b36755, fdb81a50, 35d50, 35d68, a95a) + 98
    fddc3258 __1cHMonitorEwait6Mil_i_ (35cf8, 11dce0, fff0bc00, fe3be000, 6000, 35d28) + 328
    fdf6b4fc __1cIVMThreadDrun6M_v_ (fe4124d8, fe420c84, 0, fe420c70, fe420c80, 1) + 1b4
    fe264240 __1cG_start6Fpv_0_ (11dce0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (11dce0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 5  --------------------------------
    ff29c968 _door_return (4, ff38d658, ff38d670, 3, ff38c000, 1) + 10
    ff36a358 _lwp_start (fe785d98, 0, 6000, ff325b9c, 0, 0) + 18
    ff371c2c thr_yield (0, 0, 0, 0, 0, 0) + 8c
    -----------------  lwp# 6 / thread# 5  --------------------
    ff29f428 ___lwp_cond_wait (120148, 120130, 0, ff2968f0, 0, 1) + 8
    fdeb4768 __1cNObjectMonitorEwait6MxipnGThread__v_ (120370, 9408, 8c00, fe3be000, 11fdc0, 120390) + 59c
    fdeb5978 __1cSObjectSynchronizerEwait6FnGHandle_xpnGThread__v_ (120372, 86d4, 0, 11fdc0, 8400, fe3be000) + e8
    fdeb5628 JVM_MonitorWait (0, e7a14, 11fdc0, fe412d84, 0, 1) + 31c
    f840e22c ???????? (51, fda816ec, fda81670, ffffff80, fdd2e808, 0)
    f840e1d0 ???????? (f15817e0, b6, 0, c, 0, fda81688)
    f8405764 ???????? (f15817e0, b6, 0, f8419354, 80000000, fda81708)
    f8405764 ???????? (0, fe411508, 0, f8419770, 7bac, fda817a0)
    f8400218 ???????? (fda81888, fda81a88, a, f40901d0, f840ba20, fda819a0)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 11fdc0, fda81998, fda81898, 4, fda81a88) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 11fdc0, e7a00, e7a0c, e7a08, f0000760) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 11fdc0, e7dec, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (11fdc0, 0, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (11fdc0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (11fdc0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 7 / thread# 6  --------------------
    ff29f428 ___lwp_cond_wait (121cc8, 121cb0, 0, ff2968f0, 0, 1) + 8
    fdeb4768 __1cNObjectMonitorEwait6MxipnGThread__v_ (1203b8, 9408, 8c00, fe3be000, 121530, 1203d8) + 59c
    fdeb5978 __1cSObjectSynchronizerEwait6FnGHandle_xpnGThread__v_ (1203ba, 86d4, 0, 121530, 8400, fe3be000) + e8
    fdeb5628 JVM_MonitorWait (0, 121714, 121530, fe412d84, 0, 1) + 31c
    f840e22c ???????? (59, fb3817ec, fb381768, ffffff80, 0, 0)
    f840e1d0 ???????? (f1581760, b6, 0, c, 80000000, fb381780)
    f8405764 ???????? (f1581768, b6, 0, f8419354, 8800, fb381820)
    f8405874 ???????? (f1581768, b6, 0, f8419354, f402af58, fb3818a0)
    f8405874 ???????? (0, fe411508, 0, f8419720, 7bac, fb381920)
    f8400218 ???????? (fb381a08, fb381c08, a, f4090ab8, f840ba20, fb381b20)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 121530, fb381b18, fb381a18, 4, fb381c08) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 121530, 121700, 12170c, 121708, f0000858) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 121530, 121aec, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (121530, 0, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (121530, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (121530, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 8 / thread# 7  --------------------
    ff29f428 ___lwp_cond_wait (1b65f0, 1b65d8, fb281890, ff296918, ff29691c, fb281888) + 8
    ff296a7c _lwp_cond_timedwait (2964ae0, 44b36756, fb281910, 1b65d8, 1b65f0, 153180) + 98
    fe2606e0 __1cCosHSolarisFEventEpark6Mx_i_ (1b65b0, 6a60, fe4238d8, 1b65f0, fe406ef0, fb281910) + 1bc
    fe25ff28 __1cNObjectMonitorIraw_wait6MxipnGThread__i_ (1b65b0, fe214b70, 7bac, 0, 1b59e8, 12a188) + 20c
    fe1f4b8c __1cIJvmtiEnvORawMonitorWait6MpnPJvmtiRawMonitor_x_nKjvmtiError__ (fb2, 12a188, 0, 7d0, 4, 1b59e8) + a8
    fe7d87ec process_next_event (fe40b864, 10000, fe7f24a0, fe7f4b90, fe7f25f0, fe7f25ec) + 108
    fe7c7630 agent_bar (1b5aa4, fe7f2ae8, ff38c000, fe7f24c8, 0, 0) + 1c
    fe7c7bc4 jvmti_debug_agent (fe7f2400, 1b5aa4, 0, 1678, 1, fe7f252c) + d4
    fe211c74 __1cQJvmtiAgentThreadTcall_start_function6M_v_ (1b59e8, 7800, 8c94, fe3be000, fe4123c4, fe4163d4) + 150
    fdede29c __1cKJavaThreadDrun6M_v_ (1b59e8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b59e8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b59e8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 9 / thread# 8  --------------------
    ff29f474 _lwp_sema_wait (fb181e60, ff38c000, 0, fb181d98, 0, 0) + c
    ff3690d8 _swtch   (fb181d98, ffffffff, ff38c000, 5, 1000, fe4170c4) + 158
    ff36c860 sema_wait (fe4217e8, ff39785c, 5000, 10000, 534d, 1) + bc
    fdf61dc0 __1cVcheck_pending_signals6Fi_i_ (1, fe4124d8, fe4123c0, 1b6ab8, fe3be000, a) + 254
    fdf58ec8 __1cTsignal_thread_entry6FpnKJavaThread_pnGThread__v_ (7428, 1b6ab8, 1b7088, fe3be000, 4ff61c, 1b7080) + 34
    fdede29c __1cKJavaThreadDrun6M_v_ (1b6ab8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b6ab8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b6ab8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 10 / thread# 9  --------------------
    ff29f428 ___lwp_cond_wait (36548, 36530, fe423d60, 0, 0, ff29695c) + 8
    fddc33ec __1cHMonitorEwait6Mil_i_ (364d8, 1b7998, 1b8510, fe3be000, 1b7998, 36508) + 4bc
    fdf35d00 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (fe4124d8, 1b54c0, 1b7998, fe407360, 8c00, 364d8) + 22c
    fdede29c __1cKJavaThreadDrun6M_v_ (1b7998, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b7998, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b7998, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 11 / thread# 10  --------------------
    fdd0c200 __1cENodeSremove_dead_region6MpnIPhaseGVN_i_i_ (9132d4, fa6809f4, 1, 9132d4, 9132f4, 68ad00) + 2c
    fdd3332c __1cHMemNodeMIdeal_common6MpnIPhaseGVN_i_pnENode__ (fa6809f4, fa6809f4, 1, 800, 9132d4, 9132f4) + 3c
    fdd68a78 __1cJStoreNodeFIdeal6MpnIPhaseGVN_i_pnENode__ (9132d4, fa6809f4, 1, 0, fdd4be74, fe3fbb44) + c
    fdcf887c __1cMPhaseIterGVNNtransform_old6MpnENode__2_ (fa6809f4, 9132d4, 89e938, fdd68a6c, 20000000, c7) + 28
    fdde3b98 __1cMPhaseIterGVNIoptimize6M_v_ (fa6809f4, 14, 9132d4, 0, fa680b44, ffffffff) + ac
    fde048d8 __1cOPhaseIdealLoop2t6MrnMPhaseIterGVN_pk0i_v_ (fa68087c, ffffffff, fa680fb0, 2, 0, fa680880) + c34
    fde7fb28 __1cHCompileIOptimize6M_v_ (fa680fb0, 1b55b0, 0, 0, fe3be000, 1000) + 200
    fde82740 __1cHCompile2t6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_ (fa680fb0, 0, 36ee0, 0, fe3384f8, fa680fb0) + be8
    fde786f8 __1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_ (fa6818ac, fe324608, 9269b8, ffffffff, 36ee0, 1000001) + b0
    fde793b4 __1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_ (597690, 1b8eb4, 1b55b0, 1b88d8, fe338ea7, fa6818ac) + 4cc
    fdf35f20 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (fe4124d8, 1b5550, 1b88d8, f4368380, 1b8eb0, 0) + 44c
    fdede29c __1cKJavaThreadDrun6M_v_ (1b88d8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b88d8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b88d8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 12 / thread# 11  --------------------
    ff29f428 ___lwp_cond_wait (365d8, 365c0, fe423d60, 0, 0, ff29695c) + 8
    fddc33ec __1cHMonitorEwait6Mil_i_ (36568, 1b9818, 1ba390, fe3be000, 1b9818, 36598) + 4bc
    fdf35d00 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (fe4124d8, 1b5550, 1b9818, fe407360, 1b9df0, 36568) + 22c
    fdede29c __1cKJavaThreadDrun6M_v_ (1b9818, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1b9818, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1b9818, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 13 / thread# 12  --------------------
    ff29f428 ___lwp_cond_wait (34858, 34840, 1, 0, 1, 1bb248) + 8
    fddc309c __1cHMonitorEwait6Mil_i_ (347e8, 1bb1a0, fff0bc00, fe3be000, f4000, 34818) + 16c
    fdf64b44 __1cRLowMemoryDetectorbGlow_memory_detector_thread_entry6FpnKJavaThread_pnGThread__v_ (1bb1a0, 347e8, 9000, fe411604, fe3be000, 1) + 2ec
    fdede29c __1cKJavaThreadDrun6M_v_ (1bb1a0, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (1bb1a0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1bb1a0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 14 / thread# 13  --------------------
    ff29d600 _poll    (fe4124d8, 32, 0, 0, 8400, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, 1, 0, 2, 4) + 25c
    fdfa5194 __1cNWatcherThreadDrun6M_v_ (1bc4c8, 1b5460, 32, fe3be000, fe417db8, 0) + e8
    fe264240 __1cG_start6Fpv_0_ (1bc4c8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (1bc4c8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 15 / thread# 14  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, f1, 3baea0, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (ba8, fe406e44, 3baea0, fe412d84, 3bb180, fe3be000) + 260
    f840e22c ???????? (c9, eff818f4, eff81870, ffffff80, f00e5250, 0)
    f840e1d0 ???????? (0, b8, 0, 8, f00e5250, eff81888)
    f8405764 ???????? (2000, 2, 0, f8419754, eff0e110, eff81920)
    f8400218 ???????? (eff81a08, eff81c08, a, f423b868, f840ba20, eff81b20)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 3baea0, eff81b18, eff81a18, 4, eff81c08) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 3baea0, 254948, 254954, 254950, f01aaaf8) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 3baea0, 254d34, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (3baea0, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (3baea0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (3baea0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 16 / thread# 15  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, f0, 358da0, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (368, fe406e44, 358da0, fe412d84, 358f98, fe3be000) + 260
    f840e22c ???????? (91, efe81874, efe817f0, ffffff80, f00dfeb0, 0)
    f840e1d0 ???????? (0, b8, 0, 8, f00dfeb0, efe81808)
    f8405764 ???????? (2000, 2, 0, f8419754, efe0e110, efe818a0)
    f8400218 ???????? (efe81988, efe81b88, a, f423b868, f840ba20, efe81aa0)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 358da0, efe81a98, efe81998, 4, efe81b88) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 358da0, 254d48, 254d54, 254d50, f01b4270) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 358da0, 255134, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (358da0, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (358da0, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (358da0, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 17 / thread# 16  --------------------
    ff29d600 _poll    (fe4124d8, 0, 1, f5, 3dd8a8, fe3be000) + 8
    fdd90cd0 __1cCosFsleep6FpnGThread_xi_i_ (6e58, 6c00, fdf04534, 1, 7, 4) + 25c
    fdf0ca4c JVM_Sleep (1628, fe406e44, 3dd8a8, fe412d84, 3c5f98, fe3be000) + 260
    f840e22c ???????? (c1, efb015fc, efb01570, ffffff80, efb, 0)
    f840e1d0 ???????? (0, b8, 0, 8, 8800, efb01588)
    f8405764 ???????? (2000, 2, 0, f8419770, efa8e110, efb01620)
    f8400218 ???????? (efb01708, efb01908, a, f422e650, f840ba20, efb01820)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 3dd8a8, efb01818, efb01718, 4, efb01908) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 3dd8a8, 3568f8, 356904, 356900, f007a910) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 3dd8a8, 356ce4, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (3dd8a8, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (3dd8a8, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (3dd8a8, 0, 0, 0, 0, 0) + 40
    -----------------  lwp# 18 / thread# 17  --------------------
    ff29f428 ___lwp_cond_wait (65b7a0, 65b788, ef681128, fe4123c4, ff2968f4, ef681120) + 8
    ff296a7c _lwp_cond_timedwait (2ea2b88, 44b36759, ef681248, 65b788, 65b7a0, ff2969e4) + 98
    fdeb4974 __1cNObjectMonitorEwait6MxipnGThread__v_ (120400, fe406ef0, fe4238d8, fe3be000, 65b550, 120420) + 7a8
    fdeb5978 __1cSObjectSynchronizerEwait6FnGHandle_xpnGThread__v_ (120402, 86d4, 0, 65b550, 8400, fe3be000) + e8
    fdeb5628 JVM_MonitorWait (0, 3dcdb4, 65b550, fe412d84, 0, 1) + 31c
    f840e22c ???????? (91, ef6814b4, ef681430, ffffff80, ef6, 0)
    f840e1d0 ???????? (f1631648, b6, 0, c, f00d7f28, ef681448)
    f8405764 ???????? (f007ea58, fe4163d4, 0, f8419354, 8800, ef681508)
    f8405764 ???????? (2000, 2, 0, f84195a0, ef60e110, ef6815a0)
    f8400218 ???????? (ef681688, ef681888, a, f456d6b8, f840ba20, ef6817a0)
    fdd9a808 __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (1, 65b550, ef681798, ef681698, 4, ef681888) + 5a0
    fdec319c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (7a9c, 65b550, 3dcda0, 3dcdac, 3dcda8, f007ea58) + 188
    fdee2700 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (f402fc58, 65b550, 3dd18c, fe424640, fe424710, fe424130) + 134
    fdede29c __1cKJavaThreadDrun6M_v_ (65b550, 18, fe40c790, 6c00, 7dd4, 7c00) + 1d8
    fe264240 __1cG_start6Fpv_0_ (65b550, 3b62, fe3be000, 0, 4f24, 4c00) + 208
    ff37b01c _thread_start (65b550, 0, 0, 0, 0, 0) + 40
    --------------------------  thread# 3  --------------------
    ff36d9b8 _reap_wait (ff390980, 1e924, 0, ff38c000, 0, 0) + 38
    ff36d710 _reaper  (ff38ce00, ff392708, ff390980, ff38cdd8, 1, fe400000) + 38
    ff37b01c _thread_start (0, 0, 0, 0, 0, 0) + 40

  • Java process - high CPU usage

    Hi,
    I'm describing a high CPU scenario which gets triggered randomly ( I'm not able to replicate it on my lab setup).
    There are around 120 threads which are running in my java process. The jvm is running on a high traffic (through put) site, where there are a lot of async events coming to the java process.( around 220 events per 60 seconds ). The java process works fine in this scenario, the normal CPU consumption hovers around 1.5 % to 2.0 %.
    But, at times, I've seen CPU to be as high as 43 %, and it stays at that value for hours altogther. In those situations, I usually do a failover to standby java process. I tried debugging the issue to see which java thread could be causing the issue, but, I could not come to a conclusion or replicate the situation in lab environment.
    Here are the details of the execution environment
    java -version
    java version "1.4.2_11"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_11-b06)
    Java HotSpot(TM) Client VM (build 1.4.2_11-b06, mixed mode)
    prstat during high CPU
    PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
    10485 root 120 10 0 570M 381M cpu1 268:10 43.64% java
    prstat -Lm -p output
    PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
    10485 root 53 0.5 0.2 0.0 0.0 30 0.2 16 69 1K 118 0 java/2
    10485 root 31 0.0 0.1 0.0 0.0 53 0.2 16 23 778 93 0 java/26
    10485 root 0.4 0.0 0.0 0.0 0.0 99 0.0 0.1 10 16 106 0 java/12
    10485 root 0.1 0.0 0.0 0.0 0.0 100 0.0 0.0 3 2 7 0 java/15
    10485 root 0.1 0.0 0.0 0.0 0.0 97 0.0 2.4 120 3 128 0 java/41
    10485 root 0.1 0.0 0.0 0.0 0.0 97 0.0 2.5 120 4 131 0 java/410
    Some more points about the last prstat -Lm output.
    java/2 is "VM Thread" ( responsible for GC). "VM Thread" is having a NORMAL priority ( 5 )
    java/26 is a "Worker" thread, with priority MINIMUM ( 1 ).
    Could you suggest what could be issue, and what other information I could collect to find out the issue. Its difficult to profile the process because the problem scenario is difficult to ascertain and the process is running on a production setup.
    Any help is appreciated.
    Thanks
    Sanjay

    Hi,
    Thanks for your response. Both, the production setup and lab setup have have 2 physical CPUs.
    Actually, there are two java threads ( machine is solaris 10) one is "VM Thread" and other is my applications worker thread. (there are 10 of them with priority 1). If you look at the top two lwps in the prstat -Lm , both are showing high value of ICX.
    I'm still not able to drill down to my code level. (Worker thread is waiting on a queue to de-queue server request). Could you give some hint to move foward?
    rgds
    Sanjay

  • Java process table empty after upgrade solution manager 7.1 Sp10

    Hi Experts,
    I have installed solution manager 7.1 SP3 and completed the SP10 patch upgrade using SUM tool. The upgrade was completed successfully. After upgrade igswd was red. Then i copied the Igs files in kernel files. Now ABAP instance and Igs is working now but JAVA is not coming up. JAVA process table also empty. Can any one help me to solve this issue. Attached the error for reference.
    trc file: "dev_disp", trc level: 1, release: "721"
    sysno      00
    sid        VSM
    systemid   562 (PC with Windows NT)
    relno      7210
    patchlevel 0
    patchno    201
    intno      20020600
    make       multithreaded, Unicode, 64 bit, optimized
    profile    \\VKCSOL\sapmnt\VSM\SYS\profile\VSM_DVEBMGS00_VKCSOL
    pid        5448
    kernel runs with dp version 138000(ext=120000) (@(#) DPLIB-INT-VERSION-138000-UC)
    length of sys_adm_ext is 592 bytes
    *** SWITCH TRC-HIDE on ***
    ***LOG Q00=> DpSapEnvInit, DPStart (00 5448) [dpxxdisp.c   1316]
      shared lib "dw_xml.dll" version 201 successfully loaded
      shared lib "dw_xtc.dll" version 201 successfully loaded
      shared lib "dw_stl.dll" version 201 successfully loaded
      shared lib "dw_gui.dll" version 201 successfully loaded
      shared lib "dw_mdm.dll" version 201 successfully loaded
      shared lib "dw_rndrt.dll" version 201 successfully loaded
      shared lib "dw_abp.dll" version 201 successfully loaded
      shared lib "dw_sym.dll" version 201 successfully loaded
      shared lib "dw_aci.dll" version 201 successfully loaded
    rdisp/softcancel_sequence :  -> 0,5,-1
    use internal message server connection to port 3900
    rdisp/dynamic_wp_check : 1
    rdisp/calculateLoadAverage : 1
    Tue Apr 01 19:02:09 2014
    *** WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 6 seconds
    ***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c  6479]
    MtxInit: 30000 0 0
    DpSysAdmExtInit: ABAP is active
    DpSysAdmExtInit: VMC (JAVA VM in WP) is not active
    display_tcp_ip: 0
    DpIPCInit2: write dp-profile-values into sys_adm_ext
    DpIPCInit2: start server >VKCSOL_VSM_00                           <
    DpShMCreate: sizeof(wp_adm) 42864 (2256)
    DpShMCreate: sizeof(tm_adm) 5517056 (27448)
    DpShMCreate: sizeof(wp_ca_adm) 64000 (64)
    DpShMCreate: sizeof(appc_ca_adm) 64000 (64)
    DpCommTableSize: max/headSize/ftSize/tableSize=500/16/1384064/1384080
    DpShMCreate: sizeof(comm_adm) 1384080 (2744)
    DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=0/0/0/0/0
    DpShMCreate: sizeof(slock_adm) 0 (296)
    DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0
    DpShMCreate: sizeof(file_adm) 0 (80)
    DpShMCreate: sizeof(vmc_adm) 0 (2160)
    DpShMCreate: sizeof(wall_adm) (41664/42896/64/192)
    DpShMCreate: sizeof(gw_adm) 48
    DpShMCreate: sizeof(j2ee_adm) 3952
    DpShMCreate: SHM_DP_ADM_KEY (addr: 000000000DDE0050, size: 7174832)
    DpShMCreate: allocated sys_adm at 000000000DDE0060
    DpShMCreate: allocated wp_adm_list at 000000000DDE30B0
    DpShMCreate: allocated wp_adm at 000000000DDE32A0
    DpShMCreate: allocated tm_adm_list at 000000000DDEDA20
    DpShMCreate: allocated tm_adm at 000000000DDEDA70
    DpShMCreate: allocated wp_ca_adm at 000000000E330980
    DpShMCreate: allocated appc_ca_adm at 000000000E340390
    DpShMCreate: allocated comm_adm at 000000000E34FDA0
    DpShMCreate: system runs without slock table
    DpShMCreate: system runs without file table
    DpShMCreate: allocated vmc_adm_list at 000000000E4A1C40
    DpShMCreate: system runs without vmc_adm
    DpShMCreate: allocated gw_adm at 000000000E4A1CF0
    DpShMCreate: allocated j2ee_adm at 000000000E4A1D30
    DpShMCreate: allocated ca_info at 000000000E4A2CB0
    DpShMCreate: allocated wall_adm at 000000000E4A2D40
    DpCommAttachTable: attached comm table (header=000000000E34FDA0/ft=000000000E34FDB0)
    DpSysAdmIntInit: initialize sys_adm
    rdisp/test_roll : roll strategy is DP_NORMAL_ROLL
    dia token check not active (10 token)
    MBUF state OFF
    DpCommInitTable: init table for 500 entries
    DpRqQInit: keep protect_queue / slots_per_queue 0 / 2001 in sys_adm
    rdisp/queue_size_check_value :  -> on,50,30,40,500,50,500,80
    EmInit: MmSetImplementation( 2 ).
    MM global diagnostic options set: 0
    <ES> client 0 initializing ....
    <ES> EsILock: use spinlock for locking
    <ES> InitFreeList
    <ES> block size is 4096 kByte.
    <ES> Info: em/initial_size_MB( 16383MB) not multiple of em/blocksize_KB( 4096KB)
    <ES> Info: em/initial_size_MB rounded up to 16384MB
    Using implementation view
    <EsNT> Using memory model view.
    <EsNT> Memory Reset disabled as NT default
    <ES> 4095 blocks reserved for free list.
    ES initialized.
    mm.dump: set maximum dump mem to 192 MB
    mm.dump: set global maximum dump mem to 192 MB
    EsRegisterEmCheck: Register EmGetEsHandles at 000000013FC48640
    DpVmcSetActive: set vmc state DP_VMC_NOT_ACTIVE
    MPI: dynamic quotas disabled.
    MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%
    J2EE server info
      start = TRUE
      state = STARTED
      pid = 5824
      argv[0] = D:\usr\sap\VSM\DVEBMGS00\exe\jcontrol.EXE
      argv[1] = D:\usr\sap\VSM\DVEBMGS00\exe\jcontrol.EXE
      argv[2] = pf=\\VKCSOL\sapmnt\VSM\SYS\profile\VSM_DVEBMGS00_VKCSOL
      argv[3] = -DSAPSTART=1
      argv[4] = -DCONNECT_PORT=65000
      argv[5] = -DSAPSYSTEM=00
      argv[6] = -DSAPSYSTEMNAME=VSM
      argv[7] = -DSAPMYNAME=VKCSOL_VSM_00
      argv[8] = -DSAPPROFILE=\\VKCSOL\sapmnt\VSM\SYS\profile\VSM_DVEBMGS00_VKCSOL
      argv[9] = -DFRFC_FALLBACK=ON
      argv[10] = -DFRFC_FALLBACK_HOST=localhost
      start_lazy = 0
      start_control = SAP J2EE startup framework
    DpJ2eeStart: j2ee state = STARTED
    rdisp/http_min_wait_dia_wp : 1 -> 1
    ***LOG CPS=> DpLoopInit, ICU ( 4.0.1 4.0.1 5.1) [dpxxdisp.c   1702]
    ***LOG Q0K=> DpMsAttach, mscon ( VKCSOL) [dpxxdisp.c   12707]
    MBUF state LOADING
    DpStartStopMsg: send start message (myname is >VKCSOL_VSM_00                           <)
    DpStartStopMsg: start msg sent
    CCMS: Initialized monitoring segment of size 60000000.
    CCMS: Initialized CCMS Headers in the shared monitoring segment.
    CCMS: Checking Downtime Configuration of Monitoring Segment.
    CCMS: AlMsUpload called by wp 1024.
    Tue Apr 01 19:02:10 2014
    01.04.2014 13:32:10.628 PID=5448, TID=5744 SapTimer info. QPC/Windows time mismatch. System suspended, busy, missing ticks. Mismatch QpcTime-WindowsTime:  -168 millisec (max +0). Slept 1000 millisec, Qpc elapsed: +1000 millisec, WindowsUtc elapsed +1168 millisec. SapTime-WindowsUtcTime=-1 seconds. TotalTimeCorrection=+0 millisec, QpcTime-WindowsUtcTime=-1028 millisec (*).
    Tue Apr 01 19:02:11 2014
    CCMS: AlMsUpload successful for D:\usr\sap\VSM\DVEBMGS00\log\ALMTTREE.DAT (2189 MTEs).
    Tue Apr 01 19:02:13 2014
    CCMS: start to initalize 3.X shared alert area (first segment).
    DpJ2eeLogin: j2ee state = CONNECTED
    DpMBufHwIdSet: set Hardware-ID
    ***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1296]
    MBUF state ACTIVE
    DpWpBlksLow: max wp blocks in queue is 800 (80 %)
    MBUF component UP
    DpMsgProcess: 1 server in MBUF
    DpAppcBlksLow: max appc blocks in queue is 500 (50 %)
    Tue Apr 01 19:02:21 2014
    DpModState: change server state from STARTING to ACTIVE
    Tue Apr 01 19:02:23 2014
    ***LOG Q0I=> NiIRead: P=127.0.0.1:61107; L=127.0.0.1:65000: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 5086]
    *** ERROR => NiIRead: SiRecv failed for hdl 33/sock 1008
        (SI_ECONN_BROKEN/10054; I4; ST; P=127.0.0.1:61107; L=127.0.0.1:65000) [nixxi.cpp    5086]
    DpJ2eeMsgProcess: j2ee state = CONNECTED (NIECONN_BROKEN)
    DpIJ2eeShutdown: send SIGINT to SAP J2EE startup framework (pid=5824)
    *** ERROR => DpProcKill: kill failed [dpntdisp.c   397]
    DpIJ2eeShutdown: j2ee state = SHUTDOWN
    Regards
    Jose H.

    Hi Ashutosh,
    Thanks for your immediate reply. Please find the log below.
    trc file: "dev_server0", trc level: 1, release: "721"
    node name   : ID3506750
    pid         : 8068
    system name : VSM
    system nr.  : 00
    started at  : Sat Mar 29 13:41:08 2014
    arguments  :
       arg[00] : D:\usr\sap\VSM\DVEBMGS00\exe\jlaunch.exe
       arg[01] : pf=D:\usr\sap\VSM\SYS\profile\VSM_DVEBMGS00_VKCSOL
       arg[02] : -DSAPINFO=VSM_00_server
       arg[03] : pf=D:\usr\sap\VSM\SYS\profile\VSM_DVEBMGS00_VKCSOL
       arg[04] : -DSAPSTART=1
    Used property files
    -> files [00] : D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\instance.properties
    Bootstrap nodes
    -> [00] bootstrap            : D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\instance.properties
    -> [01] bootstrap_ID3506700  : D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\instance.properties
    -> [02] bootstrap_ID3506750  : D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\instance.properties
    Worker nodes
    -> [00] ID3506700            : D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\instance.properties
    -> [01] ID3506750            : D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\instance.properties
    JStartupReadInstanceProperties: read instance properties [D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\instance.properties]
    -> ms host    : VKCSOL
    -> ms port    : 3901
    -> OS libs    : D:\usr\sap\VSM\DVEBMGS00\j2ee\os_libs
    -> Admin URL  :
    -> run mode   : safe
    -> run action : UPGRADE
    -> enabled    : yes
    [Thr 19300] Sat Mar 29 13:41:08 2014
    [Thr 19300] JLaunchRequestQueueInit: create named pipe for ipc
    [Thr 19300] JLaunchRequestQueueInit: create pipe listener thread
    [Thr 19300] JLaunchInitSignalHandling: signal handling is disabled
    [Thr 4928] WaitSyncSemThread: Thread 4928 started as semaphore monitor thread.
    [Thr 23444] JLaunchRequestFunc: Thread 23444 started as listener thread for np messages.
    [Thr 19300] NiInit3: NI already initialized; param 'maxHandles' ignored (init=2;par=32768;cur=2048)
    [Thr 19300] MPI: dynamic quotas disabled.
    [Thr 19300] MPI init: pipes=4000 buffers=1279 reserved=383 quota=10%
    [Thr 19300] CCMS: Detected Double Stack (parameter "rdisp/j2ee_start" is active)
    [Thr 19300] CCMS: SemInMgt: Semaphore Management initialized by AlAttachShm_Doublestack.
    [Thr 19300] CCMS: SemInit: Semaphore 38 initialized by AlAttachShm_Doublestack.
    [Thr 19300] CCMS: CCMS Monitoring Initialization finished, rc=0.
    [Thr 19300] *** WARNING => Maximum Java heap size specified twice (through maxHeapSize and in javaParameters) - using -Xmx2048m [jstartxx.c   2800]
    [Thr 19300] JStartupICheckFrameworkPackage: can't find framework package D:\usr\sap\VSM\DVEBMGS00\exe\jvmx.jar
    JStartupIReadSection: read node properties [ID3506750]
    -> node name          : server0
    -> node type          : server
    -> node execute       : yes
    -> jlaunch parameters :
    -> java path          : D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4
    -> java parameters    : -XX:MaxNewSize=600M -XX:NewSize=600M -XX:MaxPermSize=512M -XX:PermSize=512M -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:SurvivorRatio=4 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -XX:SoftRefLRUPolicyMSPerMB=1 -XX:HeapDumpPath=OOM.hprof -XX:+HeapDumpOnOutOfMemoryError -Djco.jarm=1 -Djava.awt.headless=true -Djava.security.policy=./java.policy -Djava.security.egd=file:/dev/urandom -Dsun.io.useCanonCaches=false -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy -Dorg.omg.PortableInterceptor.ORBInitializerClass.com.sap.engine.services.ts.jts.ots.PortableInterceptor.JTSInitializer -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy -Dcom.sap.jvm.scenario=j2ee -XX:+JavaMonitorsInStackTrace -Xbootclasspath/p:D:/usr/sap/DAA/SMDA97/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.8.2.3.5-2011-09-13/wily/connectors/AutoProbeConnector.jar;D:/usr/sap/DAA/SMDA97/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.8.2.3.5-2011-09-13/wily/Agent.jar -Dcom.wily.introscope.agent.agentName=VSM_DVEBMGS00_server0 -Dcom.wily.introscope.agentProfile=D:/usr/sap/DAA/SMDA97/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.8.2.3.5-2011-09-13/wily/IntroscopeAgent.profile -Dstartup.mode=SAFE -Dstartup.action=UPGRADE
    -> java vm version    : 4.1.024 1.6-b04
    -> java vm vendor     : SAP Java Server VM (SAP AG)
    -> java vm type       : server
    -> java vm cpu        : amd64
    -> heap size          : 2048M
    -> init heap size     : 2048M
    -> stack size         : 2M
    -> root path          : D:\usr\sap\VSM\DVEBMGS00\j2ee\cluster\server0
    -> class path         : .\bin\boot\boot.jar;.\bin\boot\jaas.jar;.\bin\system\bytecode.jar;.
    -> OS libs path       : D:\usr\sap\VSM\DVEBMGS00\j2ee\os_libs
    -> main class         : com.sap.engine.boot.Start
    -> framework class    : com.sap.bc.proj.jstartup.JStartupFramework
    -> registr. class     : com.sap.bc.proj.jstartup.JStartupNatives
    -> framework path     : D:\usr\sap\VSM\DVEBMGS00\exe\jstartup.jar;D:\usr\sap\VSM\DVEBMGS00\exe\jvmx.jar
    -> shutdown class     : com.sap.engine.boot.Start
    -> parameters         :
    -> debuggable         : no
    -> debug mode         : no
    -> debug port         : 50021
    -> shutdown timeout   : 120 sec.
    [Thr 19300] JLaunchISetDebugMode: set debug mode [no]
    [Thr 9092] JLaunchIStartFunc: Thread 9092 started as Java VM thread.
    JHVM_LoadJavaVM: VM Arguments of node [server0]
    -> stack   : 2097152 Bytes
    -> arg[  0]: exit
    -> arg[  1]: abort
    -> arg[  2]: vfprintf
    -> arg[  3]: -XX:MaxNewSize=600M
    -> arg[  4]: -XX:NewSize=600M
    -> arg[  5]: -XX:MaxPermSize=512M
    -> arg[  6]: -XX:PermSize=512M
    -> arg[  7]: -XX:+UseConcMarkSweepGC
    -> arg[  8]: -XX:+DisableExplicitGC
    -> arg[  9]: -XX:TargetSurvivorRatio=90
    -> arg[ 10]: -XX:SurvivorRatio=4
    -> arg[ 11]: -XX:+PrintGCDetails
    -> arg[ 12]: -XX:+PrintGCTimeStamps
    -> arg[ 13]: -verbose:gc
    -> arg[ 14]: -XX:SoftRefLRUPolicyMSPerMB=1
    -> arg[ 15]: -XX:HeapDumpPath=OOM.hprof
    -> arg[ 16]: -XX:+HeapDumpOnOutOfMemoryError
    -> arg[ 17]: -Djco.jarm=1
    -> arg[ 18]: -Djava.awt.headless=true
    -> arg[ 19]: -Djava.security.policy=./java.policy
    -> arg[ 20]: -Djava.security.egd=file:/dev/urandom
    -> arg[ 21]: -Dsun.io.useCanonCaches=false
    -> arg[ 22]: -Dorg.omg.CORBA.ORBClass=com.sap.engine.system.ORBProxy
    -> arg[ 23]: -Dorg.omg.CORBA.ORBSingletonClass=com.sap.engine.system.ORBSingletonProxy
    -> arg[ 24]: -Dorg.omg.PortableInterceptor.ORBInitializerClass.com.sap.engine.services.ts.jts.ots.PortableInterceptor.JTSInitializer
    -> arg[ 25]: -Djavax.rmi.CORBA.PortableRemoteObjectClass=com.sap.engine.system.PortableRemoteObjectProxy
    -> arg[ 26]: -Dcom.sap.jvm.scenario=j2ee
    -> arg[ 27]: -XX:+JavaMonitorsInStackTrace
    -> arg[ 28]: -Xbootclasspath/p:D:/usr/sap/DAA/SMDA97/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.8.2.3.5-2011-09-13/wily/connectors/AutoProbeConnector.jar;D:/usr/sap/DAA/SMDA97/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.8.2.3.5-2011-09-13/wily/Agent.jar
    -> arg[ 29]: -Dcom.wily.introscope.agent.agentName=VSM_DVEBMGS00_server0
    -> arg[ 30]: -Dcom.wily.introscope.agentProfile=D:/usr/sap/DAA/SMDA97/SMDAgent/applications.config/com.sap.smd.agent.application.wily/BytecodeAgent/ISAGENT.8.2.3.5-2011-09-13/wily/IntroscopeAgent.profile
    -> arg[ 31]: -Dstartup.mode=SAFE
    -> arg[ 32]: -Dstartup.action=UPGRADE
    -> arg[ 33]: -Dsys.global.dir=D:\usr\sap\VSM\SYS\global
    -> arg[ 34]: -Dapplication.home=D:\usr\sap\VSM\DVEBMGS00\exe
    -> arg[ 35]: -Djava.class.path=D:\usr\sap\VSM\DVEBMGS00\exe\jstartup.jar;D:\usr\sap\VSM\DVEBMGS00\exe\jvmx.jar;.\bin\boot\boot.jar;.\bin\boot\jaas.jar;.\bin\system\bytecode.jar;.
    -> arg[ 36]: -Djava.library.path=D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4\jre\bin\server;D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4\jre\bin;D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4\bin;D:\usr\sap\VSM\DVEBMGS00\j2ee\os_libs;D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4\jre\bin\server;D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4\jre\bin;D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4\bin;D:\usr\sap\VSM\DVEBMGS00\j2ee\os_libs;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\;D:\usr\sap\VSM\DVEBMGS00\exe;D:\usr\sap\VSM\SYS\exe\uc\NTAMD64
    -> arg[ 37]: -Dmemory.manager=2048M
    -> arg[ 38]: -Xmx2048M
    -> arg[ 39]: -Xms2048M
    -> arg[ 40]: -Xss2M
    -> arg[ 41]: -DLoadBalanceRestricted=no
    -> arg[ 42]: -Djstartup.mode=JCONTROL
    -> arg[ 43]: -Djstartup.ownProcessId=8068
    -> arg[ 44]: -Djstartup.ownHardwareId=I0649086862
    -> arg[ 45]: -Djstartup.whoami=server
    -> arg[ 46]: -Djstartup.debuggable=no
    -> arg[ 47]: -Xjvmx
    -> arg[ 48]: -XsapSystem=00
    -> arg[ 49]: -XmonGcCallback
    -> arg[ 50]: -DSAPINFO=VSM_00_server
    -> arg[ 51]: -DSAPSTART=1
    -> arg[ 52]: -DSAPSTARTUP=1
    -> arg[ 53]: -DSAPSYSTEM=00
    -> arg[ 54]: -DSAPSYSTEMNAME=VSM
    -> arg[ 55]: -DSAPMYNAME=VKCSOL_VSM_00
    -> arg[ 56]: -DSAPDBHOST=VKCSOL
    -> arg[ 57]: -Dj2ee.dbhost=VKCSOL
    (CompilerOracle read from file D:\usr\sap\VSM\DVEBMGS00\exe\sapjvm_4\jre\.hotspot_compiler)
    [Thr 9092] JHVM_LoadJavaVM: Java VM created OK.
    JHVM_BuildArgumentList: main method arguments of node [server0]
    [Thr 20084] Sat Mar 29 13:41:10 2014
    [Thr 20084] JHVM_RegisterNatives: com.sap.bc.krn.perf.PerfTimes
    2.781: [GC 2.781: [ParNew: 409600K->2306K(512000K), 0.0126041 secs] 409600K->2306K(1994752K), 0.0127881 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
    [Thr 20084] Sat Mar 29 13:41:11 2014
    [Thr 20084] JHVM_RegisterNatives: com.sap.bc.proj.jstartup.JStartupFramework
    [Thr 20084] JLaunchISetClusterId: set cluster id 3506750
    [Thr 20084] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)]
    [Thr 20084] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)]
    Sat Mar 29 13:41:23 2014
    15.071: [GC 15.071: [ParNew: 411906K->16315K(512000K), 0.0306641 secs] 411906K->16315K(1994752K), 0.0309011 secs] [Times: user=0.13 sys=0.00, real=0.03 secs]
    Sat Mar 29 13:41:32 2014
    24.118: [GC 24.118: [ParNew: 425915K->29114K(512000K), 0.0363407 secs] 425915K->29114K(1994752K), 0.0365634 secs] [Times: user=0.11 sys=0.02, real=0.04 secs]
    [Thr 18736] Sat Mar 29 13:41:38 2014
    [Thr 18736] JHVM_RegisterNatives: com.sap.mona.api.JMonAPI
    [Thr 18736] CCMS: openMonitoringSegment(): inside static JMON Lib: already connected.
    [Thr 17644] JHVM_RegisterNatives: com.sap.mw.rfc.driver.CpicDriver
    [Thr 19340] JHVM_RegisterNatives: com.sap.security.core.server.vsi.service.jni.VirusScanInterface
    [Thr 7816] Sat Mar 29 13:41:39 2014
    [Thr 7816] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7816] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 25/sock 7884
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17936] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17936] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 17/sock 7888
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    31.816: [GC 31.816: [ParNew[Thr 21936] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 21936] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 1/sock 7856
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 2800] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 2800] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 9/sock 7860
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    : 438714K->67978K(512000K), 0.0626636 secs] 438714K->67978K(1994752K), 0.0628963 secs] [Times: user=0.11 sys=0.08, real=0.06 secs]
    [Thr 7276] Sat Mar 29 13:41:40 2014
    [Thr 7276] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7276] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 7988
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 24572] Sat Mar 29 13:41:42 2014
    [Thr 24572] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 24572] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 8132
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 5092] Sat Mar 29 13:41:43 2014
    [Thr 5092] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 5092] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 8088
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 20084] JLaunchISetState: change state from [Starting (2)] to [Starting applications (10)]
    [Thr 24276] Sat Mar 29 13:41:44 2014
    [Thr 24276] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 24276] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 8480
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 1316] Sat Mar 29 13:41:45 2014
    [Thr 1316] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 1316] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 9292
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 13068] Sat Mar 29 13:41:46 2014
    [Thr 13068] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 13068] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 9736
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 19652] JLaunchISetState: change state from [Starting applications (10)] to [Running (3)]
    [Thr 17680] Sat Mar 29 13:41:47 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 9736
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 23760] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 23760] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 10/sock 5976
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:41:48 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 9736
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:41:49 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 9736
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:41:50 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 6272
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 5348] Sat Mar 29 13:41:51 2014
    [Thr 5348] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 5348] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 6336
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17548] Sat Mar 29 13:41:52 2014
    [Thr 17548] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17548] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 6548
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 4216] Sat Mar 29 13:41:53 2014
    [Thr 4216] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 4216] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 6424
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 23788] Sat Mar 29 13:41:54 2014
    [Thr 23788] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 23788] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 6844
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 6100] Sat Mar 29 13:41:55 2014
    [Thr 6100] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 6100] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 7448
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 6828] Sat Mar 29 13:41:56 2014
    [Thr 6828] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 6828] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 6216
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 644] Sat Mar 29 13:41:57 2014
    [Thr 644] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 644] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 5992
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 7376] Sat Mar 29 13:41:58 2014
    [Thr 7376] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7376] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 6136
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 24096] Sat Mar 29 13:41:59 2014
    [Thr 24096] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 24096] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 6004
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 24460] Sat Mar 29 13:42:00 2014
    [Thr 24460] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 24460] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 6276
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 13452] Sat Mar 29 13:42:01 2014
    [Thr 13452] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 13452] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 7328
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17884] Sat Mar 29 13:42:03 2014
    [Thr 17884] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17884] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 7372
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 6616] Sat Mar 29 13:42:04 2014
    [Thr 6616] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 6616] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 7200
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 3136] Sat Mar 29 13:42:05 2014
    [Thr 3136] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 3136] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 6344
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 8284] Sat Mar 29 13:42:06 2014
    [Thr 8284] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 8284] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 7380
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 19240] Sat Mar 29 13:42:07 2014
    [Thr 19240] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 19240] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 6940
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 21316] Sat Mar 29 13:42:08 2014
    [Thr 21316] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 21316] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 5988
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 22436] Sat Mar 29 13:42:09 2014
    [Thr 22436] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 22436] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 7368
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 20936] Sat Mar 29 13:42:10 2014
    [Thr 20936] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 20936] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 7528
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 18784] Sat Mar 29 13:42:11 2014
    [Thr 18784] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 18784] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 4460
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 21320] Sat Mar 29 13:42:12 2014
    [Thr 21320] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 21320] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 6148
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 19632] Sat Mar 29 13:42:13 2014
    [Thr 19632] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 19632] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 5948
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 8684] Sat Mar 29 13:42:14 2014
    [Thr 8684] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 8684] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 6012
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 7468] Sat Mar 29 13:42:15 2014
    [Thr 7468] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7468] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 6032
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 5588] Sat Mar 29 13:42:16 2014
    [Thr 5588] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 5588] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 1032
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 2036] Sat Mar 29 13:42:17 2014
    [Thr 2036] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 2036] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 6052
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 6692] Sat Mar 29 13:42:18 2014
    [Thr 6692] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 6692] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 6188
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 5184] Sat Mar 29 13:42:19 2014
    [Thr 5184] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 5184] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 6192
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 5604] Sat Mar 29 13:42:20 2014
    [Thr 5604] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 5604] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 6220
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 15996] Sat Mar 29 13:42:21 2014
    [Thr 15996] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 15996] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 6236
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:42:22 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 6236
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:42:23 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 6236
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:42:24 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 6236
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:42:26 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 6064
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 19856] Sat Mar 29 13:42:27 2014
    [Thr 19856] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 19856] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 6184
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 24128] Sat Mar 29 13:42:28 2014
    [Thr 24128] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 24128] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 6248
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 13684] Sat Mar 29 13:42:29 2014
    [Thr 13684] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 13684] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 6260
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 7384] Sat Mar 29 13:42:30 2014
    [Thr 7384] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7384] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 6264
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 7432] Sat Mar 29 13:42:31 2014
    [Thr 7432] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7432] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 6292
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 6700] Sat Mar 29 13:42:32 2014
    [Thr 6700] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 6700] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 6304
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 7272] Sat Mar 29 13:42:33 2014
    [Thr 7272] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7272] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 6340
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 19864] Sat Mar 29 13:42:34 2014
    [Thr 19864] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 19864] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 6392
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 5608] Sat Mar 29 13:42:35 2014
    [Thr 5608] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 5608] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 6404
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 7160] Sat Mar 29 13:42:36 2014
    [Thr 7160] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7160] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 6416
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 14536] Sat Mar 29 13:42:37 2014
    [Thr 14536] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 14536] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 7340
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 24404] Sat Mar 29 13:42:38 2014
    [Thr 24404] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 24404] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 6232
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 22756] Sat Mar 29 13:42:39 2014
    [Thr 22756] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 22756] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 6076
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 6284] Sat Mar 29 13:42:40 2014
    [Thr 6284] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 6284] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 7472
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 12696] Sat Mar 29 13:42:41 2014
    [Thr 12696] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 12696] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 35/sock 6380
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 24180] Sat Mar 29 13:42:42 2014
    [Thr 24180] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 24180] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 36/sock 6420
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 23444] Sat Mar 29 13:42:43 2014
    [Thr 23444] JLaunchRequestFunc: receive command:17, argument:0 from pid:20668
    [Thr 23444] JLaunchIShutdownInvoke: set shutdown interval (stop:1396080763/end:1396080883/TO:120)
    [Thr 23444] JLaunchProcessCommand: Invoke VM Shutdown
    [Thr 23444] JHVM_FrameworkShutdownDirect: invoke direct shutdown
    [Thr 8780] JLaunchISetState: change state from [Running (3)] to [Waiting for stop (4)]
    [Thr 8780] JLaunchISetState: change state from [Waiting for stop (4)] to [Stopping (5)]
    [Thr 5188] JHVM_RegisterNatives: com.sap.bc.proj.jstartup.sadm.ShmCache
    [Thr 7240] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 7240] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 37/sock 6376
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 6536] Sat Mar 29 13:42:44 2014
    [Thr 6536] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 6536] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 38/sock 6852
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 18244] Sat Mar 29 13:42:46 2014
    [Thr 18244] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 18244] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 39/sock 6348
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:42:47 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 32/sock 6348
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    [Thr 17680] Sat Mar 29 13:42:48 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 127.0.0.1:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 33/sock 6348
        (SI_ECONN_REFUSE/10061; I4; ST; 127.0.0.1:3300) [nixxi.cpp    3285]
    [Thr 22796] CCMS: closeMonitoringSegment(): inside static JMON Lib: nothing to do.
    [Thr 8780] JLaunchISetState: change state from [Stopping (5)] to [Stopped (6)]
    [Thr 17680] Sat Mar 29 13:42:49 2014
    [Thr 17680] ***LOG Q0I=> NiPConnect2: 192.168.10.62:3300: connect (10061: WSAECONNREFUSED: Connection refused) [nixxi.cpp 3285]
    [Thr 17680] *** ERROR => NiPConnect2: SiPeekPendConn failed for hdl 34/sock 6348
        (SI_ECONN_REFUSE/10061; I4; ST; 192.168.10.62:3300) [nixxi.cpp    3285]
    Heap
    par new generation   reserved 614400K, committed 614400K, used 313963K [0x000000007fff0000, 0x00000000a57f0000, 0x00000000a57f0000)
      eden space 409600K,  60% used [0x000000007fff0000, 0x000000008f028658, 0x0000000098ff0000)
      from space 102400K,  66% used [0x0000000098ff0000, 0x000000009d2528e8, 0x000000009f3f0000)
      to   space 102400K,   0% used [0x000000009f3f0000, 0x000000009f3f0000, 0x00000000a57f0000)
    concurrent mark-sweep generation reserved 1482752K, committed 1482752K, used 0K [0x00000000a57f0000, 0x00000000ffff0000, 0x00000000ffff0000)
    concurrent-mark-sweep perm gen reserved 524288K, committed 524288K, used 70349K [0x00000000ffff0000, 0x000000011fff0000, 0x000000011fff0000)
    [Thr 18896] JLaunchIExitJava: exit hook is called (rc = 0)
    [Thr 18896] JLaunchCloseProgram: good bye (exitcode = 0)
    Regards
    Jose H.

  • Starting New process (continous) from another java process

    Hi,
    I am running into a trouble in starting a new java process (continous process it is like daemon process ) from another java process.
    Eg;
    Caller.java runs on one JVM instance, from this Caller.java i need to start a daemon java process ConnectionManager.java (My caller.java should have to know whether ConnectionManager was successfully started as a seperate java process or not)
    Also Caller.java has to stop ConnectionManager.java(which is running as seperate process)
    Any help would be appeciated.
    Thank you very much.

    First of all the design is unusual. Your issue is basically intercommunication between two processes written in java. There are various ways to do that:
    1. Use Persistent system(File/Database) : Easiest but have external dependency.
    2. Use RPC calls both JVM,s little bit complex.
    3. Run new JVM in debug mode and connect on bootstrap port.
    4. Communicate using Sockets.
    Whatever suits you...

  • PRODUCTION ISSUE - Java process terminates unexpectedly

    Hello,
    I realize this is a BEA forum, however, we are perplexed by this issue and perhaps someone here has experienced something similar. Apologies if this topic is out of context to BEA.
    We are trying to troubleshoot a production issue where the JAVA process terminates without any debugging information available to try and understand the cause. This is a Windows 2K3 Server running JAVA 1.4.2_04 using native C++. (We are **NOT** running Weblogic on this server)
    Can anyone provide insight into how to debug this problem? Specifically, do any management beans exist to where we can track the crashing process or perhaps any tools? This is occurring frequently and we have no means of debugging this issue all while this is occurring on a PRODUCTION server.
    Any help would be greatly appreciated!
    Regards,
    -Nader

    what heap size are you providing while running your app ?

  • Java Process Consuming 100% of Server CPU

    Hi,
    We have a new instance of CQ5 (version 5.4) running on a Windows 2003 SP2 Server. For some reason, which we can't determine, after a period of time a Java process on the system begins to consume 100% of the CPU and the system becomes totally unresponsive. Has anyone seen this before? Is there a fix for this?
    RK         

    Does the it eventually recover on it's own? It's possible that the JVM is doing full garbage collection.  How big is your max heap size?  I suggest to add some garbage collection debugging in the JVM_OPTS to see if it is Java's garbage collection or not.  Might as well check the heap too, to see if it's running our of memory.
    Example;
    -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -Xloggc:/tmp/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp
    Ron

  • Long time to start on  Java process dispatcher

    Hi ,
    I am trying to Install Solution Manager. When I attempt to start my sap system, the Java process dispatcher of instance/stage SM1 (solution manager) takes too much time and finally doesn't start showing this intallation error.
    ERROR 2008-09-24 11:01:11.314
    CJS-30151  Java process dispatcher of instance SM1/DVEBMGS00 [ABAP: ACTIVE, Java: (dispatcher: UNKNOWN, server0: RUNNING)] did not start after 2:00 minutes. Giving up.
    ERROR 2008-09-24 11:01:11.502
    FCO-00011  The step startJava with step key |NW_Onehost|ind|ind|ind|ind|0|0|NW_Onehost_System|ind|ind|ind|ind|1|0|NW_CI_Instance|ind|ind|ind|ind|11|0|NW_CI_Instance_StartJava|ind|ind|ind|ind|5|0|startJava was executed with status ERROR .
    Can anyone please let me know what is missing in this procedure? Any idea?
    tkanks in advance.

    Hi Matias,
    What are the parameters you set for your dispatcher?
    Regards
    Val

  • All Java Processes crash on RH Linux 7.3 / 6.3 without any comment

    We have the following Problem on two Machines. We installed j2sdk1.4.0 first, then jdk-1.3.1_04 (Because of other Problems with 1.4.0).
    Now all running java Processes crash after up to two hours without any comment, any error file. Looks like they were killed by the 'kill' Command. The running java Processes are different Programs, that do very different things.
    The OS on the two machines are RH Linux 7.3 and 6.3 - normal installation without any essential changes.
    I would be very happy, if someone have an idea, how to solve this Problem. If you need additional Information, ask for.
    Holger

    ... if someone have an idea, how to solve this Problem.Seems like I remember seeing a problem like that.
    It turned out that indeed it was the 'kill' command. Another process was running, looking for the app, and killing it when it found it.
    We spend quite a bit of time trying to figure out why our app was 'crashing' before deciding to look elsewhere (when we discovered the other process.)

  • How do I stop Java processes from appearing on the dock?

    I have a Java application that has a fairly extensive build process using ant. It builds fine on OSX, but constantly annoys me. There are a large number of unit tests that run using JUnit, and each test starts in a new Java process. The problem is that when each process starts (and there are about 100 during the course of the build), a new Java icon appears on the dock and stays until the process finishes. First of all, that slows down the build (since it has to do all that graphics work to keep changing the appearance of the dock). Secondly, every time a new process starts, it takes away the keyboard focus from whatever I'm currently working on. I either have to click in the window I want again, or else wait for the process to finish.
    On Windows, this is handled with an alternative version of the JVM called "javaw.exe" that does not get a foreground window. However, I see no "javaw" in the Java directory on OSX. Is there some option to start Java with that will stop this dock updating? Note that there are times when I want the dock to be updated (when I'm running a "normal" Java app).

    Even when the Guest Account / Access is disabled, the login option / and lock screen, still have a guest login.
    This is because in Preferences/ Security & Privacy/General the bottom item is turned on.
    Turn on: Disable restarting to Safari when screen is locked

  • How to restart a java process in Oracle AS 10g

    Hi
    In Oracle AS 10g, the java process consumes 50% cpu resource due to the report invoked by a user. Now I need to restart the java back ground process?, please reply me with the syntax or with examples.
    ps -eaf
    CPU PROCESS
    50% /ORACLE_HOME/jdk/jre/bin/java -server -cp /report/oracle
    Thanks in advance
    Bala

    The copy method being using is unsupported, hence the target Portal would be in an unsupported state.
    This is unsupported as per the following note:
    Note 333867.1 - Portal Export and Import Utility Supportability Scenarios :
    Copy operations after an object is renamed:
    Exporting and importing a page group, then renaming the page group on the target system (to create a copy) and repeating the export and import operation.
    OR:
    Exporting and importing a page group, then renaming the same in source or target and repeating the export and import operation.
    Not supported.
    Please do not rename objects if you plan to perform export and import operations.
    Note: The display name can be changed, but internal name change is considered a renaming operation.
    In 10.1.4, renaming is supported, but renaming page group and pages with the intention of copying them is not supported..."

  • Java Process with High CPU

    I have a java process running on HP-UX with a 1.4.2 JVM. This program is using some third-party code (quartz) to act as a batch scheduler similar to cron. When running in production, the process runs fine for several days and then starts maxing out the CPU. The server has 8 cpu's and the last time the problem occured, 4 of the cpu's were 100% in use with this process.
    I've tried sending a "kill -3 pid" but I can't seem to get a thread dump. I've eventually gotten some 700Meg core files. Is there anyway to use these core files to tell me what is going on?
    Any ideas on how to trouble shoot this problem? We've used some profilers but we can't duplicate the problem on test servers so we aren't learning anything new.
    Thanks,
    Brian

    Try pstack to read the stacks of the threads in the process.
    Try pldd to determine the shared libraries in the process then
    use truss to trace calls in any of these libraries to determine where
    the cycles are being spent.

  • Ps -p running very slow (1-2 seconds) on Java process

    Hi Solaris gurus:
    I encountered a issue that running ps -p on java process running very slow, it took almost 2 seconds to complete.
    I issued a truss on the "ps -p " command, the following is part of the output:
    /1: 0.0001 fstat(1, 0xFFFFFFFF7FFFE1F0) = 0
    /1: 0.0000 write(1, " U I D P".., 55) = 55
    /1: 0.0002 open("/proc/19299/psinfo", O_RDONLY) = 3
    */1: 1.3170 read(3, "02\0\0\0\0\0\011\0\0 K c".., 416) = 416*
    /1: 1.2401 close(3) = 0
    /1: 0.0002 stat("/dev/tty", 0xFFFFFFFF7FFFE830) = 0
    It seems that the read() spent the most time.
    Anyone can help?

    Not enough memory, page-outs is too large
    8.91 GB    Page-outs
    After removing adware, and do a safe boot of Safari and remove extensions, then reset cache history etc. You need to do a boot into Recovery Mode and run Disk Repair from there. Also boot the system in Safe Mode.
    On startup it sounds like you have a problem with the directory which would also account for long startups and checking the directory. Along with or instead of DU Repair Disk you can use Single User Mode and fsck -fy to try to fix the directory but in some cases that may not be enough.
    Backups from before you got this adware and problems helps and should always be ready and able to restore a system from known good backups or system restore image.
    4GB of RAM may have been fine originally but "Early 2011" is now 5 years and 4 OS version changes. You can upgrade the memory and while at it consider a nice SSD internal drive which will help as well. Take a look and see what prices and options there are from http://www.macsales.com for your 2011 MacBook Pro.
    http://www.everymac.com
    Community for MBP MacBook Pro

  • JConsole not picking up local Java processes

    Hi,
    Using JDK 1.5.0_06 JConsole is not picking up my local Java processes. I have set the -Dcom.sun.management.jmxremote as an arg to the JVM startup. My Windows username does not have an underscore in it and my default temporary directory is on NTFS. When I run the jps command I see my java processes but I get process information unavailable for each one except the jps command.
    Anybody have any ideas as to what else I should look at.

    Found the answer I was looking for - What a pain - http://forum.java.sun.com/thread.jspa?threadID=730397&messageID=4308804

  • SunOS 5.10 spawns multiple PID's for same java process

    On SunOS 5.10 I see same java process having multiple PID's. ps -ef lists out the same java process running from same path with multiple PID's. What can be the reason for this?
    Please let me know if you need some more information.....
    Edited by: sayanb on Jan 5, 2010 9:29 AM

    Hi Chris,
    Thank you for this update.
    Again this seems to be a known issue on SunOS 10.Also I will try to take the stack strace for the specific java process.
    http://forums.sun.com/thread.jspa?forumID=546&threadID=5297465
    http://groups.google.com/group/bojug/browse_thread/thread/19d722275c4384b8
    Basically the issue seems to be with one of the fork system calls which confuses with active and actual threads running.The above link speaks about the same , faced by similar users.
    Any updates will be appreciated.Meanwhile will try to get the stack trace.
    Regards
    Sayan

Maybe you are looking for

  • Changing domain users desktop wallpaper periodically

    Hi  I know we can set a wallpaper for users from gpedit> Administrative Templates> Desktop.  but is there any way to do this automatically each day or week ? Tanx

  • Multi-layered buttons used psd

    Newb here... I have a multi-layed psd file that I would like to use as button. Simply says Lesson 1. One layer is yellow, then white, then red for normal, active, and selected. What do I do to make each layer show as a button as I go from normal (yel

  • Large mp3 collection painfully slow to open in Mus...

    I know my mp3 collection is relatively large (full 16gb class 6 micro sd card) so I expect a bit of lag but im finding access to my mp3's on my E72 frustratingly slow.  For example opening my list of Artists isn't so bad but accessing an individual a

  • Image transfer from Nikon D810 to Lightroom fail

    I have just bought a Nikon D810 and tried to download images directly from the camera to Lightroom 4.4 by using "Copy as DNG" and "Copy". Both failed and no images were imported. I have tried it with a card reader (hama) as well. Same result - zero i

  • Import ECC or BW index to the EP (IVIEW)

    Hello, So far I have made a Index for a data source to look for documents, and within BW I have created a Business Object, added a search class an created and Index to allo TREX to look for that information. Is it possible for the EP to look for inde