Dispatcher Died after not able to reach message server

Question

Hi Gurus,
My Dispatcher is not able to connect to MS. Please note I am using windows/oracle. After som time disp+work died. Here I am posting latest dev_disp log. Please suggest.
regards-JASWANT
trc file: "dev_disp", trc level: 1, release: "700"
sysno      00
sid        EC5
systemid   560 (PC with Windows NT)
relno      7000
patchlevel 0
patchno    144
intno      20050900
make:      multithreaded, Unicode, optimized
pid        3096
Mon Apr 27 11:01:58 2009
kernel runs with dp version 232000(ext=109000) (@(#) DPLIB-INT-VERSION-232000-UC)
length of sys_adm_ext is 576 bytes
SWITCH TRC-HIDE on ***
***LOG Q00=> DpSapEnvInit, DPStart (00 3096) [dpxxdisp.c   1243]
     shared lib "dw_xml.dll" version 144 successfully loaded
     shared lib "dw_xtc.dll" version 144 successfully loaded
     shared lib "dw_stl.dll" version 144 successfully loaded
     shared lib "dw_gui.dll" version 144 successfully loaded
     shared lib "dw_mdm.dll" version 144 successfully loaded
rdisp/softcancel_sequence :  -> 0,5,-1
Mon Apr 27 11:02:03 2009
WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 4 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and adjust the DNS settings [dpxxtool2.c  5371]
MtxInit: 30000 0 0
DpSysAdmExtInit: ABAP is active
DpSysAdmExtInit: VMC (JAVA VM in WP) is not active
DpIPCInit2: start server >hpnwecc_EC5_00                          <
DpShMCreate: sizeof(wp_adm)          18672     (1436)
DpShMCreate: sizeof(tm_adm)          4270848     (21248)
DpShMCreate: sizeof(wp_ca_adm)          24000     (80)
DpShMCreate: sizeof(appc_ca_adm)     8000     (80)
DpCommTableSize: max/headSize/ftSize/tableSize=500/8/528056/528064
DpShMCreate: sizeof(comm_adm)          528064     (1048)
DpSlockTableSize: max/headSize/ftSize/fiSize/tableSize=0/0/0/0/0
DpShMCreate: sizeof(slock_adm)          0     (96)
DpFileTableSize: max/headSize/ftSize/tableSize=0/0/0/0
DpShMCreate: sizeof(file_adm)          0     (72)
DpShMCreate: sizeof(vmc_adm)          0     (1544)
DpShMCreate: sizeof(wall_adm)          (38456/34360/64/184)
DpShMCreate: sizeof(gw_adm)     48
DpShMCreate: SHM_DP_ADM_KEY          (addr: 05970040, size: 4930904)
DpShMCreate: allocated sys_adm at 05970040
DpShMCreate: allocated wp_adm at 05972090
DpShMCreate: allocated tm_adm_list at 05976980
DpShMCreate: allocated tm_adm at 059769B0
DpShMCreate: allocated wp_ca_adm at 05D894B0
DpShMCreate: allocated appc_ca_adm at 05D8F270
DpShMCreate: allocated comm_adm at 05D911B0
DpShMCreate: system runs without slock table
DpShMCreate: system runs without file table
DpShMCreate: allocated vmc_adm_list at 05E12070
DpShMCreate: allocated gw_adm at 05E120B0
DpShMCreate: system runs without vmc_adm
DpShMCreate: allocated ca_info at 05E120E0
DpShMCreate: allocated wall_adm at 05E120E8
MBUF state OFF
DpCommInitTable: init table for 500 entries
Mon Apr 27 11:02:04 2009
ThTaskStatus: rdisp/reset_online_during_debug 0
EmInit: MmSetImplementation( 2 ).
MM global diagnostic options set: 0
<ES> client 0 initializing ....
<ES> InitFreeList
<ES> block size is 1024 kByte.
Using implementation view
<EsNT> Using memory model view.
<EsNT> Memory Reset disabled as NT default
<ES> 755 blocks reserved for free list.
ES initialized.
J2EE server info
  start = TRUE
  state = STARTED
  pid = 5940
  argv[0] = E:\usr\sap\EC5\DVEBMGS00\exe\jcontrol.EXE
  argv[1] = E:\usr\sap\EC5\DVEBMGS00\exe\jcontrol.EXE
  argv[2] = pf=E:\usr\sap\EC5\SYS\profile\EC5_DVEBMGS00_hpnwecc
  argv[3] = -DSAPSTART=1
  argv[4] = -DCONNECT_PORT=65000
  argv[5] = -DSAPSYSTEM=00
  argv[6] = -DSAPSYSTEMNAME=EC5
  argv[7] = -DSAPMYNAME=hpnwecc_EC5_00
  argv[8] = -DSAPPROFILE=E:\usr\sap\EC5\SYS\profile\EC5_DVEBMGS00_hpnwecc
  argv[9] = -DFRFC_FALLBACK=ON
  argv[10] = -DFRFC_FALLBACK_HOST=localhost
  start_lazy = 0
  start_control = SAP J2EE startup framework
Continued......

Answer 1

.....continued
DpJ2eeStart: j2ee state = STARTED
rdisp/http_min_wait_dia_wp : 1 -> 1
***LOG CPS=> DpLoopInit, ICU ( 3.0 3.0 4.0.1) [dpxxdisp.c   1633]
***LOG Q0F=> DpLoopInit, nomstry () [dpxxdisp.c   1720]
ERROR => Connection to Message Server failed - check installation [dpxxdisp.c   1721]
CCMS: AlInitGlobals : alert/use_sema_lock = TRUE.
CCMS: start to initalize 3.X shared alert area (first segment).
DpJ2eeLogin: j2ee state = CONNECTED
Mon Apr 27 11:02:07 2009
***LOG Q0I=> NiIRead: recv (10054: WSAECONNRESET: Connection reset by peer) [nixxi.cpp 4424]
ERROR => NiIRead: SiRecv failed for hdl 3 / sock 1500
    (SI_ECONN_BROKEN/10054; I4; ST; 127.0.0.1:3319) [nixxi.cpp    4424]
DpJ2eeMsgProcess: j2ee state = CONNECTED (NIECONN_BROKEN)
DpIJ2eeShutdown: send SIGINT to SAP J2EE startup framework (pid=5940)
ERROR => DpProcKill: kill failed [dpntdisp.c   371]
DpIJ2eeShutdown: j2ee state = SHUTDOWN
Mon Apr 27 11:02:44 2009
ERROR => DpHdlDeadWp: W0 (pid 5680) died [dpxxdisp.c   14532]
ERROR => DpHdlDeadWp: W1 (pid 4888) died [dpxxdisp.c   14532]
ERROR => DpHdlDeadWp: W2 (pid 4208) died [dpxxdisp.c   14532]
ERROR => DpHdlDeadWp: W3 (pid 444) died [dpxxdisp.c   14532]
ERROR => DpHdlDeadWp: W4 (pid 5312) died [dpxxdisp.c   14532]
ERROR => DpHdlDeadWp: W5 (pid 4880) died [dpxxdisp.c   14532]
my types changed after wp death/restart 0xbf --> 0xbe
ERROR => MsIModTypes: not_attached [msxxi.c      1834]
ERROR => DpHdlDeadWp: W6 (pid 6120) died [dpxxdisp.c   14532]
my types changed after wp death/restart 0xbe --> 0xbc
ERROR => MsIModTypes: not_attached [msxxi.c      1834]
ERROR => DpHdlDeadWp: W7 (pid 2604) died [dpxxdisp.c   14532]
my types changed after wp death/restart 0xbc --> 0xb8
ERROR => MsIModTypes: not_attached [msxxi.c      1834]
ERROR => DpHdlDeadWp: W8 (pid 4688) died [dpxxdisp.c   14532]
ERROR => DpHdlDeadWp: W9 (pid 2636) died [dpxxdisp.c   14532]
ERROR => DpHdlDeadWp: W10 (pid 6012) died [dpxxdisp.c   14532]
my types changed after wp death/restart 0xb8 --> 0xb0
ERROR => MsIModTypes: not_attached [msxxi.c      1834]
ERROR => DpHdlDeadWp: W11 (pid 2428) died [dpxxdisp.c   14532]
my types changed after wp death/restart 0xb0 --> 0xa0
ERROR => MsIModTypes: not_attached [msxxi.c      1834]
ERROR => DpHdlDeadWp: W12 (pid 5256) died [dpxxdisp.c   14532]
my types changed after wp death/restart 0xa0 --> 0x80
ERROR => MsIModTypes: not_attached [msxxi.c      1834]
DP_FATAL_ERROR => DpWPCheck: no more work processes
DISPATCHER EMERGENCY SHUTDOWN ***
increase tracelevel of WPs
NiWait: sleep (10000ms) ...
NiISelect: timeout 10000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:02:54 2009
NiISelect: TIMEOUT occured (10000ms)
dump system status
Workprocess Table (long)               Mon Apr 27 05:32:54 2009
========================
No Ty. Pid      Status  Cause Start Err Sem CPU    Time  Program          Cl  User         Action                    Table
0 DIA     5680 Ended         no      1   0        0                                                                         
1 DIA     4888 Ended         no      1   0        0                                                                         
2 DIA     4208 Ended         no      1   0        0                                                                         
3 DIA      444 Ended         no      1   0        0                                                                         
4 DIA     5312 Ended         no      1   0        0                                                                         
5 DIA     4880 Ended         no      1   0        0                                                                         
6 UPD     6120 Ended         no      1   0        0                                                                         
7 ENQ     2604 Ended         no      1   0        0                                                                         
8 BTC     4688 Ended         no      1   0        0                                                                         
9 BTC     2636 Ended         no      1   0        0                                                                         
10 BTC     6012 Ended         no      1   0        0                                                                         
11 SPO     2428 Ended         no      1   0        0                                                                         
12 UP2     5256 Ended         no      1   0        0                                                                         
Dispatcher Queue Statistics               Mon Apr 27 05:32:54 2009
===========================
--------++++--
+
Typ
now
high
max
writes
reads
--------++++--
+
NOWP
0
2
2000
6
6
--------++++--
+
DIA
5
5
2000
5
0
--------++++--
+
UPD
0
0
2000
0
0
--------++++--
+
ENQ
0
0
2000
0
0
--------++++--
+
BTC
0
0
2000
0
0
--------++++--
+
SPO
0
0
2000
0
0
--------++++--
+
UP2
0
0
2000
0
0
--------++++--
+
max_rq_id          9
wake_evt_udp_now     0
wake events           total     7,  udp     6 ( 85%),  shm     1 ( 14%)
since last update     total     7,  udp     6 ( 85%),  shm     1 ( 14%)
Dump of tm_adm structure:               Mon Apr 27 05:32:54 2009
=========================
Term    uid  man user    term   lastop  mod wp  ta   a/i (modes)
RM-T13, U14,                 , tr-12-05, 11:02:30, M0, W-1,     , 1/0
RM-T14, U15,                 , tr-12-08, 11:02:31, M0, W-1,     , 1/0
Workprocess Comm. Area Blocks               Mon Apr 27 05:32:54 2009
=============================
Slots: 300, Used: 2, Max: 1
--------++--
+
id
owner
pid
eyecatcher
--------++--
+
0
DISPATCHER
-1
WPCAAD000
1
DISPATCHER
-1
WPCAAD001
NiWait: sleep (5000ms) ...
NiISelect: timeout 5000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:02:59 2009
NiISelect: TIMEOUT occured (5000ms)
DpHalt: shutdown server >hpnwecc_EC5_00                          < (normal)
DpJ2eeDisableRestart
Switch off Shared memory profiling
ShmProtect( 57, 3 )
ShmProtect(SHM_PROFILE, SHM_PROT_RW
ShmProtect( 57, 1 )
ShmProtect(SHM_PROFILE, SHM_PROT_RD
DpWakeUpWps: wake up all wp's
Stop work processes
Stop gateway
killing process (4884) (SOFT_KILL)
Stop icman
killing process (4708) (SOFT_KILL)
Terminate gui connections
send SHUTDOWN to REM TM 13
blks_in_queue/wp_ca_blk_no/wp_max_no = 2/300/13
LOCK WP ca_blk 2
return errno (-17) to T13
errormsg without overhead: take mode 0
NiBufIAlloc: malloc NiBufadm, to 0 bytes
try to send 14 to M0
NiBufSend starting
NiIWrite: hdl 3 sent data (wrt=14,pac=1,MESG_IO)
REL WP ca_blk 2
set status of T13 to TM_DISCONNECTED
NiBufISelUpdate: new MODE -- (r-) for hdl 3 in set0
SiSelNSet: set events of sock 1500 to: ---
NiBufISelRemove: remove hdl 3 from set0
SiSelNRemove: removed sock 1500 (pos=2)
SiSelNRemove: removed sock 1500
NiSelIRemove: removed hdl 3
DpDelSocketInfo: del info for socket 3 (type=3)
NiICloseHandle: shutdown and close hdl 3 / sock 1500
NiBufIClose: clear extension for hdl 3
dp_tm_adm[13].stat = DP_SLOT_FREE
DpGetSchedule: next schedule 1240810384/282000
DpGetSchedule: no schedule found
DpITmSlotRelease: release slot 13
DpListInsert: insert elem 13 into tmadm_free_list (at begin)
DpListInsert: 187 elems in tmadm_free_list
DpListRemove: remove elem 13 from tmadm_inuse_list
DpListRemove: 14 elems in tmadm_inuse_list
send SHUTDOWN to REM TM 14
blks_in_queue/wp_ca_blk_no/wp_max_no = 2/300/13
LOCK WP ca_blk 2
return errno (-17) to T14
errormsg without overhead: take mode 0
NiBufIAlloc: malloc NiBufadm, to 0 bytes
try to send 14 to M0
NiBufSend starting
NiIWrite: hdl 5 sent data (wrt=14,pac=1,MESG_IO)
REL WP ca_blk 2
set status of T14 to TM_DISCONNECTED
NiBufISelUpdate: new MODE -- (r-) for hdl 5 in set0
SiSelNSet: set events of sock 1452 to: ---
NiBufISelRemove: remove hdl 5 from set0
SiSelNRemove: removed sock 1452
NiSelIRemove: removed hdl 5
DpDelSocketInfo: del info for socket 5 (type=3)
NiICloseHandle: shutdown and close hdl 5 / sock 1452
NiBufIClose: clear extension for hdl 5
dp_tm_adm[14].stat = DP_SLOT_FREE
DpGetSchedule: next schedule 1240810384/282000
DpGetSchedule: no schedule found
DpITmSlotRelease: release slot 14
DpListInsert: insert elem 14 into tmadm_free_list (at begin)
DpListInsert: 188 elems in tmadm_free_list
DpListRemove: remove elem 14 from tmadm_inuse_list
DpListRemove: 13 elems in tmadm_inuse_list
wait for end of work processes
wait for end of gateway
[DpProcDied] Process lives  (PID:4884  HANDLE:1568)
waiting for termination of gateway ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:03:00 2009
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process died  (PID:4884  HANDLE:1568)
wait for end of icman
[DpProcDied] Process lives  (PID:4708  HANDLE:1572)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:03:01 2009
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives  (PID:4708  HANDLE:1572)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:03:02 2009
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives  (PID:4708  HANDLE:1572)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:03:03 2009
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives  (PID:4708  HANDLE:1572)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:03:04 2009
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives  (PID:4708  HANDLE:1572)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:03:05 2009
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process lives  (PID:4708  HANDLE:1572)
waiting for termination of icman ...
NiWait: sleep (1000ms) ...
NiISelect: timeout 1000ms
NiISelect: maximum fd=1597
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Mon Apr 27 11:03:06 2009
NiISelect: TIMEOUT occured (1000ms)
[DpProcDied] Process died  (PID:4708  HANDLE:1572)
[DpProcDied] Process died  (PID:5940  HANDLE:1552)
not attached to the message server
cleanup EM
EsCleanup ....
EmCleanup() -> 0
Es2Cleanup: Cleanup ES2
***LOG Q05=> DpHalt, DPStop ( 3096) [dpxxdisp.c   10421]
Good Bye .....