Dispatcher won't stay up. (disp+work.exe)

Question

I have an SAP BW 3.5 system on WIndows 2003 with an Oracle database.
It was working fine for some time.  Now the system won't come online.  When I start the system everything goes green temporarily and then shuts down as the Dispatcher stops and won't come back.
I can't figure out how to fix this.  I have a trace file from the dev_disp log that I will paste below.  Would really appreciate someone with some basis experience to walk me through how to fix this.  Your help is greatly appreciated.  Thank you in advance.
-start of log--
trc file: "dev_disp", trc level: 1, release: "640"
Thu Jan 18 10:26:14 2007
kernel runs with dp version 128(ext=102) (@(#) DPLIB-INT-VERSION-128)
length of sys_adm_ext is 312 bytes
systemid   560 (PC with Windows NT)
relno      6400
patchlevel 0
patchno    21
intno      20020600
make:      multithreaded, ASCII
pid        3816
***LOG Q00=> DpSapEnvInit, DPStart (02 3816) [dpxxdisp.c   1100]
     shared lib "dw_xml.dll" version 21 successfully loaded
     shared lib "dw_xtc.dll" version 21 successfully loaded
     shared lib "dw_stl.dll" version 21 successfully loaded
     shared lib "dw_gui.dll" version 21 successfully loaded
Thu Jan 18 10:26:18 2007
WARNING => DpNetCheck: NiAddrToHost(1.0.0.0) took 4 seconds
***LOG GZZ=> 1 possible network problems detected - check tracefile and
adjust the DNS settings [dpxxtool2.c  3886]
MtxInit: -2 0 0
DpSysAdmExtInit: ABAP is active
DpSysAdmExtInit: JAVA is not active
DpShMCreate: sizeof(wp_adm)          6624     (828)
DpShMCreate: sizeof(tm_adm)          2219848     (11044)
DpShMCreate: sizeof(wp_ca_adm)          18000     (60)
DpShMCreate: sizeof(appc_ca_adm)     6000     (60)
DpShMCreate: sizeof(comm_adm)          192000     (384)
DpShMCreate: sizeof(vmc_adm)          0     (320)
DpShMCreate: sizeof(wall_adm)          (22440/34344/56/100)
DpShMCreate: SHM_DP_ADM_KEY          (addr: 04960040, size: 2505416)
DpShMCreate: allocated sys_adm at 04960040
DpShMCreate: allocated wp_adm at 049617A8
DpShMCreate: allocated tm_adm_list at 04963188
DpShMCreate: allocated tm_adm at 049631B0
DpShMCreate: allocated wp_ca_adm at 04B810F8
DpShMCreate: allocated appc_ca_adm at 04B85748
DpShMCreate: allocated comm_adm_list at 04B86EB8
DpShMCreate: allocated comm_adm at 04B86ED0
DpShMCreate: allocated vmc_adm_list at 04BB5CD0
DpShMCreate: system runs without vmc_adm
DpShMCreate: allocated ca_info at 04BB5CF8
DpShMCreate: allocated wall_adm at 04BB5D00
MBUF state OFF
Thu Jan 18 10:26:19 2007
EmInit: MmSetImplementation( 2 ).
<ES> client 0 initializing ....
<ES> InitFreeList
<ES> block size is 1024 kByte.
Using implementation std
<EsNT> Memory Reset enabled as NT default
<EsNT> EsIUnamFileMapInit: Initialize the memory 500 MB
<ES> 499 blocks reserved for free list.
ES initialized.
rdisp/http_min_wait_dia_wp : 1 -> 1
***LOG Q0K=> DpMsAttach, mscon ( SAPSERV-2) [dpxxdisp.c   9719]
CCMS: Initalizing shared memory of size 40000000 for monitoring segment.
Thu Jan 18 10:26:20 2007
CCMS: start to initalize 3.X shared alert area (first segment).
DpMsgAdmin: Set release to 6400, patchlevel 0
MBUF state PREPARED
MBUF component UP
DpMBufHwIdSet: set Hardware-ID
***LOG Q1C=> DpMBufHwIdSet [dpxxmbuf.c   1025]
DpMsgAdmin: Set patchno for this platform to 21
Release check o.K.
Thu Jan 18 10:26:59 2007
ERROR => W0 (pid 2544) died [dpxxdisp.c   12170]
ERROR => W1 (pid 4040) died [dpxxdisp.c   12170]
my types changed after wp death/restart 0xbf --> 0xbe
ERROR => W2 (pid 3848) died [dpxxdisp.c   12170]
my types changed after wp death/restart 0xbe --> 0xbc
ERROR => W3 (pid 3308) died [dpxxdisp.c   12170]
my types changed after wp death/restart 0xbc --> 0xb8
ERROR => W4 (pid 3996) died [dpxxdisp.c   12170]
ERROR => W5 (pid 2964) died [dpxxdisp.c   12170]
my types changed after wp death/restart 0xb8 --> 0xb0
ERROR => W6 (pid 2700) died [dpxxdisp.c   12170]
my types changed after wp death/restart 0xb0 --> 0xa0
ERROR => W7 (pid 3260) died [dpxxdisp.c   12170]
my types changed after wp death/restart 0xa0 --> 0x80
DP_FATAL_ERROR => DpWPCheck: no more work processes
DISPATCHER EMERGENCY SHUTDOWN ***
increase tracelevel of WPs
killing W0-2544 (SIGUSR2)
ERROR => DpWpKill(2544, SIGUSR2) failed [dpxxtool.c   2468]
killing W1-4040 (SIGUSR2)
ERROR => DpWpKill(4040, SIGUSR2) failed [dpxxtool.c   2468]
killing W2-3848 (SIGUSR2)
ERROR => DpWpKill(3848, SIGUSR2) failed [dpxxtool.c   2468]
killing W3-3308 (SIGUSR2)
ERROR => DpWpKill(3308, SIGUSR2) failed [dpxxtool.c   2468]
killing W4-3996 (SIGUSR2)
ERROR => DpWpKill(3996, SIGUSR2) failed [dpxxtool.c   2468]
killing W5-2964 (SIGUSR2)
ERROR => DpWpKill(2964, SIGUSR2) failed [dpxxtool.c   2468]
killing W6-2700 (SIGUSR2)
ERROR => DpWpKill(2700, SIGUSR2) failed [dpxxtool.c   2468]
killing W7-3260 (SIGUSR2)
ERROR => DpWpKill(3260, SIGUSR2) failed [dpxxtool.c   2468]
NiWait: sleep (10000 msecs) ...
NiISelect: timeout 10000 ms
NiISelect: maximum fd=1637
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Thu Jan 18 10:27:09 2007
NiISelect: TIMEOUT occured (10000 ms)
dump system status
Workprocess Table (long)               Thu Jan 18 15:27:09 2007
========================
No Ty. Pid      Status  Cause Start Err Sem CPU    Time  Program  Cl  User  
       Action                    Table
0 DIA     2544 Ended         no      1   0             0
1 DIA     4040 Ended         no      1   0             0
2 UPD     3848 Ended         no      1   0             0
3 ENQ     3308 Ended         no      1   0             0
4 BTC     3996 Ended         no      1   0             0
5 BTC     2964 Ended         no      1   0             0
6 SPO     2700 Ended         no      1   0             0
7 UP2     3260 Ended         no      1   0             0
Dispatcher Queue Statistics               Thu Jan 18 15:27:09 2007
===========================
--------++++--
+
Typ
now
high
max
writes
reads
--------++++--
+
NOWP
0
3
2000
10
10
--------++++--
+
DIA
6
6
2000
6
0
--------++++--
+
UPD
0
0
2000
0
0
--------++++--
+
ENQ
0
0
2000
0
0
--------++++--
+
BTC
0
0
2000
0
0
--------++++--
+
SPO
0
0
2000
0
0
--------++++--
+
UP2
0
0
2000
0
0
--------++++--
+
max_rq_id          14
wake_evt_udp_now     0
wake events           total     9,  udp     8 ( 88%),  shm     1 ( 11%)
since last update     total     9,  udp     8 ( 88%),  shm     1 ( 11%)
Dump of tm_adm structure:               Thu Jan 18 15:27:09 2007
=========================
Term    uid  man user    term   lastop  mod wp  ta   a/i (modes)
RM-T8, U9,     , bigdaddy, 10:26:49, M0, W-1,     , 1/0
Workprocess Comm. Area Blocks               Thu Jan 18 15:27:09 2007
=============================
Slots: 300, Used: 2, Max: 1
--------++--
+
id
owner
pid
eyecatcher
--------++--
+
0
DISPATCHER
-1
WPCAAD000
1
DISPATCHER
-1
WPCAAD001
NiWait: sleep (5000 msecs) ...
NiISelect: timeout 5000 ms
NiISelect: maximum fd=1637
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Thu Jan 18 10:27:14 2007
NiISelect: TIMEOUT occured (5000 ms)
Shutdown server ...
DpModState: buffer in state MBUF_PREPARED
NiBufSend starting
NiIWrite: write 110, 1 packs, MESG_IO, hdl 3, data complete
MsINiWrite: sent 110 bytes
MsIModState: change state to SHUTDOWN
DpModState: change server state from STARTING to SHUTDOWN
Switch off Shared memory profiling
ShmProtect( 57, 3 )
ShmProtect(SHM_PROFILE, SHM_PROT_RW
ShmProtect( 57, 1 )
ShmProtect(SHM_PROFILE, SHM_PROT_RD
DpWakeUpWps: wake up all wp's
Stop work processes...
Stop gateway
killing process (2952) (SOFT_KILL)
Stop icman
killing process (3548) (SOFT_KILL)
Terminate gui connections
send SHUTDOWN to REM TM 8
blks_in_queue/wp_ca_blk_no/wp_max_no = 2/300/8
LOCK WP ca_blk 2
return errno (-17) to T8
errormsg without overhead: take mode 0
NiBufIAlloc: malloc NiBufadm, to 0 bytes
try to send 14 to M0
NiBufSend starting
NiIWrite: write 14, 1 packs, MESG_IO, hdl 7, data complete
REL WP ca_blk 2
set status of T8 to TM_DISCONNECTED
NiSelClear: removed hdl 7 from selectset
DpDelSocketInfo: del info for socket 7 (type=3)
NiBufClose: clear extensions for hdl 7
NiBufSetStat: bufstat of hdl 7 changed from OK to OFF
NiICloseHandle: shutdown and close hdl 7 / socket 1468
dp_tm_adm[8].stat = DP_SLOT_FREE
DpGetSchedule: next schedule 1169134039
DpGetSchedule: no schedule found
DpITmSlotRelease: release slot 8
DpListInsert: insert elem 8 into tmadm_free_list (at begin)
DpListInsert: 193 elems in tmadm_free_list
DpListRemove: remove elem 8 from tmadm_inuse_list
DpListRemove: 8 elems in tmadm_inuse_list
[DpProcDied] Process lives  (PID:2952  HANDLE:1616)
waiting for termination of gateway
NiWait: sleep (1000 msecs) ...
NiISelect: timeout 1000 ms
NiISelect: maximum fd=1637
NiISelect: read-mask is NULL
NiISelect: write-mask is NULL
Thu Jan 18 10:27:15 2007
NiISelect: TIMEOUT occured (1000 ms)
[DpProcDied] Process died  (PID:2952  HANDLE:1616)
[DpProcDied] Process died  (PID:3548  HANDLE:1608)
DpHalt: cancel all lcom connections
MPI CancelAll 2 -> 0
MPI DeleteAll 2 -> 0
NiIMyHostName: hostname = 'SAPSERV-2'
AdGetSelfIdentRecord: >                                                     
                      <
AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 4 (AD_STARTSTOP), ser 0, ex 0, errno 0
DpConvertRequest: net size = 163 bytes
NiBufSend starting
NiIWrite: write 562, 1 packs, MESG_IO, hdl 3, data complete
MsINiWrite: sent 562 bytes
send msg (len 110+452) to name          -, type 4, key -
detach from message server
***LOG Q0M=> DpMsDetach, ms_detach () [dpxxdisp.c   9945]
NiBufSend starting
NiIWrite: write 110, 1 packs, MESG_IO, hdl 3, data complete
MsINiWrite: sent 110 bytes
MsIDetach: send logout to msg_server
MsIDetach: call exit function
DpMsShutdownHook called
NiSelClear: removed hdl 3 from selectset
MBUF state OFF
AdGetSelfIdentRecord: >                                                     
                      <
AdCvtRecToExt: opcode 60 (AD_SELFIDENT), ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
AdCvtRecToExt: opcode 40 (AD_MSBUF), ser 0, ex 0, errno 0
blks_in_queue/wp_ca_blk_no/wp_max_no = 2/300/8
LOCK WP ca_blk 2
make DISP owner of wp_ca_blk 2
DpRqPutIntoQueue: put request into queue (reqtype 1, prio LOW, rq_id 20)
MBUF component DOWN
NiBufClose: clear extensions for hdl 3
NiBufSetStat: bufstat of hdl 3 changed from OK to OFF
NiICloseHandle: shutdown and close hdl 3 / socket 1620
MsIDetach: detach MS-system
EsCleanup ....
***LOG Q05=> DpHalt, DPStop ( 3816) [dpxxdisp.c   8478]
Good Bye .....

Answer 1

Hi Siddhesh,
Thanks for trying to help.  I have found the dev_w0 trace file and pasted it below:
start file ---
trc file: "dev_w0", trc level: 1, release: "640"
ACTIVE TRACE LEVEL           1
ACTIVE TRACE COMPONENTS      all, M
B
B Thu Jan 18 12:30:04 2007
B  create_con (con_name=R/3)
B  Loading DB library 'C:\usr\sap\B01\SYS\exe\run\dboraslib.dll' ...
B  Library 'C:\usr\sap\B01\SYS\exe\run\dboraslib.dll' loaded
B  Version of 'C:\usr\sap\B01\SYS\exe\run\dboraslib.dll' is "640.00",
patchlevel (0.19)
B  New connection 0 created
M systemid   560 (PC with Windows NT)
M relno      6400
M patchlevel 0
M patchno    21
M intno      20020600
M make:      multithreaded, ASCII
M pid        3724
M
M  ***LOG Q0Q=> tskh_init, WPStart (Workproc 0 3724) [dpxxdisp.c   1160]
I  MtxInit: -2 0 0
M  DpSysAdmExtCreate: ABAP is active
M  DpSysAdmExtCreate: JAVA is not active
M  DpShMCreate: sizeof(wp_adm)          6624     (828)
M  DpShMCreate: sizeof(tm_adm)          2219848     (11044)
M  DpShMCreate: sizeof(wp_ca_adm)          18000     (60)
M  DpShMCreate: sizeof(appc_ca_adm)     6000     (60)
M  DpShMCreate: sizeof(comm_adm)          192000     (384)
M  DpShMCreate: sizeof(vmc_adm)          0     (320)
M  DpShMCreate: sizeof(wall_adm)          (22440/34344/56/100)
M  DpShMCreate: SHM_DP_ADM_KEY          (addr: 05160040, size: 2505416)
M  DpShMCreate: allocated sys_adm at 05160040
M  DpShMCreate: allocated wp_adm at 051617A8
M  DpShMCreate: allocated tm_adm_list at 05163188
M  DpShMCreate: allocated tm_adm at 051631B0
M  DpShMCreate: allocated wp_ca_adm at 053810F8
M  DpShMCreate: allocated appc_ca_adm at 05385748
M  DpShMCreate: allocated comm_adm_list at 05386EB8
M  DpShMCreate: allocated comm_adm at 05386ED0
M  DpShMCreate: allocated vmc_adm_list at 053B5CD0
M  DpShMCreate: system runs without vmc_adm
M  DpShMCreate: allocated ca_info at 053B5CF8
M  DpShMCreate: allocated wall_adm at 053B5D00
X  EmInit: MmSetImplementation( 2 ).
X  <ES> client 0 initializing ....
X  Using implementation std
M  <EsNT> Memory Reset enabled as NT default
X  ES initialized.
M
M Thu Jan 18 12:30:05 2007
M  calling db_connect ...
C  Got ORACLE_HOME=E:\oracle\ora92 from environment
C  Client NLS settings: AMERICAN_AMERICA.WE8DEC
C  Logon as OPS$-user to get SAPB01's password
C  Connecting as /@S01 on connection 0 ...
C  Attaching to DB Server S01 (con_hdl=0,svchp=06E14A4C,svrhp=037C27C4)
C  Starting user session
(con_hdl=0,svchp=06E14A4C,srvhp=037C27C4,usrhp=037CAFA4)
C  *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033
[dboci.c      3721]
C  Detaching from DB Server (con_hdl=0,svchp=06E14A4C,srvhp=037C27C4)
C  *** ERROR => CONNECT failed with sql error '1033'
[dbsloci.c    9753]
C  Try to connect with default password
C  Connecting as SAPB01/<pwd>@S01 on connection 0 ...
C  Attaching to DB Server S01 (con_hdl=0,svchp=06E14A4C,svrhp=037C27C4)
C  Starting user session
(con_hdl=0,svchp=06E14A4C,srvhp=037C27C4,usrhp=037CAFA4)
C  *** ERROR => OCI-call 'OCISessionBegin' failed: rc = 1033
[dboci.c      3721]
C  Detaching from DB Server (con_hdl=0,svchp=06E14A4C,srvhp=037C27C4)
C  *** ERROR => CONNECT failed with sql error '1033'
[dbsloci.c    9753]
B  ***LOG BV3=> severe db error 1033      ; work process is stopped [dbsh#2
@ 1195] [dbsh    1195 ]
B  ***LOG BY2=> sql error 1033   performing CON [dblink#1 @ 419] [dblink 
0419 ]
B  ***LOG BY0=> ORA-01033: ORACLE initialization or shutdown in progress
[dblink#1 @ 419] [dblink  0419 ]
M  ***LOG R19=> tskh_init, db_connect ( DB-Connect 000256) [thxxhead.c  
1271]
M  in_ThErrHandle: 1
M  *** ERROR => tskh_init: db_connect (step 1, th_errno 13, action 3, level
1) [thxxhead.c   9379]
M
M  Info for wp 0
M
M    stat = 4
M    reqtype = 1
M    act_reqtype = -1
M    rq_info = 0
M    tid = -1
M    mode = 255
M    len = -1
M    rq_id = 65535
M    rq_source = 255
M    last_tid = 0
M    last_mode = 0
M    int_checked_resource(RFC) = 0
M    ext_checked_resource(RFC) = 0
M    int_checked_resource(HTTP) = 0
M    ext_checked_resource(HTTP) = 0
M    report = >                                        <
M    action = 0
M    tab_name = >                              <
M
M 
M  *
M  *  LOCATION    SAP-Server SAPSERV-2_B01_02 on host SAPSERV-2 (wp 0)
M  *  ERROR       tskh_init: db_connect
M  *
M  *  TIME        Thu Jan 18 12:30:05 2007
M  *  RELEASE     640
M  *  COMPONENT   Taskhandler
M  *  VERSION     1
M  *  RC          13
M  *  MODULE      thxxhead.c
M  *  LINE        9555
M  *  COUNTER     1
M  *
M 
M
M  PfStatDisconnect: disconnect statistics
M  Entering TH_CALLHOOKS
M  ThCallHooks: call hook >ThrSaveSPAFields< for event BEFORE_DUMP
M  *** ERROR => ThrSaveSPAFields: no valid thr_wpadm [thxxrun1.c   730]
M  *** ERROR => ThCallHooks: event handler ThrSaveSPAFields for event
BEFORE_DUMP failed [thxxtool3.c  246]
M  Entering ThSetStatError
M  Entering ThReadDetachMode
M  call ThrShutDown (1)...
M  ***LOG Q02=> wp_halt, WPStop (Workproc 0 3724) [dpnttool.c   357]