Dispatcher died and all WP being occupied by RFCUSER

Question

Hi Team,
Dispatcher went down and unable to start the SAP MMC.
I found that all work processes are being occupied by two porgrams
SAPLSOPX and SAPLIRFC by RFCUSER due to that none of the user's can't able to login into sap.
when i go to SM 37 and see below jobs are in active state and i have tried to delete the old spool jobs as the program SAPLSOPX called by RFC user seems to spool process.
QRFC:C0A8010411E049AC7BE502EE
QRFC:C0A8010411E049AC7C080321
QRFC:C0A8010411E049AC7C100333
Inspite of killing the work processes they are immediately occupied by these jobs. Can some please suggest some thing on priority since my CRM production server is down.
Pleae see the enclosed developer traces and much appriciated for any suggessions.
trc file: "dev_w2", trc level: 1, release: "640"
*S Tue Mar 03 08:45:19 2009
S  *** init spool environment
S  initialize debug system
T  Stack direction is downwards.
T  debug control: prepare exclude for printer trace
T  new memory block 20E3A200
S  spool kernel/ddic check: Ok
S  using table TSP02FX for frontend printing
S  1 spool work process(es) found
S  frontend print via spool service enabled
S  printer list size is 150
S  printer type list size is 50
S  queue size (profile)   = 300
S  hostspool list size = 3000
S  option list size is 30
S      found processing queue enabled
S  found spool memory service RSPO-RCLOCKS at 28D00098
S  doing lock recovery
S  setting server cache root
S  found spool memory service RSPO-SERVERCACHE at 28D00680
S    using messages for server info
S  size of spec char cache entry: 165020 bytes (timeout 100 sec)
S  size of open spool request entry: 1152 bytes
S  immediate print option for implicitely closed spool requests is disabled
A 
A  -PXA--
A  PXA INITIALIZATION
A  PXA: Fragment Size too small: 170 MB, reducing # of fragments
A  System page size: 4kb, total admin_size: 11748kb, dir_size: 11688kb.
A  Attached to PXA (address 3C550040, size 350000K)
A  abap/pxa = shared protect gen_remote
A  PXA INITIALIZATION FINISHED
A  -PXA--
A 
A  ABAP ShmAdm attached (addr=687F5000 leng=20955136 end=69BF1000)
A  >> Shm MMADM area (addr=68B5D300 leng=134720 end=68B7E140)
A  >> Shm MMDAT area (addr=68B7F000 leng=17244160 end=69BF1000)
A  RFC rfc/signon_error_log = -1
A  RFC rfc/dump_connection_info = 0
A  RFC rfc/dump_client_info = 0
A  RFC rfc/cp_convert/ignore_error = 1
A  RFC rfc/cp_convert/conversion_char = 23
A  RFC rfc/wan_compress/threshold = 251
A  RFC rfc/recorder_pcs not set, use defaule value: 1
A  RFC rfc/delta_trc_level not set, use default value: 0
A  RFC rfc/no_uuid_check not set, use default value: 0
A  RFC abap/SIGCHILD_default not set, use default value: 0
A  RFC rfc/bc_ignore_thcmaccp_retcode not set, use default value: 0
A  RFC Method> initialize RemObjDriver for ABAP Objects
A  Hotpackage version: 59
M  ThrCreateShObjects          allocated 27108 bytes at 0FF40040
N  SsfSapSecin: putenv(SECUDIR=E:\usr\sap\PCR\DVEBMGS00\sec): ok
N 
Tue Mar 03 08:50:22 2009
S  **** ERROR => rspogus__get_uniq_spoolnumber: Spool voll? [rspospno.c   351]*
*S  *** ERROR => error 1 set in spool C call [rspoabap.c   485]*
*S  *** ERROR => error message 'spool overflow' in spool C call [rspoabap.c*   714]
M 
M Tue Mar 03 08:50:56 2009
M  ThNoLockEntry: tried to lock Mutex 11 times
M 
M Tue Mar 03 08:54:26 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 9
M 
M Tue Mar 03 08:56:27 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 8
M 
M Tue Mar 03 08:58:29 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 7
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 9
M 
M Tue Mar 03 09:04:30 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 8
M 
M Tue Mar 03 09:06:30 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 7
M 
M Tue Mar 03 09:08:32 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 6
C 
C Tue Mar 03 09:10:28 2009
C  SQLBREAK: DBSL_CMD_SQLBREAK: CbOnCancel was not set. rc: 1
M 
M Tue Mar 03 09:10:33 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 5
M 
M Tue Mar 03 09:14:26 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 9
M 
M Tue Mar 03 09:16:27 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 8
M 
M Tue Mar 03 09:18:28 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 7
M 
M Tue Mar 03 09:20:29 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 6
C 
C Tue Mar 03 09:22:26 2009
C  SQLBREAK: DBSL_CMD_SQLBREAK: CbOnCancel was not set. rc: 1
M 
M Tue Mar 03 09:22:30 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 5
M Tue Mar 03 09:28:26 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
M Tue Mar 03 09:41:59 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 8
M 
M Tue Mar 03 09:44:01 2009
M  *** ERROR => ThNoLockEntry: timeout locking buffer entry for [thxxnum.c    5655]
M     cli/obj/subobj/range = 000/SPO_NUM   /      /01
M  *** ERROR => ThINoGet: failed to lock interval (rc=21) [thxxnum.c    3063]
M  return number range rc 14
M  *** WARNING => ThNoSpool: get from object (cli/obj/subobj/range = 000/SPO_NUM   /      /01) returned rc 14
S  ThNoSpool failed -> retries left 7
M 
S Tue Mar 03 09:45:50 2009
S  *** ERROR => rspogus__get_uniq_spoolnumber: Spool voll? [rspospno.c   351]
S  *** ERROR => error 1 set in spool C call [rspoabap.c   485]
S  *** ERROR => error message 'spool overflow' in spool C call [rspoabap.c   714]
Thanks & Regards,
Vinod

Answer 1

Hi Subhash,
Since all work processes are busy with report SAPLSOPX by RFC user and couldn't login into sap.
Is there any way that we can clear outstanding spool request at OS level ? and we could see even all ground processes are stopped with SAPLIRFC program.
Any idea what this program does SAPLSOPX and SAPLIRFC ?? do you think it would be better if we lock the user "RFCUSER" time being until we fix this issue?
Inspite of killing the work processes they are immediately occupied by these jobs...
Many Thanks,
Vinod