Ehp5 Implementation - System fails to start at MAIN_NEWBAS/STARTSAP_NBAS

Question

Hello,
I'm running an Ehp5 implementation from ERP 6.0. Ehpi is stuck in step Downtime during phase MAIN_NEWBAS/STARTSAP_NBAS with the following error:
Last error code set: RFC call to subst_get_uvers failed with key RFC_ERROR_COMMUNICATION (open): SAP_CMINIT3 : rc=20 > Connect to SAP gateway failed
Connect_PM GWHOST=sapapt01, GWSERV=sapgw00, SYSNR=00 
LOCATION CPIC (TCP/IP) on local host with Unicode
ERROR partner '10.10.1.21:3300' not reached
TIME Mon Jan 16 23:03:11 2012
RELEASE 720
COMPONENT NI (network interface)
VERSION 40 RC -10
MODULE nixxi.cpp LINE 3286
DETAIL NiPConnect2: 10.10.1.21:3300
SYSTEM CALL connect
ERRNO 111
ERRNO TEXT Connection refused
COUNTER 226 
System start failed
I checked the system instance and it was down. When I try to start up it manually with startsap, I do not get any error:
sapapt01:ippadm 16> startsap
Checking IPP Database
Database is running
Starting Startup Agent sapstartsrv
OK
Instance Service on host sapapt01 started
starting SAP Instance DVEBMGS00
Startup-Log is written to /home/ippadm/startsap_DVEBMGS00.log
/usr/sap/IPP/SYS/exe/run/sapcontrol -prot NI_HTTP -nr 00 -function Start
Instance on host sapapt01 started.
The problem is that no work process is started nor dispatcher or message server. When I look at the instance work directory I see that the only files that are written are the following:
available.log
sapstartsrv.log
logsave.bin
dev_tp (but it's empty)
The content of sapstartsrv.log is the following:
trc file: "sapstartsrv.log", trc level: 0, release: "720"
pid        29149
[Thr 47879647924784] Tue Jan 17 00:51:54 2012
No halib defined => HA support disabled
CCMS agent initialization: return code 0.
CCMS agent start: return code 0.
Initializing SAPControl Webservice
Starting AutoRestart thread
SapSSLInit failed => https support disabled
[Thr 1079777600] Tue Jan 17 00:51:56 2012
AutoRestart thread started
Starting WebService thread
Webservice thread started, listening on port 50013
Trusted http connect via Unix domain socket '/tmp/.sapstream50013' enabled.
trusted unix domain socket user is starting SAP System at 2012/01/17  0:52:04
The startsap_DVEBMGS00.log at the sidadm home directory shows the following:
Trace of system startup/check of SAP System IPP on Tue Jan 17 00:51:54 ART 2012
Called command: startsap start
00:51:54
Starting Startup Agent sapstartsrv
00:52:04
Instance Service on host sapapt01 started
00:52:04
starting SAP Instance DVEBMGS00
00:52:04
Startup-Log is written to /home/ippadm/startsap_DVEBMGS00.log
00:52:04
/usr/sap/IPP/SYS/exe/run/sapcontrol -prot NI_HTTP -nr 00 -function Start
17.01.2012 00:52:05
Start
OK
00:52:15
Instance on host sapapt01 started
The /var/log/messages content is the following:
Jan 17 00:51:56 sapapt01 SAPIPP_00[29149]: SAP Service SAPIPP_00 successfully started.
I can't find any clue why the system processes are not starting. The phase is the next after kernel upgrade from 700 to 720, so I believe it is related to the switch and maybe my kernel is not correctly updated or something is missing.
Please let me know any suggestion to know why the system is not starting.
Regards,
Alejandro

Answer 1

Hi,
No object was removed with the cleanipc command, anyway I've rebooted the server and then started db and sap again. Now the dev traces are written.
Output of the cleanipc command:
sapapt01:ippadm 42> ./cleanipc 00
Show/Cleanup SAP-IPC-Objects V2.3, 94/01/20
===========================================
+----+
| To inhibit destruction of valid SAP-IPC-Objects |
| during shutdown, this program requires now      |
| special options for removing IPC-Objects.       |
+----+
Running SAP-Systems (Nr)...:
---- Show IPC-Objects of Sap-System  0 -
Number of IPC-Objects...........:    0
Summary of all Shared Memory....:     8802.0 MB (may be incomplete when not in superuser mode)
Number of SAP_ES files found:.............:    0
Number of SAP_ES files removed:...........:    0
dev_w0 trace file:
trc file: "dev_w0", trc level: 1, release: "700"
ACTIVE TRACE LEVEL           1
ACTIVE TRACE COMPONENTS      all, MJ
M sysno      00
M sid        IPP
M systemid   390 (AMD/Intel x86_64 with Linux)
M relno      7000
M patchlevel 0
M patchno    185
M intno      20050900
M make:      single threaded, Unicode, 64 bit, optimized
M pid        29183
M
M
M Tue Jan 17 03:23:58 2012
M  kernel runs with dp version 241000(ext=110000) (@(#) DPLIB-INT-VERSION-241000
-UC)
M  length of sys_adm_ext is 576 bytes
M  ***LOG Q01=> ThInit, WPStart (Workproc 0 1 29183) [thxxhead.c   1290]
M  ThInit: running on host sapapt01
M  calling db_connect ...
B  create_con (con_name=R/3)
B  Loading DB library '/usr/sap/IPP/SYS/exe/run/dboraslib.so' ...
B  Library '/usr/sap/IPP/SYS/exe/run/dboraslib.so' loaded
*B  *** ERROR => Version mismatch between kernel (700.08) and DBSL shared library*
M  ***LOG R19=> ThInit, db_connect ( DB-Connect 008192) [thxxhead.c   1449]
M  in_ThErrHandle: 1
M  *** ERROR => ThInit: db_connect (step 1, th_errno 13, action 3, level 1) [thx
xhead.c   10524]
M
M  Info for wp 0
M
M    pid = 29183
M    severity = 0
M    status = 0
M    stat = WP_RUN
M    waiting_for = NO_WAITING
M    reqtype = DP_RQ_DIAWP
M    act_reqtype = NO_REQTYPE
M    rq_info = 0
M    tid = -1
M    mode = 255
M    len = -1
M    rq_id = 65535
M    rq_source =
M    last_tid = 0
M    last_mode = 0
M    semaphore = 0
M    act_cs_count = 0
M    csTrack = 0
M    csTrackRwExcl = 0
M    csTrackRwShrd = 0
M    mode_cleaned_counter = 0
The kernel version seems to be 700 here, I don't know why because when I check the disp+work -version  is 720.
Regards,
AN