Very unstable application server on Linux : work process are dying

Question

Hello,
SAP release : SAP R/3 Enterprise
Linux : x86_64 x86_64 x86_64 GNU/Linux
Oracle : 9.2.0.7.0
Our production environnement consists of one Central Instance server (HP-UXia 64) and five application servers (Linux x86_64). All the application servers are exatly the same, same hardware, same OS, CPU, RAM ...etc
But only applications server "number 4" is giving us problems, it is not restarted automatically with the others servers, all the work processes appear in red in transaction sm51, the workprocesses (DIA,BGD,UPD) are dying one after another. Yesterday two workprocesses were stopped, then four, and this morning eight workprocecess
We don't have any performance problems :
each application server has 32Gb of physical memory ( 24Gb of physical free memory)
each applications erver 4 CPU (idle = 80%)
It doesn't seem to be an SAP problem.
At first we thought, ot was an OS issue, we desinstalled the applivcation server, and asked our system administrataros to reintall the OS, then we reintalled the application server.
But it didn't help, we're still facing the same problems.
Half of our workprocecesses are now stopped, the workprocesses log can not be utilized for analysis :
There is no information about what happened this morning :
Tue Jun  3 17:32:27 2008
===...sucessfully completed.
=================================================
MskiInitLogonTicketCacheHandle: Logon Ticket cache pointer retrieved from shared m
MskiInitLogonTicketCacheHandle: Workprocess runs with Logon Ticket cache.
=================================================
=== ipl_Init() called
   ITSP Running against db release 620!
   ITSP Disable Kernel Web GUI functionality
=== ipl_Init() returns 2, ITSPE_DISABLED: Service is disabled (sapparam)
=================================================
                                                                                Wed Jun  4 00:30:38 2008
dbtran INFO (init_connection '<DEFAULT>' [ORACLE:640.00]):
  max_blocking_factor =   5,  max_in_blocking_factor      =   5,
  min_blocking_factor =   5,  min_in_blocking_factor      =   5,
  prefer_union_all    =   0,  prefer_union_for_select_all =   0,
  prefer_fix_blocking =   0,  prefer_in_itab_opt          =   1,
  convert AVG         =   0,  alias table FUPD            =   0,
  escape_as_literal   =   1,  opt GE LE to BETWEEN        =   0,
  select *            =0x0f,  character encoding          =SBCS / <none>:-,
  use_hints           = abap->1, dbif->0x1, upto->2147483647, rule_in->0,
                        rule_fae->0, concat_fae->0, concat_fae_or->0
SecAudit(RsauShmInit): WP attached to existing shared memory.
SecAudit(RsauShmInit): addr of SCSA........... = 0x2aa017f000
SecAudit(RsauShmInit): addr of RSAUSHM........ = 0x2aa017f450
SecAudit(RsauShmInit): addr of RSAUSLOTINFO... = 0x2aa017f488
SecAudit(RsauShmInit): addr of RSAUSLOTS...... = 0x2aa017f494
login/password_change_for_SSO : 1 -> 1
                                                                                Wed Jun  4 00:30:39 2008
handle memory type is RSTSPROMMM                                                 
We have checked the profile with the sappfpar command :
Nr of operating system shared memory segments: 28
Shared memory resource requirements estimated
================================================================
Total Nr of shared segments required.....:         28
System-imposed number of shared memories.:       1000
Shared memory segment size required min..: 1073741924 (1024.0 MB)
System-imposed maximum segment size......: 1140850688 (1088.0 MB)
Max. shared memory segment size advised..: 2147483648 (2048.0 MB)
Swap space requirements estimated
================================================
Shared memory....................: 3030.2 MB
..in pool 10  121.8 MB,   96% used
..in pool 40  169.7 MB,   96% used
..not in pool: 2730.7 MB
Processes........................:  378.8 MB
Extended Memory .................: 16384.0 MB
Total, minimum requirement.......: 19793.0 MB
Process local heaps, worst case..: 1907.3 MB
Total, worst case requirement....: 21700.3 MB
Errors detected..................:    0
Warnings detected................:    0
Any idea ?
Any useful help would be highly appreciated ?
Best regards.
Raoul

Answer 1

Hi there,
Your physical memory is totally adequate. To make sure, you can check the buffer tune transaction ST04 for issues like swap.
Is the work process log you pasted here complete?? After restart of dialog instance, do all wp come up in Wait state?
Apart from work process logs, please check the system log for errors. Also, check the database connectivity from your app server.
regards, Sean.