Shutting down database hangs

Question

Dear Gurus,
I m facing a database hang issue while performing the scheduled offline backup.I m able to take the online backup.When the backup starts , the database starts to shutdown.When checked in the log of backup it shows in tail as shutting down database.When checked for user trace , it shows message as "Attempting to kill pid xxxxxxx" and keeps on retrying for n no. of times.Then I have to manually kill the backup and restart the application.Then again try for an offline backup.This time it happens smoothly.This activity we did for couple of weeks .But now from past 3 days even after restarting the backup, it goes to hang mode.In other words unable to perform the backup activity.
The message had been raised to SAP from almost 15 days at very high.They every time ask the logs , and say that the data in log is nt providing much info for analysing the issue or cause of hang.Also enabled the traces of 10400 for trace level 1 ,10 some more traces also.But still the output is not helping even them to analyse the root cause.
Its my observation that this issue is occuring only on ABAP+JAVA systems(CRM &BIW)not in ECC.We are on oracle database 10.2.0.0.4 with latest patch set as suggested by SAP for the same issue.
This issue is h
Please suggest me at the earliest.As need to resolve this at the earliest.
Bye.
Amit

Answer 1

Hi,
No "archive process shutdown avoided" message is there in the log file.
In usertrace i get messages as mentioned below.
bslcrmprddb #more bcp_ora_1257502.trc
Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_1257502.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/BCP/102_64
System name:    AIX
Node name:      bslcrmprddb
Release:        3
Version:        5
Machine:        00C6C4024C00
Instance name: BCP
Redo thread mounted by this instance: 1
Oracle process number: 0
Unix process pid: 1257502, image: oracle@bslcrmprddb
2009-06-05 08:01:36.875
Oracle Instance Startup operation failed. Another process may be attempting to startup or shutdown
Failed to acquire instance startup/shutdown serialization primitive
bslcrmprddb #more bcp_ora_2453676.trc
Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_2453676.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/BCP/102_64
System name:    AIX
Node name:      bslcrmprddb
Release:        3
Version:        5
Machine:        00C6C4024C00
Instance name: BCP
Redo thread mounted by this instance: 0 <none>
Oracle process number: 0
Unix process pid: 2453676, image: oracle@bslcrmprddb
2009-06-06 07:04:22.494
Dynamic strand is set to TRUE
Running with 2 shared and 88 private strand(s). Zero-copy redo is FALSE
bslcrmprddb #more bcp_ora_1495266.trc
Dump file /oracle/BCP/saptrace/usertrace/bcp_ora_1495266.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/BCP/102_64
System name:    AIX
Node name:      bslcrmprddb
Release:        3
Version:        5
Machine:        00C6C4024C00
Instance name: BCP
Redo thread mounted by this instance: 1
Oracle process number: 213
Unix process pid: 1495266, image: oracle@bslcrmprddb (TNS V1-V3)
ACTION NAME:() 2009-06-06 03:00:22.223
MODULE NAME:(sqlplus@bslcrmprddb (TNS V1-V3)) 2009-06-06 03:00:22.223
SERVICE NAME:(SYS$USERS) 2009-06-06 03:00:22.223
SESSION ID:(517.13229) 2009-06-06 03:00:22.223
ksukia: Starting kill, force = 0
ksukia: killed 274 out of 274 processes.
ksukia: Starting kill, force = 0
ksukia: killed 0 out of 2 processes.
ksukia: Starting kill, force = 0
ksukia: killed 0 out of 1 processes.
ksukia: Starting kill, force = 0
ksukia: killed 0 out of 1 processes.
ksukia: Starting kill, force = 0
Please suggest.
Regards,
Amit