DB recovery crashes when executed from script, finishes OK manually

Question

Hello everybody,
I have a DB restore script which I used/tested a lot of times. On the last run it crashed in the archivelog recovery phase, but I was able to open the instance when applying the remaining commands manually.
DB is a 10g2 on Solaris 10, I have a full backup made with EMC NetWorker. After taking the backup, the machine was re-installed from scratch (OS, additional 3rd party software, Oracle software and backup client). A new, empty DB instance was also created.
Restore script performs the following:
- checks/creates directory structure needed by Oracle instance;
- stops the listener;
- shuts down the new instance;
- startup mount exclusive, enable restricted session;
- drops database instance via RMAN
- recovers the orapw, spfile, tnsnames.ora and listener.ora files from the backupset;
- stars the instance in nomount;
- replicates the controlfile from a saved copy in the backupset;
- mounts the instance;
- starts the following RMAN sequence:
run {
set until $LAST_SCN_IN_BACKUPSET;
allocate channel c3 type 'sbt_tape' parms 'ENV=(....)';
restore database check readonly;
recover database check readonly;
release channel c3;
sql 'alter database open resetlogs';
- starts the listener.
Almost all is performed OK, until the "recover database check readonly" command, where the recover crashes with the following messages:
Starting recover at 18-FEB-09
starting media recovery
channel c3: starting archive log restore to default destination
channel c3: restoring archive log
archive log thread=1 sequence=17
channel c3: reading from backup piece 09k7hb2h_1_1
channel c3: restored backup piece 1
piece handle=09k7hb2h_1_1 tag=TAG20090216T181752
channel c3: restore complete, elapsed time: 00:00:36
archive log filename=<...>/oracle/oradata/SNM/arch/arch_1_17_678643049.arc thread=1 sequence=17
released channel: c3
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 02/18/2009 19:58:19
ORA-00283: recovery session canceled due to errors
RMAN-11003: failure during parse/execution of SQL statement: alter database recover logfile '<...>/oracle/oradata/SNM/arch/arch_1_17_678643049.arc'
ORA-00283: recovery session canceled due to errors
ORA-19755: could not open change tracking file
ORA-19750: change tracking file: '/alcatel/oracle/oradata/SNM/bct.dbf'
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3
Recovery Manager complete.
The curious thing is that when I ran the following commands manually (from RMAN), the recovery was OK:
run {
set until scn $LAST_SCN_IN_BACKUPSET;
recover database check readonly;
(finished in a couple of minutes)
sql 'alter database open resetlogs';
(finished after a few minutes)
lsnrctl start (from shell) - OK.
Could somebody point me to the possible causes of this behavior?
Thank you all for your time,
Adrian

Answer 1

Hi Werner,
Thank you for your reply.
Yes, I'm using the block change tracking file. As the DB instance was re-created when the machine was installed, the bct must have been created also. However, in my script I dropped the DB instance, and the bct was, most likely, deleted, as were all the other datafiles.
After the restore script crashed, I had no bct file in the expected location (which is the location of all the datafiles).
I'm wondering why the bct could not have been created during the run of the restore script.
Looking through the alert log, I realise I tried an "alter database open resetlogs" before running the recovery sequence, and at that point, the bct was created.
Then, the second attempt to open the instance was successfull - please see below the messages from the alert log:
Thu Feb 19 09:43:47 2009
alter database open
Thu Feb 19 09:43:47 2009
CHANGE TRACKING is enabled for this database, but the
change tracking file can not be found. Recreating the file.
Change tracking file recreated.
Block change tracking file is current.
ORA-1589 signalled during: alter database open...
Thu Feb 19 09:43:54 2009
alter database open resetlogs
ORA-1196 signalled during: alter database open resetlogs...
Thu Feb 19 09:48:04 2009
alter database recover datafile list clear
Thu Feb 19 09:48:04 2009
Completed: alter database recover datafile list clear
Thu Feb 19 09:48:04 2009
alter database recover datafile list
1 , 2 , 3 , 4 , 5 , 6 , 7 , 8
Completed: alter database recover datafile list
1 , 2 , 3 , 4 , 5 , 6 , 7 , 8
Thu Feb 19 09:48:04 2009
alter database recover if needed
start until change 7033941 using backup controlfile
Media Recovery Start
parallel recovery started with 7 processes
ORA-279 signalled during: alter database recover if needed
start until change 7033941 using backup controlfile
Thu Feb 19 09:48:05 2009
alter database recover logfile '/alcatel/oracle/oradata/SNM/arch/arch_1_17_678643049.arc'
Thu Feb 19 09:48:05 2009
Media Recovery Log /alcatel/oracle/oradata/SNM/arch/arch_1_17_678643049.arc
Thu Feb 19 09:48:43 2009
Incomplete Recovery applied until change 7033941
Thu Feb 19 09:48:43 2009
Media Recovery Complete (SNM)
Completed: alter database recover logfile '/alcatel/oracle/oradata/SNM/arch/arch_1_17_678643049.arc'
Thu Feb 19 09:49:10 2009
alter database open resetlogs
<....>
Thu Feb 19 09:50:47 2009
LOGSTDBY: Validation complete
Starting control autobackup
Control autobackup written to DISK device
handle '/alcatel/oracle/oradata/SNM/flash_recovery_area/SNM/autobackup/2009_02_19/o1_mf_s_679225849_4st3tswj_.bkp'
Completed: alter database open resetlogs
So, to recap:
- bct file not created at first try (restore script);
- bct file created during the failed attempt to open the db (manual command);
- second attempt to open the db successfull (manual command).
The behavior doesn't seem to be systematic, as I don't think this happens every time. In this case, I'm starting to wonder if it wouldn't be a good idea to disable the bct before starting the restore, and then to enable it back when the db is opened.
Thank you again for your idea,
Adrian