Recovery session canceled due to errors. Can't start DB after kernel panic.

Question

Hi all,
I have a cluster Oracle 10 DB that was incorrectly shutdown (node 1 got a kernel panic when I disconnected the cluster interconnect and after a while node 2 also had a panic).
Now the DB doesn't start and I already tried an endless loop of solutions that don't work and lead me right to the start. I was hoping someone could help me fix this bloody thing please. This DB was running in archive log mode.
/export/home/oracle
oracle@dbsce066/1 - TSP/(CLIENT_10.2.0)/SERVER_10.2.0/(CRS_10.2.0)
sqlplus '/ as sysdba'SQL*Plus: Release 10.2.0.3.0 - Production on Mon Jul 18 23:18:24 2011
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 9680453632 bytes
Fixed Size 2162704 bytes
Variable Size 1570701296 bytes
Database Buffers 7969177600 bytes
Redo Buffers 138412032 bytes
Database mounted.
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
SQL> shutdown
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.
Total System Global Area 9680453632 bytes
Fixed Size 2162704 bytes
Variable Size 1553924080 bytes
Database Buffers 7985954816 bytes
Redo Buffers 138412032 bytes
Database mounted.
SQL> recover database using backup controlfile until cancel;
ORA-00283: recovery session canceled due to errors
ORA-01110: data file 87: '/export/home/oracle/dev/DX/dx_index10_logging3'
ORA-01110: data file 86: '/export/home/oracle/dev/DX/dx_index10_logging2'
ORA-00372: file 86 cannot be modified at this time
ORA-01110: data file 86: '/export/home/oracle/dev/DX/dx_index10_logging2'
SQL> select name,status,file# from v$datafile where name like '%index10_logging%';
NAME
STATUS FILE#
/export/home/oracle/dev/DX/dx_index10_logging1
ONLINE 85
/export/home/oracle/dev/DX/dx_index10_logging2
RECOVER 86
/export/home/oracle/dev/DX/dx_index10_logging3
ONLINE 87
SQL> ALTER DATABASE OPEN resetlogs ;
ALTER DATABASE OPEN resetlogs
ERROR at line 1:
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/export/home/oracle/dev/DX/dx_system'
SQL>
I also saw the "_allow_resetlogs_corruption". But I would like to leave this as last resort.
Thanks in advance

Answer 1

Thank you everyone for the advices. I used this tutorial to implement the allow reset logs corruption, but still it didn't allow me to startup the DB normally without errors or recover the datafile 86.
@@ How to use "_allow_resetlogs_corruption"
@@ http://dba-oracle.fr/dba/content/view/14/9/
vi /export/home/oracle/products/10.2.0/dbs/initDX1.ora
"/export/home/oracle/products/10.2.0/dbs/initDX1.ora" 94 lines, 4520 characters
DX1.undo_tablespace=dx_undo1
DX1.log_archive_dest_1= 'LOCATION=/export/home/oracle/dev/DX/archivelog1 REOPEN=10'
DX1.local_listener="(address=(protocol=tcp)(HOST=localhost)(PORT=1531))"
### cluster specific parameters
*.cluster_database = true
*.cluster_database_instances = 2
allowresetlogs_corruption=TRUE
"/export/home/oracle/products/10.2.0/dbs/initDX1.ora" 95 lines, 4553 characters
SQL> STARTUP MOUNT
ORACLE instance started.
Total System Global Area 9680453632 bytes
Fixed Size 2162704 bytes
Variable Size 1570701296 bytes
Database Buffers 7969177600 bytes
Redo Buffers 138412032 bytes
Database mounted.
SQL> ALTER DATABASE OPEN RESETLOGS;
ALTER DATABASE OPEN RESETLOGS
ERROR at line 1:
ORA-01110: data file 86: '/export/home/oracle/dev/DX/dx_index10_logging2'
ORA-00372: file 86 cannot be modified at this time
ORA-01110: data file 86: '/export/home/oracle/dev/DX/dx_index10_logging2'
SQL> SELECT 'alter database datafile /export/home/oracle/dev/DX/dx_index10_logging2 END BACKUP;' from v$datafile;
'ALTERDATABASEDATAFILE/EXPO
alter database datafile /export/home/oracle/dev/DX/dx_index10_logging2 END BACKUP;
alter database datafile /export/home/oracle/dev/DX/dx_index10_logging2 END BACKUP;
alter database datafile /export/home/oracle/dev/DX/dx_index10_logging2 END BACKUP;
SQL> recover database until cancel using backup controlfile;
ORA-00283: recovery session canceled due to errors
ORA-01110: data file 87: '/export/home/oracle/dev/DX/dx_index10_logging3'
ORA-01110: data file 86: '/export/home/oracle/dev/DX/dx_index10_logging2'
ORA-00372: file 86 cannot be modified at this time
ORA-01110: data file 86: '/export/home/oracle/dev/DX/dx_index10_logging2'
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done
SQL> alter database open;
alter database open
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
SQL> shut
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
root@dbsce066> ls -l /export/home/oracle/dev/DX/
total 450
lrwxrwxrwx 1 root other 29 Apr 13 2010 dx_index10_logging1 -> /dev/md/oraindex1dg/rdsk/d651
lrwxrwxrwx 1 root other 29 Apr 13 2010 dx_index10_logging10 -> /dev/md/oraindex1dg/rdsk/d660
lrwxrwxrwx 1 root other 29 Apr 13 2010 dx_index10_logging2 -> /dev/md/oraindex1dg/rdsk/d652
I can access the file and it is readable:
root@dbsce066> ls -ltr /export/home/oracle/dev/DX/dx_index10_logging2
lrwxrwxrwx 1 root other 29 Apr 13 2010 /export/home/oracle/dev/DX/dx_index10_logging2 -> /dev/md/oraindex1dg/rdsk/d652
root@dbsce066> ls -ltr /dev/md/oraindex1dg/rdsk/d652
lrwxrwxrwx 1 root root 44 Aug 2 17:59 /dev/md/oraindex1dg/rdsk/d652 -> ../../../../../devices/pseudo/md@0:3,652,raw
root@dbsce066> dd if=/dev/md/oraindex1dg/rdsk/d652 of=/dev/null
^C18564+0 records in
18564+0 records out
root@dbsce066>
SQL> select * from v$datafile where FILE#=86;
FILE# CREATION_CHANGE# CREATION_TIME TS# RFILE# STATUS
ENABLED CHECKPOINT_CHANGE# CHECKPOINT_TIME UNRECOVERABLE_CHANGE#
UNRECOVERABLE_TIME LAST_CHANGE# LAST_TIME OFFLINE_CHANGE#
ONLINE_CHANGE# ONLINE_TIME BYTES BLOCKS CREATE_BYTES BLOCK_SIZE
NAME
PLUGGED_IN BLOCK1_OFFSET
AUX_NAME
FIRST_NONLOGGED_SCN FIRST_NONLOGGED_TIM
86 184062 2010-04-13:17:18:24 62 86 RECOVER
READ ONLY 498989920 2011-07-18:15:28:29 0
485512497
496226794 2011-07-16:00:01:53 660602880 80640 660602880 8192
/export/home/oracle/dev/DX/dx_index10_logging2
0 8192
NONE
0
select STATUS,ENABLED,NAME from v$datafile where FILE#=86
STATUS ENABLED NAME
RECOVER READ ONLY /export/home/oracle/dev/DX/dx_index10_logging2
SQL> select name from v$tablespace where TS#=62;
NAME
DX_INDEX10_LOGGING2
SQL> alter tablespace DX_INDEX10_LOGGING2 read write;
alter tablespace DX_INDEX10_LOGGING2 read write
ERROR at line 1:
ORA-01109: database not open
SQL> ALTER DATABASE DATAFILE 86 ONLINE;
Database altered.
SQL> select STATUS,ENABLED,NAME from v$datafile where FILE#=86;
STATUS ENABLED
NAME
RECOVER READ ONLY
/export/home/oracle/dev/DX/dx_index10_logging2
SQL>
SQL> alter tablespace DX_INDEX10_LOGGING2 end backup;
alter tablespace DX_INDEX10_LOGGING2 end backup
ERROR at line 1:
ORA-01235: END BACKUP failed for 1 file(s) and succeeded for 0
ORA-01199: file 86 is not in online backup mode
ORA-01110: data file 86: '/export/home/oracle/dev/DX/dx_index10_logging2'
SQL> select log_mode from v$database;
LOG_MODE
ARCHIVELOG
SQL> SELECT FILE#, ERROR, ONLINE_STATUS, CHANGE#, TIME FROM V$RECOVER_FILE;
FILE# ERROR ONLINE_ CHANGE# TIME
86 ONLINE 498989920 2011-07-18:15:28:29
SQL> recover database until cancel using backup controlfile;
ORA-00279: change 497920329 generated at 07/18/2011 12:53:10 needed for thread 2
ORA-00289: suggestion : /export/home/oracle/dev/DX/archivelog1/LOG_DX_8574_2_716223168.ARC
ORA-00280: change 497920329 for thread 2 is in sequence #8574
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
auto
ORA-00308: cannot open archived log '/export/home/oracle/dev/DX/archivelog1/LOG_DX_8574_2_716223168.ARC'
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/export/home/oracle/dev/DX/dx_system'
@@ Attempt to start DB without that datafile:
SQL> ALTER DATABASE DATAFILE '/export/home/oracle/dev/DX/dx_index10_logging2' OFFLINE DROP;
Database altered.
SQL> SHUTDOWN IMMEDIATE;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> SQL> startup
ORACLE instance started.
Total System Global Area 9680453632 bytes
Fixed Size 2162704 bytes
Variable Size 1570701296 bytes
Database Buffers 7969177600 bytes
Redo Buffers 138412032 bytes
Database mounted.
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
SQL> ALTER DATABASE OPEN NORESETLOGS;
ALTER DATABASE OPEN NORESETLOGS
ERROR at line 1:
ORA-01588: must use RESETLOGS option for database open
SQL> ALTER DATABASE OPEN RESETLOGS;
ALTER DATABASE OPEN RESETLOGS
ERROR at line 1:
ORA-01245: offline file 76 will be lost if RESETLOGS is done
ORA-01110: data file 76: '/export/home/oracle/dev/DX/dx_data10_logging2'
SQL>
@@ Nothing works. I can't even start the DB without that datafile. So I give up of this recovery scenario. Re-install from scratch.