Applying logs in a semi-standby instance

Question

Oracle Version: 10.2.0.3 Standard Edition
OS: Windows 2003 Server
I setup oracle standby (followed the steps in metalink doc 432514.1)..
I am running 10.2.0.3 standard edition.. so teh standby have to be done manually (which is fine)..
I shutdown production last night with shutdown immediate.. copy all the logs, datafiles, pfile..etc to the standby server.. I recreated the controlfile on the standby (NORESETLOGS and NOARCHIVELOGS).. the database is mounted and it's ok:
SQL> select status from v$instance;
STATUS
MOUNTED
SQL>
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6764 52428800 2 NO INACTIVE 2437870303 07-MAY-09
3 1 6765 52428800 2 NO INACTIVE 2437871111 07-MAY-09
2 1 6766 52428800 2 NO CURRENT 2437886254 07-MAY-09
SQL>
then after that I restarted production, after a while, I did a couple of switch logs and generated 2 archive logs of which I transferred to the standby to apply them.. now at this stage prod redo logs are:
SQL> select * from v$log
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6767 52428800 2 YES INACTIVE 2438366118 08-MAY-09
2 1 6766 52428800 2 YES INACTIVE 2437886254 07-MAY-09
3 1 6768 52428800 2 NO CURRENT 2438369450 08-MAY-09
SQL>
On the standby, I logged in as sysdba and tried to just apply the logs as you do normally:
C:\scheduled_scripts>sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.3.0 - Production on Fri May 8 08:58:03 2009
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Release 10.2.0.3.0 - Production
SQL> select status from v$instance;
STATUS
MOUNTED
SQL>
SQL>
SQL> recover database using backup controlfile until cancel;
ORA-00279: change 2438369450 generated at 05/08/2009 07:45:07 needed for thread
1
ORA-00289: suggestion :
D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6768_%U_.ARC
ORA-00280: change 2438369450 for thread 1 is in sequence #6768
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6767_508KCMM0_.ARC
ORA-00310: archived log contains sequence 6767; sequence 6768 required
ORA-00334: archived log:
'D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6767_508KCMM0_.ARC'
SQL>
So my question is why isn't it applying the 6767 change? (keep in mind that at this stage production has not yet generated archive log 6768).. shouldn't it be applying 6767? or is it bcoz it's Inactive, it doesn't need to do that?
More over, I did the following:
I did a switch log on production to create the 6768 archive log:
SQL> select * from v$log
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6767 52428800 2 YES INACTIVE 2438366118 08-MAY-09
2 1 6766 52428800 2 YES INACTIVE 2437886254 07-MAY-09
3 1 6768 52428800 2 NO CURRENT 2438369450 08-MAY-09
SQL>
SQL>
SQL> alter system switch logfile;
System altered.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6767 52428800 2 YES INACTIVE 2438366118 08-MAY-09
2 1 6769 52428800 2 NO CURRENT 2438387289 08-MAY-09
3 1 6768 52428800 2 YES ACTIVE 2438369450 08-MAY-09
this generated teh archive log: O1_MF_1_6768_508VGSS5_.ARC
I copied the archive log to the standby serve.. checked teh current sequence (database only mounted on standby):
SQL> select * from v$log
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6764 52428800 2 NO INACTIVE 2437870303 07-MAY-09
3 1 6765 52428800 2 NO INACTIVE 2437871111 07-MAY-09
2 1 6766 52428800 2 NO CURRENT 2437886254 07-MAY-09
C:\scheduled_scripts>sqlplus "/as sysdba"
SQL*Plus: Release 10.2.0.3.0 - Production on Fri May 8 10:46:38 2009
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Release 10.2.0.3.0 - Production
SQL> recover database using backup controlfile until cancel;
ORA-00279: change 2438369450 generated at 05/08/2009 07:45:07 needed for thread1
ORA-00289: suggestion :
D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6768_%U_.ARC
ORA-00280: change 2438369450 for thread 1 is in sequence #6768
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00279: change 2438387289 generated at 05/08/2009 10:37:29 needed for thread1
ORA-00289: suggestion :
D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6769_%U_.ARC
ORA-00280: change 2438387289 for thread 1 is in sequence #6769
ORA-00278: log file
'D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6768_508VGSS5_.ARC' no longer needed for this recovery
ORA-00308: cannot open archived log
'D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6769_%U_.ARC'
ORA-27041: unable to open file
OSD-04002: unable to open file
O/S-Error: (OS 2) The system cannot find the file specified.
and as you can see.. now that it has 6768, it's doing the same thing by not wanting it anymore and wanting the next one in teh sequence which is 6769 (which has not been generated on production yet).. I'm just not making any sense out of this!!
now afer few changes, the production sequence is:
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6773 52428800 2 YES ACTIVE *2438396976* 08-MAY-09
2 1 6772 52428800 2 YES INACTIVE 2438394852 08-MAY-09
3 1 6774 52428800 2 NO CURRENT 2438398862 08-MAY-09
standby sequence is:
SQL> SELECT * FROM V$LOG;
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6764 52428800 2 NO INACTIVE 2437870303 07-MAY-09
3 1 6765 52428800 2 NO INACTIVE 2437871111 07-MAY-09
2 1 6766 52428800 2 NO CURRENT 2437886254 07-MAY-09
If I try to do the following on standby (archived has been shipped to standby at this stage) and do the AUTO option, it doesnt like the file:
SQL> RECOVER DATABASE using backup controlfile UNTIL CHANGE 2438396976;
ORA-00279: change 2438396976 generated at 05/08/2009 11:45:02 needed for thread 1
ORA-00289: suggestion : D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6773_%U_.ARC
ORA-00280: change 2438396976 for thread 1 is in sequence #6773
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00308: cannot open archived log 'D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6773_%U_.ARC'
ORA-27041: unable to open file
OSD-04002: unable to open file
O/S-Error: (OS 2) The system cannot find the file specified.
ORA-00308: cannot open archived log 'D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6773_%U_.ARC'
ORA-27041: unable to open file
OSD-04002: unable to open file
O/S-Error: (OS 2) The system cannot find the file specified.
but when i try to do the same and I actuially give it the proper file name (no _%U) then it's ok and does the recovery.. Is there a way of telling Oracle to grab the acual file name, not O1_MF_1_6773_50909L7L_.ARC instead of O1_MF_1_6773_%U_.ARC?
SQL> RECOVER DATABASE using backup controlfile UNTIL CHANGE *2438396976*;
ORA-00279: change 2438396976 generated at 05/08/2009 11:45:02 needed for thread 1
ORA-00289: suggestion : D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6773_%U_.ARC
ORA-00280: change 2438396976 for thread 1 is in sequence #6773
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\ORACLE\PRODUCT\FLASH_RECOVERY_AREA\EBCP01\ARCHIVELOG\2009_05_08\O1_MF_1_6773_50909L7L_.ARC
Log applied.
Media recovery complete.
hmm.. now when would it actually update the sequence numbers on the logs, I am asking this bcoz the sequence on teh standby still reads:
SQL> SELECT * FROM V$LOG;
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
1 1 6764 52428800 2 NO INACTIVE 2437870303 07-MAY-09
3 1 6765 52428800 2 NO INACTIVE 2437871111 07-MAY-09
2 1 6766 52428800 2 NO CURRENT 2437886254 07-MAY-09
I am lost here!!! and I'm not sure what to do about this! Your help is really appreciated.
Thanks

Answer 1

The "readme" for Patch 9369783 (which covers the AprilCPU for our 11.1.0.7 HPUX-IA64 environment) includes this short reference to DataGuard:
If you are using a Data Guard Physical Standby database, you must first install this patch on the primary database before installing the patch on the physical standby database. It is not supported to install this patch on the physical standby database before installing the patch on the primary database. For more information, see My Oracle Support Note 278641.1.
When checking that note 278641.1 we see that it also appears to only cover 10.2. Although this note has more detail, it is clearly the same procedure as discussed in 813445.1. Therefore, the conclusion that I make is: OPatch works exactly the same with DataGuard in 11g as it did in 10g.
We will be upgrading our DataGuard enviornment to 11g in one-month. At this point, I am completely expecting our OPatch procedures to remain unchanged from what we have done for years with 9i and 10g. I would also note that the upgrade procedures we have tested (involving DG from 10.2.0.4 to 11.1.0.7) are nearly identical to the above mentioned support notes.
Hope that helps,
Stan