Standby database failover flashback restart as standby gap 1-100

Question

Hi,
I have setup a physical standby database for a 3 node RAC primary database and configured the database to run in MAXIMUM AVAILABILITY mode. Both the primary and standby databases use ASM for managing the IO. I have enabled FLASHBACK ON on both the databases.
I did failover to standby database with primary db SHUTDOWN for doing some testing. After testing is done, I flashbacked the (standby, currently primary)database to the point when it became primary and start it as standby again and the old primary as primary.
Following are the steps:
Failover steps
.Shutdown the primary database
To finisish the recovery at standby
SQL>alter database recover managed standby database finish;
To start as primary
SQL>alter database commit to switchover to primary;
do some DML on the new primary
SQL>flashback database to scn <standby_became_primary_scn>;
SQL>alter database create standby controlfile as '/backup/standby.ctl';
Changed the spfile to point control_files to '/backup/standby.ctl';
Restarted the standby db in mount mode and
created the required stanbdy logs on it.
Then issued
SQL>alter database recover managed standby database using current logfile disconnect;
I have the following message in v$dataguard_status at standby db;
Fetching gap sequence in thread 1 branch(resetlogs_id) 653234404, gap seq 1-100
DBID 2651011616 branch 653234404
GAP - thread 1 sequence 1-100
FAL[client]: Failed to request gap sequence
FAL[client]: All defined FAL servers have been attempted.
The same error is found in alert log file also.
The status of the varioud managed standby db are as follows.
SQL> select process, status from v$managed_standby;
PROCESS STATUS
ARCH CLOSING
ARCH CLOSING
ARCH CLOSING
MRP0 WAIT_FOR_GAP
RFS IDLE
RFS IDLE
RFS IDLE
RFS IDLE
RFS IDLE
RFS IDLE
RFS IDLE
PROCESS STATUS
RFS IDLE
RFS IDLE
RFS IDLE
RFS IDLE
15 rows selected.
SQL>
Could you please help me how to resolve the above error.
Thanks,
Sreekanth

Answer 1

Sreekanth,
I think your steps are right on for your requirement, except instead of
alter database create standby controlfile as '/backup/standby.ctl';
you can do this:
ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
(although perhaps it achieves the same thing internally but you don't need to repoint to new controlfile).
So far as the GAP - thread 1 sequence 1-100
Let me guess, your current sequences are probably well ahead of this reported gap?
If so, this is a bogus gap in a sense that your standby has long ago applied them, but for some reason standby controlfile gets a little confused and starts asking for them again.
I have had this issue and opened multiple TAR's with Oracle since 10.2.0.1, got a few patches and advises to upgrade, but still occasionally get it now in 10.2.0.3
I strongly believe this is related to combination of RAC (i.e. multiple threads of redo) and dataguard.
It is also pretty hard to reproduce (I have not figured out how to reproduce this reliably, if you have, let me know), so I can understand why Oracle has so far been unable to fix this.
Anyway, the workaround is to re-create standby controlfile with a copy generated on primary, and move it in place of the existing (and confused) controlfiles on standby.