Archive log missing on standby: FAL[client]: Failed to request gap sequence

Question

My current environment is Oracle 10.2.0.4 with ASM 10.2.0.4 on a 2 node RAC in production and a standby that is the same setup. I'm also running on Oracle Linux 5. Almost daily now an archivelog doesnt make it to the standby and oracle doesnt seem to resolve the gap sequence from the primary. If I stop and restart recovery it gets the logfile and continues recovery just fine. I have checked my fal_client and fal_server settings and they look good. The logs after this error do continue to get written to the standby but the standby wont continue recovery until I stop and restart recovery and it fetches this missing log.
The only thing I know thats happening is that the firewall people are disconnecting any connections that are inactive for 60 minutes and recently did an upgrade that they are claiming didnt change anything:)  I dont know if this is causing this problem or not. Any thoughts on what might be happening?
Error in standby alert.log:
Tue Jun 29 23:15:35 2010
RFS[258]: Possible network disconnect with primary database
Tue Jun 29 23:15:36 2010
Fetching gap sequence in thread 2, gap sequence 9206-9206
Tue Jun 29 23:16:46 2010
FAL[client]: Failed to request gap sequence
GAP - thread 2 sequence 9206-9206
DBID 661398854 branch 714087609
FAL[client]: All defined FAL servers have been attempted.
Error on primary alert.log:
Tue Jun 29 23:00:07 2010
ARC0: Creating remote archive destination LOG_ARCHIVE_DEST_2: 'WSSPRDB' (thread 1 sequence 9265)
(WSSPRD1)
ARC0: Transmitting activation ID 0x29c37469
Tue Jun 29 23:00:07 2010
Errors in file /u01/app/oracle/admin/WSSPRD/bdump/wssprd1_arc0_14024.trc:
ORA-03135: connection lost contact
FAL[server, ARC0]: FAL archive failed, see trace file.
Tue Jun 29 23:00:07 2010
Errors in file /u01/app/oracle/admin/WSSPRD/bdump/wssprd1_arc0_14024.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Tue Jun 29 23:00:07 2010
ORACLE Instance WSSPRD1 - Archival Error. Archiver continuing.
Tue Jun 29 23:00:41 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
Tue Jun 29 23:00:41 2010
FAL[server, ARC2]: Begin FAL archive (dbid 0 branch 714087609 thread 2 sequence 9206 dest WSSPRDB)
FAL[server, ARC2]: FAL archive failed, see trace file.
Tue Jun 29 23:00:43 2010
Errors in file /u01/app/oracle/admin/WSSPRD/bdump/wssprd1_arc2_14028.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Tue Jun 29 23:00:43 2010
ORACLE Instance WSSPRD1 - Archival Error. Archiver continuing.
Tue Jun 29 23:01:16 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
Tue Jun 29 23:15:01 2010
Thread 1 advanced to log sequence 9267 (LGWR switch)
I have checked the trace files that get spit out but they arent anything meaningful to me as to whats really happening. Snipit of the trace file:
tkcrrwkx: Starting to process work request
tkcrfgli: SRL header: 0
tkcrfgli: SRL tail: 0
tkcrfgli: ORL to arch: 4
tkcrfgli: le# seq thr for bck tba flags
tkcrfgli: 1 359 1 2 0 3 0x0008 ORL active cur
tkcrfgli: 2 358 1 0 1 1 0x0000 ORL active
tkcrfgli: 3 361 2 4 0 0 0x0008 ORL active cur
tkcrfgli: 4 360 2 0 3 2 0x0000 ORL active
tkcrfgli: 5 -- entry deleted --
tkcrfgli: 6 -- entry deleted --
tkcrfgli: 7 -- entry deleted --
tkcrfgli: 8 -- entry deleted --
tkcrfgli: 9 -- entry deleted --
tkcrfgli: 191 -- entry deleted --
tkcrfgli: 192 -- entry deleted --
*** 2010-03-27 01:30:32.603 20998 kcrr.c
tkcrrwkx: Request from LGWR to perform: <startup>
tkcrrcrlc: Starting CRL ARCH check
*** 2010-03-27 01:30:32.603 66085 kcrr.c
Beginning controlfile transaction 0x0x7fffd0b53198 [kcrr.c:20395 (14011)]
*** 2010-03-27 01:30:32.645 66173 kcrr.c
Acquired controlfile transaction 0x0x7fffd0b53198 [kcrr.c:20395 (14024)]
*** 2010-03-27 01:30:32.649 66394 kcrr.c
Ending controlfile transaction 0x0x7fffd0b53198 [kcrr.c:20397]
tkcrrasgn: Checking for 'no FAL', 'no SRL', and 'HB' ARCH process
# HB NoF NoS CRL Name
29 NO NO NO NO ARC0
28 NO YES YES NO ARC1
27 NO NO NO NO ARC2
26 NO NO NO NO ARC3
25 YES NO NO NO ARC4
24 NO NO NO NO ARC5
23 NO NO NO NO ARC6
22 NO NO NO NO ARC7
21 NO NO NO NO ARC8
20 NO NO NO NO ARC9
Thanks.
Kristi

Answer 1

It's the network that's messing up; unlikely due to firewall timeout as it waits for 60 minutes and you are switching every 15 minutes. There may be some other network glitch that needs rectified.
In any case - arch file missing/ corrupt / halfway through - FAL setting should have refetched the problematic archive log automatically.
As many had suggested already, the best way to resolve RFS issues I believe is to use real-time apply by configuring standby redo logs. It's very easy to configure it and you can opt for real-time apply even in max-performance mode that you are using right now.
Even though you are maintaining (I guess) 1-1 between primary & standby instances, you can provide both primary instances in fal_server (like fal_server=string1,string2). See if that helps.
lastly, check if you are having simiar issue at other times as well that might be getting rectified automatically as expected.
col message for a80
col time for a20
select message, to_char(timestamp,'dd-mon-rr hh24:mi:ss') time
from v$dataguard_status
where severity in ('Error','Fatal')
order by timestamp;
Cheers.