All of sudden primary alert log filled

Question

ORA-00270: error creating archive log
FAL[server, ARC2]: FAL archive failed, see trace file.
Errors in file /mv1/oracle/admin/PRD/diag/diag/rdbms/prd/PRD1/trace/PRD1_arc2_909440.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance PRD1 - Archival Error. Archiver continuing.
DB version is 11.1.0.7, and nothing has been changed except I just increased db recovery area in standby side.
Anybody can provide me ways to debug?
Thanks

Answer 1

NAME DISPLAY_VALUE
db_file_name_convert
db_name PRD
db_unique_name PRD
dg_broker_config_file1 /u01/orahome_a/dbs/dr1PRD.dat
dg_broker_config_file2 /u01/orahome_a/dbs/dr2PRD.dat
dg_broker_start FALSE
fal_client
fal_server
local_listener listener_oracle
log_archive_config
log_archive_dest_2 service=PRD_standby.PROD
log_archive_dest_state_2 ENABLE
log_archive_max_processes 4
log_file_name_convert /oradata/dy, +PRD_ DAT
remote_login_passwordfile EXCLUSIVE
standby_archive_dest ?/dbs/arch
standby_file_management auto
SQL> col name for a10
SQL> col DATABASE_ROLE for a10
SQL> SELECT name,db_unique_name,protection_mode,DATABASE_ROLE,OPEN_MODE,switchover_status from v$database;
NAME DB_UNIQUE_NAME PROTECTION_MODE DATABASE_R OPEN_MODE SWITCHOVER_STATUS
PRD PRD MAXIMUM PERFORMANCE PRIMARY READ WRITE SESSIONS ACTIVE
SQL> select thread#,max(sequence#) from v$archived_log group by thread#;
THREAD# MAX(SEQUENCE#)
1 78035
2 77624
SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"
2 FROM
3 (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
4 (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
5 WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;
Thread Last Sequence Received Last Sequence Applied Difference
1 78035 78035 0
1 78035 78035 0
2 77624 77624 0
2 77624 77624 0
SQL> col severity for a15
SQL> col message for a70
SQL> col timestamp for a20
SQL> select severity,error_code,to_char(timestamp,'DD-MON-YYYY HH24:MI:SS') "timestamp" , message from v$dataguard_status where dest_id=2;
SEVERITY ERROR_CODE timestamp MESSAGE
Warning 270 06-JUL-2012 18:46:02 ARC3: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (270) )
Warning 270 06-JUL-2012 18:46:02 ARC3: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Error 270 06-JUL-2012 18:46:02 FAL[server, ARC3]: Error 270 creating remote archivelog file 'PRD_ standby.PROD'
Error 270 06-JUL-2012 18:46:05 FAL[server, ARC3]: Error 270 creating remote archivelog file 'PRD_ standby.PROD'
SQL> select ds.dest_id id
2 , ad.status
3 , ds.database_mode db_mode
4 , ad.archiver type
5 , ds.recovery_mode
6 , ds.protection_mode
7 , ds.standby_logfile_count "SRLs"
8 , ds.standby_logfile_active active
9 , ds.archived_seq#
10 from v$archive_dest_status ds
11 , v$archive_dest ad
12 where ds.dest_id = ad.dest_id
13 and ad.status != 'INACTIVE'
14 order by
15 ds.dest_id;
ID STATUS DB_MODE TYPE RECOVERY_MODE PROTECTION_MODE SRLs ACTIVE ARCHIVED_SEQ#
1 VALID OPEN ARCH IDLE MAXIMUM PERFORMANCE 0 0 78035
2 ERROR MOUNTED-STANDBY LGWR IDLE MAXIMUM PERFORMANCE 0 0 77578
3 VALID OPEN ARCH IDLE MAXIMUM PERFORMANCE 0 0 78035
From standby side:
NAME DISPLAY_VALUE
db_file_name_convert
db_name PRD
db_unique_name PRD
dg_broker_config_file1 /u01/orahome_a/dbs/dr1PRD.dat
dg_broker_config_file2 /u01/orahome_a/dbs/dr2PRD.dat
dg_broker_start FALSE
fal_client
fal_server
local_listener listener_oracle4
log_archive_config
log_archive_dest_2
log_archive_dest_state_2 enable
log_archive_max_processes 4
log_file_name_convert
remote_login_passwordfile EXCLUSIVE
standby_archive_dest ?/dbs/arch
standby_file_management auto
SQL> SQL> SQL> col name for a30
select * from v$dataguard_stats;
select * from v$archive_gap;
col name format a60
select name
, floor(space_limit / 1024 / 1024) "Size MB"
, ceil(space_used / 1024 / 1024) "Used MB"
from v$recovery_file_dest
order by name;
spool off
NAME DB_UNIQUE_NAME PROTECTION_MODE DATABASE_R OPEN_MODE
PRD PRD MAXIMUM PERFORM PHYSICAL S MOUNTED
ANCE TANDBY
SQL>
THREAD# MAX(SEQUENCE#)
1 77989
2 77577
SQL>
PROCESS STATUS THREAD# SEQUENCE#
ARCH CONNECTED 0 0
ARCH CONNECTED 0 0
ARCH CONNECTED 0 0
ARCH CONNECTED 0 0
SQL> SQL>
NAME VALUE UNIT TIME_COMPUTED
apply finish time day(2) to second(3) interval
apply lag day(2) to second(0) interval
estimated startup time 12 second
standby has been open N
transport lag day(2) to second(0) interval
SQL> SQL> SQL> 2 3 4 5
NAME Size MB Used MB
/oracle_arch/_FRA 30720 99216