Archive log miss : How to restart capture
Hi Gurus,
I configured hotlog CDC distributed on 10.2.0.4 Databases.
I make a mistake: in my source Db I have deleted an Archive log,
and now the state of Capture process in V_$STREAM_CAPTURE is "WAITING FOR REDO: LAST SCN MINED 930696".
Now I'd like to restart the capture process from the next archive (just after the missed archive)
How is it possible?
tnk Fabio
I'm sorry to tell you that, but it's not possible. (Just as it's not possible to recover database with missing logs...)
You will have to recreate the capture process and to re-instantiate the replicated tables.
Regards,
Similar Messages
-
Capture process issue...archive log missing!!!!!
Hi,
Oracle Streams capture process is alternating between INITIALIZING and DICTIONARY INITIALIZATION state and not proceeding after this state to capture updates made on table.
we have accidentally missing archivelogs and no backup archive logs.
Now I am going to recreate the capture process again.
How I can start the the capture process from new SCN ?
And Waht is the batter way to remove the archive log files from central server, because
SCN used by capture processes?
Thanks,
Faziarain
Edited by: [email protected] on Aug 12, 2009 12:27 AMUsing dbms_Streams_Adm to add a capture, perform also a dbms_capture_adm.build. You will see in v$archived_log at the column dictionary_begin a 'yes', which means that the first_change# of this archivelog is first suitable SCN for starting capture.
'rman' is the prefered way in 10g+ to remove the archives as it is aware of streams constraints. If you can't use rman to purge the archives, then you need to check the min required SCN in your system by script and act accordingly.
Since 10g, I recommend to use rman, but nevertheless, here is the script I made in 9i in the old time were rman was eating the archives needed by Streams with appetite.
#!/usr/bin/ksh
# program : watch_arc.sh
# purpose : check your archive directory and if actual percentage is > MAX_PERC
# then undertake the action coded by -a param
# Author : Bernard Polarski
# Date : 01-08-2000
# 12-09-2005 : added option -s MAX_SIZE
# 20-11-2005 : added option -f to check if an archive is applied on data guard site before deleting it
# 20-12-2005 : added option -z to check if an archive is still needed by logminer in a streams operation
# set -xv
#--------------------------- default values if not defined --------------
# put here default values if you don't want to code then at run time
MAX_PERC=85
ARC_DIR=
ACTION=
LOG=/tmp/watch_arch.log
EXT_ARC=
PART=2
#------------------------- Function section -----------------------------
get_perc_occup()
cd $ARC_DIR
if [ $MAX_SIZE -gt 0 ];then
# size is given in mb, we calculate all in K
TOTAL_DISK=`expr $MAX_SIZE \* 1024`
USED=`du -ks . | tail -1| awk '{print $1}'` # in Kb!
else
USED=`df -k . | tail -1| awk '{print $3}'` # in Kb!
if [ `uname -a | awk '{print $1}'` = HP-UX ] ;then
TOTAL_DISK=`df -b . | cut -f2 -d: | awk '{print $1}'`
elif [ `uname -s` = AIX ] ;then
TOTAL_DISK=`df -k . | tail -1| awk '{print $2}'`
elif [ `uname -s` = ReliantUNIX-N ] ;then
TOTAL_DISK=`df -k . | tail -1| awk '{print $2}'`
else
# works on Sun
TOTAL_DISK=`df -b . | sed '/avail/d' | awk '{print $2}'`
fi
fi
USED100=`expr $USED \* 100`
USG_PERC=`expr $USED100 / $TOTAL_DISK`
echo $USG_PERC
#------------------------ Main process ------------------------------------------
usage()
cat <<EOF
Usage : watch_arc.sh -h
watch_arc.sh -p <MAX_PERC> -e <EXTENTION> -l -d -m <TARGET_DIR> -r <PART>
-t <ARCHIVE_DIR> -c <gzip|compress> -v <LOGFILE>
-s <MAX_SIZE (meg)> -i <SID> -g -f
Note :
-c compress file after move using either compress or gzip (if available)
if -c is given without -m then file will be compressed in ARCHIVE DIR
-d Delete selected files
-e Extention of files to be processed
-f Check if log has been applied, required -i <sid> and -g if v8
-g Version 8 (use svrmgrl instead of sqlplus /
-i Oracle SID
-l List file that will be processing using -d or -m
-h help
-m move file to TARGET_DIR
-p Max percentage above wich action is triggered.
Actions are of type -l, -d or -m
-t ARCHIVE_DIR
-s Perform action if size of target dir is bigger than MAX_SIZE (meg)
-v report action performed in LOGFILE
-r Part of files that will be affected by action :
2=half, 3=a third, 4=a quater .... [ default=2 ]
-z Check if log is still needed by logminer (used in streams),
it requires -i <sid> and also -g for Oracle 8i
This program list, delete or move half of all file whose extention is given [ or default 'arc']
It check the size of the archive directory and if the percentage occupancy is above the given limit
then it performs the action on the half older files.
How to use this prg :
run this file from the crontab, say, each hour.
example
1) Delete archive that is sharing common arch disk, when you are at 85% of 2500 mega perform delete half of the files
whose extention is 'arc' using default affected file (default is -r 2)
0,30 * * * * /usr/local/bin/watch_arc.sh -e arc -t /arc/POLDEV -s 2500 -p 85 -d -v /var/tmp/watch_arc.POLDEV.log
2) Delete archive that is sharing common disk with oother DB in /archive, act when 90% of 140G, affect by deleting
a quater of all files (-r 4) whose extention is 'dbf' but connect before as sysdba in POLDEV db (-i) if they are
applied (-f is a dataguard option)
watch_arc.sh -e dbf -t /archive/standby/CITSPRD -s 140000 -p 90 -d -f -i POLDEV -r 4 -v /tmp/watch_arc.POLDEV.log
3) Delete archive of DB POLDEV when it reaches 75% affect 1/3 third of files, but connect in DB to check if
logminer do not need this archive (-z). this is usefull in 9iR2 when using Rman as rman do not support delete input
in connection to Logminer.
watch_arc.sh -e arc -t /archive/standby/CITSPRD -p 75 -d -z -i POLDEV -r 3 -v /tmp/watch_arc.POLDEV.log
EOF
#------------------------- Function section -----------------------------
if [ "x-$1" = "x-" ];then
usage
exit
fi
MAX_SIZE=-1 # disable this feature if it is not specificaly selected
while getopts c:e:p:m:r:s:i:t:v:dhlfgz ARG
do
case $ARG in
e ) EXT_ARC=$OPTARG ;;
f ) CHECK_APPLIED=YES ;;
g ) VERSION8=TRUE;;
i ) ORACLE_SID=$OPTARG;;
h ) usage
exit ;;
c ) COMPRESS_PRG=$OPTARG ;;
p ) MAX_PERC=$OPTARG ;;
d ) ACTION=delete ;;
l ) ACTION=list ;;
m ) ACTION=move
TARGET_DIR=$OPTARG
if [ ! -d $TARGET_DIR ] ;then
echo "Dir $TARGET_DIR does not exits"
exit
fi;;
r) PART=$OPTARG ;;
s) MAX_SIZE=$OPTARG ;;
t) ARC_DIR=$OPTARG ;;
v) VERBOSE=TRUE
LOG=$OPTARG
if [ ! -f $LOG ];then
> $LOG
fi ;;
z) LOGMINER=TRUE;;
esac
done
if [ "x-$ARC_DIR" = "x-" ];then
echo "NO ARC_DIR : aborting"
exit
fi
if [ "x-$EXT_ARC" = "x-" ];then
echo "NO EXT_ARC : aborting"
exit
fi
if [ "x-$ACTION" = "x-" ];then
echo "NO ACTION : aborting"
exit
fi
if [ ! "x-$COMPRESS_PRG" = "x-" ];then
if [ ! "x-$ACTION" = "x-move" ];then
ACTION=compress
fi
fi
if [ "$CHECK_APPLIED" = "YES" ];then
if [ -n "$ORACLE_SID" ];then
export PATH=$PATH:/usr/local/bin
export ORAENV_ASK=NO
export ORACLE_SID=$ORACLE_SID
. /usr/local/bin/oraenv
fi
if [ "$VERSION8" = "TRUE" ];then
ret=`svrmgrl <<EOF
connect internal
select max(sequence#) from v\\$log_history ;
EOF`
LAST_APPLIED=`echo $ret | sed 's/.*------ \([^ ][^ ]* \).*/\1/' | awk '{print $1}'`
else
ret=`sqlplus -s '/ as sysdba' <<EOF
set pagesize 0 head off pause off
select max(SEQUENCE#) FROM V\\$ARCHIVED_LOG where applied = 'YES';
EOF`
LAST_APPLIED=`echo $ret | awk '{print $1}'`
fi
elif [ "$LOGMINER" = "TRUE" ];then
if [ -n "$ORACLE_SID" ];then
export PATH=$PATH:/usr/local/bin
export ORAENV_ASK=NO
export ORACLE_SID=$ORACLE_SID
. /usr/local/bin/oraenv
fi
var=`sqlplus -s '/ as sysdba' <<EOF
set pagesize 0 head off pause off serveroutput on
DECLARE
hScn number := 0;
lScn number := 0;
sScn number;
ascn number;
alog varchar2(1000);
begin
select min(start_scn), min(applied_scn) into sScn, ascn from dba_capture ;
DBMS_OUTPUT.ENABLE(2000);
for cr in (select distinct(a.ckpt_scn)
from system.logmnr_restart_ckpt\\$ a
where a.ckpt_scn <= ascn and a.valid = 1
and exists (select * from system.logmnr_log\\$ l
where a.ckpt_scn between l.first_change# and l.next_change#)
order by a.ckpt_scn desc)
loop
if (hScn = 0) then
hScn := cr.ckpt_scn;
else
lScn := cr.ckpt_scn;
exit;
end if;
end loop;
if lScn = 0 then
lScn := sScn;
end if;
select min(sequence#) into alog from v\\$archived_log where lScn between first_change# and next_change#;
dbms_output.put_line(alog);
end;
EOF`
# if there are no mandatory keep archive, instead of a number we just get the "PLS/SQL successfull"
ret=`echo $var | awk '{print $1}'`
if [ ! "$ret" = "PL/SQL" ];then
LAST_APPLIED=$ret
else
unset LOGMINER
fi
fi
PERC_NOW=`get_perc_occup`
if [ $PERC_NOW -gt $MAX_PERC ];then
cd $ARC_DIR
cpt=`ls -tr *.$EXT_ARC | wc -w`
if [ ! "x-$cpt" = "x-" ];then
MID=`expr $cpt / $PART`
cpt=0
ls -tr *.$EXT_ARC |while read ARC
do
cpt=`expr $cpt + 1`
if [ $cpt -gt $MID ];then
break
fi
if [ "$CHECK_APPLIED" = "YES" -o "$LOGMINER" = "TRUE" ];then
VAR=`echo $ARC | sed 's/.*_\([0-9][0-9]*\)\..*/\1/' | sed 's/[^0-9][^0-9].*//'`
if [ $VAR -gt $LAST_APPLIED ];then
continue
fi
fi
case $ACTION in
'compress' ) $COMPRESS_PRG $ARC_DIR/$ARC
if [ "x-$VERBOSE" = "x-TRUE" ];then
echo " `date +%d-%m-%Y' '%H:%M` : $ARC compressed using $COMPRESS_PRG" >> $LOG
fi ;;
'delete' ) rm $ARC_DIR/$ARC
if [ "x-$VERBOSE" = "x-TRUE" ];then
echo " `date +%d-%m-%Y' '%H:%M` : $ARC deleted" >> $LOG
fi ;;
'list' ) ls -l $ARC_DIR/$ARC ;;
'move' ) mv $ARC_DIR/$ARC $TARGET_DIR
if [ ! "x-$COMPRESS_PRG" = "x-" ];then
$COMPRESS_PRG $TARGET_DIR/$ARC
if [ "x-$VERBOSE" = "x-TRUE" ];then
echo " `date +%d-%m-%Y' '%H:%M` : $ARC moved to $TARGET_DIR and compressed" >> $LOG
fi
else
if [ "x-$VERBOSE" = "x-TRUE" ];then
echo " `date +%d-%m-%Y' '%H:%M` : $ARC moved to $TARGET_DIR" >> $LOG
fi
fi ;;
esac
done
else
echo "Warning : The filesystem is not full due to archive logs !"
exit
fi
elif [ "x-$VERBOSE" = "x-TRUE" ];then
echo "Nothing to do at `date +%d-%m-%Y' '%H:%M`" >> $LOG
fi -
Archived log missed in standby database
Hi,
OS; Windows 2003 server
Oracle: 10.2.0.4
Data Guard: Max Performance
Dataguard missed some of the archivelog files and but latest log files are applying. standby database is not in sync with primary.
SELECT LOCAL.THREAD#, LOCAL.SEQUENCE# FROM (SELECT THREAD#, SEQUENCE# FROM V$ARCHIVED_LOG WHERE DEST_ID=1) LOCAL WHERE LOCAL.SEQUENCE# NOT IN (SELECT SEQUENCE# FROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND THREAD# = LOCAL.THREAD#);
I queried above command and I found some files are missed in standby.
select status, type, database_mode, recovery_mode,protection_mode, srl, synchronization_status,synchronized from V$ARCHIVE_DEST_STATUS where dest_id=2;
STATUS TYPE DATABASE_MODE RECOVERY_MODE PROTECTION_MODE SRL SYNCHRONIZATION_STATUS SYN
VALID PHYSICAL MOUNTED-STANDBY MANAGED MAXIMUM PERFORMANCE NO CHECK CONFIGURATION NO
Anyone can tell me how to apply those missed archive log files.
Thanks in advacneDeccan Charger wrote:
I got below error.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION
ERROR at line 1:
ORA-01153: an incompatible media recovery is activeYou need to essentially do the following.
1) Stop managed recovery on the standby.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;2) Resolve the archive log gap - if you have configured FAL_SERVER and FAL_CLIENT Oracle should do this when you follow step 3 below, as you've manually copied the missed logs you should be OK
3) restart managed recovery using the command shown above.
You can monitor archive log catchup using the alert.log or your original query.
Niall Litchfield
http://www.orawin.info/
Edited by: Niall Litchfield on May 4, 2010 2:29 PM
missed tag -
Archive log missing on standby: FAL[client]: Failed to request gap sequence
My current environment is Oracle 10.2.0.4 with ASM 10.2.0.4 on a 2 node RAC in production and a standby that is the same setup. I'm also running on Oracle Linux 5. Almost daily now an archivelog doesnt make it to the standby and oracle doesnt seem to resolve the gap sequence from the primary. If I stop and restart recovery it gets the logfile and continues recovery just fine. I have checked my fal_client and fal_server settings and they look good. The logs after this error do continue to get written to the standby but the standby wont continue recovery until I stop and restart recovery and it fetches this missing log.
The only thing I know thats happening is that the firewall people are disconnecting any connections that are inactive for 60 minutes and recently did an upgrade that they are claiming didnt change anything:) I dont know if this is causing this problem or not. Any thoughts on what might be happening?
Error in standby alert.log:
Tue Jun 29 23:15:35 2010
RFS[258]: Possible network disconnect with primary database
Tue Jun 29 23:15:36 2010
Fetching gap sequence in thread 2, gap sequence 9206-9206
Tue Jun 29 23:16:46 2010
FAL[client]: Failed to request gap sequence
GAP - thread 2 sequence 9206-9206
DBID 661398854 branch 714087609
FAL[client]: All defined FAL servers have been attempted.
Error on primary alert.log:
Tue Jun 29 23:00:07 2010
ARC0: Creating remote archive destination LOG_ARCHIVE_DEST_2: 'WSSPRDB' (thread 1 sequence 9265)
(WSSPRD1)
ARC0: Transmitting activation ID 0x29c37469
Tue Jun 29 23:00:07 2010
Errors in file /u01/app/oracle/admin/WSSPRD/bdump/wssprd1_arc0_14024.trc:
ORA-03135: connection lost contact
FAL[server, ARC0]: FAL archive failed, see trace file.
Tue Jun 29 23:00:07 2010
Errors in file /u01/app/oracle/admin/WSSPRD/bdump/wssprd1_arc0_14024.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Tue Jun 29 23:00:07 2010
ORACLE Instance WSSPRD1 - Archival Error. Archiver continuing.
Tue Jun 29 23:00:41 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
Tue Jun 29 23:00:41 2010
FAL[server, ARC2]: Begin FAL archive (dbid 0 branch 714087609 thread 2 sequence 9206 dest WSSPRDB)
FAL[server, ARC2]: FAL archive failed, see trace file.
Tue Jun 29 23:00:43 2010
Errors in file /u01/app/oracle/admin/WSSPRD/bdump/wssprd1_arc2_14028.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Tue Jun 29 23:00:43 2010
ORACLE Instance WSSPRD1 - Archival Error. Archiver continuing.
Tue Jun 29 23:01:16 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
Tue Jun 29 23:15:01 2010
Thread 1 advanced to log sequence 9267 (LGWR switch)
I have checked the trace files that get spit out but they arent anything meaningful to me as to whats really happening. Snipit of the trace file:
tkcrrwkx: Starting to process work request
tkcrfgli: SRL header: 0
tkcrfgli: SRL tail: 0
tkcrfgli: ORL to arch: 4
tkcrfgli: le# seq thr for bck tba flags
tkcrfgli: 1 359 1 2 0 3 0x0008 ORL active cur
tkcrfgli: 2 358 1 0 1 1 0x0000 ORL active
tkcrfgli: 3 361 2 4 0 0 0x0008 ORL active cur
tkcrfgli: 4 360 2 0 3 2 0x0000 ORL active
tkcrfgli: 5 -- entry deleted --
tkcrfgli: 6 -- entry deleted --
tkcrfgli: 7 -- entry deleted --
tkcrfgli: 8 -- entry deleted --
tkcrfgli: 9 -- entry deleted --
tkcrfgli: 191 -- entry deleted --
tkcrfgli: 192 -- entry deleted --
*** 2010-03-27 01:30:32.603 20998 kcrr.c
tkcrrwkx: Request from LGWR to perform: <startup>
tkcrrcrlc: Starting CRL ARCH check
*** 2010-03-27 01:30:32.603 66085 kcrr.c
Beginning controlfile transaction 0x0x7fffd0b53198 [kcrr.c:20395 (14011)]
*** 2010-03-27 01:30:32.645 66173 kcrr.c
Acquired controlfile transaction 0x0x7fffd0b53198 [kcrr.c:20395 (14024)]
*** 2010-03-27 01:30:32.649 66394 kcrr.c
Ending controlfile transaction 0x0x7fffd0b53198 [kcrr.c:20397]
tkcrrasgn: Checking for 'no FAL', 'no SRL', and 'HB' ARCH process
# HB NoF NoS CRL Name
29 NO NO NO NO ARC0
28 NO YES YES NO ARC1
27 NO NO NO NO ARC2
26 NO NO NO NO ARC3
25 YES NO NO NO ARC4
24 NO NO NO NO ARC5
23 NO NO NO NO ARC6
22 NO NO NO NO ARC7
21 NO NO NO NO ARC8
20 NO NO NO NO ARC9
Thanks.
KristiIt's the network that's messing up; unlikely due to firewall timeout as it waits for 60 minutes and you are switching every 15 minutes. There may be some other network glitch that needs rectified.
In any case - arch file missing/ corrupt / halfway through - FAL setting should have refetched the problematic archive log automatically.
As many had suggested already, the best way to resolve RFS issues I believe is to use real-time apply by configuring standby redo logs. It's very easy to configure it and you can opt for real-time apply even in max-performance mode that you are using right now.
Even though you are maintaining (I guess) 1-1 between primary & standby instances, you can provide both primary instances in fal_server (like fal_server=string1,string2). See if that helps.
lastly, check if you are having simiar issue at other times as well that might be getting rectified automatically as expected.
col message for a80
col time for a20
select message, to_char(timestamp,'dd-mon-rr hh24:mi:ss') time
from v$dataguard_status
where severity in ('Error','Fatal')
order by timestamp;
Cheers. -
Hi Gurus
While Configuring Data Guard for ORacle 10g (10.2.0.4) 64 bits on Windows 2007 Server 64 bits.
I got few questions
1. What is the Default mode of Standby Database?
2. Should we Always Start Physical Standby Database to Recover Missing Redo Archive Log?
SQL> startup mount;
ORACLE instance started.
Total System Global Area 591396864 bytes
Fixed Size 2067496 bytes
Variable Size 163578840 bytes
Database Buffers 419430400 bytes
Redo Buffers 6320128 bytes
Database mounted.
SQL> alter databse recover managed standby database disconnect from session;
Database altered.
3. When there are missing Redo Log Archives e.g.
----On Standby Database--------
SQL> SELECT RESETLOGS_ID,SEQUENCE#,STATUS,ARCHIVED FROM V$ARCHIVED_LOG
2 ORDER BY RESETLOGS_ID,SEQUENCE#;
RESETLOGS_ID SEQUENCE# S ARC
812980008 15 A YES
812980008 16 A YES
812980008 17 A YES
812980008 18 A YES
812980008 19 A YES
812980008 20 A YES
812980008 21 A YES
812980008 22 A YES
812980008 23 A YES
812980008 24 A YES
812980008 25 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 26 A YES
812980008 27 A YES
812980008 28 A YES
812980008 29 A YES
812980008 30 A YES
812980008 31 A YES
812980008 32 A YES
812980008 33 A YES
812980008 34 A YES
812980008 35 A YES
812980008 36 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 37 A YES
812980008 38 A YES
812980008 39 A YES
812980008 40 A YES
812980008 41 A YES
812980008 42 A YES
812980008 43 A YES
29 rows selected.
---------------On Primary Database---------------------
SQL> SELECT RESETLOGS_ID,SEQUENCE#,STATUS,ARCHIVED FROM V$ARCHIVED_LOG
2 ORDER BY RESETLOGS_ID,SEQUENCE# ;
RESETLOGS_ID SEQUENCE# S ARC
*812980008 8 A YES*
*812980008 9 A YES*
*812980008 10 A YES*
*812980008 11 A YES*
*812980008 12 A YES*
*812980008 13 A YES*
*812980008 14 A YES*
812980008 15 A YES
812980008 15 A YES
812980008 16 A YES
812980008 16 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 17 A YES
812980008 17 A YES
812980008 18 A YES
812980008 18 A YES
812980008 19 A YES
812980008 19 A YES
812980008 20 A YES
812980008 20 A YES
812980008 21 A YES
812980008 21 A YES
812980008 22 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 22 A YES
812980008 23 A YES
812980008 23 A YES
812980008 24 A YES
812980008 24 A YES
812980008 25 A YES
812980008 25 A YES
812980008 26 A YES
812980008 26 A YES
812980008 27 A YES
812980008 27 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 28 A YES
812980008 28 A YES
812980008 29 A YES
812980008 29 A YES
812980008 30 A YES
812980008 30 A YES
812980008 31 A YES
812980008 31 A YES
812980008 32 A YES
812980008 32 A YES
812980008 33 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 33 A YES
812980008 34 A YES
812980008 34 A YES
812980008 35 A YES
812980008 35 A YES
812980008 36 A YES
812980008 36 A YES
812980008 37 A YES
812980008 37 A YES
812980008 38 A YES
812980008 38 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 39 A YES
812980008 39 A YES
812980008 40 A YES
812980008 40 A YES
812980008 41 A YES
812980008 41 A YES
812980008 42 A YES
812980008 42 A YES
812980008 43 A YES
812980008 43 A YES
65 rows selected.
Log 8, 9, 10, 11, 12, 13, 14, 15 are missing.
How to Apply / Recover These Logs on Standby Database?
Regards
Thunder2777Hi
Thunder2777 wrote:
Hi Gurus
While Configuring Data Guard for ORacle 10g (10.2.0.4) 64 bits on Windows 2007 Server 64 bits.
I got few questions
1. What is the Default mode of Standby Database?
What is default mode? I think you want ask in what mode standby database is apply redo logs.
Standby database can apply received redo only MOUNT mode, (your version is 10g, after 11g can apply open mode with READ ONLY WITH APPLY)
2. Should we Always Start Physical Standby Database to Recover Missing Redo Archive Log?
If Standby database opened mount mode, then database can receive redo.
If you are start Redo Apply then MPR can request from primary for missing redo logs.
SQL> startup mount;
ORACLE instance started.
Total System Global Area 591396864 bytes
Fixed Size 2067496 bytes
Variable Size 163578840 bytes
Database Buffers 419430400 bytes
Redo Buffers 6320128 bytes
Database mounted.
SQL> alter databse recover managed standby database disconnect from session;
Database altered.
It is started recovery, in other words Redo Apply (MRP0 process)
>
3. When there are missing Redo Log Archives e.g.
----On Standby Database-------->
SQL> SELECT RESETLOGS_ID,SEQUENCE#,STATUS,ARCHIVED FROM V$ARCHIVED_LOG
2 ORDER BY RESETLOGS_ID,SEQUENCE#;
RESETLOGS_ID SEQUENCE# S ARC
812980008 15 A YES
812980008 16 A YES
812980008 17 A YES
812980008 18 A YES
812980008 19 A YES
812980008 20 A YES
812980008 21 A YES
812980008 22 A YES
812980008 23 A YES
812980008 24 A YES
812980008 25 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 26 A YES
812980008 27 A YES
812980008 28 A YES
812980008 29 A YES
812980008 30 A YES
812980008 31 A YES
812980008 32 A YES
812980008 33 A YES
812980008 34 A YES
812980008 35 A YES
812980008 36 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 37 A YES
812980008 38 A YES
812980008 39 A YES
812980008 40 A YES
812980008 41 A YES
812980008 42 A YES
812980008 43 A YES
29 rows selected.
---------------On Primary Database---------------------
SQL> SELECT RESETLOGS_ID,SEQUENCE#,STATUS,ARCHIVED FROM V$ARCHIVED_LOG
2 ORDER BY RESETLOGS_ID,SEQUENCE# ;
RESETLOGS_ID SEQUENCE# S ARC
*812980008 8 A YES*
*812980008 9 A YES*
*812980008 10 A YES*
*812980008 11 A YES*
*812980008 12 A YES*
*812980008 13 A YES*
*812980008 14 A YES*
812980008 15 A YES
812980008 15 A YES
812980008 16 A YES
812980008 16 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 17 A YES
812980008 17 A YES
812980008 18 A YES
812980008 18 A YES
812980008 19 A YES
812980008 19 A YES
812980008 20 A YES
812980008 20 A YES
812980008 21 A YES
812980008 21 A YES
812980008 22 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 22 A YES
812980008 23 A YES
812980008 23 A YES
812980008 24 A YES
812980008 24 A YES
812980008 25 A YES
812980008 25 A YES
812980008 26 A YES
812980008 26 A YES
812980008 27 A YES
812980008 27 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 28 A YES
812980008 28 A YES
812980008 29 A YES
812980008 29 A YES
812980008 30 A YES
812980008 30 A YES
812980008 31 A YES
812980008 31 A YES
812980008 32 A YES
812980008 32 A YES
812980008 33 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 33 A YES
812980008 34 A YES
812980008 34 A YES
812980008 35 A YES
812980008 35 A YES
812980008 36 A YES
812980008 36 A YES
812980008 37 A YES
812980008 37 A YES
812980008 38 A YES
812980008 38 A YES
RESETLOGS_ID SEQUENCE# S ARC
812980008 39 A YES
812980008 39 A YES
812980008 40 A YES
812980008 40 A YES
812980008 41 A YES
812980008 41 A YES
812980008 42 A YES
812980008 42 A YES
812980008 43 A YES
812980008 43 A YES
65 rows selected.
Log 8, 9, 10, 11, 12, 13, 14, 15 are missing.It is no missing, you are created standby database, after sequence 15.
As you know , if a sequence redo is not applied, then after is sequenced redo log is cannot apply to standby database.
It means GAP.
There have 43 archived redo log, and your last sequenced archive log received by standby database
and applied.
You can check with following scripts, too
select max(Sequence#) from v$archived_log; -- on primary
select max(Sequence#) from v$archived_log where applied = 'YES'; - on standby Regards
Mahir M. Quluzade -
Logical standby | archive log deleted | how to remove gap ???
hi gurus...
i have problem on logical standby
by mistake standby log coming to logical standby has been deleted , now how to fill up the gap ???
ON STANDBY
SEQUENCE# FIRST_CHANGE# NEXT_CHANGE# APPLIED
228 674847 674872 YES
229 674872 674973 CURRENT
230 674973 674997 NO
231 674997 675023 NO
232 675023 675048 NO
233 675048 675109 NO
234 675109 675135 NO
235 675135 675160 NO
236 675160 675183 NO
237 675183 675208 NO
238 675208 675232 NO
239 675232 675257 NO
240 675257 675282 NO
241 675282 675382 NO
242 675382 675383 NO
243 675383 675650 NO
244 675650 675652 NO
245 675652 675670 NO
246 675670 675688 NO
247 675688 675791 NO
248 675791 678524 NO
archive log are shipping to standby location also and getting registered
ALERT LOG OF STANDBY
Fri May 7 12:25:36 2010
Primary database is in MAXIMUM PERFORMANCE mode
RFS[21]: Successfully opened standby log 5: '/u01/app/oracle/oradata/BEST/redo05.log'
Fri May 7 12:25:37 2010
RFS LogMiner: Registered logfile [u01/app/oracle/flash_recovery_area/BEST/archivelog/archBEST_248_1_715617824.dbf] to LogMiner session id [1]
but i dont have standby log after 229 sequence ...
ON PRIMARY
SYS@TEST AS SYSDBA> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /u01/app/oracle/flash_recovery_area/TEST/standlogOldest online log sequence 247
Next log sequence to archive 249
Current log sequence 249
what to do next to apply sequences and bring both in sync.
please help me ,,,,
Edited by: user12281508 on May 7, 2010 9:45 AMthanks for response.
no its pure logical standby
i have tried to ftp the archive logs of primary to standby and applied manually
SYS@BEST AS SYSDBA> alter database register logfile '/u01/app/oracle/flash_recovery_area/BEST/archivelog/archBEST_230_1_715617824.dbf';
alter database register logfile '/u01/app/oracle/flash_recovery_area/BEST/archivelog/archBEST_230_1_715617824.dbf'
ERROR at line 1:
ORA-01289: cannot add duplicate logfile
SYS@BEST AS SYSDBA> alter database register logfile '/u01/app/home/archTEST_230_1_715617824.dbf';
alter database register logfile '/u01/app/home/archTEST_230_1_715617824.dbf'
ERROR at line 1:
ORA-01289: cannot add duplicate logfile
any other way ???? -
How to find out which archived logs needed to recover a hot backup?
I'm using Oracle 11gR2 (11.2.0.1.0).
I have backed up a database when it is online using the following backup script through RMAN
connect target /
run {
allocate channel d1 type disk;
backup
incremental level=0 cumulative
filesperset 4
format '/san/u01/app/backup/DB_%d_%T_%u_%c.rman'
database
}The backup set contains the backup of datafiles and control file. I have copied all the backup pieces to another server where I will restore/recover the database but I don't know which archived logs are needed in order to restore/recover the database to a consistent state.
I have not deleted any archived log.
How can I find out which archived logs are needed to recover the hot backup to a consistent state? Can this be done by querying V$BACKUP_DATAFILE and V$ARCHIVED_LOG? If yes, which columns should I query?
Thanks for any help.A few ways :
1a. Get the timestamps when the BACKUP ... DATABASE began and ended.
1b. Review the alert.log of the database that was backed up.
1c. From the alert.log identify the first Archivelog that was generated after the begin of the BACKUP ... DATABASE and the first Archivelog that was generated after the end of the BACKUP .. DATABASE.
1d. These (from 1c) are the minimal Archivelogs that you need to RECOVER with. You can choose to apply additional Archivelogs that were generated at the source database to contininue to "roll-forward"
2a. Do a RESTORE DATABASE alone.
2b. Query V$DATAFILE on the restored database for the lowest CHECKPOINT_CHANGE# and CHECKPOINT_TIME. Also query for the highest CHECKPOINT_CHANGE# and CHECKPOINT_TIME.
2c. Go back to the source database and query V$ARCHIVED_LOG (FIRST_CHANGE#) to identify the first Archivelog that has a higher SCN (FIRST_CHANGE#) than the lowest CHECKPOINT_CHANGE# from 2b above. Also query for the first Archivelog that has a higher SCN (FIRST_CHANGE#) than the highest CHECKPOINT_CHANGE# from 2b above.
2d. These (from 2c) are the minimal Archivelogs that you need to RECOVER with.
(why do you need to query V$ARCHIVED_LOG at the source ? If RESTORE a controlfile backup that was generated after the first Archivelog switch after the end of the BACKUP ... DATABASE, you would be able to query V$ARCHIVED_LOG at the restored database as well. That is why it is important to force an archivelog (log switch) after a BACKUP ... DATABASE and then backup the controlfile after this -- i.e. last. That way, the controlfile that you have restored to the new server has all the information needed).
3. RESTORE DATABASE PREVIEW in RMAN if you have the archivelogs and subsequent controlfile in the backup itself !
Hemant K Chitale -
How can I set destination for archived logs?
I would like to know:
how to set destination for archived logs?
how to identify the init.ora that is used for my database?
With rman using compressed backupset by default and and making
backup database;
What does it backup exactly?Another thing I am wondering, when I make a backup with rman : backup database.
It saves the backups in the directory autobackup from the flash_recovery_area but it seems that it only saves the data files and the control files.Isn't there a way to sava archived logs files, control files, datafiles in a single backup?
In fact I would like to make a full backup using rman on sunday of everything and a incremental backup all days of the week how can I acomplish this with a retention of 7 days? -
Status of db backup taken after 2 logs archived went missing
Hi ,
what would be the effect or issues of the full db backup & archivelog backup of 9i db taken after 2 archive logs missing that were never backed up, since backup, or crosscheck did't alerted any error messages. will this be still a valid backup to restore or recovery in case.
posting the backup script details;
note:we are not using catalog repository
run {
allocate channel dev1 type disk format 'Z:\Rbackup\DMP\db_t%t_s%s_p%p';
allocate channel dev2 type disk format 'Z:\Rbackup\DMP\db_t%t_s%s_p%p';
backup database;
backup archivelog all;
delete noprompt archivelog until time 'sysdate-1' ;
crosscheck archivelog all;
release channel dev1 ;
release channel dev2 ;
}logs missing date is Jan 14th and no backups were taken on this day. In that case, the "BACKUP ARCHIVELOG ALL" on the 15th would have generated an error flagging the missing archivelogs because the "ALL" specification is a direction to backup all known ArchiveLogs that are available for backup. The only reasons why it wouldn't have flagged the missing ArchiveLogs would be :
a. You have another job that deletes archivelogs and this one ran before the backup on 15th Jan
b. The ArchiveLogs are in the FRA and Oracle had automatically purged ArchiveLogs when the FRA was running low on free space.
The Backup on 15th is good for restore and recovery. Use the "VALIDATE" command in RMAN to validate the database and archivelogs.
See http://hemantoracledba.blogspot.com/2010/03/misinterpreting-restore-database.html
However, as two archivelogs are missing and not backed up, you will not be able to do a Point In Time Recovery to any point in time from the time of the first missing archivelog to the end of the backup of the 15th.
Hemant K Chitale -
How to find thead 2 archive log when i recover from 2-node RAC to single
I backup 2-ndoe RAC and restore to single - node
Control file created.
SQL> recover database iuckup controlfile
SQL> recover database using backup controlfIle;
ORA-00279: change 12100176131169 generated at 07/06/2013 16:36:57 needed for
thread 1
ORA-00289: suggestion : /arch/hop1_566085708_1_212692.dat -- Oracle suggest
ORA-00280: change 12100176131169 for thread 1 is in sequence #212692
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
arch/sch1_566085708_1_212692.dat --- I input
ORA-00279: change 12100176131169 generated at needed for thread 2
==> Oracle didn't suggest thread 2 archive log file/
How can I find ?You have to query V$ARCHIVED_LOG for THREAD# 2 on the source database to find the first archivelog where 12100176131169 is less than the NEXT_CHANGE#
select SEQEUNCE# from V$ARCHIVED_LOG where THREAD#=2 and 12100176131169 between FIRST_CHANGE# and NEXT_CHANGE#
Once you provide the Sequence# and file name for the first Thread 2 file, the RECOVER command will automatically generate the expected file name for the subsequent archivelogs (of both threads).
Hemant K Chitale -
Archive Log Gap between Disaster recovery server & Production Server
Hi
Can anybody provide me the script to find out the archive log missing on Production Server means that archive log backed up & deleted from the production server & rest of the Archive logs has been shipped on to the DR server. Means it will shot the gap between production server & DR server until that log which has been deleted from the production server , but next logs has been shipped to the DR server. But rest of the logs only be applied when delete log be restored on the production server & shipped on the DR server.
thanks.
Regards
Ravi Kant AryaRavi,
The question is how are you getting the archive logs shipped over to your standby database? If you are doing it via FTP or scripts, then you may want to look at configuring DataGuard to do it for you. Look at "Setting Up Oracle Dataguard for SAP" at:
http://www.oracle.com/us/solutions/sap/wp-ora4sap-dataguard11g-303811.pdf
Good Luck.
Mike Kennedy -
RMAN-6089: archive log not found or out of sync with catalog
Hi:
I am using OEM, not RMAN, to do a DB backup and got the above error message. I think the problem is caused by some archive logs missing from my NT machine.
I tried to manually copy from an existing archive log and rename it to the missing archive log but it didn't work.
I then tried to backup the archive logs only, not whole DB, and specified "delete input" option, and the process completed successfully.
However, when I tried to re-do the whole DB backup, it failed with the same error.
I looked at v$archived_log and I can see all archive logs, including the ones from previous incarnation and the missing ones from the current incarnation. How can I delete them or change their status (from OEM not from RMAN) from not deleted (NO) to deleted (YES) in order to perform a DB backup?Hi,
I'm using controlfile for rman. I've issued "change archivelog all validate" before a full backup Before the full backup, I had a incremental backup process crashed(don't know why), which left some entres in v$archived_log. I have no problem completing the rman backup, but "restore archivelog all validate" command gives me "falure of restore..." and all those seq can be found in v$archived_log with "yes" for deleted column. Does that mean controlfile is still not in sync with the system? I don't need those target seq because those were before the full backup. Will there be potential problem later on during recovery? How do I (or when Oracle server will) get rid of those entries in v$archived_log? Thanks.
RMAN> restore archivelog all validate;
Starting restore at 11-AUG-05
using target database controlfile instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=28 devtype=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 08/11/2005 10:20:24
RMAN-06026: some targets not found - aborting restore
RMAN-06025: no backup of log thread 1 seq 1048 scn 122640071 found to restore
RMAN-06025: no backup of log thread 1 seq 1047 scn 122572176 found to restore
RMAN-06025: no backup of log thread 1 seq 1046 scn 122431604 found to restore
SQL> select name, completion_time,deleted from v$archived_log;
NAME COMPLETIO DEL
/u04/arch/arch_1_1046.arc 10-AUG-05 YES
/u04/arch/arch_1_1047.arc 10-AUG-05 YES
/u04/arch/arch_1_1048.arc 11-AUG-05 YES
/u04/arch/arch_1_1049.arc 11-AUG-05 YES
/u04/arch/arch_1_1050.arc 11-AUG-05 YES -
Duplexing the Archive Log - is there a potential performance hit
Good Afternoon Oracle People -
I apologize if this question is silly or out of place.
Basically, we are looking at options for implementing a (cheap) DR solution for our Oracle Database.
Bottom line objective is to have a second copy of our production system (not running, offline) with a usable archive log to recover from at a remote site with a similar set of disk technologies etc.
The sites are linked by 100MBPs link - so any access to secondary site is relatively fast.
What I was thinking of doing is creating an iSCSI target on the destination SAN and adding this as a disk into the production database. Then, I was going to go in and define a duplex archive log destination to this iSCSI target.
My fear is that if the archiver waits for the write to the destination, I would imagine that this would slow down database access. Is this a valid concern? Or does Oracle treat the Duplex as a replicated (slower) copy?
Again, sorry if this question is stupid - i have tried the google machine but couldn't find anything, and it isn't all that clear from the documentation.
Kind Regards,
AlekseiOr does Oracle treat the Duplex as a replicated (slower) copy?
Oracle treats it in a way you tell Oracle to treat it – please take a look at LOG_ARCHIVE_DEST_n parameters such as MANDATORY and OPTIONAL, MAX_FAILURE, etc
There are two major things to consider:
->Is the throughput of the destination enough to handle max archive log generation?
You need to take a look at the size of archive logs and how frequently they are
generated during peak load.
->What happens when the destination is not available?
HTH,
Iordan Iotzov -
Backup archive logs problem using RMAN
Hi guys
I got failure when using RMAN to backup archive log files:
Starting backup at 20-APR-06
current log archived
released channel: c1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 04/20/2006 21:53:57
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file /opt/oracle/flash_recovery_area/DB10G/archivelog/2006_03_17/o1_mf_1_1_21p5c251_.arc
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
RMAN> **end-of-file**
My archive log files location:
SQL> show parameter log_archive_dest
NAME TYPE VALUE
log_archive_dest_1 string LOCATION=/opt/oracle/oradata/DB10G/arch/
Current archive log files:
$ ls /opt/oracle/oradata/DB10G/arch
1_17_586191737.dbf 1_21_586191737.dbf 1_25_586191737.dbf 1_29_586191737.dbf 1_33_586191737.dbf
1_18_586191737.dbf 1_22_586191737.dbf 1_26_586191737.dbf 1_30_586191737.dbf 1_34_586191737.dbf
1_19_586191737.dbf 1_23_586191737.dbf 1_27_586191737.dbf 1_31_586191737.dbf 1_35_586191737.dbf
1_20_586191737.dbf 1_24_586191737.dbf 1_28_586191737.dbf 1_32_586191737.dbf afiedt.buf
$
But when I check v$archived_log:
SQL> select name,status,deleted from v$archived_log;
NAME S DEL
D YES
D YES
D YES
D YES
D YES
D YES
D YES
D YES
D YES
D YES
D YES
/opt/oracle/flash_recovery_area/DB10G/archivelog/2006_03_17/ A NO
/opt/oracle/flash_recovery_area/DB10G/archivelog/2006_03_18/ A NO
/opt/oracle/flash_recovery_area/DB10G/archivelog/2006_03_19/ A NO
/opt/oracle/oradata/DB10G/redo01.log A NO
/opt/oracle/oradata/DB10G/redo02.log A NO
/opt/oracle/oradata/DB10G/redo03.log A NO
/opt/oracle/oradata/DB10G/arch/1_5_585926175.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_6_585926175.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_7_585926175.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_1_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_2_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_3_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_4_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_5_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_6_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_7_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_8_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_9_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_10_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_11_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_12_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_13_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_14_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_15_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_16_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_17_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_18_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_19_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_20_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_21_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_22_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_23_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_24_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_25_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_26_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_27_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_28_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_29_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_30_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_31_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_32_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_33_586191737.dbf A NO
/opt/oracle/oradata/DB10G/arch/1_34_586191737.dbf A NO
More records than actual archived logs. How could it happen? How to solve?
THanks in advance.
SharonHi,
Use
RMAN>crosscheck archivelog all; cmd -
Hi,
In our database suddenly we are seeing the rapid growth of archive logs.
How to find which session is causing the growth,
thanks in advacne.
Regards
JMFound Re: How to check user/session generating a lot of archivelogs? using this
Maybe you are looking for
-
Help - Downloading the par file into NWDS
Hi, Can anyone please tell me how to download the par file in NWDS to customize iviews. What is the pre-requisites? Is there any step by step procedure available to download the EP iViews and put our codes into that. Regards, Nirmal
-
Right click context menu is not working in flash player 10 and above
In right click custom context menu i have create like "A" if i click "A" i have attached one movie clip in that movie clip right click, i have custom context menu like "Remove A" this is working fine in flash player 9 and below. But flash player 10 a
-
My account can't buy adobe cloud photographer.
That error is "There was a problem processing your order, please contact our Customer Service team for assistance." Help me please.
-
ok, so if anyone can help me I would greatly appreciate it. I have a "notification" that goes off every 5 minutes...there is no notification sign to let me know what it is...I have shut off everything I can think of and it is driving me crazy! Any
-
How do I get Illustrator to save out flat PDF/X-1a:2001 files again?
Suddenly when I save out a PDF/X-1a:2001 Acrobat 4 (PDF 1.3) it remains layered and keeps fonts editable. It no longer flattens the file. No other Acrobat version that I select flattens a file either. I'm on a PC with Illustrator CC 2014. Any ideas?