Oracle 3113 creating a standby with rman

Question

Hello,
environment is 11gR2 on Linux.
Tested on CentOS 5.5 as OS and local LAN as interconnect and the duplicate command terminates successfully in some minutes.
Same script but using oel 5.5 and a wan (3Mbit/s) and I keep getting 3113 errors after about 2h or 3h.
I suspect some timeout on wan (that is over vpn) but I would also like to trace something at my side.
I run a shell script with nohup and send it to background and this is the log, from which you can see the rman script:
Recovery Manager: Release 11.2.0.1.0 - Production on Tue Sep 7 14:50:11 2010
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
RMAN>
connected to target database: TESDB1 (DBID=11492743)
RMAN>
connected to auxiliary database: DUMMY (not mounted)
RMAN>
RMAN> run{
2> allocate channel prmy1 type disk;
3> allocate auxiliary channel stby1 type disk;
4> duplicate target database for standby from active database
5> spfile
6> parameter_value_convert 'TESDB1','TESDB1DR'
7> set db_file_name_convert='/TESDB1/','/TESDB1DR/'
8> set log_file_name_convert='/TESDB1/','/TESDB1DR/'
9> set 'db_unique_name'='TESDB1DR'
10> set control_files='/app/oradata/TESDB1DR/control01.ctl','/app/oradata/flash_recovery_area/TESDB1DR/control02.ctl'
11> set db_recovery_file_dest='/app/oradata/flash_recovery_area'
12> set DB_RECOVERY_FILE_DEST_SIZE='4977M'
13> nofilenamecheck;
14> }
using target database control file instead of recovery catalog
allocated channel: prmy1
channel prmy1: SID=12 device type=DISK
allocated channel: stby1
channel stby1: SID=135 device type=DISK
Starting Duplicate Db at 07-SEP-10
executing Memory Script
executing command: SET NEWNAME
renamed tempfile 1 to /app/oradata/TESDB1DR/temp01.dbf in control file
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
Starting backup at 07-SEP-10
channel prmy1: starting datafile copy
input datafile file number=00003 name=/app/oradata/TESDB1/undotbs01.dbf
.. after about three hours (and 3.2Gb transferred) it continues with:
RMAN-03009: failure of backup command on prmy1 channel at 09/07/2010 17:53:49
RMAN-06136: ORACLE error from auxiliary database: ORA-03113: end-of-file on communication channel
Process ID: 3187
Session ID: 192 Serial number: 3
continuing other job steps, job failed will not be re-run
.. after 15 minutes:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 09/07/2010 18:09:35
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of backup command on clone_default channel at 09/07/2010 18:09:34
RMAN-10038: database session for channel clone_default terminated unexpectedly
RMAN> **end-of-file**
RMAN>
Recovery Manager complete.
the undo tbs is 3.2 Gb so I'm not sure if the 3113 happens at the end of copy... but the tablespace doesn't succeed
and the other jobs don't start even if it says "continuing other job steps"..
rman started at 14:50 and in listener at dr site I have:
Tue Sep 07 14:50:55 2010
07-SEP-2010 14:50:55 * service_update * TESDB1DR * 0
07-SEP-2010 14:50:55 * (CONNECT_DATA=(SERVICE_NAME=TESDB1DR)(CID=(PROGRAM=oracle)(HOST=lb-orajde.boiron.ntm)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.49.1.93)(PORT=35240)) * establish * TESDB1DR * 0
Tue Sep 07 15:00:58 2010
07-SEP-2010 15:00:58 * service_update * TESDB1DR * 0
Tue Sep 07 16:35:04 2010
07-SEP-2010 16:35:04 * service_update * TESDB1DR * 0
Tue Sep 07 17:02:04 2010
07-SEP-2010 17:02:04 * service_update * TESDB1DR * 0
07-SEP-2010 17:02:10 * service_update * TESDB1DR * 0
Tue Sep 07 17:53:17 2010
07-SEP-2010 17:53:17 * service_update * TESDB1DR * 0
Tue Sep 07 18:09:00 2010
07-SEP-2010 18:09:00 * (CONNECT_DATA=(SERVICE_NAME=TESDB1DR)(CID=(PROGRAM=rman)(HOST=lb-orajde.boiron.ntm)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.49.1.93)(PORT=32610)) * establish * TESDB1DR * 0
In alert.log at dr site I have at 17:02 (about 50 minutes before error):
Tue Sep 07 15:05:01 2010
db_recovery_file_dest_size of 4977 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Tue Sep 07 17:02:03 2010
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 07-SEP-2010 17:02:03
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505
TNS-00505: Operation timed out
nt secondary err code: 110
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.49.1.93)(PORT=35226))
some times (always at 17:02) and nothing more
In alert.log at primary site:
Tue Sep 07 14:50:36 2010
Clearing standby activation ID 11529351 (0xafec87)
The primary database controlfile was created using the
'MAXLOGFILES 16' clause.
There is space for up to 13 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 52428800;
Tue Sep 07 16:00:40 2010
Thread 1 advanced to log sequence 759 (LGWR switch)
Current log# 3 seq# 759 mem# 0: /app/oradata/TESDB1/redo03.log
Tue Sep 07 16:00:41 2010
Archived Log entry 528 added for thread 1 sequence 758 ID 0xafec87 dest 1:
In sqlnet.log of primary site:
Tue Sep 07 17:53:49 2010
Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Linux: Version 11.2.0.1.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
Time: 07-SEP-2010 17:53:49
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505
TNS-00505: Operation timed out
nt secondary err code: 110
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.12.41)(PORT=1521))
and the same at 18:09
No other errors.
Any hints?
BTW: what is the minimum size of undo tbs? Can I have compression enabled also during duplication if transport service is configured with COMPRESSION=ENABLED, or does it count only when copying redo files?
Thanks in advance,
Gianluca

Answer 1

Thanks for your reply.
I will try your suggestion as it could come useful some time.
So with this alternate method at the end I would have standby aligned as of backup time, correct?
And after running something like this on standby:
SQL> alter database recover managed standby database disconnect from session;
It will begin to receive, archive and apply redo logs, correct?
Some answers regarding the original question. It was confirmed that there was a 2hours cut-off time at firewall level.
The first datafile that the rman session began to copy was 3.2Gb in size, taking more than 2 hours completing.
Monitoring with netstat on source and target I saw this at source during the duplicate process:
tcp 0 0 10.49.1.93:46626 192.168.12.41:1521 ESTABLISHED
tcp 0 0 10.49.1.93:46627 192.168.12.41:1521 ESTABLISHED
tcp 0 0 10.49.1.93:46629 192.168.12.41:1521 ESTABLISHED
tcp 0 369240 10.49.1.93:50371 192.168.12.41:1521 ESTABLISHED
and the session bound to port 50371 was the only one pushing data (send queue not empty).
The other 3 ones I presume are for control of rman session. I get 3 sessions even if I open only one channel on primary side.
After the first datafile completes copy, a new session (with port 14871) is established substituting the other one.
The three initial ones instead remain for all the duplicate process.
tcp 0 487872 10.49.1.93:14871 192.168.12.41:1521 ESTABLISHED
tcp 0 0 10.49.1.93:46626 192.168.12.41:1521 ESTABLISHED
tcp 0 0 10.49.1.93:46627 192.168.12.41:1521 ESTABLISHED
tcp 0 0 10.49.1.93:46629 192.168.12.41:1521 ESTABLISHED
So after the copy of the first file, the session come back to one or more of the initial 3 ones, and they probably make a push-ack, got a 3113 as the firewall has silently cut them after 2 hours.... and the datafile, even if completed, is not marked as completed. A new run would copy again it...
Now the time has been put to 8hours, with 24hours being the maximum value set (no disable possible as far as the network guy said...).
And the duplicate completed correctly.
Would I have to fear any problems with this cut-off due to "normal" dataguard operations or not, in your opinion?
Thanks
Gianluca