RMAN-backup slow perform

I have one 10g SE database running on Linux x86_64; I have migrated this database resently with upgrade;
I used export-import utilities for perform this task;
Some information about this database:
OS version: "Linux xxx.qqq.ru 2.6.18-194.3.1.el5 #1 SMP Sun May 2 04:17:42 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux"
SQL> select * from v$version;
BANNER
Oracle Database 10g Release 10.2.0.4.0 - 64bit Production
PL/SQL Release 10.2.0.4.0 - Production
CORE 10.2.0.4.0 Production
TNS for Linux: Version 10.2.0.4.0 - Production
NLSRTL Version 10.2.0.4.0 - Production
I noticed that rman perform backup operation too long; For example here I place piece of rman log from one database backup operation:
executing script: backup_db_nfs
allocated channel: nfs
channel nfs: sid=1275 devtype=DISK
Starting backup at 31-07-10 04:00:04
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00028 name=/db/u12/oradata/billing/excellent_02.dbf
input datafile fno=00030 name=/db/u12/oradata/billing/excellent_04.dbf
input datafile fno=00032 name=/db/u12/oradata/billing/excellent_06.dbf
input datafile fno=00034 name=/db/u12/oradata/billing/excellent_08.dbf
input datafile fno=00029 name=/db/u11/oradata/billing/excellent_03.dbf
input datafile fno=00031 name=/db/u11/oradata/billing/excellent_05.dbf
input datafile fno=00033 name=/db/u11/oradata/billing/excellent_07.dbf
input datafile fno=00035 name=/db/u11/oradata/billing/excellent_09.dbf
channel nfs: starting piece 1 at 31-07-10 04:00:05
channel nfs: finished piece 1 at 31-07-10 04:34:21
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2615.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: starting piece 2 at 31-07-10 04:34:21
channel nfs: finished piece 2 at 31-07-10 05:08:36
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2615.2.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: starting piece 3 at 31-07-10 05:08:36
channel nfs: finished piece 3 at 31-07-10 07:38:43
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2615.3.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 03:38:38
channel nfs: throttle time: 3:###:42
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00036 name=/db/u12/oradata/billing/excellent_10.dbf
input datafile fno=00038 name=/db/u12/oradata/billing/excellent_12.dbf
input datafile fno=00040 name=/db/u12/oradata/billing/excellent_14.dbf
input datafile fno=00042 name=/db/u12/oradata/billing/excellent_16.dbf
input datafile fno=00037 name=/db/u11/oradata/billing/excellent_11.dbf
input datafile fno=00039 name=/db/u11/oradata/billing/excellent_13.dbf
input datafile fno=00041 name=/db/u11/oradata/billing/excellent_15.dbf
input datafile fno=00043 name=/db/u11/oradata/billing/excellent_17.dbf
channel nfs: starting piece 1 at 31-07-10 07:38:43
channel nfs: finished piece 1 at 31-07-10 08:13:18
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2636.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: starting piece 2 at 31-07-10 08:13:18
channel nfs: finished piece 2 at 31-07-10 08:47:24
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2636.2.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: starting piece 3 at 31-07-10 08:47:24
channel nfs: finished piece 3 at 31-07-10 11:17:20
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2636.3.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 03:38:37
channel nfs: throttle time: 3:###:59
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00044 name=/db/u12/oradata/billing/excellent_18.dbf
input datafile fno=00046 name=/db/u12/oradata/billing/excellent_20.dbf
input datafile fno=00047 name=/db/u12/oradata/billing/index_all_02.dbf
input datafile fno=00049 name=/db/u12/oradata/billing/index_all_04.dbf
input datafile fno=00045 name=/db/u11/oradata/billing/excellent_19.dbf
input datafile fno=00048 name=/db/u11/oradata/billing/index_all_03.dbf
input datafile fno=00050 name=/db/u11/oradata/billing/index_all_05.dbf
input datafile fno=00052 name=/db/u11/oradata/billing/index_all_07.dbf
channel nfs: starting piece 1 at 31-07-10 11:17:20
channel nfs: finished piece 1 at 31-07-10 12:16:46
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2655.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: starting piece 2 at 31-07-10 12:16:46
channel nfs: finished piece 2 at 31-07-10 14:55:53
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2655.2.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 03:38:33
channel nfs: throttle time: 3:###:02
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00051 name=/db/u12/oradata/billing/index_all_06.dbf
input datafile fno=00053 name=/db/u12/oradata/billing/index_all_08.dbf
input datafile fno=00055 name=/db/u12/oradata/billing/index_all_10.dbf
input datafile fno=00057 name=/db/u12/oradata/billing/index_all_12.dbf
input datafile fno=00054 name=/db/u11/oradata/billing/index_all_09.dbf
input datafile fno=00056 name=/db/u11/oradata/billing/index_all_11.dbf
input datafile fno=00058 name=/db/u11/oradata/billing/index_all_13.dbf
input datafile fno=00060 name=/db/u11/oradata/billing/index_all_15.dbf
channel nfs: starting piece 1 at 31-07-10 14:55:53
channel nfs: finished piece 1 at 31-07-10 18:34:30
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2674.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 03:38:37
channel nfs: throttle time: 3:###:44
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00019 name=/db/u12/oradata/billing/excellent_big_02.dbf
input datafile fno=00021 name=/db/u12/oradata/billing/excellent_big_04.dbf
input datafile fno=00068 name=/db/u12/oradata/billing/support_excl_04.dbf
input datafile fno=00024 name=/db/u11/oradata/billing/excellent_big_07.dbf
input datafile fno=00026 name=/db/u11/oradata/billing/excellent_big_09.dbf
input datafile fno=00012 name=/db/u13/oradata/billing/pay_assist_01.dbf
input datafile fno=00004 name=/db/u13/oradata/billing/users_01.dbf
input datafile fno=00023 name=/db/u12/oradata/billing/excellent_big_06.dbf
input datafile fno=00025 name=/db/u12/oradata/billing/excellent_big_08.dbf
input datafile fno=00027 name=/db/u12/oradata/billing/excellent_big_10.dbf
channel nfs: starting piece 1 at 31-07-10 18:34:30
channel nfs: finished piece 1 at 31-07-10 21:59:27
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2695.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 03:24:57
channel nfs: throttle time: 3:###:13
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00059 name=/db/u12/oradata/billing/index_all_14.dbf
input datafile fno=00061 name=/db/u12/oradata/billing/index_all_16.dbf
input datafile fno=00063 name=/db/u12/oradata/billing/index_all_18.dbf
input datafile fno=00065 name=/db/u12/oradata/billing/index_all_20.dbf
input datafile fno=00062 name=/db/u11/oradata/billing/index_all_17.dbf
input datafile fno=00064 name=/db/u11/oradata/billing/index_all_19.dbf
input datafile fno=00015 name=/db/u12/oradata/billing/monitor_01.dbf
channel nfs: starting piece 1 at 31-07-10 21:59:27
channel nfs: finished piece 1 at 01-08-10 00:50:14
piece handle=/db/backup/billing/rman_nfs/full_20100731.BILLING.2712.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 02:50:47
channel nfs: throttle time: 2:###:07
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00002 name=/db/u12/oradata/billing/undotbs_01.dbf
input datafile fno=00011 name=/db/u12/oradata/billing/index_all_01.dbf
input datafile fno=00009 name=/db/u12/oradata/billing/mviewlog_01.dbf
input datafile fno=00008 name=/db/u11/oradata/billing/support_excl_01.dbf
input datafile fno=00010 name=/db/u11/oradata/billing/excellent_01.dbf
input datafile fno=00067 name=/db/u11/oradata/billing/support_excl_03.dbf
input datafile fno=00007 name=/db/u11/oradata/billing/tpcctab_01.dbf
input datafile fno=00013 name=/db/u11/oradata/billing/tpchtab_01.dbf
input datafile fno=00005 name=/db/u13/oradata/billing/web_01.dbf
input datafile fno=00006 name=/db/u13/oradata/billing/alien_users_01.dbf
channel nfs: starting piece 1 at 01-08-10 00:50:14
channel nfs: finished piece 1 at 01-08-10 02:08:00
piece handle=/db/backup/billing/rman_nfs/full_20100801.BILLING.2729.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: starting piece 2 at 01-08-10 02:08:00
channel nfs: finished piece 2 at 01-08-10 03:40:46
piece handle=/db/backup/billing/rman_nfs/full_20100801.BILLING.2729.2.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 02:50:32
channel nfs: throttle time: 2:###:12
channel nfs: starting full datafile backupset
channel nfs: specifying datafile(s) in backupset
input datafile fno=00016 name=/db/u12/oradata/billing/excellent_big_01.dbf
input datafile fno=00017 name=/db/u12/oradata/billing/monitor_lob_01.dbf
input datafile fno=00066 name=/db/u12/oradata/billing/support_excl_02.dbf
input datafile fno=00020 name=/db/u11/oradata/billing/excellent_big_03.dbf
input datafile fno=00022 name=/db/u11/oradata/billing/excellent_big_05.dbf
input datafile fno=00069 name=/db/u11/oradata/billing/support_excl_05.dbf
input datafile fno=00001 name=/db/u13/oradata/billing/system_01.dbf
input datafile fno=00003 name=/db/u13/oradata/billing/sysaux_01.dbf
input datafile fno=00018 name=/db/u13/oradata/billing/excl_addition_01.dbf
input datafile fno=00014 name=/db/u13/oradata/billing/arm_xml_01.dbf
channel nfs: starting piece 1 at 01-08-10 03:40:46
channel nfs: finished piece 1 at 01-08-10 06:02:03
piece handle=/db/backup/billing/rman_nfs/full_20100801.BILLING.2742.1.1 tag=BACKUP_DB_NFS comment=NONE
channel nfs: backup set complete, elapsed time: 02:21:17
channel nfs: throttle time: 2:###:51
Finished backup at 01-08-10 06:02:03
Size of this database:
SQL> select sum(user_bytes)/1024/1024/1024 from dba_data_files;
SUM(USER_BYTES)/1024/1024/1024
456.025085
I have another database that runs on the same hardware platform, eith the same configuration, and under the same OS. and oracle instance has the same configuration (except oracle version: there is 9.2.0.8);
Size of this, database is ~700Gb, and this database load is heavier that database with slow rman;
But on this database rman makes full-db backup in 10 hours, usually;
I read note 360443.1, and I checked - how long rman perform full database backup with validate option;
This time, practicaly, is equal database backup time ~24 hours;
Almost all database datafiles are splaced on disk array - stripe 10 and /db/u11 and /db/u12 - are mount points of two partitions created on this disk array;
/db/u13 - this is raid 5;
Now I wish to know in details - where time is spent and my question is: can anybody suggest me - what should I do for find it;

Instance configuration
audit_file_dest      /var/log/oracle/billing/audit
audit_sys_operations      TRUE
audit_trail      OS
background_dump_dest      /var/log/oracle/billing/bdump
compatible      10.2.0.3.0
control_files      /db/u11/oradata/billing/ctrl00.ctl, /db/u12/oradata/billing/ctrl01.ctl, /db/u00/oradata/billing/ctrl02.ctl, /db/u13/oradata/billing/ctrl03.ctl
core_dump_dest      /var/log/oracle/billing/cdump
db_block_checking      TRUE
db_block_checksum      TRUE
db_block_size      8192
db_cache_advice      ON
db_cache_size      24696061952
db_file_multiblock_read_count      64
db_keep_cache_size      3221225472
db_name      billing
db_recycle_cache_size      3221225472
db_writer_processes      4
disk_asynch_io      FALSE
dispatchers      (protocol=tcp)(listener=mts_1522)(dispatchers=2), (protocol=tcp)(listener=mts_1523)(dispatchers=2)
filesystemio_options      DIRECTIO
global_names      FALSE
java_pool_size      1342177280
job_queue_processes      10
large_pool_size      536870912
lock_sga      TRUE
log_archive_dest      /db/archive/billing
log_archive_format      %T_%S_%r.arclog
log_buffer      144326144
log_checkpoint_interval      10000
log_checkpoints_to_alert      TRUE
log_checkpoint_timeout      0
max_dispatchers      5
max_dump_file_size      100M
max_shared_servers      350
nls_date_format      DD.MM.RR
nls_language      AMERICAN
nls_numeric_characters      .,
nls_territory      RUSSIA
open_cursors      1500
open_links      17
open_links_per_instance      34
optimizer_index_caching      90
optimizer_index_cost_adj      15
optimizer_mode      RULE
pga_aggregate_target      24696061952
processes      1200
query_rewrite_enabled      TRUE
remote_login_passwordfile      EXCLUSIVE
resource_limit      TRUE
session_cached_cursors      900
sessions      1500
sga_max_size      38654705664
sga_target      38654705664
shared_pool_size      3221225472
shared_servers      150
shared_server_sessions      1000
star_transformation_enabled      FALSE
timed_statistics      TRUE
undo_management      AUTO
undo_retention      10800
undo_tablespace      UNDOTBS1
user_dump_dest      /var/log/oracle/billing/udump
workarea_size_policy      AUTO
You can see here that parameter filesystemio_options      has value DIRECTIO;
This is becouse last week I had accident on this database and, as result, I had to turn off async io for oracle;
All time before, since database mirgate moment, this parameter had value SETALL, and disk_asynch_io parameter had value true;
And rman has kept his inappropriate behaviour since this time (e.g.: since database mirgate);

Similar Messages

RMAN backup slow

Hi,
I have a Production Database (10204) running on HP Unix (B.11.11).
Daily morning 6AM, a full RMAN backup runs. The db size is only around 225GB. But it takes 4.5 hours to complete.
The syntax which I use is given below.
RMAN> run {
     allocate channel ch1 type "sbt_tape";
     allocate channel ch2 type "sbt_tape";
     allocate channel ch3 type "sbt_tape";
send
'NB_ORA_CLIENT=jjcprd04-back.backup.ncsus.jnj.com,NB_ORA_POLICY=JJCPRD04-RMAN';
     backup
     format 'bk_%s_%p_%t'
     (database include current controlfile);
     sql 'alter system archive log current';
     backup
     format 'arch_%d_%s_%p_%t'
     (archivelog all delete input );
Is there a way to increase the speed of this backup? Currently the backup is way too slow as it takes 4.5 hours to copy 225GB of data!
Thanks!

I'll start by stating that backups directly to tape are generally quite slow.
I'd still recommend moving to disk backups though... and just archiving it to a tape...Hi Jony,
You can of course backup to disk. Thats a common solution. But you have to deal with space more than everything. In addition to that backing up to disk does not free you from moving/copying your data - depending on your backup strategy - to a long-term storage.
So you have two different and not conjunct backup operations: the rman backup itself and the job which backups the rman area to another media.
In recovery cases you might end up with RMAN requesting older backupsets which are not on disk anymore. This will most probably lead to recovery failures and manual intervention which is not desirable in recovery scenarios.
Another point is the control of the data flow. Backing up via a MML (i.e. "to tape") does not necessarily mean "to tape". Instead it means "to the backup software" which is responsible for managing the backup data. Common scenarios involve backing up the full backups via SAN directly to tape and backing up the archivelogs or incremental backups over LAN which will be stored by the backup software on a hard disk and later copied and/or moved on to tape. The data flow (to SAN / to disk / whereever) is solely controlled by the backup software.
The advantage here is one continuous backup and restore job. The backup software deals with "where is my saveset located" and requests automatically the needed tapes or disks in recovery scenarios. This makes recoveries far more flexible and comfortable.
Regarding the throughput i have set up a rather large OLTP database ( 12 TB currently) which is backed up with EMC Networker. Full backups are written via SAN directly to tape. The average speed with four LTO-3 drives is approx. 550 MB/s (the tape drives compress the data in hardware; the database itself is text-only and seems rather good compressable). I guess thats not too bad.
you can also compress the BACKUPSETS (this generates a little more CPU but less network IO)He said he is on HP-UX. These kind of CPUs tend to be not that fast. Using compression is an option worth testing but there is another point to test when doing compressed backups: The RESTORE TIME.
Ronny Egner
My Blog: http://blog.ronnyegner-consulting.de

RMAN backup problme becasue of TSM

I Joined one call today , topic was RMAN backups problem because of TSM, I am wondering what is TSM, i guess Tivoli Storage manager...Not sure, and how it can effect RMAN Backup & its performance...Just trying to get general idea....

Yes, TSM is Tivoli Storage Manager which is IBM's backup/recovery/storage management product. RMAN needs a media manager interface to write to tapes, this is where TSM comes into play. It acts as this media manager passing off the data to write to tape. This is made possible via a library which TSM provides (an Oracle library link file is replaced with one pointing to the TSM library file) to inform RMAN of it's presence and enable it's use. There are also configuration settings (network timeout, throughput, client name, media server name, etc.) which to a great deal affect the performance of RMAN and TSM to the extent where backups may not be possible if setup incorrectly.
For more information you can look at the IBM site.
Hope this helps.

RMAN backup deletion by BP key

Hello,
We are having 2 node 11g RAC (ASM) cluster & had set up one ASM disk as FRA on which daily rman backup is performed now as FRA is running out of space we are required to backup FRA so I backup FRA using backup recovery area; now after that all my backups in list backupset summary command are coming as below
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
263 B F A * 28-MAR-11 1 2 YES BACKUP_PRODAILY_032811100000
264 B F A * 28-MAR-11 1 2 YES BACKUP_PRODAILY_032811100000
265 B F A * 28-MAR-11 1 2 YES BACKUP_PRODAILY_032811100000
& for each backup peice I am having two copies one on tape & other on FRA so now I want to remove one copy of every peice i.e FRA's one so taht FRA can be empty, Please tell me how can I delete one copy of each backupset as both copies are having same backupset set but hav different BP KEY as below
RMAN> list backupset 263;
List of Backup Sets
===================
BS Key Type LV Size
263 Full 1001.36M
List of Datafiles in backup set 263
File LV Type Ckp SCN Ckp Time Name
10 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/system.278.742062987
12 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/undotbs1.258.742063665
15 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/apps_ts_media.271.742062841
18 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/apps_ts_queues.285.742063093
27 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/apps_ts_tx_idx.259.742062485
29 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/apps_ts_tx_idx.266.742062695
30 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/ctxd.295.742063149
42 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/apps_ts_media.327.742654423
50 Full 124174871 28-MAR-11 +DATAPROD/prod/datafile/xxmpcd_ts_tx_idx.336.742838847
Backup Set Copy #1 of backup set 263
Device Type Elapsed Time Completion Time Compressed Tag
DISK 00:07:47 28-MAR-11 YES BACKUP_PRODAILY_032811100000
List of Backup Pieces for backup set 263 Copy #1
BP Key Pc# Status Piece Name
651 1 AVAILABLE +FRAPROD/prod/backupset/2011_03_28/nnndf0_backup_prodaily_032811100000_0.393.747007719
Backup Set Copy #2 of backup set 263
Device Type Elapsed Time Completion Time Compressed Tag
SBT_TAPE 00:07:47 25-APR-11 YES BACKUP_PRODAILY_032811100000
List of Backup Pieces for backup set 263 Copy #2
BP Key Pc# Status Media Piece Name
1036 1 AVAILABLE 3764a8c0:4daeb8e0:06c8:0001 FRA_BKP<PROD_361:747007718:1>.dbf
So please tell me how can I remove just one copy so that FRA can be empty.
Thanks & Regards
Saransh Soni

Maybe you may set retention policy to 1,
RMAN> CONFIGURE RETENTION POLICY TO REDUNDANCY 1;
End after taking a backup, you issue the following command so previous backup will be deleted:
RMAN> DELETE OBSOLETE;
Regards
Gokhan

RMAN backup using OSB too slow

Hi all,
We are exploring Oracle Secure Backup in our environment as legato networker alternative. But RMAN backup in OSB is very slow due to sbt wait event(sbtwrite2). it can backup 5GB/hour. We think this not normal behavior. we have checked network throughput between media server and client server. It can send 50 GB data by 30 min.Could you please help us to resolve the problem.
Thanks,

Hello,
What OS do you use? This is a Windows OS or UNIX ? Above all, we need more information about the Hardware and the others plans that ran in this environment.It can be also competition from other jobs.
Kind regards,
Bruno Reis

RMAN BACKUP PERFORMANCE.

Hello Sir,
Oracle Version : 9.2.0.1 /10.2.0.1
Operating System : ANY
This is just for knowledge :
What will be the performance issue if I have more than 1db (for example 10 Dbs.) and I start RMAN backups of all Dbs simultaneously and only the default channel to disk is being used ?
If there is any performance issue i.e. slowdown, what will be the best remedy for that and how ?
Please guide
Thanks & Regards Neeraj

Hi,
Please take a look on this links:
- http://blogs.sun.com/glennf/entry/parallelizing_oracle_backup_with_rman
- [Advise On How To Improve Rman Performance - Note:579158.1|https://metalink.oracle.com/metalink/plsql/f?p=130:14:725432850963062762::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,579158.1,1,1,1,helvetica]
- [RMAN Performance Tuning Diagnostics - Note:311068.1|https://metalink.oracle.com/metalink/plsql/f?p=130:14:725432850963062762::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,311068.1,1,1,1,helvetica]
- [Known RMAN Performance Problems - Note:247611.1|https://metalink.oracle.com/metalink/plsql/f?p=130:14:725432850963062762::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,247611.1,1,1,1,helvetica]
Do you have access to Metalink?
Cheers,
Francisco Munoz Alvarez
http://www.oraclenz.com
Edited by: F. Munoz Alvarez on Sep 17, 2008 9:43 PM

Slow rman backup and slow database

Hello all,
Whenever I run rman backup to sbt_tape, the complete database becomes slow. Also RMAN takes more than 12 hours to complete.
RMAN script is:
run {
allocate channel t1 type 'sbt_tape' parms 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';
sql 'alter system switch logfile';
backup incremental level 2 tag 'db_level2_backup'
format'%d/INC2/%t/%s/%p'
database diskratio=0;
backup
format '%d/LOG_INC2/%t/%s/%p'
archivelog all
delete all input;
release channel t1;
}The AWR shows following "Top 5 timed events"
Event                                      Waits     Time(s)     Avg Wait(ms)     % Total Call Time     Wait Class
db file sequential read                    817,467     17,419         21                         28.6         User I/O
CPU time                                  14,294                              23.5
log file sync                            141,642     8,570         61                         14.1          Commit
enq: TX - row lock contention              3,371     7,171       2,127                          11.8         Application
log file parallel write                     149,486     5,356         36                           8.8           System I/OWhat can be done? Can anyone tell what wrong is happening here?
Database is 10.2.0.4
OS is AIX 5.3
SGA_TARGET is 6G
PGA_AGGREGATE_TARGET is 2G
Total physical RAM is 12G
EBiz application is running in the same server
Regards,
SA

On my server, we needed to adjust the vmm on aix for oracle to run best. Before I did that it was really slow.
http://www.ibm.com/developerworks/aix/library/au-aixoracle/index.html
Specifically these as outlined in the document linked above:
Listing 3. Changing the default setting for the lru_file_repage parameter using vmo
root@lpar21ml16ed_pub[] > vmo -o lru_file_repage=0
Setting lru_file_repage to 0
root@lpar21ml16ed_pub[] >
Setting this to 0 tells the VMM that you want to steal only file pages and not computational pages. As this will change if the numperm < minperm or > maxperm, we will make maxperm high and minperm very low. Years ago, before the lru_file_repage parameter was introduced, we used to make maxperm low. If we did this now, we would stop the application caching programs that are currently running.
Listing 4 shows how we'll set these parameters:
Listing 4. Setting the minperm, maxperm and maxclient parameters
vmo -p -o minperm%=5
vmo -p -o maxperm%=90
vmo -p -o maxclient%=90
We also want to take a look at minfree and maxfree. When the pages on our free list fall below minfree, the VMM will start to steal pages, which we don't want to happen until the free list has beefed up the number in maxfree. The values should be similar to the ones shown in Listing 5.
Listing 5. Setting the minfree and maxfree parameters
vmo -p -o minfree=960
vmo -p -o maxfree=1088
Edited by: user455434 on Oct 12, 2010 1:14 PM

RMAN backups are slow with TSM

Hi,
I encountered a problem using RMAN with TSM to backup a database. It seems that the RMAN channel doesn't use the entire bandwidth to the TSM (or whatever) and is idle most of the time. This makes the database backup really slow.
As a comparison, we backup also filesystem and SQL Server, they both use very few channels and each channel backup about X5 more data per second than Oracle. For example, SQL Server uses 5 channels to the TSM, and we get a total of around 100MB/s. When we configure Oracle to backup using 5 channels we get about 20MB/s and we see that the SQL Server is writing to the TSM all the time while Oracle is idle a large portion of the time.
Any ideas? Known configuration issues?
I hope I'm clear about the problem.
Thanks,
Liron

Hi,
Thanks for everybody for your responses, I'll try to sum the answers here:
1. Yes, we backup the filesystem, and everything works fine. The problem is only with RMAN backups
2. Resources are not an issue, it happens on strong and weak servers (and several O/S as well)
3. Database versions are not the issue, it happens on 10.2 as well as 11.2
My feeling is that this has something to do with the integration, either oracle setting or TSM setting, but I can't be sure until I solve this.
For those of you who backup with TSM, did you notice that filesystem / SQL Servers are being backed up faster? Did you check the Oracle wait events and TSM status while you run the backup?
Thanks
BTW Girish, great comment, I wish I had all the answers... :)

RMAN backup performance tuning

Hi All,
We have oracle 9iR2 two nodes RAC on solaris 9, I am looking for some helpful material on RMAN backup performance tuning, though i went through oracle's official RMAN performance tuning guide, still if there is some really nice material please let me know!
TIA

Oracle Material is the best.
Anyway, the one given below is also a good P&T book.
http://www.dba-oracle.com/bp/s_oracle_tuning_book.htm
I am looking for some helpful material on RMAN backup performance tuningTip: Give a good value for large_pool_size buffer for a better RMAN performance. For RAC configure channels on multiple nodes for parallel backup process through RMAN,
CONFIGURE DEVICE TYPE sbt PARALLELISM 3;
CONFIGURE DEFAULT DEVICE TYPE TO sbt;
CONFIGURE CHANNEL 1 DEVICE TYPE sbt CONNECT 'usr1/pwd1@n1';
CONFIGURE CHANNEL 2 DEVICE TYPE sbt CONNECT 'usr2/pwd2@n2';
CONFIGURE CHANNEL 3 DEVICE TYPE sbt CONNECT 'usr3/pwd3@n3';
BACKUP DATABASE PLUS ARCHIVELOG DELETE INPUT;
[email protected]

Trace dumps when performing RMAN backups

Hi,
10g release 2
I'm not very experinced with oracle 10g, but on 9i I've never seen this before:
Every time i do a RMAN backup errors is written in the alert log pointing to the instances' trace file in the udump dir.
*** ACTION NAME:(0000018 STARTED16) 2006-09-01 12:07:50.993
*** MODULE NAME:(backup full datafile) 2006-09-01 12:07:50.993
*** SERVICE NAME:(instance_name) 2006-09-01 12:07:50.993
*** SESSION ID:(140.2064) 2006-09-01 12:07:50.993
*** ACTION NAME:(0000022 STARTED111) 2006-09-01 12:07:52.098
*** 2006-09-01 12:09:56.409
*** ACTION NAME:(0000094 STARTED111) 2006-09-01 12:09:56.409
*** MODULE NAME:(backup archivelog) 2006-09-01 12:09:56.409
*** 2006-09-01 12:14:22.122
*** ACTION NAME:(0000117 STARTED16) 2006-09-01 12:14:22.122
*** MODULE NAME:(backup full datafile) 2006-09-01 12:14:22.122
*** ACTION NAME:(0000121 STARTED111) 2006-09-01 12:14:23.238
to me it looks only informational, but i was wondering if anyone been experiencing similar behavior before, and if there are a way to avoid this? or is this an error i should take action on.
any ideas would be very much appricated!

This is a known bug # 4596065 with RMAN when controlfile autobackup is on and flash recovery area is enabled. Workaround is to ignore the trce files.

RMAN backups running slow with Catalog , Running fine using control file.

I am facing a weird scenario
RMAN backups are running fine with Control file but are failing using Catalog
There are other databases configured on the same catalog and they are running fine leaves us to suspect this is issue with Database.
Can you please suggest what need to be checked in such scenario
DB: 11.2.0.2
OS: Aix 6.1
Catalog : 11.2.0.2

Hi,
Basically its not only with backups, simple list incarnation also taking a lot of timeDo sql tracing on your catalog session and target db session while running the 'list incarnation' command for your problem dbs and a normal dbs.
Regards,
Tycho

Rman backup is too slow

Hi
I have oracle database (11gR1) 64 bit installed on Windows 2003 R2 64 bit server. MY DB size is almost 152 GB and its running in archive log mode. When i run RMAN online backup , Its taking too much time almost it has taken 22 Hours and still its running , I am not sure why it is consuming this much time ..
I am using below RMAN script ...
run {
allocate channel ch1 type disk;
backup FILESPERSET 32
format 'F:\RMAN_BACKUP\%d_t%t_s%s_FULL' tag Full_DATABASE_07_july_13
(database)
CURRENT CONTROLFILE SPFILE;
SQL "alter system archive log current";
backup FILESPERSET 200
format 'F:\RMAN_BACKUP\ARC_%d_%s_%t'
(archivelog all);
release channel ch1;
Kindly help me and whats wrong in this please tell me . Also tell me if any other information needed ..
Thanks & Regards,
Vikash Jain

hi, Till now RMAN backup not completed for single time so i cant tell how long it takes.
I tried with parallelism 2 also but it is also not working.
CONFIGURE DEFAULT DEVICE TYPE TO disk;
# backup goes to disk
CONFIGURE DEVICE TYPE disk PARALLELISM 2;
# three channels used in parallel
CONFIGURE CONTROLFILE AUTOBACKUP ON;
#controlfile and spfile take backup automatically
CONFIGURE CHANNEL 1 DEVICE TYPE DISK FORMAT 'F:\RMAN_BACKUP\%d_t%t_s%s_FULL' MAXPIECESIZE 10G; # 1st channel
CONFIGURE CHANNEL 2 DEVICE TYPE DISK FORMAT 'F:\RMAN_BACKUP\%d_t%t_s%s_FULL' MAXPIECESIZE 10G; # 2nd channel
run {
      allocate channel chnnel1 type disk;
    allocate channel chnnel2 type disk;
backup database tag Full_DB_090713_With_CH_02;
SQL "alter system archive log current";
    backup archivelog all format 'F:\RMAN_BACKUP\ARC_%d_%s_%t';
release channel chnnel1;
release channel chnnel2;
this also having below error ..
RMAN-03009: failure of backup command on chnnel1 channel at 07/12/2013 10:38:31
ORA-19502: write error on file "D:\APP\ADMINISTRATOR\PRODUCT\11.1.0\DB_1\DATABASE\86OEH1S5_1_1", block number 1476225 (block size=8192)
ORA-27072: File I/O error
OSD-04008: WriteFile() failure, unable to write to file
O/S-Error: (OS 112) There is not enough space on the disk.
channel chnnel1 disabled, job failed on it will be run on another channel
released channel: chnnel1
released channel: chnnel2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on chnnel2 channel at 07/12/2013 10:42:08
ORA-19502: write error on file "D:\APP\ADMINISTRATOR\PRODUCT\11.1.0\DB_1\DATABASE\87OEHG94_1_1", block number 1476097 (block size=8192)
ORA-27072: File I/O error
OSD-04008: WriteFile() failure, unable to write to file
O/S-Error: (OS 112) There is not enough space on the disk.
I checked there is enough space on F drive but still its showing the above error , Also i cant see backup pieces in "F:\RMAN_BACKUP\" location ... I am sure whats wrong with the above script ...
Thanks & Regards,
Vikash Jain

Alter system flushed shared pool in RMAN backup

Hi,
I am trying to take RMAN backup of 11.2.0.1 Database in IBM AIX 6.1 server.
The RMAN is hanging .
Though the backup gets completed, The channels allocated doesnt get released and the RMAN gets hanging.
In earlier RMAN backup Scripts,
the DBA was using alter system flush shared pool in RMAN backup script and the backup was getting succesful.
Now my question is , is using ALTER SYSTEM FLUSH SHARED POOL have any performance impact on the database.
Regards,
TEJAS

TEJAS_DBA wrote:
Hi,
I am trying to take RMAN backup of 11.2.0.1 Database in IBM AIX 6.1 server.
The RMAN is hanging .
Though the backup gets completed, The channels allocated doesnt get released and the RMAN gets hanging.Are you setting the large pool? If you don't, then rman uses the shared pool. Read about tuning rman performance in the docs.
>
In earlier RMAN backup Scripts,
the DBA was using alter system flush shared pool in RMAN backup script and the backup was getting succesful.
Now my question is , is using ALTER SYSTEM FLUSH SHARED POOL have any performance impact on the database.Yes, you are allowing the components in there to be loaded in the random order of whatever is called first. This may have a good impact if you had some fragmentation in there, or it could be mildly bad if everything was well sorted, or it could be very bad if you are unlucky or have some pattern of invalidations or should be pinning something or who-knows-what. It generally is considered not a good thing to do as a habit. You wind up with [url http://tkyte.blogspot.com/2012/05/another-debugging-story.html]rainy Monday scenarios.
Edit: I notice there are some bugs, including very slow performance when using a catalog. When you say hang, how long are you waiting? Have you considered current patches?
Edited by: jgarry on Aug 8, 2012 11:09 AM

GZIP RMAN BACKUP

Dear All
We are running Oracle 10g(10.2.0.4) on Solaris 10
I want to gzip up my rman backups, which are to disk.
How do I add the command to my script so after end of backup it starts gzipping it.
Please find the script below
export ORACLE_BASE=/app/oradataerp/oraprod
export ORACLE_HOME=/app/oradataerp/oraprod/db/tech_st/10.2.0
export ORACLE_SID=PROD
export LD_LIBRARY_PATH=${ORACLE_HOME}/lib
export PATH=${ORACLE_HOME}/bin:${ORACLE_HOME}/lib:${PATH}
export DATE=`/usr/bin/date +%Y%m%d`
rman target sys/***** nocatalog msglog /nfs-bkp-erp/PROD-ERP/PROD-DB/RMAN.${DATE}.log <<EOF
crosscheck archivelog all;
#backup database format '/nfs-bkp-erp/PROD-ERP/PROD-DB/PROD_t%t_s%s_p%p%c';
#backup current controlfile format '/nfs-bkp-erp/PROD-ERP/PROD-DB/PROD_cntrl_t%t_s%s_p%p%c';
#delete noprompt archivelog All complEted before 'SYSDATE-10';
crosscheck backup;
#delete expired archivelog all;
delete obsolete;
crosscheck backup;
EXIT
EOF;
Regards
Musaddaq

When using compression, you will need to actually test to see which backup compression gives you the best compression/performance. Higher compression = slower performance, Lower compression ratio = more performance.
To make your life a lot easier when trying to do backups, make sure you do rman backup to disk, then sweep these backups to tape. The restore would then be much faster.

Unable to restore and recover my Database from RMAN Backup

Dear All,
Iam practicing RMAN in Windows environment.
My configurations are
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION ON; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO 'F:\Backup\%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'E:\ORACLE\DATABASE\SNCFORCL.ORA';
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT 'F:\Backup\%U.bak';
Then i took the RMAN backup, now iam shutdown my DB and removed one logfile..
then i tried to startup the DB but it is saying
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: 'E:\ORADATA\ORCL\REDO03.LOG'
So i tried to restore the Database with my rman backup
RMAN> restore database;
But i am unable to restore my logfile in the Datafiles location where as all other datafiles are restored but redo logfiles are not restored.
So, i am unable to recover my DB
Please tell me whether i am doing any mistake, what is the solution for my problem.
Thanks in advance
Mahi

All you restored was your datafiles.
Now issue recover database and that will apply the archive logs.
This is in the oracle documentation.
"You must restore backups of these archived logs for recovery to proceed. You can restore either to the default or nondefault location. If you do not have backups, then you must performing incomplete recovery up to an SCN before the first missing redo log and open RESETLOGS."
http://download.oracle.com/docs/cd/B28359_01/backup.111/b28270/osrecvry.htm
In this case after issuing restore database in RMAN, then issue this.
"Recover database"
It will then be an incomplete recovery. They should have been multiplexed. Did you restore your controlfile with this backup? You may have to issue an "Alter database open resetlogs;"

RMAN-backup slow perform

Similar Messages

Maybe you are looking for