Trying boost the performance of RMAN on AIX5L

Question

Hi all,
I need to find the best "number" for my test environment:
I'm looking for opinions based on knowledge of who has experience with RMAN.
I'm trying find better numbers of channels, maxopenfiles and additional parameters.
I accept the recommendation of changing the parameters of Operating System or Database
Environment Info:
DB:  10.2.0.5
ASM: 10.2.0.5
OS:  AIX 5L
CPUs: 16 
DISKGROUP: 1 (with 16 ASMDISK)
Adapter: 4 HBA (4GBIT)I want to find out what the maximum rate that can be achieved, then this test I will use only "BACKUP VALIDATE" When I get the best number i'll go to TSM the backup will be done via LAN-FREE.
Below the results of the tests using BACKUP VALIDATE:
=====================================================
1° Test
Database Parameter
  backup_tape_io_slaves=TRUE
  dbwr_io_slaves=0
  db_writer_processes=6
  tape_asynch_io= TRUE
  disk_asynch_io= TRUE
RMAN Parameter
  4 Channel
  4 MAXOPENFILES for each Channel
TIME: 01:08:13
SIZE IN:892.11GB     
RATE: 223MB/s
=====================================================
=====================================================
2° Test
Database Parameter
   backup_tape_io_slaves=TRUE
   dbwr_io_slaves=0
   db_writer_processes=6
   tape_asynch_io= TRUE
   disk_asynch_io= TRUE
RMAN Parameter
   4 Channel
   3 MAXOPENFILES for each Channel
TIME: 00:57:42
SIZE IN:892.11GB     
RATE: 264MB/s
=====================================================
=====================================================
3° Test - Here my best number
Database Parameter
  backup_tape_io_slaves=TRUE
  dbwr_io_slaves=0
  db_writer_processes=6
  tape_asynch_io= TRUE
  disk_asynch_io= TRUE
RMAN Parameter
  4 Channel
  2 MAXOPENFILES for each Channel
TIME: 00:53:01
SIZE IN:892.11GB     
RATE: 287MB/s
=====================================================I ran other test with 2 Channels and 3 MAXOPENFILES and perfomance was like 3° Test - about 54 minutes
I could not understand why using the maximum of 2 MAXOPENFILES and 4 Channels = 8 MAXOPENFILES is more performatic than if I increase the number parallelism of the datafiles reads.
My 1° test i can get 784 GB/hour
My 3° test i can get 1009 GB/hour
It's a big difference, because this test database (~ 900GB) is small compared with the production.
What makes sense for this to be optimal number (8 MAXOPENFILES)?
Thanks,
Levi Pereira

Answer 1

Hi Dude,
Thanks for the input.
Dude wrote:
I think the difference could be memory and process related combined with hardware limitations. RMAN will read the datafiles from disk into memory. Doesn't parallelism introduce additional channels and each channel writes to a separate backupset? From what I understand, too many I/O channels can reduce overall performance and cause additional overhead in particular if the media or disk is the bottleneck. On the other hand, If the cpu was the bottleneck, more cpu or processes can boost performance. When processes run on different cpu's, caching is less effective. The same applies to your HBA adapter. I monitored the whole process of backup and saw that the issue was always associated with performance of I/O the RMAN not was waiting for the CPU and also had no waits on disks. I knew I could increase my throughput but I do not know how. (i.e Resources was left over but did not know how exploring them.)
>
As a rule, the number of channels used in carrying out an individual RMAN command should match the number of physical devices accessed in carrying out that command. Striped disk configurations involve hardware multiplexing, so that the level of RMAN multiplexing does not need to be as high and a smaller MAXOPENFILES setting can results in faster performance.
I can use up to 4 Drives (4 Channels) that are LTO-5 that may have an acceptable rate of 120MB/s, so I can go up to 480Mb/s I know it's not exact, there are several extra factors.
I wish be sure which was the highest rate I could get by using four channels.
It remains unknown (x =?) this math.
16 (asmdisk) + 2 (HBA) + 256 (Datafiles) its ok to use 4 Channel + 2 MAXOPENFILES + 12 DB_WRITE
If I knew the math to find the ideal number of channels and maxopenfiles would be my great discovery.
I guess you already checked the Tuning Backup and Recovery chapter of the Database Backup and Recovery Advanced User's Guide, which also shows that you can use the V$BACKUP_SYNC_IO and V$BACKUP_ASYNC_IO views to determine the source of backup or restore bottlenecks and to see detailed progress of backup jobs. http://download.oracle.com/docs/cd/B19306_01/backup.102/b14191/rcmtunin.htm#i1006195
I add more notes than I used:
RMAN Performance Tuning Using Buffer Memory Parameters [ID 1072545.1]
RMAN Performance Tuning Diagnostics [ID 311068.1]
Advise On How To Improve Rman Performance [ID 579158.1]
Using V$BACKUP_ASYNC_IO / V$BACKUP_SYNC_IO to Monitor RMAN Performance [ID 237083.1]
http://levipereira.wordpress.com/2010/11/20/tuning-oracle-rman-jobs/
I tried one more change:
On large workloads, the database writer may become a bottleneck. If it does, then increase the value of DB_WRITER_PROCESSES. As a general rule, do not increase the number of database writer processes above one for each pair of CPUs in the system or partition.
http://download.oracle.com/docs/cd/B28359_01/server.111/b32009/appa_aix.htm#BEHGGBAJ
I changed DB_WRITE_PROCESSES from 6 to 12.
My last RATE: 305MB/s.
Cheers,
Levi Pereira