Howto increase log recovery speed?

Question

We created a copy of our production MaxDB (2,8 TB) to have a shadow database.
When I apply the logs (via script) I find out, that the shadow database is not able to "keep up" with the number of logs produced in the production. I figured out so far this is caused by the fact that the initial recover_start with the last but one log needs between 9 and 11 minutes until the first log recover is actually started; only 64 pager are used during this process despite having 100 datafiles (with 30 GB each) connected to a SAN with 248 disks.
I first recovered only one log (256 MB/32000 pages) by using a script like
db_admin
db_connect
recover_start logsich log 1000
recover_replace logsich /archivelog/log 1001
recover_cancel
This took roughly 10 - 12 minutes for each of them in average which is too long.
I switched then the script to recover more than one log at a time like
db_admin
db_connect
recover_start logsich log 1000
recover_replace logsich /archivelog/log 1001
recover_replace logsich /archivelog/log 1002
recover_replace logsich /archivelog/log 1003
recover_replace logsich /archivelog/log 1004
recover_cancel
using 10 logs. This process takes in avg. about 90 minutes which is too long.
The following parameters were changed already:
MaxServerTasks  1500
MaxUserTasks  300
MaxDataVolumes 130
However, I'm not able to keep up with the number of logs produced in the production, the backlog is always 10 - 15 logs. If we now wanted to stop the log recover to do a full database backup of the shadow database instead of the production we will never be able to make that backup current.
It seems to me that two things are mainly responsible for this:
- the long startup between "recover_start" and "recover_replace" (from db_offline to db_online it takes even without crashrecovery 12 - 15 minutes)
- the fact that there's not really a full parallelism used to apply the log (whatever that means)
Any hints on how to speed up that process are appreciated.
Markus

Answer 1

I don't believe the memory thing (CacheMemorySize) is really an issue at this point.
If I check the memory usage vs. CacheSize I see that even after the application of 8 logs not all memory is used:
PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
23963 maxdb    143  53    2   11G 7320M sleep   21:36  7.38% /sapdb/SHA/db/pgm/kernel SHA -diagpipe 18
I also notice, that despite having 1500 potential threads to read and write the data the I/O subsystem is not really leveraged to its fullest:
capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
dbdataSHA   2.71T   507G    374      0  46.5M      0
dbdataSHA   2.71T   507G     75      0  9.41M      0
dbdataSHA   2.71T   507G    135      0  16.6M      0
dbdataSHA   2.71T   507G    154      0  18.8M      0
dbdataSHA   2.71T   507G    139      0  16.9M      0
dbdataSHA   2.71T   507G    141      0  17.5M      0
dbdataSHA   2.71T   507G    164      0  19.8M      0
dbdataSHA   2.71T   507G    122      0  14.9M      0
dbdataSHA   2.71T   507G     74      0  9.35M      0
So we read roughly 15 - 20 MB/sec though the I/O subsystem can deliver and write up to 180 - 200 MB/sec (e. g. during a normal savepoint operation).
I guess this is yet-another-MaxDB-random-8kb-I/O-thing that can't be speed up
sigh
Markus