Is unable to write archive log fast enough the problem?

Question

Oracle 10g:
I see very very long checkpoints in alert logs:
Mon Mar 23 12:35:57 2009
Beginning log switch checkpoint up to RBA [0x2001.2.10], SCN: 2667813010
Mon Mar 23 12:35:57 2009
Thread 1 advanced to log sequence 8193 (LGWR switch)
  Current log# 3 seq# 8193 mem# 0: /u05/oradata/perf/redo03a.log
  Current log# 3 seq# 8193 mem# 1: /u06/oradata/perf/redo03b.log
Mon Mar 23 12:41:02 2009
Completed checkpoint up to RBA [0x2001.2.10], SCN: 2667813010Our redo logs are 2gb and have 6 of those.
I am not sure why checkpoints are so long. Also, does this affect the requests trying to read-write to DB?
I am seeing high response time on our customer transactions when backup runs and one of the things I see is high checkpoints. But I am not sure if that would cause 20x high response time even though we do it at night time when the traffic is relatively low. There is something wrong and I am not sure if it's chekpoints, redo logs or something else.
Edited by: user628400 on Mar 23, 2009 5:50 PM
Edited by: user628400 on Mar 24, 2009 10:09 AM

Answer 1

Everything is indicating to me this is a big database.
My understanding, possilbly flawed, is to use a backup technique of spliting off the the third mirror the tablespaces should be put in backup mode before split, and out of backup mode after the split. (I believe this should be visible by entries in the alert log).
Whilst tablespaces are in backup mode extra redo will be generated, so it is important to check this time is minimized.
If data paths have been properly separated then there should only be a small blip in performance while the mirror split is occuring,
and possibly a drop when the third mirror is silverd back in.
The fact you are on a 3 way mirror backup design means you need to have documented the design and infrastucture of that set up and
ensure backup is documented and understood and monitored. This might also help pinpoint where in the process response times are increasing.
You might also try to monitor whether your disk access wait times are increasing during backups, or whether you have increased network latency.
(At abolute worst case in my opinion bad operation of this form of backup could lead to an unrecoverable database .... some system/backup admins might not appreciate this .... )
What could however be happening is while the third mirror is being backed up elsewhere that is causing contention between that backup and your databases access to storage.
This is all infrastructure dependent.
Slightly different topic what I mean was that the /uNN mounts may be filesystems set up by a volume manager, and may be on the same physical disk. Understanding the
3 way mirror backup takes priority over this.
In summary I suggest you seek all that is known about the design of your backup and its supporting documents. In your position I think I certainly would.
(It is past my bedtime and I may have veered off topic).
It is probably worth noting how often you switch redo logs and if that rate gets more frequent at certain times of day,
(ie graph when your redo logs were created and at what size).
Use Database Control / Grid Control to view AWR reports especially watching I/O quantity and waits overnight.
Before I forget it may be worth ensuring FAST_START_MTTR_TARGET is set to non zero value. (eg) 300s. This has an effect on incremental checkpointing.
Hope some of this helps - bigdelboy