Physical standby crash leads to peculiar process behavior on primary nodes

Question

Hey all, we had an issue early this morning with our physical standby database in our production environment. It's an odd problem and I thought I'd share it to see if anyone else has come across a similar problem.
Our environment consists of a two node RAC cluster running 10.2.0.2 Enterprise on Red-Hat x86-64 Linux boxes. We have implemented a single physical standby at a remote site using real-time log apply.
Upon arrival this morning I discovered that a large portion of the primary database activity was a wait class event(ARCH wait on SENDREQ). The problem was quickly identified as a network issue between our site and the remote standby. However, the physical standby seemed to be running normal, with no messages in the alert logs regarding network disconnects. Further investigation showed that our log-apply had stopped all together early that morning. It seems the DBWR process crashed around 630am, as per this trace dump:
*** 2008-02-27 06:24:53.667
Waited for detached process: DBW0 for 300 seconds:
*** 2008-02-27 06:24:53.667
Dumping diagnostic information for DBW0:
OS pid = 10587
loadavg : 12.11 11.67 8.95
memory info: free memory = 0.00M
swap info: free = 0.00M alloc = 0.00M total = 0.00M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S oracle 2448 31813 0 78 0 - 13190 wait 06:07 ? 00:00:00 /bin/sh -c /usr/bin/gdb -quiet -x
/tmp/stacknvvano
/proc/10587/exe 10587 < /dev/null 2>&1
0 S oracle 2449 2448 0 78 0 - 38780 wait 06:07 ? 00:00:00 /usr/bin/gdb -quiet -x
/tmp/stacknvvano /proc/10587
/exe 10587
0 D oracle 10587 1 0 76 0 - 43072 - Jan04 ? 00:04:09 ora_dbw0_schclsdb
So what we did was restart the standby server and the database came up fine. We reinitiated log apply services and both databases did sync up. However
we are still seeing the "ARCH wait on SENDREQ" wait event from ARC0 and ARC1 on both the RAC instances.
We tried reseting the connection to the standby database, however did this not clear up the wait event. The system itself, as far as I can tell is not suffering, and redo logs are going across to the standby database without issue. What could be the problem with the ARC0 and ARC1 processes?
It would probably be helpful to mention that we don't actually use the ARCH processes for the log apply unless the standby falls behind. Archiving is handled by the local LNS process and remote RFS service. A quick instance restart would fix it, to be sure, but since this is happening in a production deployment in a 24x7 environment, I'd rather not have any downtime. A service request is of course filed, but since they have not gotten back to me yet and Metalink hasn't been of much help, I thought I'd see if any of you have seen this before. This giant gray blob on my grid control histogram is quite obnoxious. =) Thanks for taking the time to read. Cheers.
-Jim Henaghan

Answer 1

Just as an update to anyone else who may have this problem, we haven't found a good way to clear up our Archiver issue, so a instance restart will take place. Further research indicates the issue came about from a MRP process crash due to a known bug fixed in 10.2.0.4 (https://metalink.oracle.com/metalink/plsql/f?p=130:14:316259464254960300::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,5894403.8,1,0,1,helvetica)