How to remove error from propagation and verify replication is ok?

Question

Have a one-way schema level streams setup and the target DB is 3-node RAC (named PDAMLPR1). I run a large insert at source DB (35million rows). After committing on source, I made a failure test on the target DB by shutting down the entire database. The streams seems stopped as the heartbeat table sequence (it inserts a row each min) on target is still reflecting last night. We get the error in the dba_propagation:
ORA-02068: following severe error from PDAMLPR1
ORA-01033: ORACLE initialization or shutdown in progress
ORA-06512: at "SYS.DBMS_AQADM_SYS", line 1087
ORA-06512: at "SYS.DBMS_AQADM_SYS", line 7639
ORA-06512: at "SYS.DBMS_AQADM", line 631
ORA-06512: at line 1
08-FEB-10
while capture, propagation, and apply are all in enabled status. I restarted capture and propagation at source DB. But still see the error message above. My questions are:
1. How to delete the error from dba_propagation?
2. How to verify the streams is still running fine?
In normal test, during such large insert, the heartbeat table added a row in an hour. Very slow.
thanks for advice.

Answer 1

Well, if I can give you my point of view: I think that 35 millions LCR is totally unreasonnable. Did you really post a huge insert of 35 millions then committed that single utterly huge transaction? Don't be surprised it's going to work very very hard for a while!
With a default setup, Oracle recommends to commit every 1000 LCR (row change).
There are ways to tune Streams for large transaction but I have not done so personnaly. Look on Metalink, you will find information about that (mostly documents id 335516.1, 365648.1 and 730036.1).
One more thing: you mentionned about a failure test. Your target database is RAC. Did you read about queue ownership? queue_to_queue propagation? You might have an issue related to that.
How did you setup your environment? Did you give enough streams_pool_size? You can watch V$STREAMS_POOL_ADVICE to check what Oracle think is good for your workload.
If you want to skip the transaction, you can remove the table rule or use the IGNORETRANSACTION apply parameter.
Hope it helps
Regards,