Logical standby stuck at initializing SQL apply only coordinator process up

Question

Hi
OS: solaris 5.10
Hardware: sun sparc
Oracle database: 11.2.0.1.0
Primary database name: asadmin
Standby database name: test
I had been trying to convert a physical standby to logical standby database. Both the primary and standby reside on the same machine.
The physical standby was created with a hot backup of primary.
I had been following document id 278371.1 to convert the physical to logical standby and used the following steps:
Relevant init parameters on primary:
*.db_name='asadmin'
*.db_unique_name='asadmin'
*.log_archive_config='dg_config=(asadmin,test)'
*.log_archive_dest_1='location=/u01/asadmin/archive valid_for=(all_logfiles,all_roles) db_unique_name=asadmin'
*.log_archive_dest_2='SERVICE=test async valid_for=(online_logfiles,primary_role) db_unique_name=test'
*.log_archive_dest_state_1='enable'
*.log_archive_dest_state_2='enable'
*.fal_client='asadmin'
*.fal_server='test'
*.remote_login_passwordfile='EXCLUSIVE'
Relevant init parameters on standby database:
*.db_name='test' -- Was asadmin before I renamed the DB during conversion to logical standby
*.db_unique_name='test'
*.log_archive_dest_1='location=/u01/test/archive valid_for=(all_logfiles,all_roles) db_unique_name=test'
*.log_archive_dest_2='service=asadmin async valid_for=(online_logfiles,primary_role) db_unique_name=asadmin'
*.log_archive_dest_state_1=enable
*.log_archive_dest_state_2=defer
*.remote_login_passwordfile='EXCLUSIVE'*.fal_server=test
*.fal_client=asadmin
Steps on primary:
1) alter system set log_archive_dest_state_2=defer;
2) shutdown immediate;
3) Made sure that the physical standby has applied all of the redo sent to it following the shutdown.
4) startup mount;
5) ALTER DATABASE BACKUP CONTROLFILE to '/home/oracle/control01.ctl';
6) ALTER SYSTEM ENABLE RESTRICTED SESSION;
7) ALTER DATABASE OPEN;
8) Verified that the supplemental logging is on.
9) ALTER SYSTEM ARCHIVE LOG CURRENT;
10) Checked for the checkpoint change no. at this point which is 72403818 and is present in archive log file 1_62_775102253.dbf
11) EXECUTE DBMS_LOGSTDBY.BUILD;
12) ALTER SYSTEM ARCHIVE LOG CURRENT;
13) Checked for the archive log containing dictionary build which is 1_64_775102253.dbf
14) ALTER SYSTEM DISABLE RESTRICTED SESSION;
Details of archive logs and related checkpoint change nos:
NAME FIRST_CHANGE# NEXT_CHANGE#
/u01/asadmin/archive/1_61_775102253.dbf 72402901 72403817
/u01/asadmin/archive/1_62_775102253.dbf 72403817 72404069
/u01/asadmin/archive/1_63_775102253.dbf 72404069 72404211
/u01/asadmin/archive/1_64_775102253.dbf 72404211 72405700
Steps on standby:
1) shutdown immediate;
2) Copy the archivelog file 61(was created at primary after apply stopped at standby), 62(contains checkpoint no. 72403818), 63 and 64(contains dictionary build). Copy the backup controlfile from step 5 above to the controlfile location in standby init.
3) startup mount;
4) Rename all datafiles and redo log files (including standby redo log files) to the correct path on standby.
5) alter database recover automatic from '/u01/test/archive' until change 72405700 using backup controlfile; -- This completed error-free
6) alter database guard all; -- this completed error free
7) alter database open resetlogs; -- this completed error free.
8) nid target=sys/oracle12 dbname=test
9) Changed the db_name in init file to new name test.
10) Added a tempfile to temp tablespaces.
11) ALTER DATABASE REGISTER LOGICAL LOGFILE '/u01/test/archive/1_61_775102253.dbf'; -- ORA-16225: Missing LogMiner session name for Streams
12) ALTER DATABASE START LOGICAL STANDBY APPLY INITIAL 72405700; -- This completed error free.
Also enabled the log_archive_dest_state_2 on primary.
After this output from some views:
SQL> SELECT SESSION_ID, STATE FROM V$LOGSTDBY_STATE;
SESSION_ID STATE
1 INITIALIZING
SQL> SELECT SID, SERIAL#, SPID, TYPE FROM V$LOGSTDBY_PROCESS;
SID SERIAL# SPID TYPE
587 22 15476 COORDINATOR
SELECT PERCENT_DONE, COMMAND
FROM V$LOGMNR_DICTIONARY_LOAD
WHERE SESSION_ID = (SELECT SESSION_ID FROM V$LOGSTDBY_STATE);
PERCENT_DONE
COMMAND
0
SQL> SELECT TYPE, HIGH_SCN, STATUS FROM V$LOGSTDBY;
TYPE HIGH_SCN STATUS
COORDINATOR ORA-16111: log mining and apply setting up
SQL> SELECT APPLIED_SCN, NEWEST_SCN FROM DBA_LOGSTDBY_PROGRESS;
APPLIED_SCN NEWEST_SCN
72405700 72411501
SELECT THREAD#, SEQUENCE#, FILE_NAME FROM DBA_LOGSTDBY_LOG L
WHERE NEXT_CHANGE# NOT IN
(SELECT FIRST_CHANGE# FROM DBA_LOGSTDBY_LOG WHERE L.THREAD# = THREAD#)
ORDER BY THREAD#,SEQUENCE#;
no rows selected
SQL> SELECT EVENT_TIME, STATUS, EVENT
FROM DBA_LOGSTDBY_EVENTS
ORDER BY EVENT_TIMESTAMP, COMMIT_SCN; 2 3
EVENT_TIME STATUS EVENT
14-FEB-12 02:00:50 ORA-16111: log mining and apply setting up
14-FEB-12 02:00:50 Apply LWM 72405699, HWM 72405699, SCN 72405699
14-FEB-12 02:20:11 ORA-16128: User initiated stop apply successfully
completed
14-FEB-12 02:20:39 ORA-16111: log mining and apply setting up
14-FEB-12 02:20:39 Apply LWM 72405699, HWM 72405699, SCN 72405699
14-FEB-12 02:54:15 ORA-16128: User initiated stop apply successfully
completed
14-FEB-12 02:57:38 ORA-16111: log mining and apply setting up
EVENT_TIME STATUS EVENT
14-FEB-12 02:57:38 Apply LWM 72405699, HWM 72405699, SCN 72405699
14-FEB-12 03:01:36 ORA-16128: User initiated stop apply successfully
completed
14-FEB-12 03:13:44 ORA-16111: log mining and apply setting up
14-FEB-12 03:13:44 Apply LWM 72405699, HWM 72405699, SCN 72405699
14-FEB-12 04:32:23 ORA-16128: User initiated stop apply successfully
completed
14-FEB-12 04:34:17 ORA-16111: log mining and apply setting up
14-FEB-12 04:34:17 Apply LWM 72405699, HWM 72405699, SCN 72405699
EVENT_TIME STATUS EVENT
14-FEB-12 04:36:16 ORA-16128: User initiated stop apply successfully
completed
14-FEB-12 04:36:21 ORA-16111: log mining and apply setting up
14-FEB-12 04:36:21 Apply LWM 72405699, HWM 72405699, SCN 72405699
14-FEB-12 05:15:22 ORA-16128: User initiated stop apply successfully
completed
14-FEB-12 05:15:29 ORA-16111: log mining and apply setting up
14-FEB-12 05:15:29 Apply LWM 72405699, HWM 72405699, SCN 72405699
I also greped for lsp and lcr processes and found that lsp is up but do not see any lcr.
The logs are getting transported to the archive destination on standby whenever they are archived on primary but are not getting applied to standby.
Also in case the standby is down while a log is generated on primary it is not automatically transported to standby once the standby is up, means gap resolution is also not working.
I see the following in alert log every time I try to restart the log apply, everything seems to be stuck at initialization.
ALTER DATABASE START LOGICAL STANDBY APPLY (test)
with optional part
IMMEDIATE
Attempt to start background Logical Standby process
Tue Feb 14 05:15:28 2012
LSP0 started with pid=28, OS id=23391
Completed: alter database start logical standby apply immediate
LOGMINER: Parameters summary for session# = 1
LOGMINER: Number of processes = 3, Transaction Chunk Size = 201
LOGMINER: Memory Size = 30M, Checkpoint interval = 150M
LOGMINER: SpillScn 0, ResetLogScn 0
-- NOTHING AFTER THIS

Answer 1

Hello;
I noticed some of your parameters seem to be wrong.
fal_client - This is Obsolete in 11.2
You have db_name='test' on the Standby, it should be 'asadmin'
fal_server=test is set like this on the standby, it should be 'asadmin'
I might consider changing VALID_FOR to this :
VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES)Would review 4.2 Step-by-Step Instructions for Creating a Logical Standby Database of Oracle Document E10700-02
Document 278371.1 is showing its age in my humble opinion.
-----Wait on this until you fix your parameters----------------------
Try restarting the SQL Apply
ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATEI don't see the parameter MAX_SERVERS, try setting it to 8 times the number of cores.
Use these statements to trouble shoot :
SELECT NAME, VALUE, UNIT FROM V$DATAGUARD_STATS;
SELECT NAME, VALUE FROM V$LOGSTDBY_STATS WHERE NAME LIKE ;TRANSACTIONS%';
SELECT COUNT(1) AS IDLE_PREPARERS FROM V$LOGSTDBY_PROCESS WHERE
TYPE = 'PREPERER' AND STATUS_CODE = 16166;Best Regards
mseberg
Edited by: mseberg on Feb 14, 2012 7:37 AM