Oracle ORA_16000 when trying to add standby instance to existing rac node

Question

I attempted to use dbca to add a new standby instance to an existing cluster. The cluster is 4 nodes, Linux RHEL 5.3 Oracle 11.1.0.7. Also using ASM, asmlib, ocfs2 and shared block devices.
ASM instances are up and functional on all nodes. current config appears to be running normally and correctly.
I have a 4 instance database running on the cluster. I also have 3 physical standby active data guard instances running on 3 of the nodes. I wanted to add a new ADG instance to the 4th node.
While running dbca I received ORA-00604 and ORA-16000.
The active data guard database was open (read only) and redo apply was on. I am using data guard broker as well, but not grid control.
Does anyone have a procedure for adding an instance in this environment? Do I need to have the standby in mount state? If dbca won't work does anyone have a manual procedure for adding a new instance?
Thanks

Answer 1

zulo
Let's say you adding node nusclust160## to you existing cluster and dbca is a pain to use.
Extend clusterware to the nusclust160## server.
re: Page 64 of Oracle® Clusterware Administration and Deployment Guide 11g Release 1 (11.1)
1a.
Add undo tablespace to support additional node.
Re-check space for DATA1 on nusclust16007 and /dbdata/ORADB on sun16109.
As of Thursday, May 21, 2009 the DATA1 asm group has 53,584M free.
As of Thursday, May 21, 2009 the /dbdata/ORADB has 77G free.
In a separate terminal window on nusclust16007 run the following in sqlplus
CREATE UNDO TABLESPACE UNDOTBS4 datafile '+DATA1' SIZE 13300M AUTOEXTEND ON ;
This will take a long time to create this tablespace. Please minimize the window after submitting the ddl and move on to the next step.
1b.
Insure .bash_profile on nusclust160## should look like this:
vi .bash_profile
export ORACLE_HOSTNAME=nusclust160##
export ORACLE_SID=ORADB4
export ORA_CRS_BASE=/apps/ocr/oracle
export ORACLE_BASE=/apps/dbs/oracle
export PATH=/usr/ccs/bin:/usr/X/bin:/usr/bin:/usr/sfw/bin:/usr/sbin:/usr/local/bin
export server=`uname -n`
export PS1="$ORACLE_SID@$HOSTNAME >"
alias cls='clear'
alias More='more'
alias ll='ls -lt | more'
Gather IP addresses for fourth node from /etc/hosts:
222.65.125.### nusclust160##
222.65.125.### nusclust160##-vip
10.333.248.### nusclust160##-priv
2. Start Oracle Universal Installer:
Go to CRS_home/oui/bin and run the addNode.sh script on one of the existing
nodes. Oracle Universal Installer runs in add node mode.
The Oracle inventory on nusclust16007, nusclust16008, and nusclust16036 are found under:
/home/oracle/oraInventory
Use a X windows enabled session (The OUI takes 33 minutes)
cd /apps/ocr/oracle/product/11.1.0/crs/oui/bin
./addNode.sh
a. In the first screen specify a new node as :
Public Node Name:          nusclust160##
Private Node Name:     nusclust160##-priv
Virtual Host Name:     nusclust160##-vip
If you receive the error:
" tar. ./bin/racgvip.orig: Permission denied"
Do the following:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
ls -al racgvip.orig
paste here:
chown root:oinstall racgvip.orig
chmod 771 racgvip.orig
should now show:
-rwxrwx--x 1 root oinstall 19213 Feb 11 08:36 racgvip.orig
As root:
a.
On nusclust160##:
cd /home/oracle/oraInventory
./orainstRoot.sh
b.
On nusclust16007:
cd /apps/ocr/oracle/product/11.1.0/crs/install
./rootaddnode.sh
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 4: nusclust160## nusclust160##-priv nusclust160##
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/apps/ocr/oracle/product/11.1.0/crs/bin/srvctl add nodeapps -n nusclust160## -A nusclust160##-vip/255.255.255.224/bge0
c.
On nusclust160##:
cd /apps/ocr/oracle/product/11.1.0/crs/
./root.sh
WARNING: directory '/apps/ocr/oracle/product/11.1.0' is not owned by root
WARNING: directory '/apps/ocr/oracle/product' is not owned by root
WARNING: directory '/apps/ocr/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
OCR LOCATIONS = /raw/ocr/ocrconf1,/raw/ocr/ocrconf2
OCR backup directory '/apps/ocr/oracle/product/11.1.0/crs/cdata/rac_cluster' does not exist. Creating now
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/apps/ocr/oracle/product/11.1.0' is not owned by root. Changing owner to root
The directory '/apps/ocr/oracle/product' is not owned by root. Changing owner to root
The directory '/apps/ocr/oracle' is not owned by root. Changing owner to root
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: nusclust16007 nusclust16007-priv nusclust16007
node 2: nusclust16008 nusclust16008-priv nusclust16008
node 3: nusclust16036 nusclust16036-priv nusclust16036
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
nusclust16007
nusclust16008
nusclust16036
nusclust160##
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
4. After this is done crs_stat -t will show nusclust160## in the crs i.e.
I see:
Name Type Target State Host
ora....160##.gsd application ONLINE ONLINE sun...160##
ora....160##.ons application ONLINE OFFLINE
ora....160##.vip application ONLINE ONLINE sun...160##
Do not be concerned about ora.nusclust160##.ons being OFFLINE, as that will be fixed shortly in a step that follows this one.
5. As oracle :
On nusclust16007:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
./racgons add_config nusclust160##:6251
This should take about one second to run.
If it says that it has already been added to the OCR you are fine.
If it hangs, you may need to reboot all servers to clear this issue.
6. Insure new node is properly added to ocr by running
On nusclust16007:
ocrdump
Check for the entries that show:
[DATABASE.ONS_HOSTS.nusclust160##.PORT]
ORATEXT : 6251
7. Check that your cluster is integrated and that the cluster is not divided into
partitions by completing the following operations:
On nusclust16007:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
./cluvfy comp clumgr -n all -verbose
Should see Verification of cluster manager integrity was successful.
8.
Use the following command to perform an integrated validation of the Oracle
Clusterware setup on all of the configured nodes, both the preexisting nodes
and the nodes that you have added:
AS oracle on nusclust16007:
cluvfy stage -post crsinst -n all -verbose
Post-check for cluster services setup was successful.
good: Post-check for cluster services setup was successful.
9.
On nusclust160## as oracle run the following:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
./crs_stat -t | grep OFFLINE
If you see this:
ora.nusclust160##.ons application ONLINE OFFLINE
then run this:
./crs_start -all
After:
./crs_stat -t
ora.nusclust160##.ons application ONLINE ONLINE nusclust160##
If you see the above then you can move on the next step.
Adding database binaries to the nusclust160## server and setting up the listener.
1.
From nusclust16007:
Open an X window (The OUI part takes 13 minutes)
cd /apps/dbs/oracle/product/11.1.0/db_1/oui/bin
./runInstaller -addNode ORACLE_HOME=/apps/dbs/oracle/product/11.1.0/db_1 $*
You should get a prompt to specify a new node, in this case you should see nusclust160## where you will need to put a check mark beside it.
2.
from nusclust160##:
Eventually you will be prompted to run the following as root on the new node
On nusclust160##
cd /apps/dbs/oracle/product/11.1.0/db_1
./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /apps/dbs/oracle/product/11.1.0/db_1
Enter the full pathname of the local bin directory: [usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /var/opt/oracle/oratab file...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
3. verification
Now set up the .bash_profile and .asm profile to on nusclust160## to support new ORADB4 and +ASM4 instances for the oracle userid.
On nusclust160##:
cp .bash_profile .bash_profile.bak
On nusclust16007:
sftp nusclust160##
put .bash_profile
On nusclust160##:
vi .bash_profile
change ORALCE_SID to ORADB4
cp .bash_profile .asm
vi .asm
change ORALCE_SID to +ASM4 in .asm file
which sqlplus
Should show the path below is $PATH environmental variable is set correctly.
/apps/dbs/oracle/product/11.1.0/db_1/bin/sqlplus
On nusclust160##:
oifcfg getif
This should show:
ce4 10.333.248.192 global cluster_interconnect
ce5 222.65.125.128 global public
4.
Run Netbackup Oracle Agent link script.
As oracle make sure ORACLE_HOME is fined.
env | grep ORACLE_HOME
then
cd /usr/openv/netbackup/bin/
./oracle_link
ls -al $ORACLE_HOME/lib/libobk.so
should show:
/apps/dbs/oracle/product/11.1.0/db_1/lib/libobk.so -> /usr/openv/netbackup/bin/libobk.so64.1
5.
On the target node, run the Net Configuration Assistant (NETCA) to add a
listener. Add a listener to the target node by running NETCA from the target node and
selecting only the target node on the Node Selection page.
I shall do the following on nusclust160## using X Windows
Now before I do this I see:
crs_stat -t
ora.nusclust160##.gsd application ONLINE ONLINE nusclust160##
ora.nusclust160##.ons application ONLINE ONLINE nusclust160##
ora.nusclust160##.vip application ONLINE ONLINE nusclust160##
Connect to nusclust160## and open up X windows session.
netca
Choose Cluster configuration.
select nusclust160## as the node to configure.
Choose Listener configuration, then Add.
When it prompts you for a listener name choose LISTENER as it will append _NUSCLUST160##(server name) to end of the LISTENER name to make a complete listener name. 
At this point you will have listener to support the new node in the crs.
now
crs_stat -t
will show:
ora....0#.lsnr application ONLINE ONLINE nusclust160##
ora.nusclust160##.gsd application ONLINE ONLINE nusclust160##
ora.nusclust160##.ons application ONLINE ONLINE nusclust160##
ora.nusclust160##.vip application ONLINE ONLINE nusclust160##
At this point the necessary crs entries for gsd, ons, vip, and the listener on nusclust160## all we need now are the ORADB4 and +ASM4 instances added.
III. 7/11/2009 7:40 AM Sat [120 min] NTTA DBA
Use NON dbca method to create additional instances on the nusclust160## server. This will involve a complete shutdown of all RAC instances.
1.
Undo tablespace creation was taken care of in Step I,1. Check on the progress of the creation of tablespace UNDOTBS4 in the minimized window. Should see tablespace on primary and physical standby databases.
2. First we shall set up the +ASM4 instance on nusclust160## and add it to the cluster.
On nusclust160##
cd $ORACLE_HOME/dbs
vi init+ASM4.ora
# Copyright (c) 1991, 2001, 2002 by Oracle Corporation
# Cluster Database
cluster_database=true
cluster_database_instances=6
# Miscellaneous
diagnostic_dest=/apps/dbs/oracle
instance_type=asm
# Pools
large_pool_size=12M
asm_diskgroups='DATA1','ARCH','REDO1','REDO2'
asm_diskstring='/raw/asm'
+ASM1.instance_number=1
+ASM2.instance_number=2
+ASM3.instance_number=3
+ASM4.instance_number=4
3.
On nusclust16007
cd $ORACLE_HOME/dbs
sftp nusclust160##
put orapw+ASM1 /apps/dbs/oracle/product/11.1.0/db_1/dbs
put orapwORADB1 /apps/dbs/oracle/product/11.1.0/db_1/dbs
4.
On nusclust160##
cd $ORACLE_HOME/dbs
cp orapw+ASM1 orapw+ASM4
cp orapwORADB1 orapwORADB4
5.
On nusclust160##
cd $HOME
. ./.asm
sqlplus '/ as sysasm'
startup
create spfile from pfile='/apps/dbs/oracle/product/11.1.0/db_1/dbs/init+ASM4.ora' ;
shutdown immediate ;
startup
show parameters spfile
6. Now that we have a running asm instance add it the cluster.
On nusclust160##
srvctl add asm -n nusclust160## -i +ASM4 -o /apps/dbs/oracle/product/11.1.0/db_1
srvctl enable asm -n nusclust160## -i +ASM4
7. Now that we have an asm instance let's set up a database instance.
On nusclust16007/ORADB1 :
alter system set cluster_database_instances=6 scope=spfile ;
alter system set instance_name=ORADB4 scope=spfile sid='ORADB4' ;
alter system set instance_number=4 scope=spfile sid='ORADB4' ;
alter system set local_listener=LISTENER_ NUSCLUST160## scope=both sid='ORADB4' ;
alter system set thread=4 scope=both sid='ORADB4' ;
alter system set undo_tablespace=UNDOTBS4 scope=both sid='ORADB4' ;
alter database add logfile thread 4 group 28 ('+REDO1', '+REDO2' ) size 100M ;
alter database add logfile thread 4 group 29 ('+REDO1', '+REDO2' ) size 100M ;
alter database add logfile thread 4 group 30 ('+REDO1', '+REDO2' ) size 100M ;
alter database add logfile thread 4 group 31 ('+REDO1', '+REDO2' ) size 100M ;
alter database enable public thread 4;
Need to add 5 groups to support standby
So at the end of the day 900M will be added to REDO1(29,577M free) and 900M will be added to REDO2 (29,577M free).
8. Set up init.ora, listener.ora, and tnsnames.ora for ORADB4 on nusclust160##.
a. init.ora set up
cd $ORACLE_HOME/dbs
vi initORADB4.ora
SPFILE='+DATA1/ORADB/spfileORADB.ora'
b. add entries to tnsnames.ora:
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
ORADB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16007-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16008-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16036-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
LISTENERS_ORADB =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16007-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16008-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16036-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
LISTENER_NUSCLUST160## =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
ORADB_PRIM =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16007-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16008-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16036-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
c. add entries to listener.ora, The entries for most of this file should be set already, just insure modifications that need to be made are made.
SID_LIST_LISTENER_NUSCLUST160## =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /apps/dbs/oracle/product/11.1.0/db_1)
(PROGRAM = extproc)
(SID_DESC =
(GLOBAL_DBNAME = ORADB)
(ORACLE_HOME = /apps/dbs/oracle/product/11.1.0/db_1)
(SID_NAME = ORADB4)
LISTENER_NUSCLUST160## =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = NUSCLUST160##-vip)(PORT = 1521)(IP = FIRST))
(ADDRESS = (PROTOCOL = TCP)(HOST = 222.65.125.###)(PORT = 1521)(IP = FIRST))
9. Reload the listener.
lsnrclt
set current_listener LISTENER_NUSCLUST160##
reload
exit
10. Check audit trail, add instance to cluster, and start db instance.
a.
Check for audit directory and start the instance.
/apps/dbs/oracle/product/11.1.0/db_1/rdbms/audit
If this audit trail directory does not exist then create it.
b.
srvctl add instance -d ORADB -i ORADB4 -n nusclust160##
srvctl modify instance -d ORADB -i ORADB4 -s +ASM4
srvctl enable instance -d ORADB -i ORADB4
Will probably show: PRKP-1017 : Instance ORADB4 already enabled.
c.
sqlplus '/ as sysdba'
startup
**Because the cluster_database_instances parameter requires the complete shutdown of all instances in the cluster, you might have an issue when it attempts to start the instance. If you receive an error then run:
srvctl stop database -d oradb
sqlplus '/ as sysdba'
startup
shutdown
srvctl start database -d oradb
shutdown
srvctl start instance -d ORADB -i ORADB4 -o open
11.
Modify spfile of ASM1, ASM2, +ASM3
On nusclust16007
. ./.asm
sqlplus '/ as sysasm'
alter system set instance_number=4 scope=spfile sid='+ASM4' ;
On nusclust16008
. ./.asm
sqlplus '/ as sysasm'
alter system set instance_number=4 scope=spfile sid='+ASM4' ;
On nusclust16036
. ./.asm
sqlplus '/ as sysasm'
alter system set instance_number=4 scope=spfile sid='+ASM4' ;
b Modify tnsnames.ora on nusclust nusclust16007, nusclust16008, and nusclust16036
On nusclust16007
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
Add the following line to the ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the LISTENERS_ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the ORADB_PRIM alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
On nusclust16008
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
Add the following line to the ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the LISTENERS_ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the ORADB_PRIM alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
On nusclust16036
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
Add the following line to the ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the LISTENERS_ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the ORADB_PRIM alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
c Add standby logs on primary to support 4th node.
alter database add standby logfile thread 4 group 32 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 33 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 34 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 35 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 36 ('+REDO1', '+REDO2' ) size 100M ;
12.
Test the cluster to make sure everything is set up correctly.
a. Shutdown resources.
On nusclust16007:
emctl stop dbconsole
ps -ef | grep perl
ps -ef | grep agent
ps -ef | grep java
On nusclust16008:
emctl stop dbconsole
On nusclust16036:
emctl stop dbconsole
On nusclust16008:
cd $HOME
. ./.rman
cd scripts
./go
shutdown immediate
cd $HOME
. ./.bash_profile
srvctl stop database -d oradb
crs_stop -all
crs_stat -t
b. Startup resources
On nusclust16007:
cd $HOME
. ./.bash_profile
crs_start -all
crs_stat -t
The command above should show everything up and running.
ocrcheck
On nusclust16008:
cd $HOME
. ./.rman
cd scripts
./go
startup
On nusclust16007:
emctl start dbconsole
On nusclust16008:
emctl start dbconsole
On nusclust16036:
emctl start dbconsole
How does that work for you?
-JR jr