Pointing existing RAC nodes to a fresh Shared Storage discarding old one
Hi,
I have a RAC Setup with the Primary Database on Oracle 10gR2.
For this setup, there is a Physical Standby Database Setup (using DataGuard configuration) also with 30min delay.
Assume that the "Shared Storage" of the Primary DB fails completely.
In the above scenario, my plan is to refresh a "fresh" shared storage device using Physical Standby Database Setup and then "point" the RAC nodes to the new "Shared Storage".
Is this possible?
Simply put, how can I refresh the Primary database using the Standby Database?
Please help with the utilities (RMAN, DatGuard, other non-Oracle products etc.) that can be used to do this.
Regards
Does following Shared Device configuration is fine for 10g RAC on Windows 2003?
. 1 SCSI drive
• Two PCI network adapters on each node in the cluster.
• Storage cables to attach the shared storage device to all computers.
regard.
Similar Messages
-
10g RAC on varitas Cluster Software & Shared storage
we are in process of making 10g RAC without using Oracle Clusterware , we will be using Varitas Cluster software and varitas shared storage , I am looking for some quick notes/article on setting up/Installing this RAC configuration.
Step-By-Step Installation of 9i RAC on VERITAS STORAGE FOUNDATION (DBE/AC) and Solaris
Doc ID: Note:254815.1
These are the notes i was looking for, Question is Only the RDBMS version will be changes , all other setup will be same as mentioned in Notes, and DBA work will start from creating DBs, right? -
Oracle ORA_16000 when trying to add standby instance to existing rac node
I attempted to use dbca to add a new standby instance to an existing cluster. The cluster is 4 nodes, Linux RHEL 5.3 Oracle 11.1.0.7. Also using ASM, asmlib, ocfs2 and shared block devices.
ASM instances are up and functional on all nodes. current config appears to be running normally and correctly.
I have a 4 instance database running on the cluster. I also have 3 physical standby active data guard instances running on 3 of the nodes. I wanted to add a new ADG instance to the 4th node.
While running dbca I received ORA-00604 and ORA-16000.
The active data guard database was open (read only) and redo apply was on. I am using data guard broker as well, but not grid control.
Does anyone have a procedure for adding an instance in this environment? Do I need to have the standby in mount state? If dbca won't work does anyone have a manual procedure for adding a new instance?
Thankszulo
Let's say you adding node nusclust160## to you existing cluster and dbca is a pain to use.
Extend clusterware to the nusclust160## server.
re: Page 64 of Oracle® Clusterware Administration and Deployment Guide 11g Release 1 (11.1)
1a.
Add undo tablespace to support additional node.
Re-check space for DATA1 on nusclust16007 and /dbdata/ORADB on sun16109.
As of Thursday, May 21, 2009 the DATA1 asm group has 53,584M free.
As of Thursday, May 21, 2009 the /dbdata/ORADB has 77G free.
In a separate terminal window on nusclust16007 run the following in sqlplus
CREATE UNDO TABLESPACE UNDOTBS4 datafile '+DATA1' SIZE 13300M AUTOEXTEND ON ;
This will take a long time to create this tablespace. Please minimize the window after submitting the ddl and move on to the next step.
1b.
Insure .bash_profile on nusclust160## should look like this:
vi .bash_profile
export ORACLE_HOSTNAME=nusclust160##
export ORACLE_SID=ORADB4
export ORA_CRS_BASE=/apps/ocr/oracle
export ORACLE_BASE=/apps/dbs/oracle
export PATH=/usr/ccs/bin:/usr/X/bin:/usr/bin:/usr/sfw/bin:/usr/sbin:/usr/local/bin
export server=`uname -n`
export PS1="$ORACLE_SID@$HOSTNAME >"
alias cls='clear'
alias More='more'
alias ll='ls -lt | more'
Gather IP addresses for fourth node from /etc/hosts:
222.65.125.### nusclust160##
222.65.125.### nusclust160##-vip
10.333.248.### nusclust160##-priv
2. Start Oracle Universal Installer:
Go to CRS_home/oui/bin and run the addNode.sh script on one of the existing
nodes. Oracle Universal Installer runs in add node mode.
The Oracle inventory on nusclust16007, nusclust16008, and nusclust16036 are found under:
/home/oracle/oraInventory
Use a X windows enabled session (The OUI takes 33 minutes)
cd /apps/ocr/oracle/product/11.1.0/crs/oui/bin
./addNode.sh
a. In the first screen specify a new node as :
Public Node Name: nusclust160##
Private Node Name: nusclust160##-priv
Virtual Host Name: nusclust160##-vip
If you receive the error:
" tar. ./bin/racgvip.orig: Permission denied"
Do the following:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
ls -al racgvip.orig
paste here:
chown root:oinstall racgvip.orig
chmod 771 racgvip.orig
should now show:
-rwxrwx--x 1 root oinstall 19213 Feb 11 08:36 racgvip.orig
As root:
a.
On nusclust160##:
cd /home/oracle/oraInventory
./orainstRoot.sh
b.
On nusclust16007:
cd /apps/ocr/oracle/product/11.1.0/crs/install
./rootaddnode.sh
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 4: nusclust160## nusclust160##-priv nusclust160##
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/apps/ocr/oracle/product/11.1.0/crs/bin/srvctl add nodeapps -n nusclust160## -A nusclust160##-vip/255.255.255.224/bge0
c.
On nusclust160##:
cd /apps/ocr/oracle/product/11.1.0/crs/
./root.sh
WARNING: directory '/apps/ocr/oracle/product/11.1.0' is not owned by root
WARNING: directory '/apps/ocr/oracle/product' is not owned by root
WARNING: directory '/apps/ocr/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
OCR LOCATIONS = /raw/ocr/ocrconf1,/raw/ocr/ocrconf2
OCR backup directory '/apps/ocr/oracle/product/11.1.0/crs/cdata/rac_cluster' does not exist. Creating now
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/apps/ocr/oracle/product/11.1.0' is not owned by root. Changing owner to root
The directory '/apps/ocr/oracle/product' is not owned by root. Changing owner to root
The directory '/apps/ocr/oracle' is not owned by root. Changing owner to root
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: nusclust16007 nusclust16007-priv nusclust16007
node 2: nusclust16008 nusclust16008-priv nusclust16008
node 3: nusclust16036 nusclust16036-priv nusclust16036
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
nusclust16007
nusclust16008
nusclust16036
nusclust160##
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
4. After this is done crs_stat -t will show nusclust160## in the crs i.e.
I see:
Name Type Target State Host
ora....160##.gsd application ONLINE ONLINE sun...160##
ora....160##.ons application ONLINE OFFLINE
ora....160##.vip application ONLINE ONLINE sun...160##
Do not be concerned about ora.nusclust160##.ons being OFFLINE, as that will be fixed shortly in a step that follows this one.
5. As oracle :
On nusclust16007:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
./racgons add_config nusclust160##:6251
This should take about one second to run.
If it says that it has already been added to the OCR you are fine.
If it hangs, you may need to reboot all servers to clear this issue.
6. Insure new node is properly added to ocr by running
On nusclust16007:
ocrdump
Check for the entries that show:
[DATABASE.ONS_HOSTS.nusclust160##.PORT]
ORATEXT : 6251
7. Check that your cluster is integrated and that the cluster is not divided into
partitions by completing the following operations:
On nusclust16007:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
./cluvfy comp clumgr -n all -verbose
Should see Verification of cluster manager integrity was successful.
8.
Use the following command to perform an integrated validation of the Oracle
Clusterware setup on all of the configured nodes, both the preexisting nodes
and the nodes that you have added:
AS oracle on nusclust16007:
cluvfy stage -post crsinst -n all -verbose
Post-check for cluster services setup was successful.
good: Post-check for cluster services setup was successful.
9.
On nusclust160## as oracle run the following:
cd /apps/ocr/oracle/product/11.1.0/crs/bin
./crs_stat -t | grep OFFLINE
If you see this:
ora.nusclust160##.ons application ONLINE OFFLINE
then run this:
./crs_start -all
After:
./crs_stat -t
ora.nusclust160##.ons application ONLINE ONLINE nusclust160##
If you see the above then you can move on the next step.
Adding database binaries to the nusclust160## server and setting up the listener.
1.
From nusclust16007:
Open an X window (The OUI part takes 13 minutes)
cd /apps/dbs/oracle/product/11.1.0/db_1/oui/bin
./runInstaller -addNode ORACLE_HOME=/apps/dbs/oracle/product/11.1.0/db_1 $*
You should get a prompt to specify a new node, in this case you should see nusclust160## where you will need to put a check mark beside it.
2.
from nusclust160##:
Eventually you will be prompted to run the following as root on the new node
On nusclust160##
cd /apps/dbs/oracle/product/11.1.0/db_1
./root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /apps/dbs/oracle/product/11.1.0/db_1
Enter the full pathname of the local bin directory: [usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /var/opt/oracle/oratab file...
Entries will be added to the /var/opt/oracle/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
3. verification
Now set up the .bash_profile and .asm profile to on nusclust160## to support new ORADB4 and +ASM4 instances for the oracle userid.
On nusclust160##:
cp .bash_profile .bash_profile.bak
On nusclust16007:
sftp nusclust160##
put .bash_profile
On nusclust160##:
vi .bash_profile
change ORALCE_SID to ORADB4
cp .bash_profile .asm
vi .asm
change ORALCE_SID to +ASM4 in .asm file
which sqlplus
Should show the path below is $PATH environmental variable is set correctly.
/apps/dbs/oracle/product/11.1.0/db_1/bin/sqlplus
On nusclust160##:
oifcfg getif
This should show:
ce4 10.333.248.192 global cluster_interconnect
ce5 222.65.125.128 global public
4.
Run Netbackup Oracle Agent link script.
As oracle make sure ORACLE_HOME is fined.
env | grep ORACLE_HOME
then
cd /usr/openv/netbackup/bin/
./oracle_link
ls -al $ORACLE_HOME/lib/libobk.so
should show:
/apps/dbs/oracle/product/11.1.0/db_1/lib/libobk.so -> /usr/openv/netbackup/bin/libobk.so64.1
5.
On the target node, run the Net Configuration Assistant (NETCA) to add a
listener. Add a listener to the target node by running NETCA from the target node and
selecting only the target node on the Node Selection page.
I shall do the following on nusclust160## using X Windows
Now before I do this I see:
crs_stat -t
ora.nusclust160##.gsd application ONLINE ONLINE nusclust160##
ora.nusclust160##.ons application ONLINE ONLINE nusclust160##
ora.nusclust160##.vip application ONLINE ONLINE nusclust160##
Connect to nusclust160## and open up X windows session.
netca
Choose Cluster configuration.
select nusclust160## as the node to configure.
Choose Listener configuration, then Add.
When it prompts you for a listener name choose LISTENER as it will append _NUSCLUST160##(server name) to end of the LISTENER name to make a complete listener name.
At this point you will have listener to support the new node in the crs.
now
crs_stat -t
will show:
ora....0#.lsnr application ONLINE ONLINE nusclust160##
ora.nusclust160##.gsd application ONLINE ONLINE nusclust160##
ora.nusclust160##.ons application ONLINE ONLINE nusclust160##
ora.nusclust160##.vip application ONLINE ONLINE nusclust160##
At this point the necessary crs entries for gsd, ons, vip, and the listener on nusclust160## all we need now are the ORADB4 and +ASM4 instances added.
III. 7/11/2009 7:40 AM Sat [120 min] NTTA DBA
Use NON dbca method to create additional instances on the nusclust160## server. This will involve a complete shutdown of all RAC instances.
1.
Undo tablespace creation was taken care of in Step I,1. Check on the progress of the creation of tablespace UNDOTBS4 in the minimized window. Should see tablespace on primary and physical standby databases.
2. First we shall set up the +ASM4 instance on nusclust160## and add it to the cluster.
On nusclust160##
cd $ORACLE_HOME/dbs
vi init+ASM4.ora
# Copyright (c) 1991, 2001, 2002 by Oracle Corporation
# Cluster Database
cluster_database=true
cluster_database_instances=6
# Miscellaneous
diagnostic_dest=/apps/dbs/oracle
instance_type=asm
# Pools
large_pool_size=12M
asm_diskgroups='DATA1','ARCH','REDO1','REDO2'
asm_diskstring='/raw/asm'
+ASM1.instance_number=1
+ASM2.instance_number=2
+ASM3.instance_number=3
+ASM4.instance_number=4
3.
On nusclust16007
cd $ORACLE_HOME/dbs
sftp nusclust160##
put orapw+ASM1 /apps/dbs/oracle/product/11.1.0/db_1/dbs
put orapwORADB1 /apps/dbs/oracle/product/11.1.0/db_1/dbs
4.
On nusclust160##
cd $ORACLE_HOME/dbs
cp orapw+ASM1 orapw+ASM4
cp orapwORADB1 orapwORADB4
5.
On nusclust160##
cd $HOME
. ./.asm
sqlplus '/ as sysasm'
startup
create spfile from pfile='/apps/dbs/oracle/product/11.1.0/db_1/dbs/init+ASM4.ora' ;
shutdown immediate ;
startup
show parameters spfile
6. Now that we have a running asm instance add it the cluster.
On nusclust160##
srvctl add asm -n nusclust160## -i +ASM4 -o /apps/dbs/oracle/product/11.1.0/db_1
srvctl enable asm -n nusclust160## -i +ASM4
7. Now that we have an asm instance let's set up a database instance.
On nusclust16007/ORADB1 :
alter system set cluster_database_instances=6 scope=spfile ;
alter system set instance_name=ORADB4 scope=spfile sid='ORADB4' ;
alter system set instance_number=4 scope=spfile sid='ORADB4' ;
alter system set local_listener=LISTENER_ NUSCLUST160## scope=both sid='ORADB4' ;
alter system set thread=4 scope=both sid='ORADB4' ;
alter system set undo_tablespace=UNDOTBS4 scope=both sid='ORADB4' ;
alter database add logfile thread 4 group 28 ('+REDO1', '+REDO2' ) size 100M ;
alter database add logfile thread 4 group 29 ('+REDO1', '+REDO2' ) size 100M ;
alter database add logfile thread 4 group 30 ('+REDO1', '+REDO2' ) size 100M ;
alter database add logfile thread 4 group 31 ('+REDO1', '+REDO2' ) size 100M ;
alter database enable public thread 4;
Need to add 5 groups to support standby
So at the end of the day 900M will be added to REDO1(29,577M free) and 900M will be added to REDO2 (29,577M free).
8. Set up init.ora, listener.ora, and tnsnames.ora for ORADB4 on nusclust160##.
a. init.ora set up
cd $ORACLE_HOME/dbs
vi initORADB4.ora
SPFILE='+DATA1/ORADB/spfileORADB.ora'
b. add entries to tnsnames.ora:
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
ORADB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16007-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16008-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16036-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
LISTENERS_ORADB =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16007-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16008-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16036-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
LISTENER_NUSCLUST160## =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
ORADB_PRIM =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16007-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16008-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust16036-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
c. add entries to listener.ora, The entries for most of this file should be set already, just insure modifications that need to be made are made.
SID_LIST_LISTENER_NUSCLUST160## =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /apps/dbs/oracle/product/11.1.0/db_1)
(PROGRAM = extproc)
(SID_DESC =
(GLOBAL_DBNAME = ORADB)
(ORACLE_HOME = /apps/dbs/oracle/product/11.1.0/db_1)
(SID_NAME = ORADB4)
LISTENER_NUSCLUST160## =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = NUSCLUST160##-vip)(PORT = 1521)(IP = FIRST))
(ADDRESS = (PROTOCOL = TCP)(HOST = 222.65.125.###)(PORT = 1521)(IP = FIRST))
9. Reload the listener.
lsnrclt
set current_listener LISTENER_NUSCLUST160##
reload
exit
10. Check audit trail, add instance to cluster, and start db instance.
a.
Check for audit directory and start the instance.
/apps/dbs/oracle/product/11.1.0/db_1/rdbms/audit
If this audit trail directory does not exist then create it.
b.
srvctl add instance -d ORADB -i ORADB4 -n nusclust160##
srvctl modify instance -d ORADB -i ORADB4 -s +ASM4
srvctl enable instance -d ORADB -i ORADB4
Will probably show: PRKP-1017 : Instance ORADB4 already enabled.
c.
sqlplus '/ as sysdba'
startup
**Because the cluster_database_instances parameter requires the complete shutdown of all instances in the cluster, you might have an issue when it attempts to start the instance. If you receive an error then run:
srvctl stop database -d oradb
sqlplus '/ as sysdba'
startup
shutdown
srvctl start database -d oradb
shutdown
srvctl start instance -d ORADB -i ORADB4 -o open
11.
Modify spfile of ASM1, ASM2, +ASM3
On nusclust16007
. ./.asm
sqlplus '/ as sysasm'
alter system set instance_number=4 scope=spfile sid='+ASM4' ;
On nusclust16008
. ./.asm
sqlplus '/ as sysasm'
alter system set instance_number=4 scope=spfile sid='+ASM4' ;
On nusclust16036
. ./.asm
sqlplus '/ as sysasm'
alter system set instance_number=4 scope=spfile sid='+ASM4' ;
b Modify tnsnames.ora on nusclust nusclust16007, nusclust16008, and nusclust16036
On nusclust16007
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
Add the following line to the ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the LISTENERS_ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the ORADB_PRIM alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
On nusclust16008
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
Add the following line to the ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the LISTENERS_ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the ORADB_PRIM alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
On nusclust16036
ORADB4 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ORADB)
(INSTANCE_NAME = ORADB4)
Add the following line to the ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the LISTENERS_ORADB alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
Add the following line to the ORADB_PRIM alias:
(ADDRESS = (PROTOCOL = TCP)(HOST = nusclust160##-vip)(PORT = 1521))
c Add standby logs on primary to support 4th node.
alter database add standby logfile thread 4 group 32 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 33 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 34 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 35 ('+REDO1', '+REDO2' ) size 100M ;
alter database add standby logfile thread 4 group 36 ('+REDO1', '+REDO2' ) size 100M ;
12.
Test the cluster to make sure everything is set up correctly.
a. Shutdown resources.
On nusclust16007:
emctl stop dbconsole
ps -ef | grep perl
ps -ef | grep agent
ps -ef | grep java
On nusclust16008:
emctl stop dbconsole
On nusclust16036:
emctl stop dbconsole
On nusclust16008:
cd $HOME
. ./.rman
cd scripts
./go
shutdown immediate
cd $HOME
. ./.bash_profile
srvctl stop database -d oradb
crs_stop -all
crs_stat -t
b. Startup resources
On nusclust16007:
cd $HOME
. ./.bash_profile
crs_start -all
crs_stat -t
The command above should show everything up and running.
ocrcheck
On nusclust16008:
cd $HOME
. ./.rman
cd scripts
./go
startup
On nusclust16007:
emctl start dbconsole
On nusclust16008:
emctl start dbconsole
On nusclust16036:
emctl start dbconsole
How does that work for you?
-JR jr -
Is there any doucment to install RAC 10G R2 with vmware shared storage?
Hello Guys,
Is there any documentation or how to available to install Oracle RAC 10G R2 on windows 2000 platform with 2 nodes and using vmware software for shared disk purpose.
Please let me know the link. I will be really greatful to you. There is a document available for windows 2003 but couldnt find any for windows 2000.
Regards,
Imran BaigHello Guys,
I was reading this article on link http://www.dizwell.com/prod/node/25 it says the following
"If you had a physical machine with two network cards installed and a second hard disk with absolutely nothing else on it, you could achieve a RAC using a physical machine"
I am in procesof installing 2 nodes RAC and have configure network requirements on each node. I am struck with shared disk storage... can i acheive a shared diak storage by adding an other hardrive to one of the nodes? Please help...
Regards,
Imran -
Can GG work in a RAC environment which has no shared storage except ASM
Gurus:
I have a 2 nodes RAC which is on ASM for database data and FRA. However, it has no other shared space on the file systems.
Can I still use GoldenGate? How can I do it.
Please advise.
dzGoldenGate needs to be able to read the online redo logs for each thread, and running on ASM requires a few extra configuration steps. You need an ASM user to be able to connect to the ASM instance (involving listener and tnsnames.ora files). Extract parameter file uses the TRANLOGOPTIONS parameter. This is covered in the installation guide ("Additional requirements for ASM," plus the section on "Additional requirements for Oracle RAC").
In 11gR2, you can use ACFS for the GoldenGate software and its files.
Oracle GoldenGate High Availability using Oracle Clusterware Technical Whitepaper
http://www.oracle.com/technetwork/middleware/goldengate/overview/ha-goldengate-whitepaper-128197.pdf -
Is it possible to move some of the capture processes to another rac node?
Hi All,
Is it possible to move some of the ODI (Oracle Data Integrator) capture processes running on node1 to node2. Once moved does it work as usual or not? If its possible please provide me with steps.
Appreciate your response
Best Regards
SK.Hi Cezar,
Thanks for your post. I have a related question regarding this,
Is it really necessary to have multiple capture and multiple apply processes? One for each schema in ODI? Because if set to automatic configuration, ODI seems to create a capture and a related apply process for each schema, which I guess leads to our specific performance problem (high cpu etc) I mentioned in my other post: Re: Is it possible to move some of the capture processes to another rac node?
Is there way to use just one capture and one apply process for all of the schemas in ODI?
Thanks a million.
Edited by: oyigit on Nov 6, 2009 5:31 AM -
How to Reorganize CSM200 Shared Storage in Solaris 10 x86 Oracle 10gR2
I could use some guidance from those who are more experienced in RAC administration in a Solaris environment with ASM. I have a three-node RAC with Oracle 10gR2 instances on top of Solaris 10 x86 where the shared storage is a Sun CSM200 disk array which looks like a single disk to the rest of the world. I'm not very familiar with the CSM200 Common Array Manager but I do have access to use it.
During initial setup, I followed the Oracle cookbook and defined a storage slice for each of the following: OCR, OCR mirror, three voting disks, and +DATA, for a total of six slices. I brought up the RAC and we've used it for a couple of weeks.
This is a Dev and QA environment, so it changes pretty fast. The new requirement is to add a +FRA and to add a mount point for a file system on the shared storage, so that all three Oracle instances can refer to the same external table(s).
However, I've already used all the available slices in the VTOC on the shared logical drive. I'm not sure how to proceed.
1) Is it necessary to use the CAM to create two logical disks out of the single existing logical disk?
2) If so, how destructive is that? I don't need to keep the contents of the database, but I do not want to reinstall CRS or ASM or the DB instances.
3) Is it possible to combine the OCR and its mirror on the same slice, thus freeing a slice for reuse?
4) Is it possible to combine all three voting disks on the same slice, thus freeing two slices for reuse?
Edited by: user12006221 on Mar 29, 2011 3:30 PM
Another question: Under 10.2.0.4, is it possible for the OCR and voting disks to be managed by ASM? I know it would be possible under 11g, but that's not an option as I am trying to match a customer's environment and they aren't going to 11g any time real soon.What you see is what happens when the Java runtime running on Solaris 10 x86 tries to load a library which is compiled for SPARC.
Because of the native parts in SAP GUI for Java, compilations and installers are required for each OS - HW combination.
The supported platforms can be seen in SAP note 954572. For Solaris only SPARC is currently supported.
Because of the effort needed for compiling, testing, support etc. it is required to focus on OS - HW combinations widely used on desktop machines and Solaris 10 on x86 currently does not seem to be one of those. -
One very basic question on Shared Storage
For setting up 11GR2 in Sun 5.10 SPARC, our sys admins have allocated block devices in shared storage and they sent us a mail mentioning the disks. This is what their mail looks like
Storage attached to Node1
/dev/rdsk/c4ikdzs3
/dev/rdsk/c4ikdzs4
.Storage attached to Node2
/dev/rdsk/c5ikdzj2
/dev/rdsk/c5ikdzj5
.When i logged in to Node1:/dev/rdsk , i can only see
/dev/rdsk/c4ikdzs3
/dev/rdsk/c4ikdzs4
.as they have mentioned. Same for Node2
But, all the raw devices mentioned above should be visible from either node's /dev/rdsk. Right? Isn't that whole point of RAC; Shared storage ?resistanceIsFruitful wrote:
But, all the raw devices mentioned above should be visible from either node's /dev/rdsk. Right? Isn't that whole point of RAC; Shared storage ?Not exactly if you are referring to device names..
Each kernel will do a h/w discovery when booting. When dealing with LUNs via a HBA (or similar), there's absolutely no guarantee that the kernels will detect the LUNs in the same sequence and assign the same scsi device names to these. A LUN can be called device-foo-1 on one server and device-foo-21 on another.
Also, many HBAs will be dual port and running dual fibre channels. So not only is the same LUN seen as a different scsi device by each kernel, but it is seen more than once. So device-foo-1 and device-foo-33 can be the same physical LUN on server 1.
To deal with this a logical device name is needed. This will be the same device name on all servers - and it will in turn transparently support the multiple I/O paths to the LUN. This is done by "special driver" software looking at the scsi disk's unique signature - called a WWID or World Wide Name. With that unique signature, the s/w can uniquely recognise a specific LUN, irrespective of which server the s/w runs on.
This s/w is called Multipath on Linux, Powerpath by EMC and so on. I would expect that you will have something similar on your servers.
The actual scsi device mapped by the kernel to the LUN is not used. In the case of multipath for example, one will use the relevant +/dev/mpath/mpath<n>+ devices. In the case of Powerpath, these will be +/dev/emcpower<driveletter>+ devices.
These are the device names that you will use for 11g Grid Infrastructure and Oracle ASM and RAC setup and configuration as shared storage. -
Shared storage and grid configuration
hello
i want to configure grid invoirenment on two p4 computers
now i am comfused about storage userd by cluster nodes
i read that only shared storage is used which is shared both clusters nodes
please tell me that is there any there cheeper option instead of shared device
if not then can i use shared storage with only one 40 GB hard drive
if there is any link related to configure grid comluting and shared storeage
i want any cheeper option to form this beacuse i am student of comluter science
and this is my university project and i am woried about it
thanks in advance
allah hafizThe cheapest storage solution is firewire. Check out:
http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_10.shtml
Bert
hello
i want to configure grid invoirenment on two p4
computers
now i am comfused about storage userd by cluster
nodes
i read that only shared storage is used which is
shared both clusters nodes
please tell me that is there any there cheeper option
instead of shared device
if not then can i use shared storage with only one 40
GB hard drive
if there is any link related to configure grid
comluting and shared storeage
i want any cheeper option to form this beacuse i am
student of comluter science
and this is my university project and i am woried
about it
thanks in advance
allah hafiz -
Hello everyone,
I have met an error,that is our RAC node auto restart with below messages.
#/u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/alert_odsdb1.log
Fri Jun 07 12:23:42 2013
Thread 1 cannot allocate new log, sequence 58363
Checkpoint not complete
Current log# 2 seq# 58362 mem# 0: +DATA/odsdb/onlinelog/group_2.265.812288839
Current log# 2 seq# 58362 mem# 1: +DATA/odsdb/onlinelog/group_2.266.812288839
Fri Jun 07 12:23:42 2013
NOTE: ASMB terminating
Errors in file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc:
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
Errors in file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc:
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
ASMB (ospid: 32641): terminating the instance due to error 15064
Fri Jun 07 12:23:44 2013
ORA-1092 : opitsk aborting process
Fri Jun 07 12:23:46 2013
ORA-1092 : opitsk aborting process
Instance terminated by ASMB, pid = 32641
Fri Jun 07 12:25:02 2013
Starting ORACLE instance (normal)
Fri Jun 07 12:25:23 2013
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.37.103, mac=00-26-55-eb-61-89, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=135.33.2.8, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
Public Interface 'eth0:1' configured from GPnP for use as a public interface.
[name='eth0:1', type=1, ip=135.33.2.13, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/dbhome_2/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_2
System name: Linux
Node name: odsdb1
Release: 2.6.18-308.el5
Version: #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine: x86_64
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0/dbhome_2/dbs/initodsdb1.ora
System parameters with non-default values:
processes = 4500
sessions = 6784
event = ""
spfile = "+DATA/odsdb/spfileodsdb.ora"
nls_language = "SIMPLIFIED CHINESE"
nls_territory = "CHINA"
memory_target = 170G
control_files = "+DATA/odsdb/controlfile/current.262.812288837"
control_files = "+DATA/odsdb/controlfile/current.261.812288837"
db_block_size = 8192
compatible = "11.2.0.0.0"
db_files = 4096
cluster_database = TRUE
db_create_file_dest = "+DATA"
db_recovery_file_dest = ""
db_recovery_file_dest_size= 38820M
thread = 1
undo_tablespace = "UNDOTBS1"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=odsdbXDB)"
remote_listener = "odsdb-cluster-scan:1521"
job_queue_processes = 1000
audit_file_dest = "/u01/app/oracle/admin/odsdb/adump"
audit_trail = "DB"
db_name = "odsdb"
open_cursors = 300
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
169.254.37.103
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Fri Jun 07 12:25:33 2013
PMON started with pid=2, OS id=22959
Fri Jun 07 12:25:33 2013
PSP0 started with pid=3, OS id=22962
Fri Jun 07 12:25:34 2013
VKTM started with pid=4, OS id=22971 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Fri Jun 07 12:25:34 2013
GEN0 started with pid=5, OS id=22977
Fri Jun 07 12:25:34 2013
DIAG started with pid=6, OS id=22979
Fri Jun 07 12:25:35 2013
DBRM started with pid=7, OS id=22981
Fri Jun 07 12:25:35 2013
PING started with pid=8, OS id=22983
Fri Jun 07 12:25:35 2013
ACMS started with pid=9, OS id=22985
Fri Jun 07 12:25:35 2013
DIA0 started with pid=10, OS id=22987
Fri Jun 07 12:25:35 2013
LMON started with pid=11, OS id=22989
Fri Jun 07 12:25:35 2013
LMD0 started with pid=12, OS id=22991
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [61440 - 81920]
Fri Jun 07 12:25:35 2013
LMS0 started with pid=13, OS id=22994 at elevated priority
Fri Jun 07 12:25:35 2013
LMS1 started with pid=14, OS id=22998 at elevated priority
Fri Jun 07 12:25:35 2013
LMS2 started with pid=15, OS id=23002 at elevated priority
Fri Jun 07 12:25:35 2013
LMS3 started with pid=16, OS id=23006 at elevated priority
Fri Jun 07 12:25:35 2013
RMS0 started with pid=17, OS id=23010
Fri Jun 07 12:25:35 2013
LMHB started with pid=18, OS id=23013
Fri Jun 07 12:25:35 2013
MMAN started with pid=19, OS id=23015
Fri Jun 07 12:25:35 2013
DBW0 started with pid=20, OS id=23017
Fri Jun 07 12:25:35 2013
DBW1 started with pid=21, OS id=23019
Fri Jun 07 12:25:35 2013
DBW2 started with pid=22, OS id=23022
Fri Jun 07 12:25:35 2013
DBW3 started with pid=23, OS id=23024
Fri Jun 07 12:25:35 2013
DBW4 started with pid=24, OS id=23026
Fri Jun 07 12:25:35 2013
DBW5 started with pid=25, OS id=23028
Fri Jun 07 12:25:35 2013
DBW6 started with pid=26, OS id=23031
Fri Jun 07 12:25:35 2013
DBW7 started with pid=27, OS id=23033
Fri Jun 07 12:25:35 2013
LGWR started with pid=28, OS id=23035
Fri Jun 07 12:25:35 2013
CKPT started with pid=29, OS id=23037
Fri Jun 07 12:25:35 2013
SMON started with pid=30, OS id=23039
Fri Jun 07 12:25:35 2013
RECO started with pid=31, OS id=23041
Fri Jun 07 12:25:35 2013
RBAL started with pid=32, OS id=23043
Fri Jun 07 12:25:35 2013
ASMB started with pid=33, OS id=23045
Fri Jun 07 12:25:35 2013
MMON started with pid=34, OS id=23048
Fri Jun 07 12:25:35 2013
MMNL started with pid=35, OS id=23052
Fri Jun 07 12:25:35 2013
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
NOTE: initiating MARK startup
starting up 1 shared server(s) ...
Starting background process MARK
Fri Jun 07 12:25:35 2013
MARK started with pid=37, OS id=23056
NOTE: MARK has subscribed
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 119)
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid according to instance 2
* domain 0 valid = 1 according to instance 2
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration started (old inc 119, new inc 121)
List of instances:
1 2 (myinst: 1)
Nested reconfiguration detected.
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Fri Jun 07 12:25:45 2013
Submitted all GCS remote-cache requests
Fri Jun 07 12:26:08 2013
Fix write in gcs resources
Reconfiguration complete
Fri Jun 07 12:26:10 2013
LCK0 started with pid=40, OS id=23632
Fri Jun 07 12:26:10 2013
Starting background process RSMN
Fri Jun 07 12:26:10 2013
RSMN started with pid=41, OS id=23646
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /u01/app/oracle
Fri Jun 07 12:26:11 2013
ALTER SYSTEM SET local_listener=' (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=135.33.2.13)(PORT=1521))))' SCOPE=MEMORY SID='odsdb1';
ALTER DATABASE MOUNT /* db agent *//* {1:9971:2} */
Fri Jun 07 12:26:11 2013
NOTE: Loaded library: System
Fri Jun 07 12:26:11 2013
SUCCESS: diskgroup DATA was mounted
Fri Jun 07 12:26:11 2013
NOTE: dependency between database odsdb and diskgroup resource ora.DATA.dg is established
Fri Jun 07 12:26:16 2013
Successful mount of redo thread 1, with mount id 3452000551
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Completed: ALTER DATABASE MOUNT /* db agent *//* {1:9971:2} */
ALTER DATABASE OPEN /* db agent *//* {1:9971:2} */
Picked broadcast on commit scheme to generate SCNs
Thread 1 advanced to log sequence 58364 (thread open)
Thread 1 opened at log sequence 58364
Current log# 2 seq# 58364 mem# 0: +DATA/odsdb/onlinelog/group_2.265.812288839
Current log# 2 seq# 58364 mem# 1: +DATA/odsdb/onlinelog/group_2.266.812288839
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jun 07 12:26:21 2013
SMON: enabling cache recovery
Fri Jun 07 12:26:23 2013
minact-scn: Inst 1 is a slave inc#:121 mmon proc-id:23048 status:0x2
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
Fri Jun 07 12:26:34 2013
[23651] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2061372614 end:2061384964 diff:12350 (123 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Fri Jun 07 12:26:34 2013
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Starting background process GTX0
Fri Jun 07 12:26:35 2013
GTX0 started with pid=45, OS id=23931
Starting background process RCBG
Fri Jun 07 12:26:35 2013
RCBG started with pid=46, OS id=23933
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Fri Jun 07 12:26:35 2013
QMNC started with pid=48, OS id=23940
Completed: ALTER DATABASE OPEN /* db agent *//* {1:9971:2} */
Fri Jun 07 12:26:38 2013
Starting background process CJQ0
Fri Jun 07 12:26:38 2013
CJQ0 started with pid=55, OS id=23977
Fri Jun 07 12:27:56 2013
Thread 1 advanced to log sequence 58365 (LGWR switch)
Current log# 1 seq# 58365 mem# 0: +DATA/odsdb/onlinelog/group_1.263.812288839
Current log# 1 seq# 58365 mem# 1: +DATA/odsdb/onlinelog/group_1.264.812288839
Fri Jun 07 12:28:18 2013
Starting background process SMCO
Fri Jun 07 12:28:18 2013
SMCO started with pid=70, OS id=25166
Fri Jun 07 12:29:01 2013
Thread 1 cannot allocate new log, sequence 58366
Trace file /u01/app/oracle/diag/rdbms/odsdb/odsdb1/trace/odsdb1_asmb_32641.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_2
System name: Linux
Node name: odsdb1
Release: 2.6.18-308.el5
Version: #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine: x86_64
Instance name: odsdb1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 33
Unix process pid: 32641, image: oracle@odsdb1 (ASMB)
*** 2013-05-14 15:37:08.705
*** SESSION ID:(3499.1) 2013-05-14 15:37:08.705
*** CLIENT ID:() 2013-05-14 15:37:08.705
*** SERVICE NAME:() 2013-05-14 15:37:08.705
*** MODULE NAME:() 2013-05-14 15:37:08.705
*** ACTION NAME:() 2013-05-14 15:37:08.705
NOTE: initiating MARK startup
*** 2013-05-14 15:37:16.835
instance health monitoring reports instance shutting down
*** 2013-06-07 12:23:42.700
NOTE: ASMB terminating
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
error 15064 detected in background process
ORA-15064: ? ASM ??????
ORA-03113: ?????????
?? ID:
?? ID: 2047 ???: 5
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+461<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+53<-ksuitm()+1332<-ksbrdp()+3344<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36
----- End of Abridged Call Stack Trace -----
*** 2013-06-07 12:23:42.783
ASMB (ospid: 32641): terminating the instance due to error 15064
/u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
NOTE: ASMB process exiting, either shutdown is in progress
NOTE: or foreground connected to ASMB was killed.
Fri Jun 07 12:23:42 2013
NOTE: client exited [14808]
Fri Jun 07 12:23:44 2013
Received an instance abort message from instance 2
Please check instance 2 alert and LMON trace files for detail.
Fri Jun 07 12:23:44 2013
Received an instance abort message from instance 2
Please check instance 2 alert and LMON trace files for detail.
LMD0 (ospid: 31201): terminating the instance due to error 481
Instance terminated by LMD0, pid = 31201
Fri Jun 07 12:24:30 2013
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.37.103, mac=00-26-55-eb-61-89, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=135.33.2.8, mac=00-26-55-eb-61-88, net=135.33.2.0/27, mask=255.255.255.224, use=public/1]
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/11.2.0.2/grid/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
[grid@odsdb1 cssd]$ file core.30481
core.30481: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'ocssd.bin'
[grid@odsdb1 cssd]$ gdb
gdb gdbserver gdbtui
[grid@odsdb1 cssd]$ gdb ocssd.bin core.30481
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-42.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /u01/app/11.2.0.2/grid/bin/ocssd.bin...(no debugging symbols found)...done.
[New Thread 30486]
[New Thread 30530]
[New Thread 30526]
[New Thread 30525]
[New Thread 30523]
[New Thread 30522]
[New Thread 30521]
[New Thread 30520]
[New Thread 30519]
[New Thread 30504]
[New Thread 30503]
[New Thread 30495]
[New Thread 30485]
[New Thread 30484]
[New Thread 30483]
[New Thread 30481]
Reading symbols from /u01/app/11.2.0.2/grid/lib/libhasgen11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libhasgen11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libocr11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocr11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libocrb11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocrb11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libocrutl11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libocrutl11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libclntsh.so.11.1...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libclntsh.so.11.1
Reading symbols from /u01/app/11.2.0.2/grid/lib/libskgxn2.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libskgxn2.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /u01/app/11.2.0.2/grid/lib/libasmclntsh11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libasmclntsh11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libcell11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libcell11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libskgxp11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libskgxp11.so
Reading symbols from /u01/app/11.2.0.2/grid/lib/libnnz11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libnnz11.so
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libaio.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /u01/app/11.2.0.2/grid/lib/libnque11.so...(no debugging symbols found)...done.
Loaded symbols for /u01/app/11.2.0.2/grid/lib/libnque11.so
Reading symbols from /opt/oracle/extapi/64/asm/orcl/1/libasm.so...(no debugging symbols found)...done.
Loaded symbols for /opt/oracle/extapi/64/asm/orcl/1/libasm.so
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff505fd000
Core was generated by `/u01/app/11.2.0.2/grid/bin/ocssd.bin '.
Program terminated with signal 6, Aborted.
#0 0x000000369ea30265 in raise () from /lib64/libc.so.6
(gdb) where
#0 0x000000369ea30265 in raise () from /lib64/libc.so.6
#1 0x000000369ea31d10 in abort () from /lib64/libc.so.6
#2 0x00002afc67f9aeda in scls_abort (flags=0) at scls.c:7088
#3 0x000000000040babd in clssscExit (thrd=0x10d325a0, status=clssscreasonSHUTNORM) at clsssc.c:2155
#4 0x0000000000446221 in clssgmClientShutdown (thrd=0x10d325a0, cmInfo=0x10b40090) at clssgmc.c:6415
#5 0x0000000000436707 in clssgmProcClientReqs (thrd=0x10d325a0, clctx=0x10b40630) at clssgmc.c:704
#6 0x0000000000436405 in clssgmclientlsnr (thrd=0x10d325a0) at clssgmc.c:644
#7 0x000000000040ac2f in clssscthrdmain (thrd=0x10d325a0) at clsssc.c:1716
#8 0x000000369fa0677d in start_thread () from /lib64/libpthread.so.0
#9 0x000000369ead49ad in clone () from /lib64/libc.so.6
(gdb)
2013-06-07 12:19:37.377: [ CSSD][1085888832]clssscSelect: cookie accept request 0x10b40630
2013-06-07 12:19:37.377: [ CSSD][1085888832]clssgmAllocProc: (0x2aaab0133ea0) allocated
2013-06-07 12:19:37.379: [ CSSD][1085888832]clssgmClientConnectMsg: properties of cmProc 0x2aaab0133ea0 - 1,2,3,4,5
2013-06-07 12:19:37.379: [ CSSD][1085888832]clssgmClientConnectMsg: Connect from con(0x6ae44fa) proc(0x2aaab0133ea0) pid(14139/14139) version 11:2:1:4, properties: 1,2,3,4,5
2013-06-07 12:19:37.379: [ CSSD][1085888832]clssgmClientConnectMsg: msg flags 0x0000
2013-06-07 12:19:37.384: [ CSSD][1085888832]clssscSelect: cookie accept request 0x2aaab0133ea0
2013-06-07 12:19:37.384: [ CSSD][1085888832]clssscevtypSHRCON: getting client with cmproc 0x2aaab0133ea0
2013-06-07 12:19:37.384: [ CSSD][1085888832]clssgmRegisterClient: proc(69/0x2aaab0133ea0), client(1/0x2aaab010c5c0)
2013-06-07 12:19:37.385: [ CSSD][1085888832]clssgmRegisterShared: grp DBODSDB, mbr 0, type 1
2013-06-07 12:19:37.385: [ CSSD][1085888832]clssgmQueueShare: (0x2aaab0085790) target global grock DBODSDB member 0 type 1 queued from client (0x2aaab010c5c0), global grock DBODSDB, refcount 23
2013-06-07 12:19:37.385: [ CSSD][1085888832]clssgmRegisterShared: global grock DBODSDB member 0 share type 1, refcount 23
2013-06-07 12:19:37.391: [ CSSD][1085888832]clssscSelect: cookie accept request 0x2aaab0133ea0
2013-06-07 12:19:37.391: [ CSSD][1085888832]clssscevtypSHRCON: getting client with cmproc 0x2aaab0133ea0
2013-06-07 12:19:37.391: [ CSSD][1085888832]clssgmRegisterClient: proc(69/0x2aaab0133ea0), client(2/0x2aaab0061f10)
what is the problem
Edited by: 徐振富 on 2013-6-7 下午6:38
Edited by: 徐振富 on 2013-6-7 下午6:45is your ASM instance up?
If not, trying bring up ASM instance up just by itself and see if it throws any error?
Post status of crsctl status cluster -all -
I have done all the steps to remove one RAC node but got stuck at the step of running rootdelete.sh file from $CRS_HOME/install directory as I don't have this file in windows environment.
What is the equivalent file for rootdelete.sh on windows platform. I want to run this to remove the node info from the clusterware entry.
Is there a good document that explains about removing the node on windows platform.Hello,
You need to run the following steps to remove a node from a RAC cluster on Windows platform:
Perform the following steps on a node other than the node you want to delete:
1. Run the Database Configuration Assistant (DBCA) utility to delete the instance.
2. Then run the Net Configuration Assistant (NetCA) to delete the listener.
3. If the node that you are deleting has ASM instance, then delete the ASM instance using the srvctl stop asm and srvctl remove asm commands.
4. Run the command srvctl stop nodeapps -n nodename of the node to be deleted to stop the node applications.
5. Run the command srvctl remove nodeapps -n nodename of the node to be deleted to remove the node applications.
6. Stop isqlplus if it is running.
7. Run the command setup.exe -updateNodeList ORACLE_HOME=Oracle_home ORACLE_HOME_NAME=Oracle_home_name CLUSTER_NODES=remaining
nodes where remaining nodes is a list of the nodes that are to remain part of the cluster.
Perform the following steps on the deleted RAC node:
1. Run the command setup.exe -updateNodeList -local -noClusterEnabled ORACLE_HOME=Oracle_home ORACLE_HOME_NAME=Oracle_home_name CLUSTER_NODES="".
Note that you do not need a value for "" after the CLUSTER_NODES= entry in this command. If you delete more than one node, then you must run this command on every deleted node to remove the Oracle home if you have a non-shared Oracle home (non-cluster file system) installation.
2. On the same node, delete the Windows Registry entries and ASM services using Oradim.
3. From the deleted RAC node, run the command Oracle_home\oui\bin\setup.exe to start the Oracle Universal Installer (OUI). Select Deinstall Products and select the Oracle home that you want to de-install.
4. Then to delete the CRS node, from a remaining node run the command crssetup del -nn node_name of the deleted node, node number
5. Then run the command setup.exe -updateNodeList ORACLE_HOME=CRS home ORACLE_HOME_NAME=CRS home name CLUSTER_NODES=remaining nodes where remaining nodes is a list of the nodes that are to remain in the cluster.
6. Then on the deleted CRS node, run the command setup.exe -updateNodeList -local -noClusterEnabled ORACLE_HOME=CRS home ORACLE_HOME_NAME=CRS home name CLUSTER_NODES=""
7. Remove the Oracle home manually from the new node if the home is not shared and then manually remove the HKLM/software/Oracle registry keys and the Oracle services. 7
8. After adding or deleting nodes from your Oracle Database 10g with RAC environment, and after you are sure that your system is functioning properly, make a backup of the contents of the voting disk using the dd.exe utility. The dd.exe utility is part of the MKS toolkit.
ASM Instance Cleanup Procedures after Node Deletion on Windows-Based Platforms
The delete node procedure requires the following additional steps on Windows-based systems to remove the ASM instances:
1. If this is the Oracle home from which the node-specific listener named LISTENER_nodename runs, then use NetCA to remove this listener and its CRS resources. If necessary, re-create this listener in another home.
2. If this is the Oracle home from which the ASM instance runs, then remove the ASM configuration by running the following command for all nodes on which this Oracle home exists:
srvctl stop asm -n node
Then run the following command for the nodes that you are removing:
srvctl remove asm -n node
3. If you are using a cluster file system for your ASM Oracle home, then run the following commands on the local node:
4. rd -s -q %ORACLE_BASE%\admin\+ASM
delete %ORACLE_HOME%\database\*ASM*
5. If you are not using a cluster file system for your ASM Oracle home, then run the delete command mentioned in the previous step on each node on which the Oracle home exists.
6. Run the following command on each node that has an ASM instance:
oradim -delete -asmsid +ASMnode_number
Source:
Oracle® Real Application Clusters Administrator's Guide
10g Release 1 (10.1)
Part Number B10765-02
Chapter 5: Adding and Deleting Nodes and Instances
Hope this helps,
Ben Prusinski, Oracle 10g OCP
http://oracle-magician.blogspot.com -
Is it possible to install Oracle RAC without shared storage
Dear All,
I would like to seek for your advice.
I got two different servers. We call it node 1 and node 2. And two different instances name.
Node 1 -> instance name as "ORCL1"
Node 2 -> instance name as "ORCL2"
For the system we need Oracle RAC active-active cluster mode. Our objective is to have 2 replicated databases, in other words we need 2 instances of the same database automatically replicated for 100% up time to the Application server. We have 2 separate database machines and 2 application server machines. We need our application server to connect to any of the databases at any point of time and be having a consistent data on both database machines. We only need the database to be in a cluster mode, we won't need the OS to be in a cluster. There is no shared storage in this case.
Can this be done? Please advice.you should review RAC concepts, and the meaning of instance and database
For the system we need Oracle RAC active-active cluster mode.RAC = single database with multiple instances all accessing the same shared storage, no replication involved
Our objective is to have 2 replicated databases, in other words we need 2 instances of the same database automatically replicated for 100% up time to the Application server.what you describe here is = multiple databases with multiple instances, replicated between each other
We have 2 separate database machines and 2 application server machines. We need our application server to connect to any of the databases at any point of time and be having a consistent data on both database machines. We only need the database to be in a cluster mode, we won't need the OS to be in a cluster. There is no shared storage in this case.no shared storage = no RAC
you will have two seperate databases synchronizing continuously
you can use for example Streams / Advanced Replication (with multi-master configuration)
if you dont insist on an active-active configuration, you can also use Data Guard for building a standby database -
Considering shared storage for Oracle RAC 10g
Hi, guys!
My Oracle RAC will be run on VMware ESXI 5.5. So, both 2 nodes and shared storage are on VM. Don't blame for this, I dont have another choice.
I am choosing shared storage for Oracle RAC. I am choosing between NFS and ISCSI server, both can be done in RedHat linux or FreeNAS.
Can u, guys, help me to do the choise?
RedHat or FreeNAS
ISCSI or NFS
Any help will be appreciated.JohnWatson написал(а):
NFS is really easy. Create your zero-filled files, set the ownership and access modes, and point your asm_diskstring at them. Much simpler than configuring an iSCSI target and initiators, and then messing about with ASMlib or udev.
I recorded a public lecture that (if I remember correctly) describes it here, Oracle ASM Free Tutorial
I will be using OCFS2 as cluster FS. Does it make any difference for NFS vs ISCSI? -
Movig MR from existing RAC db to new standalone database
Hi,
I installed MRCA and created MR on existing RAC database. (Sun X86)
I installed application server by using that MR on RAC database.
Now this application server Portal as middle tier went alive.
Now I would like to MR from existing RAC database to newly installed standalone database.(Sun SPARC 64)
Is it possible0? If yes, how? If no, Why?
source db platform is Sun X86 and destination db platform is Sun SPARC64.
Source db is using ASM file system and destination db is using normal filesystem.
After the move, my application server should work. (Because this is production system).Hi Roberto Barrera,
I’m sorry for the way I posted my queries.
The document you suggested me is really useful but I want to make sure this document will surely applicable for my case. Our concern is to move the entire database which is also MR for Oracle Application Server 10.1.2.3.0. Let me explain you more about our architecture.
Dbnode1& dbnode2 (Sun X86 – OS: Sun Solaris 10)
We have two node RAC db and created MR on that db by MRCA. (Oracle home 1)
We installed Oracle AS Infra tier Identity Manager. (including OID, HA, DIP but excluding SSO component) (oracle home 2)
Midnode-1 & Midnode2 (Sun X86- OS: Sun Solaris 10)
We installed Oracle AS Infra tier Identity Manager. (Including SSO, HA, DIP but excluding OID) (oracle home 1)
We installed Oracle AS Middle tier (Portal and wireless services). (Including Web cache and Portal along with default component j2ee, http) (Oracle home 2)
We need to move whatever running on dbnode1 and 2 to another couple of Sun SPARC64 – OS: Sun Solaris 10 machines. We know binary migration is not possible between X86 and SPARC64. Therefore, I have installed a new database with SPARC64 installation media and created fresh MR by running runRepca.sh script. In one of the document Oracle recommend to do RMAN backup to move entire data from existing (original) database to new (target) database.
http://download.oracle.com/docs/cd/B14099_19/core.1012/b13995/chginfra.htm#BGBDDDBE
(Oracle® Application Server Administrator's Guide 10g Release 2 (10.1.2) B13995-08)
Chapter 9 Changing Infrastructure Services
9.6 Changing the Metadata Repository Used by Identity Management
In this procedure, new database also must use the same Oracle home, datafile location, SID, and global database name as the original Metadata Repositorys. Our newly installed database is not RAC database with ASM but this is standalone database on qfs file system. We could not create database with the same datafile location as the original MR.
• Can we do full database export/import? (your document only talking about portal schema but in my MR there are around 60 schemas related with MR and we need to import all the schemas)
• Is there any other possible solution for case?
Thanks in Advance :)
Edited by: Padmanaban G on Jan 28, 2010 2:58 PM -
Oracle RAC Nodes getting reboot in case of preferred controller failed
When we are disconnecting both Fiber cable from preferred Controller A or plugging out Controller A card from Disk Array(IBM DS 4300), After 90 seconds both the servers are rebooting.
In this time complete RAC network is going out of service for approx 5 minutes.After reboot both servers are coming with both instances without any manual intervention
Its a critical issue for us because we are loosing High Availability, Let us know how we can resolve this critical issue.
Detail of Network:
1. Software- Oracle 10g Release2
2. OS- Redhat Linux 3 (Kernel Version-2.4.21-27.ELsmp)
3. Shared Storage- IBM DS 4300.
4. Multipathing Driver - RDAC (rdac-LINUX-09.00 A5.13)
4. Nodes- IBM 346
5. Databse on ASM
6. ASM,OCR & Voting Disk Preferred controller is A.
7. Hangcheck timer value is 210 seconds.
8. Both Server available with 2 HBA port . I HBA port is connected with Controller A and Seconfd HBA port is connected with Controller B of SAN Disk Array.
As per my understanding,
Voting disk resides in Disk Array and Controller A is preferred owner of Voting Disk LUN.. When i am disconnecting both fiber cable from preferred controller A , then Both Nodes Clusterware software trying to contact with Voting Disk, When they are unable to contact with Voting disk in specfic time period, they are going for reboot.
I tested Controller failure testing with Oracle RAC software as well without Oracle. Without Oracle its working fine and reason behind, in that time Disk Array is waiting for approx 300 seconds for changing preferred controlller from A to B.
But With Oracle, Clusterware Software reboot both nodes before Controller can shift from A to B.
So if i conclude,the tech who has good understanding of Oracle Clusterware on Linux OS & IBM RDAC multipath driver can help me.
when we install Oracle RAC on Linux, it is required to configure hangcheck timer.
Oracle recomends 180 second.
It means if one of node is hanging, then second node will wait for 180 seconds, if within 180 seconds ,it is not able to resolve this situation then it will reboot hung node.
I think Hangcheck timer configuration reuired only with Linux OS.
Configuration File
cat >> /etc/rc.d/rc.local << EOF
modprobe hangcheck-timer hangcheck_tick=15 hangcheck_margin=60Sorry
Hangcheck timer is
Configuration File
cat >> /etc/rc.d/rc.local << EOF
modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
Maybe you are looking for
-
User field in Project def.
Dear All, I am working on one Yreport which gives STATE/ Location wise output, which we are maintaining in coding mask. Two characters in coding mask denotes STATE and in same Plant/Site we can have multiple STATE. Supposing " Minnesota" is a state
-
Best manufacturing solution for SMEs The flexible, affordable and fully scalable, eresource ERP is the ideal solution for (http://www.eresourceerp.com/manufacturing-erp-systems.html) newly emerging and smaller companies, as well as for growth-oriente
-
HTTP Server Error - 403 Forbidden
Hi all i was checking the performance of my application server, in HTTP Server (AS instance, not infrastructure instance) HTTP Server error rate is high, drilling down if found that i have an unusual number of 403 - Forbidden errors. In the access.lo
-
EIS dataloads on ASO - missing data during dataloads
We build are building an ASO application with duplicate member names. The member load works fine, but when I load the data, I am missing amounts so some measures are out of balance. In trouble shooting the data, I found if I load just the Agent (leve
-
Can not logon using akype account on Samsung Smart...
Just updated Skype on my Samsung smart TV. Now i can't logon. Error is incorrect password. Checked with windows 7 and windows 8. I can log on using microsoft account, but TV wants skype name. Seems there was a similar problem a year ago. Is thi