10g RAC - Node Freeze

Hi all,
first, sorry for my poor english :-).
Well, I have this environment:
Servers : 2 DELL PowerEdge 1850
OS: Red Hat Linux Enterprise 4 (Update 2)
Cluster: Oracle Clusterware with "OCFS2"
DB: Oracle 10g Release 2.
It was installed without problem during the instalation, but......
Every day, at night, the node 2 becomes Freeze, and i found nothing in: messages, dmesg, evmd.log, crsd.log, ocssd.log.
I need some help, I can´t put this RAC in produccion.......
Thanks a lot.
Francisco Conejeros
DBA Oracle - Santiago - Chile

Does the second node freeze every night at the same time? Is there any message written into the database alert log?
If you have access to Metalink take a look at Note:362094.1 HANGFG User Guide. HANGFG automates the generation and collection of hanganalyze and systemstate trace files.

Similar Messages

10g RAC Node down Enterprise Manager

Hello
We have a 2 node Oracle 10g RAC Rel2 Linux setup. Enterprise Manager was first stalled on Node1 and we access it using http://node1:5500/em.
This node has a hardware failure and is out of commision at the moment. When I try to connect to http://node2.5500/em it does not work.
I see the dbconsole process is running onthe node2.
How can I use the Enterprise Manager if the node1 is down?
Thanks

Can you try deconfig and config again on node 2?
emctl stop dbconsol
emca -deconfig dbcontrol
emca -config dbcontrolSalman

Latest BNX2 driver crashes 10g RAC nodes

I have a situation with 10g RAC on SLES-9 on HP Proliant 580 servers where sqlnet load balancing with jumbo frames works fine until the latest bnx2 Broadcom drivers (1.7) are applied. There are intermittent node crashes after load is generated on the RAC system (11i E-Business Suite). The crashes are 100% repeatable using hammerora and a load balanced sqlnet entry.
HP is claiming the crashes are an Oracle problem, but the crashes go away when the older bnx2-1.6 drivers are reloaded.
Anybody else having issues with the latest Broadcom drivers with RAC?

Hi mdtaylor
It seems that oracle (at least with redhat on a DELL server) is using those drivers http://www.oracle.com/technology/tech/linux/validated-configurations/html/vc_dell6950-oel4-rhel4-md3000_1.html.
What about asking people on the suse forum http://forums.opensuse.org/ or if you have support make a service request at Suse?

Oracle 10g rac network status

hi,
i want to know what should be the right detail on oracle 10g rac nodes if i type
$ifconfig -a
command on each node . i want to know what entry should be on each node in rac scenario in multiple nodes.
regards

Dear frd
i am sending u status .is there any entry prob in my rac2 node beacause i am getting prob when i run command
on node 2
$crs_stat -t
$CRS-0184: Cannot communicate with the CRS daemon.
etc/hosts (on both nodes)
192.168.141.51 RAC1
192.168.141.53 RAC2
192.168.141.52 vip-RAC1
192.168.141.54 vip-RAC2
10.1.1.1 RAC1_priv
10.1.1.2 RAC2_priv
and ifconfig -a status on both nodes
ON RAC1
# ifconfig -a
en0: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT
,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
inet 192.168.141.51 netmask 0xffffff00 broadcast 192.168.141.255
inet 192.168.141.52 netmask 0xffffff00 broadcast 192.168.141.255
inet 192.168.141.54 netmask 0xffffff00 broadcast 192.168.141.255
tcp_sendspace 131072 tcp_recvspace 65536
en1: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT
,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
tcp_sendspace 131072 tcp_recvspace 65536
lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT
>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1/0
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
ON RAC2
$ ifconfig -a
en0: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT
,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
inet 192.168.141.53 netmask 0xffffff00 broadcast 192.168.141.255
tcp_sendspace 131072 tcp_recvspace 65536
en1: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT
,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
tcp_sendspace 131072 tcp_recvspace 65536
lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT
>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1/0
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

R12 with 2 node 10g RAC is not updating with autoconfig correctly.

Hi,
we are on R12.0.6 with 2 node 10g RAC. Our collegue did the RAC setup, I was not there at that moment
steps he followed is
1. Installed 10g CRS,ASM 10.2.0.3 software and bring up the two nodes.
2. Configured the RAC DB to Oracle Appplication and registered the Listener with CRS.
Name Type Target State Host
ora....D1.inst application ONLINE ONLINE erpdb1
ora....D2.inst application ONLINE ONLINE erpdb2
ora.ERPPROD.db application ONLINE ONLINE erpdb1
ora....SM1.asm application ONLINE ONLINE erpdb1
ora....B1.lsnr application ONLINE ONLINE erpdb1
ora.erpdb1.gsd application ONLINE ONLINE erpdb1
ora.erpdb1.ons application ONLINE ONLINE erpdb1
ora.erpdb1.vip application ONLINE ONLINE erpdb1
ora....SM2.asm application ONLINE ONLINE erpdb2
ora....B2.lsnr application ONLINE ONLINE erpdb2
ora.erpdb2.gsd application ONLINE ONLINE erpdb2
ora.erpdb2.ons application ONLINE ONLINE erpdb2
ora.erpdb2.vip application ONLINE ONLINE erpdb2
oracle@erpdb1 $
The issue is, in our tnsnames.ora in both the nodes.
===
ERPPROD=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
instead of
ERPPROD=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
(INSTANCE_NAME=ERPPROD2)
Node1 tnsnames.ora
oracle@erpdb1 $ cat tnsnames.ora
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#$Header: NetServiceHandler.java 120.16.12000000.4 2008/04/28 06:39:48 schagant ship $
ERPPROD=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD1=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD1_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD2=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD2)
ERPPROD2_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD2)
ERPPROD_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD2_LOCAL=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
ERPPROD1_LOCAL=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
ERPPROD_BALANCE=
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
ERPPROD_REMOTE=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
LISTENER_ERPPROD1=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
LISTENERS_ERPPROD=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
extproc_connection_data =
(DESCRIPTION=
(ADDRESS_LIST =
(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROCERPPROD1))
(CONNECT_DATA=
(SID=PLSExtProc)
(PRESENTATION = RO)
IFILE=/u01/app/oracle/product/10/db/network/admin/ERPPROD1_erpdb1/ERPPROD1_erpdb1_ifile.oraoracle@erpdb1 $
oracle@erpdb1 $
=======
Node2 tnsnames.ora
=======
oracle@erpdb2 $ cat tnsnames.ora
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink Note
# 387859.1 for assistance.
#$Header: NetServiceHandler.java 120.16.12000000.4 2008/04/28 06:39:48 schagant ship $
ERPPROD=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD1=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD1_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD2=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD2)
ERPPROD2_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD2)
ERPPROD_FO=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
(INSTANCE_NAME=ERPPROD1)
ERPPROD2_LOCAL=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
ERPPROD1_LOCAL=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
ERPPROD_BALANCE=
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=ERPPROD)
ERPPROD_REMOTE=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
LISTENER_ERPPROD2=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
LISTENERS_ERPPROD=
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb2-vip.domain.com)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=erpdb1-vip.domain.com)(PORT=1521))
extproc_connection_data =
(DESCRIPTION=
(ADDRESS_LIST =
(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROCERPPROD2))
(CONNECT_DATA=
(SID=PLSExtProc)
(PRESENTATION = RO)
IFILE=/u01/app/oracle/product/10/db/network/admin/ERPPROD2_erpdb2/ERPPROD2_erpdb2_ifile.oraoracle@erpdb2 $
=====
Pl let us know how we can achive through autoconfig.
Regards
Vasu

Vasu,
The issue is, in our tnsnames.ora in both the nodes.What is the issue?
I believe the following documents should be helpful in configuring your tnsnames.ora (and other files) with RAC implemented.
Note: 388577.1 - Using Oracle 10g Release 2 Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 12
https://metalink2.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=388577.1
Note: 756050.1 - Troubleshooting Autoconfig issues with Oracle Applications RAC Databases
https://metalink2.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=756050.1
Regards,
Hussein

ORACLE 10G 2 Node RAC on servers AB to 3 node 11GR2 RAC on new servers XYZ

Hi Gurus ,
We have a business requirement to upgrade existing oracle 10.2.0.4 2 Node RAC on servers A,B (HP UX 11.31 ) to 3 Node ORACLE 11GR2 on new Servers X,Y and Z(Linux and servers are from different vendor)
We don't have ASM.We have RAW file system
Now this has to be done with near zero down time.This is a very busy OLTP System.Golden Gate is not an option as Management is not going for it this time.
Storage is same for everything.I am thinking of the following ways.Please let me know if you have any better plan or if you want to correct my existing plan
Initially i thought of this way and i immediately answered myself
Plan A
1 ) Storage copy (BC etc..) exising 10g RAC Database files on A,B to new volumes allocated to Servers X,Y and Z ( I don't think it's possible as OS is different.I am not sure whether copying this way is possible or not , even if possible i am not sure whether new OS can identify these files or not)
2 ) upgrade 2 Node 10.2.0.4 on X,Y to 11GR2 .
3 ) Add a Database Node on Z
Plan B is
1) Build 3 Node brand new 11GR2 on X,Y and Z
2 ) Plan for how can you replicate the data ( This has to be a logical way)
a ) RMAN - Very time consuming as this is >50 TB Database
b) Golden Gate - Not an option.Even if we can use it , i see there are so many logical issues with golden gate
c) Expdp/impdp - forget about it :)
d) physical standby (Versions are different.If it is same version and Same OS then would have been best bet)
What would be the ideal way to do this with minium down time and with out golden gate
I think something can be done on the lines of PLAN A itself.
Requesting your help Gurus
Thanks

Hi,
Have you considered possibility to setup a logical standby and do rolling upgrade ?
Please have a look at the following:
1) http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-transientlogicalrollingu-1-131927.pdf
2) http://www.oracle.com/technetwork/database/features/availability/maa-wp-11g-upgrades-made-easy-131972.pdf
If you can practice on test setup, it looks a faster option than the others.
Regards,
Ganadeva

Issues while Add / Delete RAC Node in Oracle 10g R2

Hi,
I have an requirement to add a New Node in the existing 2 Node RAC at Production, where 1 Node is Active & other one is passive due to licence issue & cannot keep both the nodes as active. Due to performance issues (Memory , CPU Cores ..etc) we are adding another new node.
Right now we are planning to add a 3rd database node making the new node as active and current active one as passive which is a swap & later on after final observation delete and decommission the current passive node.
This activity is checked at the Dev database with the same infrastructure (OS + Memory ..etc) but want to check what is the best approach (or) challenges we face during the RAC Node Addition / Deletion
RAC DB Version : 10.2.o.4
OS Version : RHEL 5.8
(1) Is the approach is right one , First Adding the node & later on delete
(2) If the approach is the correct , what would be the behavious of the 3rd node in means of active (or) passive
(3) We have taken RMAN backup , OS backup , CRS , ORACLE_HOME , ASM_Home backup , OCR & VD.
(4) Could you please give detail steps for adding / deleting node in 10g R2.
(5) Are they any known bugs to us with the DB release (or) OS while performing this activity.
Since this is a production machine we want to more proactive . Please correct or add any thing i am missing out ...
With Thanks,
Rakesh

Hello Rakesh,
Please follow the following steps.
Node Addition Steps
1. Install and configure OS and hardware for new node.
2. Add Oracle Clusterware to the new node.
3. Configure ONS for the new node.
4. Add ASM home to the new node.
5. Add Databse home to the new node.
6. Add a listener to the new node.
7. Add ASM instance to the New Node.
8. Add a database instance to the new node.
Details of steps
1. run cluvfy to verify whether New node is ready for addition or not.
     $ cluvfy stage -pre crsinst -n node2
2. from node1, execute
          $/u01/app/crs11g/oui/bin/addNode.sh
3. Specify node2 vip address and follow instructions.
4. In the last of installtion it may through an wornig and will ask to click on YES. click on YES
5. from node1,
          /u01/app/crs11g/bin/racgons add_config node2:6200
6. from Node1,set ORACLE_HOME=ASM_HOME and then execute addNode.sh from $ASM_HOME/oui/bin and Follow instrusctions.
7. From node1, set ORACLE_HOME=DB_HOME and then
     /u01/app/oracle/product/11.1.0/db_1/oui/bin/addNode.sh
     and Follow instructions.
8. from node2 start NETCA and configure listener for new node. While configuring Listener select the name of new node.
9. from node1 start dbca from ASM Home to configure ASM instance for new node.
10. Again from node1 start dbca from DB Home to add DB instance
Node deletion Steps
1. Delete the Database instance on the node to be deleted.
2. Clean up the ASM instance.
3. Remove the listener from the node to be deleted.
4. Remove the node from the database.
5. Remove the node from ASM.
6. Remove ONS configuration from the node to be deleted.
7. Remove the node from the clusterware
Details of Steps
1. Remove database Instance of node2
     Dbca -> instance Management -> delete instance -> password for sys -> select node -> finish.
2. Stop asm for node2 from any nodes.
     $srvctl stop asm –n node2
3. Remove asm for node2
     $ srvctl remove asm -n node2
4. Remove Listener from Node2 using NETCA.
5. From Node2:
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node2" -local
6. From Node2, start runinstaller from Oracle_DB_Home/oui/bin, and remove "DB_HOME"
     $ ./runinstaller
     On the WELCOME Screen -> Deinstall product -> Select dbhome name (OraDb10g_Home1) -> Remove
7. From Node1:
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node1"
8. From Node2, set Oracle_Home to asm_1 and then fire:
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node2" -local
9. From Node2, start OUI and deinstall ASM Home.
10. From Node1, Set ORACLE_HOME= /u01/app/oracle/product/11.1.0/asm_1
11. From Node1: from /u01/app/oracle/product/11.1.0/asm_1/oui/bin, start OUI
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node1"
12. From Node2: as a root user (#) execute rootdelete.sh from /u01/app/crs11g/install
     # /u01/app/crs11g/install/rootdelete.sh
13.From Node-1 first find out the node numbers
     # /u01/app/crs11g/bin/olsnodes -n
     output : node1 1
          node2 2
14. From Node-1 as a root user (#):
     # /u01/app/crs11g/install/rootdeletenode.sh node2[Node_Name] 2[node_no]
     output:
          CRS nodeapps are deleted successfully
          clscfg: EXISTING configuration version 4 detected.
          clscfg: version 4 is 11 Release 1.
          Node deletion operation successful.
          'node2' deleted successfully
15. From Node2 set ORACLE_HOME=CRS_HOME and then execute
     $$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node2" CRS=TRUE -local
16. ./runInstaller and remove CRS_HOME
17. From Node-1:
     $ /u01/app/crs11g/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node1" CRS=TRUE
18. check node is deleted from ./crs_stat -t

Apps 10g singale node to 10g RAC upgrade

Hi Friends,
I am planing to upgrade my Apps 10g singale node to 10g RAC. If you have any good document for this please share with me.

Pl identify exact OS, database and EBS versions. Pl see these MOS Docs
Using Oracle 10g Release 2 Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 12 [ID 388577.1]
Step by Step Guide to configure Oracle Applications Release 11i with 10g R2 RAC and ASM [ID 731278.1]
Configuring Oracle Applications Release 11i with 10g RAC and 10g ASM [ID 312731.1]
HTH
Srini

Oracle 10g RAC on Solaris Node Eviction

Been having periodic node eviction on my server. I've found several threads regarding RAC node reboots but nothing specific.. In my case, the node eviction warning appears to be "immediate"
[cssd(9530)]CRS-1612:node mbdmb2 (0) at 50% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1612:node mbdmb2 (0) at 50% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1611:node mbdmb2 (0) at 75% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1611:node mbdmb2 (0) at 75% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1610:node mbdmb2 (0) at 90% heartbeat fatal, eviction in 0.000 seconds
[cssd(9530)]CRS-1607:CSSD evicting node mbdmb2. Details in /u01/crs/oracle/product/10.2/app/log/mbdmb1/cssd/
ocssd.log.
Other people’s: Seem to have a time to recover.. and only reboots when it eventually runs out of time..
2009-08-31 16:05:41.405
[cssd(4968)]CRS-1612:node simsd1 (1) at 50% heartbeat fatal, eviction in 29.611 seconds
2009-08-31 16:05:42.403
[cssd(4968)]CRS-1612:node simsd1 (1) at 50% heartbeat fatal, eviction in 28.613 seconds
2009-08-31 16:05:56.412
[cssd(4968)]CRS-1611:node simsd1 (1) at 75% heartbeat fatal, eviction in 14.604 seconds
2009-08-31 16:05:57.411
[cssd(4968)]CRS-1611:node simsd1 (1) at 75% heartbeat fatal, eviction in 13.605 seconds
2009-08-31 16:06:05.413
[cssd(4968)]CRS-1610:node simsd1 (1) at 90% heartbeat fatal, eviction in 5.603 seconds
2009-08-31 16:06:06.412
[cssd(4968)]CRS-1610:node simsd1 (1) at 90% heartbeat fatal, eviction in 4.604 seconds
2009-08-31 16:06:07.410
[cssd(4968)]CRS-1610:node simsd1 (1) at 90% heartbeat fatal, eviction in 3.606 seconds
2009-08-31 16:06:08.409
[cssd(4968)]CRS-1610:node simsd1 (1) at 90% heartbeat fatal, eviction in 2.607 seconds
2009-08-31 16:06:09.407
[cssd(4968)]CRS-1610:node simsd1 (1) at 90% heartbeat fatal, eviction in 1.609 seconds
2009-08-31 16:06:10.405
[cssd(4968)]CRS-1610:node simsd1 (1) at 90% heartbeat fatal, eviction in 0.611 seconds
2009-08-31 16:06:11.061
[cssd(4968)]CRS-1609:CSSD detected a network split. Details in C:\product\11.1.0\crs\log\simsd2\cssd\ocssd.log.
2009-08-31 16:14:37.873
I'm lead to think this is due to something with the setting on the heartbeat loss window. There are some threads suggesting the hangcheck-timer but it does not appear to be for solaris. Wondering where if any place I can check/change this setting.

Ah, thanks, looks like even just looking at the log yielded some thing different. I was grepping the alertlog instead which apparantly doesn't show as much (and shows time as 0). In the ocssd.log, it shows it with the time to live.
One more question, can you tell from this that whether this is a network hb or disk hb related or something else?
Thanks!
CSSD]2010-02-24 06:46:17.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:17.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:22.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:22.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:27.685 [14] >TRACE: clssgmRegisterClient: proc(17/1009503f0), client(344/10097f
7f0)
[    CSSD]2010-02-24 06:46:27.685 [14] >TRACE: clssgmExecuteClientRequest: GRKJOIN recvd from client 344 (
10097f7f0)
[    CSSD]2010-02-24 06:46:27.685 [14] >TRACE: clssgmJoinGrock: grock DG_FLASH51 new client 10097f7f0 with
con 100932430, requested num -1
[    CSSD]2010-02-24 06:46:27.685 [14] >TRACE: clssgmAddGrockMember: adding member to grock DG_FLASH51
[    CSSD]2010-02-24 06:46:27.685 [14] >TRACE: clssgmAddMember: member (2/100921830) added. pbsz(123) prsz
(42) flags 0x0 to grock (100914210/DG_FLASH51)
[    CSSD]2010-02-24 06:46:27.686 [14] >TRACE: clssgmQueueGrockEvent: groupName(DG_FLASH51) count(3) maste
r(0) event(1), incarn 208505, mbrc 3, to member 0, events 0x0, state 0x0
[    CSSD]2010-02-24 06:46:27.686 [14] >TRACE: clssgmCommonAddMember: Local member(2) node(1) flags 0x0 0x
100 grock (2/100914210/DG_FLASH51)
[    CSSD]2010-02-24 06:46:27.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:27.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:28.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 50 2.123767e-314art
beat fatal, eviction in 59.577 seconds
[    CSSD]2010-02-24 06:46:28.941 [18] >TRACE: clssnmPollingThread: node mbdmb2 (2) is impending reconfig,
flag 1, misstime 60423
[    CSSD]2010-02-24 06:46:28.941 [18] >TRACE: clssnmPollingThread: diskTimeout set to (117000)ms impendin
g reconfig status(1)
[    CSSD]2010-02-24 06:46:29.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 50 2.123767e-314art
beat fatal, eviction in 58.577 seconds
[    CSSD]2010-02-24 06:46:30.363 [17] >TRACE: clssgmDispatchCMXMSG(): msg type(12) src(2) dest(1) size(36
0) tag(00d2002a) incarnation(88)
[    CSSD]2010-02-24 06:46:32.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:32.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:37.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:37.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:42.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:42.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:47.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:47.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:52.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:57.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:46:57.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:46:58.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 75 2.123767e-314art
beat fatal, eviction in 29.577 seconds
[    CSSD]2010-02-24 06:46:59.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 75 2.123767e-314art
beat fatal, eviction in 28.577 seconds
[    CSSD]2010-02-24 06:47:02.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:47:02.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:47:07.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:47:07.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:47:12.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:47:12.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:47:16.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 11.577 seconds
[    CSSD]2010-02-24 06:47:17.725 [17] >TRACE: clssgmDispatchCMXMSG(): msg type(12) src(2) dest(1) size(36
0) tag(00d3002a) incarnation(88)
[    CSSD]2010-02-24 06:47:17.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:47:17.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 10.577 seconds
[    CSSD]2010-02-24 06:47:17.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:47:18.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 9.577 seconds
[    CSSD]2010-02-24 06:47:19.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 8.577 seconds
[    CSSD]2010-02-24 06:47:20.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 7.577 seconds
[    CSSD]2010-02-24 06:47:21.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 6.577 seconds
[    CSSD]2010-02-24 06:47:22.941 [19] >TRACE: clssnmSendingThread: sending status msg to all nodes
[    CSSD]2010-02-24 06:47:22.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 5.577 seconds
[    CSSD]2010-02-24 06:47:22.941 [19] >TRACE: clssnmSendingThread: sent 5 status msgs to all nodes
[    CSSD]2010-02-24 06:47:26.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 1.577 seconds
[    CSSD]2010-02-24 06:47:26.941 [19] >TRACE: clssnmSendingThread: sent 4 status msgs to all nodes
[    CSSD]2010-02-24 06:47:27.703 [17] >TRACE: clssgmPeerEventHndlr: receive failed, node 2 (mbdmb2) (1009
0eb90), rc 11
[    CSSD]2010-02-24 06:47:27.704 [17] >TRACE: clssgmPeerDeactivate: node 2 (mbdmb2), death 0, state 0x800
00001 connstate 0xf
[    CSSD]2010-02-24 06:47:27.704 [17] >TRACE: clssgmPeerListener: discarded 0 future msgsfor 2
[    CSSD]2010-02-24 06:47:27.941 [18] >WARNING: clssnmPollingThread: node mbdmb2 (2) at 90 2.123767e-314art
beat fatal, eviction in 0.577 seconds
[    CSSD]2010-02-24 06:47:28.521 [18] >TRACE: clssnmPollingThread: Eviction started for node mbdmb2 (2),
flags 0x0001, state 3, wt4c 0

Configring and instaling 10g RAC on two Nods win 2003 Local Area

All Experts
I want to configure and instal 10g RAC using three database server on difirent network groups (Head office , processing, stitching). i am unable to configure ip etc. ip address is as:
work group ip address computer name
HO 192.168.0.29 maindb
processing 192.168.1.92 procdb
stitching 192.168.1.92 stichdb
Please guide how i can configure and user hosts file in windows\system32\drivers\etc\hosts
Thanks in advance

Thanks
i have three server with same subnet mask. save os and db versiong windows server 2003 and 10g on all servers.
server-1
ip address : 192.168.0.29
ubnet mask : 255.255.254.0
computer name : maindb
server-2
ip address : 192.168.0.92
ubnet mask : 255.255.254.0
computer name : procdb
server-3
ip address : 192.168.0.93
ubnet mask : 255.255.254.0
computer name : stitchdb
Please guide me how i can configure system32\dirvers\etc\hosts file.
Thanks in advance.

10g RAC's listener cannot startup when the node's back from down

Hello
My 10g RAC node1 down and I reboot it ,
then everything's back except the listener of node1.
[oracle@node2 crsd]$ crs_stat -t
Name Type Target State Host
ora....SM1.asm application ONLINE ONLINE node1
ora....E1.lsnr application ONLINE OFFLINE
ora.node1.gsd application ONLINE ONLINE node1
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip application ONLINE ONLINE node1
ora....SM2.asm application ONLINE ONLINE node2
ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application ONLINE ONLINE node2
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip application ONLINE ONLINE node2
ora.v20db.db application ONLINE ONLINE node2
ora....b1.inst application ONLINE ONLINE node1
ora....b2.inst application ONLINE ONLINE node2
And I did start it manually, but failed:
[oracle@node2 crsd]$ crs_start ora.node1.LISTENER_NODE1.lsnr
Attempting to start `ora.node1.LISTENER_NODE1.lsnr` on member `node1`
`ora.node1.LISTENER_NODE1.lsnr` on member `node1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-0215: Could not start resource 'ora.node1.LISTENER_NODE1.lsnr'.
What does that mean?
and what should I do?
Thank a lot.

You should first force stop it by doing:
crs_stop -f ora.node1.LISTENER_NODE1.lsnr
and then do :
crs_start ora.node1.LISTENER_NODE1.lsnr
either the listener will start or it will show you the error which will point to the real cause.
regards

MULTIPLE USERS 10G RAC ORACLE_HOME INSTALL WITH ASM/CRS

Hi,
We need to install multiple 10g RAC databases on a two node Sun servers. Below is our configuration:
1) Sun Solaris (ver 10) with Sun Cluster 3.2
2) One ASM/CRS install (by 1 OS account)
3) Four ORACLE_HOME 10g database install (by 4 different OS user accounts)
We would like to use one ASM instance for all four databases with appropriate privileges.
OS User:           OS Group
========      =========
oraasm           dbaasm - (ASM and CRS install owner)
ora1           dbaora1 - first db owner
ora2           dbaora2 - second db owner
ora3           dbaora3 - third db owner
ora4           dbaora4 - fourth db owner
I understand that certain privileges need to be shared between ASM/CRS and DB owners. Please let me know the steps to be followed to complete this install.
Thanks in advance.

Hi
Please read that: Documentation http://download.oracle.com/docs/html/B10766_08/intro.htm
- You can install and operate multiple Oracle homes and different versions of Oracle cluster database software on the same computer as described in the following points:
-You can install multiple Oracle Database 10g RAC homes on the same node. The multiple homes feature enables you to install one or more releases on the same machine in multiple Oracle home directories. However, each node can have only one CRS home.
-In addition, you cannot install Oracle Database 10g RAC into an existing single-instance Oracle home. If you have an Oracle home for Oracle Database 10g, then use a different Oracle home, and one that is available across the entire cluster for your new installation. Similarly, if you have an Oracle home for an earlier Oracle cluster database software release, then you must also use a different home for the new installation.
If the OUI detects an earlier version of a database, then the OUI asks you about your upgrade preferences. You have the option to upgrade one of the previous-version databases with DBUA or to create a new database using DBCA. The information collected during this dialog is passed to DBUA or DBCA after the software is installed.
- You can use the OUI to complete some of the de-install and re-install steps for Oracle Database 10g Real Application Clusters if needed.
Note:
Do not move Oracle binaries from one Oracle home to another because this causes dynamic link failures.
. If you are using ASM with Oracle database instances from multiple database homes on the same node, then Oracle recommends that you run the ASM instance from an Oracle home that is distinct from the database homes. In addition, the ASM home should be installed on every cluster node. This prevents the accidental removal of ASM instances that are in use by databases from other homes during the de-installation of a database's Oracle home.

10g RAC on Linux

All,
I am having strange problem while setting up 10g RAC on Linux box. This is two node cluster (rac1 and rac2) Two probelms..
1. When I install CRS on rac1, it copies the files to rac2 and olsnodes -n works fine from both nodes, But when I run runinstaller on node 2 and try to see installed products, I don't see CRS oracle home there on rac2.
2. When I start installing 10g database on rac1 , at third step "Hardware Cluster Installation Mode:" I see my both nodes listed there, but I don't see "SELECT ALL" button on that screen. So when I finish installation, it doesn't copy files to rac2.
Does Anyone has any idea why it is behaving like this?
Thanks
Puneet

Hi,
I found one thing.. If I change ORACLE_HOME location while running OUI, it doesn't give SELECT ALL button , but if I let Installer to select oracle_HOME, it worrks fine and give SELECT ALL button too.
Don't understand why it is behaving like this?
Puneet

10g RAC on VMWare Workstation

Hi Folks,
i tried to install 10g RAC on VMWare 4.5.2 for testing.
Hostsystem is XP, ghost system are RedHat-ES-30.
I configured tho 2 nodes with 2 scsi adapters and 2 Network Card and a shared Disk.
I installed ocfs 1.0.12 on both nodes, formated the disk and it seems to be fine.
On the next step ( Installation of CRS ) the runInstaller fails after the node configuration screen with the message:
Oracle Cluster Registry File can only be saved on a shared filesystem file or on a RAW Device.
Any Ideas ?
Thanks
Martin

Hi Laurent,
Thanks for your reply.
I checked that and i can't find the Processes or any think else witch is discribed in thr article !
The Installer pops a message during the creation of the ocr file.
Have any body tried it with RAW Devices ?
Martin

CRS-0210: Could not find resource while start ASM on 10G RAC

Dear All
I'm trying to install 10g RAC on Redhat 4.1. At the point at which I try to start ASM it fails with the message
ORA-03113: end-of-file on communication channel
select value from v$parameter where name='instance_type'
ERROR at line 1:
ORA-01034: ORACLE not available
This seems to be due to a failure CRS. The trace file shows:
[Thread-11] [17:14:2:907] [HAOperationImpl.runCommand:1254] CRS cmd is: /u01/app/oracle/product/10.2.0/crs/bin/crs_stat -u ora.bfhxx-sql012.ASM1.asm
[Thread-41] [17:14:2:964] [StreamReader.run:65] OUTPUT>CRS-0210: Could not find resource ora.bfhxx-sql012.ASM1.asm.
Executing
/u01/app/oracle/product/10.2.0/crs/bin/crs_stat
produces:
NAME=ora.bfhxx-sql012.LISTENER_BFHXX-SQL012.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on bfhxx-sql012
NAME=ora.bfhxx-sql012.gsd
TYPE=application
TARGET=ONLINE
STATE=ONLINE on bfhxx-sql012
NAME=ora.bfhxx-sql012.ons
TYPE=application
TARGET=ONLINE
STATE=ONLINE on bfhxx-sql012
NAME=ora.bfhxx-sql012.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE on bfhxx-sql012
Is this a red herring or is it the reason ASM won't start? At what point should this entry have been created (ie should it already exist when ASM starts or should ASM create it)? Can I manually create the entry using crsstat.
Many thanks
Paul

It all depends on what your mapping problem was and what you did to resolve it. Here are a couple of things you could try:
-Make sure the orc.loc file on ebdb1 has the same entry as ebdb2. Also, say you are using shared raw devices for CRS: make sure your raw devices are consistent on both nodes.
If the above does not work
-Make sure the shared devices are consistent on both nodes and restore one of the OCR backups. I am sure at least one of the backups 4hrs/8hrs/1day/2day/1week some thing must be a worthwhile. If you do restore OCR from a backup make sure to run cluvfy before you start your CRS.
Always a good practise to take a backup of status quo before you restore from the backups.
It would be interesting to know what lead to your disk mapping problem in the first palce.
Good Luck!

10g RAC - Node Freeze

Similar Messages

Maybe you are looking for