Installation of Oracle Database 11g R2 Grid is failing

Question

Hi Guys,
I've been trying to install Oracle Grid Infrastructure 11g R2 (11.2.0.2) for a long time, but, failing every time. This is what I've tried so far:
Things being used:
OS: Oracle Enterprise Linux 5.2
VM Software: VMware Workstation 7.1 and VMware Server 2
Oracle SW: Oracle Grid Infrastructure 11g R2 - 11.2.0.2
VMs: Three RAC nodes - rac1, rac2, rac3
I've tried to follow the link given below to on both VMware Server and VMware Workstation all goes well until it asks us to run two scripts as root.
http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnOEL5UsingVMwareServer2.php
The script "/u01/app/grid/11.2.0/root.sh" completes successfully on the first node, rac1, however, it fails on the rest of them with the following error:
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Start of resource "ora.crsd" failed
CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'rac2'
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Clusterware stack
Failed to start Cluster Ready Services at /u01/app/grid/11.2.0/crs/install/crsconfig_lib.pm line 1065.
/u01/app/grid/11.2.0/perl/bin/perl -I/u01/app/grid/11.2.0/perl/lib -I/u01/app/grid/11.2.0/crs/install /u01/app/grid/11.2.0/crs/install/rootcrs.pl execution failed
Instead of using shared disk as "independent-persistent" disks I've also used openfiler 2.99 to allocate storage by following below given link. With that the issue was that the iSCSI disks were not persistent and no solution worked to fix that.
http://www.jobacle.nl/?p=957
I also tried to use OCFS2 for OCR and VOTING disks, the installation completes successfully, however, the asm instance, registry, and few other services doesn't start. When I tried to reboot all the machines not a single oracle service came up. The crs just died because of OCR and VOTING disks.
I've tried so many things but nothing works.
My last installation was as follows:
==============================
VM: rac1, rac2, rac3
Config of each VM:
Memory: 4GB
Processors: 4
HDD (SCSI): 25GB
/dev/sda1 > /boot (100M)
/dev/sda2 > swap (8G)
/dev/sda3 > / (17G)HDD2 (SCSI): 60GB (Persistent)
/dev/sdb1 > ocrvot1 (2G)
/dev/sdb2 > ocrvot2 (2G)
/dev/sdb3 > ocrvot3 (2G)
/dev/sdb4 > data1 (10G)
/dev/sdb5 > data2 (10G)
/dev/sdb6 > data3 (10G)
/dev/sdb7 > fast1 (10G)
/dev/sdb8 > fast2 (10G)HDD3 (SCSI): 60GB (Persistent)
/dev/sdc1 > ocrvot4 (2G)
/dev/sdc2 > ocrvot5 (2G)
/dev/sdc3 > ocrvot6 (2G)
/dev/sdc4 > data4 (10G)
/dev/sdc5 > data5 (10G)
/dev/sdc6 > fast3 (10G)
/dev/sdc7 > fast4 (10G)
/dev/sdc8 > fast5 (10G)NIC 1: Bridge
NIC 2: Bridge
Pre-requisites completed.
ASM Disks labeled.
runcluvfy succeeds for all nodes
All nodes can see and write to the ASM disks.During the installation of GI 11.2.0.2 at prerequisites test got one error:
PRVF-5449 : Check of Voting Disk location "ORCL:(ORCL:)" failed
However, according to Metalink, note ID 1267569.1, its a bug and its going to be fixed in 11.2.0.3. The note gives a workaround which is the manual testing of the ASM disks from all the nodes and as long it succeeds we can ignore the error. Therefore, I just ignored the error and carried on. When prompted I executed the first script, "/u01/app/oraInventory/orainstRoot.sh ", on all the nodes successfully. The second script, "/u01/app/grid/11.2.0/root.sh" completes successfully on rac1, however, fails on the rest of the nodes.
On rac2 the alertrac2.log contains the following errors:
CRS-5019:All OCR locations are on ASM disk groups [OCRVOT], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/grid/11.2.0/log/rac2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".and oraagent_oracle.log contains the following errors:
2011-07-15 22:53:47.748: [ora.asm][1090414912] {0:0:215} [check] checkCrsStat 2 CLSCRS_STAT ret: 1842011-07-15 22:53:47.748: [ora.asm][1090414912] {0:0:215} [check] clsnUtils::error Exception type=2 string=
2011-07-15 22:53:47.748: [ora.asm][1090414912] {0:0:215} [check] AsmAgent::checkCbk: Exception UserErrorException
2011-07-15 22:53:47.748: [ora.asm][1090414912] {0:0:215} [check]
2011-07-15 22:53:47.748: [ora.asm][1090414912] {0:0:215} [check] InstAgent::check checkCounter 0 prev clsagfw_status 1 current clsagfw_status 4
2011-07-15 22:53:48.599: [    AGFW][1117755712] {0:0:166} Agent received the message: AGENT_HB[Engine] ID 12293:1089
2011-07-15 22:53:48.755: [ COMMCRS][1128261952]clsc_connect: (0x10c5d5b0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
On rac1:
========
[root@rac1 bin]# ./crsctl status res -t
NAME TARGET STATE SERVER STATE_DETAILS
Local Resources
ora.OCRVOT.dg
ONLINE ONLINE rac1
ora.asm
ONLINE ONLINE rac1 Started
ora.gsd
OFFLINE OFFLINE rac1
ora.net1.network
ONLINE ONLINE rac1
ora.ons
ONLINE ONLINE rac1
ora.registry.acfs
ONLINE ONLINE rac1
Cluster Resources
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.cvu
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.scan1.vip
1 ONLINE ONLINE rac1
On rac2
========
The alert+ASM2.log has the following errors:
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:215} */
NOTE: Diskgroup used for Voting files is:
OCRVOT
Diskgroup used for OCR is:OCRVOT
NOTE: cache registered group OCRVOT number=1 incarn=0xee7b8cf5
NOTE: cache began mount (not first) of group OCRVOT number=1 incarn=0xee7b8cf5
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (ORCL:OCRVOT1)
NOTE: Assigning number (1,1) to disk (ORCL:OCRVOT2)
NOTE: Assigning number (1,2) to disk (ORCL:OCRVOT3)
NOTE: Assigning number (1,3) to disk (ORCL:OCRVOT4)
NOTE: Assigning number (1,4) to disk (ORCL:OCRVOT5)
NOTE: Assigning number (1,5) to disk (ORCL:OCRVOT6)
ERROR: no PST quorum in group: required 3, found 0
NOTE: cache dismounting (clean) group 1/0xEE7B8CF5 (OCRVOT)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0xEE7B8CF5 (OCRVOT)
NOTE: cache ending mount (fail) of group OCRVOT number=1 incarn=0xee7b8cf5
NOTE: cache deleting context for group OCRVOT 1/0xee7b8cf5
GMON dismounting group 1 at 2 for pid 23, osid 19831
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup OCRVOT was not mounted
WARNING: Disk Group OCRVOT containing configured OCR is not mounted
WARNING: Disk Group OCRVOT containing voting files is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "OCRVOT" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "OCRVOT"
ERROR: ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:215} */
SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:0:215} */
Any help would be appreciated. Thanks.
Still miss those days when Oracle used to release easy to use and easy to install softwares, but now thats not the case. Grid Infrastructure, Grid Control, and other similar technologies have become so complicated and difficult for us to install on VMs and play with them.
Kashif.
Updates:
Metalink doc, Oracle Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1], shows similar errors that I've in my log files. Therefore, I thought Multicast could be the culprit. Nonetheless:
[oracle@rac3 mcasttest]$ ./mcasttest.pl -n rac1,rac2,rac3 -i eth0,eth1
########### Setup for node rac1 ##########
Checking node access 'rac1'
Checking node login 'rac1'
Checking/Creating Directory /tmp/mcasttest for binary on node 'rac1'
Distributing mcast2 binary to node 'rac1'
########### Setup for node rac2 ##########
Checking node access 'rac2'
Checking node login 'rac2'
Checking/Creating Directory /tmp/mcasttest for binary on node 'rac2'
Distributing mcast2 binary to node 'rac2'
########### Setup for node rac3 ##########
Checking node access 'rac3'
Checking node login 'rac3'
Checking/Creating Directory /tmp/mcasttest for binary on node 'rac3'
Distributing mcast2 binary to node 'rac3'
########### testing Multicast on all nodes ##########
Test for Multicast address 230.0.1.0
Jul 15 23:57:30 | Multicast Succeeded for eth0 using address 230.0.1.0:42000
Jul 15 23:57:31 | Multicast Succeeded for eth1 using address 230.0.1.0:42001
Test for Multicast address 224.0.0.251
Jul 15 23:57:32 | Multicast Succeeded for eth0 using address 224.0.0.251:42002
Jul 15 23:57:33 | Multicast Succeeded for eth1 using address 224.0.0.251:42003
Hence, the issue is not multicast related.
Kashif.
Edited by: Kashif Khan on Jul 15, 2011 9:00 PM

Answer 1

Kashif Khan wrote:
THE ISSUE HAS BEEN RESOLVED!
With the same configuration the installation completed successfully on Oracle's Virtual Box. I had a lot of issues while installing Grid Control 11g on VMware, but, that got installed successfully as well on the Virtual Box.
Hence proved, VMware is no good for Oracle virtualization any more.
Hi,
Good if you have it resolved. Now could you please mark the thread as answered to avoid confusion
Cheers