Asm createdisk error in root.sh

Question

I get the following error in root.sh during RAC install
DiskGroup OCR creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 1 regular failure groups, discovered only 0
Configuration of ASM failed, see logs for details
Did not succssfully configure and start ASM
CRS-2500: Cannot stop resource 'ora.crsd' as it is not running
CRS-4000: Command Stop failed, or completed with errors.
Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
Stop of resource "ora.crsd -init" failed
In /opt/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_gaalt-ota-odc.log ....
2010-07-02 06:31:21: Successfully created CRS resources for cluster daemon and ASM
2010-07-02 06:31:21: Checking if initial configuration has been performed
2010-07-02 06:31:21: Starting CSS in exclusive mode
2010-07-02 06:31:47: CRS-2672: Attempting to start 'ora.gipcd' on 'gaalt-ota-odc'
2010-07-02 06:31:47: CRS-2672: Attempting to start 'ora.mdnsd' on 'gaalt-ota-odc'
2010-07-02 06:31:47: CRS-2676: Start of 'ora.gipcd' on 'gaalt-ota-odc' succeeded
2010-07-02 06:31:47: CRS-2676: Start of 'ora.mdnsd' on 'gaalt-ota-odc' succeeded
2010-07-02 06:31:47: CRS-2672: Attempting to start 'ora.gpnpd' on 'gaalt-ota-odc'
2010-07-02 06:31:47: CRS-2676: Start of 'ora.gpnpd' on 'gaalt-ota-odc' succeeded
2010-07-02 06:31:47: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'gaalt-ota-odc'
2010-07-02 06:31:47: CRS-2676: Start of 'ora.cssdmonitor' on 'gaalt-ota-odc' succeeded
2010-07-02 06:31:47: CRS-2672: Attempting to start 'ora.cssd' on 'gaalt-ota-odc'
2010-07-02 06:31:47: CRS-2672: Attempting to start 'ora.diskmon' on 'gaalt-ota-odc'
2010-07-02 06:31:47: CRS-2676: Start of 'ora.diskmon' on 'gaalt-ota-odc' succeeded
2010-07-02 06:31:47: CRS-2676: Start of 'ora.cssd' on 'gaalt-ota-odc' succeeded
2010-07-02 06:31:47: Querying for existing CSS voting disks
2010-07-02 06:31:47: Performing initial configuration for cluster
2010-07-02 06:31:48: Start of resource "ora.ctssd -init" Succeeded
2010-07-02 06:31:48: Configuring ASM via ASMCA
2010-07-02 06:31:48: Executing as grid: /opt/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList ORCL:OCR -redundancy EXTERNAL -configureLocalASM
2010-07-02 06:31:48: Running as user grid: /opt/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList ORCL:OCR -redundancy EXTERNAL -configureLocalASM
2010-07-02 06:31:48: Invoking "/opt/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList ORCL:OCR -redundancy EXTERNAL -configureLocalASM" as user "grid"
2010-07-02 06:31:56: Configuration of ASM failed, see logs for details
2010-07-02 06:31:56: Did not succssfully configure and start ASM
2010-07-02 06:31:56: Exiting exclusive mode
2010-07-02 06:31:57: Command return code of 1 (256) from command: /opt/app/11.2.0/grid/bin/crsctl stop resource ora.crsd -init
2010-07-02 06:31:57: Stop of resource "ora.crsd -init" failed
2010-07-02 06:31:57: Failed to stop CRSD
2010-07-02 06:32:32: Initial cluster configuration failed. See /opt/app/11.2.0/grid/cfgtoollogs/crsconfig/rootcrs_gaalt-ota-odc.log for details
[root@GAALT-OTA-ODC oracledisks]#
So I tried the command manually , first I started CSS in exclusive mode and then ran the command from the logs above.....
crsctl start resource ora.cssd -init -env CSSD_MODE=-X
Then....
[grid@GAALT-OTA-ODC ~]$ /opt/app/11.2.0/grid/bin/asmca -silent -diskGroupName OCR -diskList ORCL:OCR -redundancy EXTERNAL -configureLocalASM
Error...
DiskGroup OCR creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 1 regular failure groups, discovered only 0
I think the issue lies somewhere in my oracleasm OCR disk creation step. The create command during root.sh works when I delete the disk and use oracleasm to recreate it locally . But then the create fails on the other node. In other words the node that uses oracleasm scandisks to pick up OCR will always fail with the error above
In other words, if I run "/usr/sbin/oracleasm createdisk OCR /dev/asm/OCR " on node1, root.sh wll fail with the above error on node2. Even though scandisks detects the OCR on node2
If I run this command on node2 first, root.sh will fail on node1 . Even though scandisks detects the OCR on node1
This is a nightmare would appreciate some advice on diagnosing where the problem is with my configuration of ASM before I roll back the root.sh again !
Thanks

Answer 1

Thanks Sebastian
I tried this but I also needed to recreate the OCR disk as root.sh throws an error that it is already created, so i ran a dd on the device , then deleted OCR using oraclasm
Then I tried to recreate on first node and got
/usr/sbin/oracleasm createdisk OCR /dev/mapper/mpath3
Unable to open device "/dev/mapper/mpath3": Device or resource busy
Tried it on other node and it worked
I rolled back root.sh and then ran it again and it thows the following error
DiskGroup OCR creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15017: diskgroup "OCR" cannot be mounted
ORA-15003: diskgroup "OCR" already mounted in another lock name space
I decided to reboot the machines to get around the busy device. Once rebooted I was able to run oracleasm createdisk without any errors. root.sh also did not throw errors but I am not 100% confident that the root.sh was ok on the second node. Anyway I have lost my install GUI during reboot so could not complete the install anyway
So now I will deinstall everything and start again , and I will update the forum to if there was no issue and problem resolved
Thanks a lot for the pointers and your time