RAC does not start after trying to move storage

Question

Hello,
We have a two nodes RAC Oracle 10g and ASM groups defined on one external storage.
RAC does not start after we reformated the storage to RAID 10.
What we did:
1) deleted the database
2) reformated the external storage to RAID 10 and defined needed volumes for OCR, VotingDisk and ASM
3) on both nodes
- recreated VotingDisk
node1:root$ /u01/oracle/product/10gr2/crs/bin/crsctl add css votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3 -force
Now formatting voting disk: /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3
successful addition of votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3.
node2:root$ /u01/oracle/product/10gr2/crs/bin/crsctl add css votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3 -force
Now formatting voting disk: /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3
successful addition of votedisk /dev/rdsk/c0t600A0B8000347E7D00000EE74A5AF386d0s3.
- tried to move OCR but was able only on one node:
/u01/oracle/product/10gr2/crs/bin/ocrconfig -replace ocrmirror /dev/rdsk/c7t30d0s3
/u01/oracle/product/10gr2/crs/bin/ocrconfig -replace ocr /dev/rdsk/c7t29d0s3
/u01/oracle/product/10gr2/crs/bin/ocrconfig -replace ocr
These return ok for both node1 and node2:
node1:oracle$ cluvfy stage -post hwos -n node1
node1:oracle$ cluvfy stage -post crsinst -n node1,node2 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "node1"
Destination Node                      Reachable?
node1                                 yes
node2                                 yes
Result: Node reachability check passed from node "node1".
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name                             Comment
node2                                 passed
node1                                 passed
Result: User equivalence check passed for user "oracle".
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name                             Status
node2                                 not running
node1                                 not running
Result: Daemon status check failed for "CSS daemon".
Cluster manager integrity check failed.
Checking cluster integrity...
Cluster is divided into 2 partitions.
Partition 1 consists of the following members:
Node Name
node1
Partition 2 consists of the following members:
Node Name
node1
node2
Cluster integrity check failed. Cluster is divided into 2 partition(s).
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
WARNING:
CSS is probably working with a non-clustered, local-only configuration on nodes:
node1
Verification will proceed with nodes:
node2
Uniqueness check for OCR device passed.
Checking the version of OCR...
OCR of correct Version "2" exists.
Checking data integrity of OCR...
Data integrity check for OCR passed.
OCR integrity check failed.
Checking CRS integrity...
Checking daemon liveness...
Check: Liveness for "CRS daemon"
Node Name                             Running
node2                                 no
node1                                 yes
Result: Liveness check failed for "CRS daemon".
Checking daemon liveness...
Check: Liveness for "CSS daemon"
Node Name                             Running
node2                                 no
node1                                 no
Result: Liveness check failed for "CSS daemon".
Checking daemon liveness...
Check: Liveness for "EVM daemon"
Node Name                             Running
node2                                 no
node1                                 yes
Result: Liveness check failed for "EVM daemon".
Liveness of all the daemons
Node Name     CRS daemon                CSS daemon                EVM daemon
node2         no                        no                        no
node1         yes                       no                        yes
Checking CRS health...
Check: Health of CRS
Node Name                             CRS OK?
node1                                 unknown
Result: CRS health check failed.
CRS integrity check failed.
Post-check for cluster services setup was unsuccessful on all the nodes.
Please help.

Answer 1

Thanks for the answer.
Problem is that the RAC is not healty, I know how to start it but now it doesn't start and I cannot find helpful logs.
You can see that cluster is not healty from cluvfy output.
When I tried to start crs whith crsctl start crs but:
- on node2 none of the demons starts (CSS, CRS or EVM)
- on node1 CSS and EVM demons start but CRS is not starting and I cannot find helpful logs
Problem 2: Cannot change OCR location
node2:root$ cat /var/opt/oracle/ocr.loc
+#Device/file /dev/rdsk/c7t30d0s3 getting replaced by device /dev/rdsk/c7t30d0s3+
ocrconfig_loc=/dev/rdsk/c7t30d0s3
local_only=falsenode2:root$ ocrconfig -replace ocr /dev/rdsk/c0t600A0B8000347EAD00000F314A5AF5B8d0s3
PROT-1: Failed to initialize ocrconfig
node1:root$ cat /var/opt/oracle/ocr.loc
ocrconfig_loc=/u01/oracle/product/10gr2/db/cdata/localhost/local.ocr
local_only=TRUE
node1:root$ ocrconfig -replace ocr /dev/rdsk/c0t600A0B8000347EAD00000F314A5AF5B8d0s3
PROT-1: Failed to initialize ocrconfig
Problem 3: On node 1 I wrongly issued a command
node1:root$ /u01/oracle/product/10gr2/db/bin/localconfig reset
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS service is started