X86 cluster 3.2

Question

Hi
I have a 2 node cluster  with one RG     configured
Currently the  RG and  one of the resource status is showing as Faulted as below
# clrg status
=== Cluster Resource Groups ===
Group Name   Node Name                         Suspended   Status
oracle-rg    cert1.cin.com   No          Online_faulted
                 cert2.cin.com   No          Offline
# clrs status
=== Cluster Resources ===
Resource Name         Node Name                         State     Status Message
oracle-storage-res    cert1.cin.com   Online    Faulted - I/O timed out on path /dev/md/ora-data/rdsk/d9
                              cert2.cin.com   Offline   Offline
THere are   5 disks in the ora-data diskset
Output of "cldevice show"  command from cert2.cin.com server shows the physical path of d9 disk above as accessible but
Running cluster status command on cert2.cin.com hangs midway  after the   resource status output
I am not able to login to cert1.cin.com where the RG is active . It does not come to the login prompt but the server is pinging
I am not sure why the resource is showing online while as the message is Faulted and the  node also is hung
Please help me in how to clear the Faulted state and recover from this condition
Can I switch the RG  to cert2.cin.com safely
Appreciate any help TIA

Answer 1

Hi ra*326096*ul,
it seems there was a timeout on device /dev/md/ora-data/rdsk/d9 on node cert1. Maybe the /var/adm/messages file give more information for the cause of this timeout. Certainly, you need to login to cert1 to check this. If you are not able to login via the network maybe you can access the cert1 via the console port and check the status of the network and the SVM d9 device? And what are the underlying cluster ‘did’ devices of the SVM d9 device? You can use metastat if you can access cert1 via console port.
It’s not sure if you can switch the RG safely to cert2 due to the fact that the ‘cluster status’ command is not finishing successful. Because after the resources output it looks to the cluster ‘did’ devices. So, if you know which are the underlying cluster ‘did’ devices of the SVM d9 device then you can check if the relevant cluster ‘did’ devices are ok. The ‘cldevice show’ command shows the cluster ‘did’ devices but not the SVM d9 device. So, there is no need that cluster ‘did’ device d9 is part of SVM d9 device. If you know the cluster ‘did’ devices which are part of SVM d9 device you check if the physical device of these’ did’ devices are ok. Maybe ‘scdidadm -L’ is also helpful to show a summary of all nodes for cluster ‘did’ devices. When you have checked that the physical devices of the SVM d9 device are ok on cert2 then I believe you can switch the RG successful.  Does 'scstat -D' work on cert2?
Hth,
  Juergen