Failing to create HA nfs storage on a shared 3310 HW Raid cluster 3.2

Question

Hi,
I'm working on testing clustering on a couple v240s, running identitcal Sol10 10/08 and Sun Cluster 3.2. In trying things, I may have messed up the cluster. I may want to backout the cluster and start over. Is that possible, or do I need to install Solaris fresh.
But first, the problem. I have the array connect to both machines and working. I mount 1 LUN on /global/nfs using the device /dev/did/dsk/d4s0. Then I ran the commands:
# clrt register SUNW.nfs
# clrt register SUNW.HAStoragePlus
# clrt list -v
Resource Type Node List
SUNW.LogicalHostname:2 <All>
SUNW.SharedAddress:2 <All>
SUNW.nfs:3.2 <All>
SUNW.HAStoragePlus:6 <All>
# clrg create -n stnv240a,stnv240b -p PathPrefix=/global/nfs/admin nfs-rg
I enabled them just now so:
# clrg status
Cluster Resource Groups ===
Group Name Node Name Suspended Status
nfs-rg stnv240a No Online
stnv240b No Offline
Then:
# clrslh create -g nfs-rg cluster
# clrslh status
Cluster Resources ===
Resource Name Node Name State Status Message
cluster stnv240a Online Online - LogicalHostname online.
stnv240b Offline Offline
I'm guessing that 'b' is offline because it's the backup.
Finally, I get:
# clrs create -t HAStoragePlus -g nfs-rg -p AffinityOn=true -p FilesystemMountPoints=/global/nfs nfs-stor
clrs: stnv240b - Invalid global device path /dev/did/dsk/d4s0 detected.
clrs: (C189917) VALIDATE on resource nfs-stor, resource group nfs-rg, exited with non-zero exit status.
clrs: (C720144) Validation of resource nfs-stor in resource group nfs-rg on node stnv240b failed.
clrs: (C891200) Failed to create resource "nfs-stor".
On stnv240a:
# df -h /global/nfs
Filesystem size used avail capacity Mounted on
/dev/did/dsk/d4s0 49G 20G 29G 41% /global/nfs
and on stnv240b:
# df -h /global/nfs
Filesystem size used avail capacity Mounted on
/dev/did/dsk/d4s0 49G 20G 29G 41% /global/nfs
Any help? Like I said, this is a test setup. I've started over once. So I can start over if I did something irreversible.

Answer 1

I still have the issue. I reinstalled from scratch and installed the cluster. Then I did the following:
$ vi /etc/default/nfs
GRACE_PERIOD=10
$ ls /global//nfs
$ mount /global/nfs
$ df -h
Filesystem size used avail capacity Mounted on
/dev/global/dsk/d4s0 49G 20G 29G 41% /global/nfs
$ clrt register SUNW.nfs
$ clrt register SUNW.HAStoragePlus
$ clrt list -v
Resource Type Node List
SUNW.LogicalHostname:2 <All>
SUNW.SharedAddress:2 <All>
SUNW.nfs:3.2 <All>
SUNW.HAStoragePlus:6 <All>
$ clrg create -n stnv240a,stnv240b -p PathPrefix=/global/nfs/admin nfs-rg
$ clrslh create -g nfs-rg patience
clrslh: IP Address 204.155.141.146 is already plumbed at host: stnv240b
$ grep cluster /etc/hosts
204.155.141.140 stnv240a stnv240a.mns.qintra.com # global - cluster
204.155.141.141 cluster cluster.mns.qintra.com # cluster virtual address
204.155.141.146 stnv240b stnv240b.mns.qintra.com patience patience.mns.qintra.com # global v240 - cluster test
$ clrslh create -g nfs-rg cluster
$ clrs create -t HAStoragePlus -g nfs-rg -p AffinityOn=true -p FilesystemMountPoints=/global/nfs nfs-stor
clrs: stnv240b - Failed to analyze the device special file associated with file system mount point /global/nfs: No such file or directory.
clrs: (C189917) VALIDATE on resource nfs-stor, resource group nfs-rg, exited with non-zero exit status.
clrs: (C720144) Validation of resource nfs-stor in resource group nfs-rg on node stnv240b failed.
clrs: (C891200) Failed to create resource "nfs-stor".
Now, on the second machine (stnv240b), /dev/global does not exist, but the file system mounts anyway. I guess that's cluster magic?
$ cat /etc/vfstab
/dev/global/dsk/d4s0 /dev/global/dsk/d4s0 /global/nfs ufs 1 yes global
$ df -h /global/nfs
Filesystem size used avail capacity Mounted on
/dev/global/dsk/d4s0 49G 20G 29G 41% /global/nfs
$ ls -l /dev/global
/dev/global: No such file or directory
I followed the other thread. devfsadm and scgdevs
One other thing I notice. Both nodes mount my global on node@1
/dev/md/dsk/d6 723M 3.5M 662M 1% /global/.devices/node@1
/dev/md/dsk/d6 723M 3.5M 662M 1% /global/.devices/node@1