Errors when adding resources to rg in zone cluster

Question

Hi guys,
I managed to create and bring up a zone cluster, create a rg and add a HAStoragePlus resource (zpool), but getting errors when I want to add a lh resource. Here's the output I find relevant:
root@node1:~# zpool list
NAME           SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
rpool         24.6G  10.0G  14.6G  40%  1.00x  ONLINE  -
zclusterpool   187M  98.5K   187M   0%  1.00x  ONLINE  -
root@node1:~# clzonecluster show ztestcluster
=== Zone Clusters ===
Zone Cluster Name:                              ztestcluster
  zonename:                                        ztestcluster
  zonepath:                                        /zcluster/ztestcluster
  autoboot:                                        TRUE
  brand:                                           solaris
  bootargs:                                        <NULL>
  pool:                                            <NULL>
  limitpriv:                                       <NULL>
  scheduling-class:                                <NULL>
  ip-type:                                         shared
  enable_priv_net:                                 TRUE
  resource_security:                               SECURE
  --- Solaris Resources for ztestcluster ---
  Resource Name:                                net
    address:                                       192.168.10.55
    physical:                                      auto
  Resource Name:                                dataset
    name:                                          zclusterpool
  --- Zone Cluster Nodes for ztestcluster ---
  Node Name:                                    node2
    physical-host:                                 node2
    hostname:                                      zclnode2
    --- Solaris Resources for node2 ---
  Node Name:                                    node1
    physical-host:                                 node1
    hostname:                                      zclnode1
    --- Solaris Resources for node1 ---
Now I want to add  a lh (zclusterip - 192.168.10.55) to a resource group named z-test-rg.
root@zclnode2:~# cat /etc/hosts
# Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
# Internet host table
::1 localhost
127.0.0.1 localhost loghost
#zone cluster
192.168.10.51   zclnode1
192.168.10.52   zclnode2
192.168.10.55   zclusterip
root@zclnode2:~# cluster status
=== Cluster Resource Groups ===
Group Name       Node Name       Suspended      State
z-test-rg        zclnode1        No             Online
                 zclnode2        No             Offline
=== Cluster Resources ===
Resource Name        Node Name      State       Status Message
zclusterpool-rs      zclnode1       Online      Online
                     zclnode2       Offline     Offline
root@zclnode2:~# clrg show
=== Resource Groups and Resources ===
Resource Group:                                 z-test-rg
  RG_description:                                  <NULL>
  RG_mode:                                         Failover
  RG_state:                                        Managed
  Failback:                                        False
  Nodelist:                                        zclnode1 zclnode2
  --- Resources for Group z-test-rg ---
  Resource:                                     zclusterpool-rs
    Type:                                          SUNW.HAStoragePlus:10
    Type_version:                                  10
    Group:                                         z-test-rg
    R_description:
    Resource_project_name:                         default
    Enabled{zclnode1}:                             True
    Enabled{zclnode2}:                             True
    Monitored{zclnode1}:                           True
    Monitored{zclnode2}:                           True
The error, for lh resource:
root@zclnode2:~# clrslh create -g z-test-rg -h zclusterip zclusterip-rs
clrslh:  No IPMP group on zclnode1 matches prefix and IP version for zclusterip
Any ideas?
Much appreciated!

Answer 1

Hello,
First of all, I detected a mistake in my previous config: instead of adding an ipmp, a "simple" NIC was added to cluster. I rectified that (I created zclusteripmp0 ipmp out of net11):
root@node1:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
clprivnet0        ip         ok           --         --
   clprivnet0/?   static     ok           --         172.16.3.66/26
   clprivnet0/?   static     ok           --         172.16.2.2/24
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128
   lo0/zoneadmd-v4 static    ok           --         127.0.0.1/8
   lo0/zoneadmd-v6 static    ok           --         ::1/128
net0              ip         ok           sc_ipmp0   --
net1              ip         ok           sc_ipmp1   --
net2              ip         ok           --         --
   net2/?         static     ok           --         172.16.0.66/26
net3              ip         ok           --         --
   net3/?         static     ok           --         172.16.0.130/26
net4              ip         ok           sc_ipmp2   --
net5              ip         ok           sc_ipmp2   --
net11             ip         ok           zclusteripmp0 --
sc_ipmp0          ipmp       ok           --         --
   sc_ipmp0/out   dhcp       ok           --         192.168.1.3/24
sc_ipmp1          ipmp       ok           --         --
   sc_ipmp1/static1 static   ok           --         192.168.10.11/24
sc_ipmp2          ipmp       ok           --         --
   sc_ipmp2/static1 static   ok           --         192.168.30.11/24
   sc_ipmp2/static2 static   ok           --         192.168.30.12/24
zclusteripmp0     ipmp       ok           --         --
   zclusteripmp0/zoneadmd-v4 static ok    --         192.168.10.51/24
root@node1:~# clzonecluster export ztestcluster
create -b
set zonepath=/zcluster/ztestcluster
set brand=solaris
set autoboot=true
set enable_priv_net=true
set ip-type=shared
add net
set address=192.168.10.55
set physical=auto
end
add dataset
set name=zclusterpool
end
add attr
set name=cluster
set type=boolean
set value=true
end
add node
set physical-host=node2
set hostname=zclnode2
add net
set address=192.168.10.52
set physical=zclusteripmp0
end
end
add node
set physical-host=node1
set hostname=zclnode1
add net
set address=192.168.10.51
set physical=zclusteripmp0
end
end
An then I tried again to add the lh, but getting the same error:
root@node2:~# zlogin -C ztestcluster
[Connected to zone 'ztestcluster' console]
zclnode2 console login: root
Password:
Last login: Mon Jan 19 15:28:28 on console
Jan 19 19:17:24 zclnode2 login: ROOT LOGIN /dev/console
Oracle Corporation      SunOS 5.11      11.2    June 2014
root@zclnode2:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
clprivnet0        ip         ok           --         --
   clprivnet0/?   inherited  ok           --         172.16.3.65/26
lo0               loopback   ok           --         --
   lo0/?          inherited  ok           --         127.0.0.1/8
   lo0/?          inherited  ok           --         ::1/128
zclusteripmp0     ipmp       ok           --         --
   zclusteripmp0/? inherited ok           --         192.168.10.52/24
root@zclnode2:~# cluster status
=== Cluster Resource Groups ===
Group Name       Node Name       Suspended      State
z-test-rg        zclnode1        No             Offline
                 zclnode2        No             Online
=== Cluster Resources ===
Resource Name        Node Name      State       Status Message
zclusterpool-rs      zclnode1       Offline     Offline
                     zclnode2       Online      Online
root@zclnode2:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
clprivnet0        ip         ok           --         --
   clprivnet0/?   inherited  ok           --         172.16.3.65/26
lo0               loopback   ok           --         --
   lo0/?          inherited  ok           --         127.0.0.1/8
   lo0/?          inherited  ok           --         ::1/128
zclusteripmp0     ipmp       ok           --         --
   zclusteripmp0/? inherited ok           --         192.168.10.52/24
root@zclnode2:~# clreslogicalhostname create -g z-test-rg -h zclusterip zcluste
rip-rs
clreslogicalhostname:  No IPMP group on zclnode1 matches prefix and IP version for zclusterip
root@zclnode2:~#
To answer your first question, yes - all global nodes and zone cluster nodes have entries for zclusterip:
root@zclnode2:~# cat /etc/hosts
# Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
# Internet host table
::1 localhost
127.0.0.1 localhost loghost
#zone cluster
192.168.10.51   zclnode1
192.168.10.52   zclnode2
192.168.10.55   zclusterip
root@zclnode2:~# ping zclnode1
zclnode1 is alive
When I tried the command you mentioned, first it gave me an error ( there was a space between interfaces), then I changed the rg group to fit mine (z-test-rg) and it  (partially) worked:
root@zclnode2:~# clrs create -g z-test-rg -t LogicalHostname -p Netiflist=sc_ip
mp0@1,sc_ipmp0@2 -p Hostnamelist=zclusterip zclusterip-rs
root@zclnode2:~# clrg show
=== Resource Groups and Resources ===
Resource Group:                                 z-test-rg
  RG_description:                                  <NULL>
  RG_mode:                                         Failover
  RG_state:                                        Managed
  Failback:                                        False
  Nodelist:                                        zclnode1 zclnode2
  --- Resources for Group z-test-rg ---
  Resource:                                     zclusterpool-rs
    Type:                                          SUNW.HAStoragePlus:10
    Type_version:                                  10
    Group:                                         z-test-rg
    R_description:
    Resource_project_name:                         default
    Enabled{zclnode1}:                             True
    Enabled{zclnode2}:                             True
    Monitored{zclnode1}:                           True
    Monitored{zclnode2}:                           True
  Resource:                                     zclusterip-rs
    Type:                                          SUNW.LogicalHostname:5
    Type_version:                                  5
    Group:                                         z-test-rg
    R_description:
    Resource_project_name:                         default
    Enabled{zclnode1}:                             True
    Enabled{zclnode2}:                             True
    Monitored{zclnode1}:                           True
    Monitored{zclnode2}:                           True
root@zclnode2:~# cluster status
=== Cluster Resource Groups ===
Group Name       Node Name       Suspended      State
z-test-rg        zclnode1        No             Offline
                 zclnode2        No             Online
=== Cluster Resources ===
Resource Name        Node Name      State       Status Message
zclusterip-rs        zclnode1       Offline     Offline
                     zclnode2       Online      Online - LogicalHostname online.
zclusterpool-rs      zclnode1       Offline     Offline
                     zclnode2       Online      Online
root@zclnode2:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
clprivnet0        ip         ok           --         --
   clprivnet0/?   inherited  ok           --         172.16.3.65/26
lo0               loopback   ok           --         --
   lo0/?          inherited  ok           --         127.0.0.1/8
   lo0/?          inherited  ok           --         ::1/128
sc_ipmp0          ipmp       ok           --         --
   sc_ipmp0/?     inherited  ok           --         192.168.10.55/24
zclusteripmp0     ipmp       ok           --         --
   zclusteripmp0/? inherited ok           --         192.168.10.52/24
root@zclnode2:~# ping zclusterip
zclusterip is alive
root@zclnode2:~# clrg switch -n zclnode1 z-test-rg
root@zclnode2:~# cluster status
=== Cluster Resource Groups ===
Group Name       Node Name       Suspended      State
z-test-rg        zclnode1        No             Online
                 zclnode2        No             Offline
=== Cluster Resources ===
Resource Name        Node Name      State       Status Message
zclusterip-rs        zclnode1       Online      Online - LogicalHostname online.
                     zclnode2       Offline     Offline - LogicalHostname offline.
zclusterpool-rs      zclnode1       Online      Online
                     zclnode2       Offline     Offline
root@zclnode2:~# ping zclusterip
no answer from zclusterip
root@zclnode2:~# ping zclusterip
no answer from zclusterip
root@zclnode2:~#
So, the lh was added, the rg can switch over to the other node, but zclusterip is pingable only from that cluster zone; I cannot ping zcluster ip from the cluster zone that does not hold the rg, nor from any global cluster node (node1, node2)...