Error while joining server to Srv Pool. Visible only on 1 server out of 9

Question

Hello,
I am hiaving issues with joining a server to the server pool
I have checked the 02cb ond ocfs2 config and they seem to be fine and in line with the other VM servers which are already added to the pool...
I have checked the DMESG and ovs-operation log and found the below errors.
The error logs are as follows..
DMESG
o2net: accepted connection from node vmclkxtt3bvms09.esptest.aurtest.national.com.au (num 8) at 10.12.72.89:7777
o2net: accepted connection from node vmclkxtt3bvms07.esptest.aurtest.national.com.au (num 6) at 10.12.72.87:7777
OCFS2 1.4.4
o2dlm: Nodes in domain 55465DB6F3604834910621BF4D229455: 0 1 2 3 4 5 6 7 8
(mount.ocfs2,11035,0):ocfs2_find_slot:243 ERROR: no free slots available!
(mount.ocfs2,11035,0):ocfs2_mount_volume:1461 ERROR: status = -22
ocfs2: Unmounting device (253,12) on (node 4)
o2net: no longer connected to node vmclkxtt3bvms01.esptest.aurtest.national.com.au (num 0) at 10.12.72.81:7777
o2net: no longer connected to node vmclkxtt3bvms02.esptest.aurtest.national.com.au (num 1) at 10.12.72.82:7777
o2net: no longer connected to node vmclkxtt3bvms03.esptest.aurtest.national.com.au (num 2) at 10.12.72.83:7777
ovs-operation.log
"2011-08-15 11:52:40" INFO=> ha_set_shutdown_mode: inform master agent, leave shutdown mode.
"2011-08-15 11:52:40" DEBUG=> ha_set_shutdown_mode: no master access information available.
"2011-08-15 11:52:40" INFO=> ha_set_shutdown_mode: remove shutdown mode flag.
"2011-08-15 12:14:36" INFO=> registered_hostname: hostname=10.12.72.85 => {"status": "SUCC"}
"2011-08-15 12:14:39" INFO=> clusterm_start_o2cb_service: reloading config from /etc/ocfs2/cluster.conf ...
"2011-08-15 12:14:40" INFO=> clusterm_start_o2cb_service: => {"status": "SUCC"}
"2011-08-15 12:14:48" INFO=> ha_get_hostname_ip: hostname=vmclkxtt3bvms05.esptest.aurtest.national.com.au;ip=10.12.72.85
"2011-08-15 12:14:49" INFO=> ha_check_ovs_mount: success.
"2011-08-15 12:14:50" INFO=> ha_get_ocfs2_config: success. nodes=ocfs2:8:10.12.72.89:7777,ocfs2:7:10.12.72.88:7777,ocfs2:6:10.12.7
2.87:7777,ocfs2:5:10.12.72.86:7777,ocfs2:4:10.12.72.85:7777,ocfs2:3:10.12.72.84:7777,ocfs2:2:10.12.72.83:7777,ocfs2:1:10.12.72.82:
7777,ocfs2:0:10.12.72.81:7777
"2011-08-15 12:15:05" ERROR=> clusterm_init_root_sr: <Exception: mount.ocfs2: Invalid argument while mounting /dev/mapper/mpath1p1
on /var/ovs/mount/DD741D59C40B42E797C193A6B8442B21. Check 'dmesg' for more information on this error.StackTrace:
File "/opt/ovs-agent-2.3/OVSXCluster.py", line 115, in clusterm_init_root_sr
sr.initialize()
File "/opt/ovs-agent-2.3/_storage/OVSFileSR.py", line 127, in initialize
self.sp.mount(mp)
File "/opt/ovs-agent-2.3/_storage/plugins/OVSFileSP.py", line 237, in mount
raise Exception(output)
"2011-08-15 12:15:33" INFO=> unregister_hostname: success.
"2011-08-15 12:15:33" INFO=> clusterm_teardown: success.
"2011-08-15 12:26:33" INFO=> ha_set_shutdown_mode: inform master agent, leave shutdown mode.
"2011-08-15 12:26:33" DEBUG=> ha_set_shutdown_mode: no master access information available.
"2011-08-15 12:26:33" INFO=> ha_set_shutdown_mode: remove shutdown mode flag.
"2011-08-15 12:39:50" INFO=> registered_hostname: hostname=10.12.72.85 => {"status": "SUCC"}
"2011-08-15 12:39:54" INFO=> clusterm_start_o2cb_service: reloading config from /etc/ocfs2/cluster.conf ...
"2011-08-15 12:39:55" INFO=> clusterm_start_o2cb_service: => {"status": "SUCC"}
"2011-08-16 12:35:20" INFO=> clusterm_init_sr: success.
"2011-08-16 12:35:34" ERROR=> clusterm_init_root_sr: <Exception: mount.ocfs2: Invalid argument while mounting /dev/mapper/OVS1p1 o
/var/ovs/mount/DD741D59C40B42E797C193A6B8442B21. Check 'dmesg' for more information on this error.StackTrace:
File "/opt/ovs-agent-2.3/OVSXCluster.py", line 115, in clusterm_init_root_sr
sr.initialize()
File "/opt/ovs-agent-2.3/_storage/OVSFileSR.py", line 127, in initialize
self.sp.mount(mp)
File "/opt/ovs-agent-2.3/_storage/plugins/OVSFileSP.py", line 237, in mount
raise Exception(output)
"2011-08-16 12:35:34" INFO=> ha_set_shutdown_mode: inform master agent, leave shutdown mode.
"2011-08-16 12:35:34" DEBUG=> ha_set_shutdown_mode: no master access information available.
"2011-08-16 12:35:34" INFO=> ha_set_shutdown_mode: remove shutdown mode flag.
"2011-08-17 10:57:43" INFO=> unregister_hostname: success.
"2011-08-17 10:57:43" INFO=> clusterm_teardown: success.
"2011-08-17 11:02:54" INFO=> registered_hostname: hostname=10.12.72.85 => {"status": "SUCC"}
"2011-08-17 11:02:57" INFO=> clusterm_start_o2cb_service: reloading config from /etc/ocfs2/cluster.conf ...
"2011-08-17 11:02:58" INFO=> clusterm_start_o2cb_service: => {"status": "SUCC"}
"2011-08-17 11:03:06" INFO=> ha_get_hostname_ip: hostname=vmclkxtt3bvms05.esptest.aurtest.national.com.au;ip=10.12.72.85
"2011-08-17 11:03:07" INFO=> ha_check_ovs_mount: success.
"2011-08-17 11:03:07" INFO=> ha_get_ocfs2_config: success. nodes=ocfs2:8:10.12.72.89:7777,ocfs2:7:10.12.72.88:7777,ocfs2:6:10.12.7
2.87:7777,ocfs2:5:10.12.72.86:7777,ocfs2:4:10.12.72.85:7777,ocfs2:3:10.12.72.84:7777,ocfs2:2:10.12.72.83:7777,ocfs2:1:10.12.72.82:
7777,ocfs2:0:10.12.72.81:7777
"2011-08-17 11:03:22" ERROR=> clusterm_init_root_sr: <Exception: mount.ocfs2: Invalid argument while mounting /dev/mapper/OVS1p1 o
/var/ovs/mount/DD741D59C40B42E797C193A6B8442B21. Check 'dmesg' for more information on this error.StackTrace:
File "/opt/ovs-agent-2.3/OVSXCluster.py", line 115, in clusterm_init_root_sr
sr.initialize()
File "/opt/ovs-agent-2.3/_storage/OVSFileSR.py", line 127, in initialize
self.sp.mount(mp)
File "/opt/ovs-agent-2.3/_storage/plugins/OVSFileSP.py", line 237, in mount
raise Exception(output)
"2011-08-18 09:31:32" INFO=> unregister_hostname: success.
"2011-08-18 09:31:32" INFO=> clusterm_teardown: success.
I ALSO rebooted the rest of the servers and ran an fsck.ocfs2 on all the shared repos. and the output is as below.. (just incase helps in TS..)
[root@vmclkxtt3bvms05 ~]# fsck.ocfs2 /dev/mapper/OVS1p1
fsck.ocfs2 1.4.3
fsck.ocfs2: Could not create domain while initializing the DLM
[root@vmclkxtt3bvms05 ~]# fsck.ocfs2 /dev/mapper/OVS2p1
fsck.ocfs2 1.4.3
Checking OCFS2 filesystem in /dev/mapper/OVS2p1:
label: <NONE>
uuid: f2 1e e8 69 21 6d 4e df a7 7e bd 12 55 3c 23 71
number of blocks: 142731492
bytes per block: 4096
number of clusters: 142731492
bytes per cluster: 4096
max slots: 8
/dev/mapper/OVS2p1 is clean. It will be checked after 20 additional mounts.
[root@vmclkxtt3bvms05 ~]# fsck.ocfs2 /dev/mapper/OVS1p1
fsck.ocfs2 1.4.3
fsck.ocfs2: Could not create domain while initializing the DLM
[root@vmclkxtt3bvms05 ~]# fsck.ocfs2 /dev/mapper/OVS3p1
fsck.ocfs2 1.4.3
Checking OCFS2 filesystem in /dev/mapper/OVS3p1:
label: <NONE>
uuid: 9d 08 3e e1 c5 e2 4e d8 a5 19 2e 1d 47 3a 0c 8e
number of blocks: 142731492
bytes per block: 4096
number of clusters: 142731492
bytes per cluster: 4096
max slots: 8
/dev/mapper/OVS3p1 is clean. It will be checked after 20 additional mounts.
[root@vmclkxtt3bvms05 ~]#
[root@vmclkxtt3bvms05 ~]# fsck.ocfs2 /dev/mapper/OVS4p1
fsck.ocfs2 1.4.3
Checking OCFS2 filesystem in /dev/mapper/OVS4p1:
label: <NONE>
uuid: 08 ee 37 92 5b 1c 4e a0 97 6b 39 6a 67 d5 6d 7f
number of blocks: 142731492
bytes per block: 4096
number of clusters: 142731492
bytes per cluster: 4096
max slots: 8
/dev/mapper/OVS4p1 is clean. It will be checked after 20 additional mounts.
Aug 18 09:50:27 vmclkxtt3bvms05 kernel: o2net: accepted connection from node vmclkxtt3bvms06.esptest.aurtest.national.com.au (num 5) at 10.12.72.86:7777
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: (o2net,4013,0):o2net_connect_expired:1667 ERROR: no connection established with node 8 after 30.0 seconds, giving up and returning errors.
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4101,0):dlm_request_join:1043 ERROR: Error -107 when sending message 510 (key 0x666c6172) to node 8
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4101,0):dlm_try_to_join_domain:1219 ERROR: status = -107
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4101,0):dlm_join_domain:1497 ERROR: status = -107
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4101,0):dlm_register_domain:1763 ERROR: status = -107
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4101,0):user_dlm_register_context:673 ERROR: status = -107
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4101,0):dlmfs_mkdir:493 ERROR: Error -107 could not register domain "55465DB6F3604834910621BF4D229455"
Aug 18 09:50:56 vmclkxtt3bvms05 kernel: o2net: no longer connected to node vmclkxtt3bvms06.esptest.aurtest.national.com.au (num 5) at 10.12.72.86:7777
Aug 18 09:51:09 vmclkxtt3bvms05 kernel: o2dlm: Nodes in domain F21EE869216D4EDFA77EBD12553C2371: 4
Aug 18 09:51:12 vmclkxtt3bvms05 puppetd[10246]: Could not request certificate: getaddrinfo: Temporary failure in name resolution
Aug 18 09:51:23 vmclkxtt3bvms05 kernel: o2net: accepted connection from node vmclkxtt3bvms06.esptest.aurtest.national.com.au (num 5) at 10.12.72.86:7777
Aug 18 09:51:50 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4113,0):dlm_join_domain:1477 Timed out joining dlm domain 55465DB6F3604834910621BF4D229455 after 92000 msecs
Aug 18 09:51:50 vmclkxtt3bvms05 kernel: (fsck.ocfs2,4113,0):dlmfs_mkdir:493 ERROR: Error -512 could not register domain "55465DB6F3604834910621BF4D229455"
Aug 18 09:51:51 vmclkxtt3bvms05 kernel: o2net: no longer connected to node vmclkxtt3bvms06.esptest.aurtest.national.com.au (num 5) at 10.12.72.86:7777
Aug 18 09:52:02 vmclkxtt3bvms05 kernel: o2dlm: Nodes in domain 9D083EE1C5E24ED8A5192E1D473A0C8E: 4
Aug 18 09:52:17 vmclkxtt3bvms05 kernel: o2dlm: Nodes in domain 08EE37925B1C4EA0976B396A67D56D7F: 4
PLS help. Thanks.

Answer 1

Deven Shanbhag wrote:
I have checked the DMESG and ovs-operation log and found the below errors.
(mount.ocfs2,11035,0):ocfs2_find_slot:243 ERROR: no free slots available!Your OCFS2 volume only has 8 node slots, so you can't add a 9th server. You need to use tune.ocfs2 to increase the number of node slots for the OCFS2 filesystem.