UDLM rac-udlm-rs failing to come online

Question

Hi all,
I'm setting up a zonecluster that will be running RAC. This has gone well, up to the point where, using clsetup, I attempt to create the rac-framework-rg (and associated rac-udlm-rs, rac-framework-rs).
In attempting to create the resourcegroup, the rac-udlm-rs fails to come online, causing the rest of the framework to fail.
The error in the logfile for UDLM (/var/cluster/ucmm/dlm_<zone>/logs/dlm.log is:
CONNECT ERROR: 'Invalid argument', family=0, port=6009, in=0.0.0.0
This repeats 5 times per second on both zonecluster nodes. regularly the error changes port (i.e. it cycles through the ports).
I've tried rebuilding the zonecluster, to no avail. I am at a loss as to what the error might be, or where to start looking, since there is no evidence anyone has ever hit this error before.
Any suggestions on where to look and what I can try to fix this is will be appreciated.
Possibly related, or even the cause, is that the zonecluster has been configured with:
add net
     address=xxx.xxx.xxx.xxx
end
at the zonecluster level, but these addresses are not showing up under 'ifconfig -a'. Maybe I'm misunderstanding how they are used, and when they should show up though.
Some notes:
the zonecluster in all other respects seems fine - I have QFS resources configured and these appear to successfully work in all circumstances.
the ORCLudlm package is installed (the one that comes with Oracle 11.2.0.3)

Answer 1

Right, in case this helps someone else in future:
I found the problem.   In a fully working zonecluster configuration the /etc/nsswitch.conf file includes these lines:
hosts:  cluster files
netmasks: cluster files
but for sad, sad, reasons on the system I was setting up this file had:
hosts:  files
netmasks: files
and you know what, this allows you to go:
ping clusternode1-priv
and this does a lookup and resolves correctly. Now, if this doesn't resolve correctly, basically the UDLM daemon cannot communicate with the other cluster nodes, because it must do a nameserver lookup on clusternode1-priv internally.
So, in essence, make sure your nsswitch.conf file looks up the cluster first for hostname lookups.
This issue shows itself if you do a traceroute to the IP of the other cluster node (from one of the zonecluster nodes) using the private interconnect IP address. In a working system it'll show the name (clusternode1-priv). In a non-working system it shows the IP only.