Coherence multicast issue - nodes leaves, cannot join back

Question

We have six nodes in a cluster. We use multicast with a TTL=1. The nodes are all part of same VLAN.
After certain period of inactivity, we see multiple nodes leaving the cluster with messages like below...
2011-11-22 18:47:52,973 ERROR [STDERR] 2011-11-22 18:47:52.972/1840.837 Oracle Coherence GE 3.5.1/461 <Error> (thread=PacketListenerN, member=2): Stopping cluster due to unhandled exception: com.tangosol.net.messaging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=10.137.2.64:8088}, MulticastUdpSocket{State=STATE_OPEN, address:port=224.3.5.7:35473, InterfaceAddress=10.137.2.64, TimeToLive=1}, TcpSocketAccepter{State=STATE_OPEN, ServerSocket=10.137.2.64:8088}]; last failed socket: MulticastUdpSocket{State=STATE_OPEN, address:port=224.3.5.7:35473, InterfaceAddress=10.137.2.64, TimeToLive=1}
     at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
     at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluster.CDB:9)
     at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
     at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
     at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
     at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Receive timed out
     at java.net.PlainDatagramSocketImpl.receive0(Native Method)
     at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145)
     at java.net.DatagramSocket.receive(DatagramSocket.java:725)
     at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
     at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
     at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketListener.CDB:19)
     at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
     at java.lang.Thread.run(Thread.java:662)
2011-11-22 18:47:52,973 ERROR [STDERR] 2011-11-22 18:47:52.973/1840.838 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=2): Service Cluster left the cluster
2011-11-22 18:47:53,093 ERROR [STDERR] 2011-11-22 18:47:53.090/1840.955 Oracle Coherence GE 3.5.1/461 <D5> (thread=DistributedCache, member=2): Service DistributedCache left the cluster+
The exact same message appears on 4 out of 6 nodes. Other two nodes then join back and form a smaller cluster.
When the next request goes to one of the "dead" node, that node tries to join back with the original cluster and it fails with the message below. The same kind of message happens on another server as well. Thus neither can join back the original cluster or form a new one.
Questions:
*1. Why would the members leave?"*
*2. When they try to join back why do they get exceptions trying to join back to the cluster?"*
*3. What are the recommendation to ensure that the members can join back? How can we increase the retry count so that they don't give up on retrying after 3 attempts?"*
*4. Is there something to be concerned about when it says "Service DistributedCache left the cluster"? What is its meaning?*
2011-11-23 14:11:31,559 ERROR [STDERR] 2011-11-23 14:11:31.559/71659.424 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
2011-11-23 14:11:58,045 ERROR [STDERR] 2011-11-23 14:11:58.045/71685.910 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2011-11-23 14:11:31.499, Address=10.137.2.64:8088, MachineId=33088, Location=site:company.net,machine:host2,process:3527) has been attempting to join the cluster at address 224.3.5.7:35473 with TTL 1 for 26 seconds without success; this could indicate a mis-configured TTL value, or it may simply be the result of a busy cluster or active failover.
2011-11-23 14:11:58,047 ERROR [STDERR] 2011-11-23 14:11:58.047/71685.912 Oracle Coherence GE 3.5.1/461 <Warning> (thread=Cluster, member=n/a): Received a discovery message that indicates the presence of an existing cluster that does not respond to join requests; this is usually caused by a network layer failure:
Message "SeniorMemberHeartbeat"
FromMember=Member(Id=1, Timestamp=2011-11-22 18:15:41.108, Address=10.137.2.63:8088, MachineId=33087, Location=site:ie.intuit.net,machine:qypprdfisap01,process:13982, Role=JavaLangThread)
FromMessageId=0
Internal=false
MessagePartCount=1
PendingCount=0
MessageType=17
ToPollId=0
Poll=null
Packets
[000]=Broadcast{PacketType=0x0DDF00D2, ToId=0, FromId=1, Direction=Incoming, ReceivedMillis=14:11:58.45, MessageType=17, MessagePartCount=1, MessagePartIndex=0, Body=0x0000000133CE3505340A89023F00000000000000000000000040001F980000813F0001050104040E636C75737465723A3078433541420D69652E696E747569742E6E6574400D71797070726466697361703031053133393832400E4A6176614C616E67546872656164000000000133D27C381B010000000500, Body.length=121}
Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.5}
ToMemberSet=null
NotifySent=false
LastRecvTimestamp=Wed Nov 23 14:11:56 PST 2011
MemberSet=MemberSet(Size=2, BitSetCount=1, ids=[1, 3])
2011-11-23 13:24:03,630 ERROR [STDERR] 2011-11-23 13:24:03.628/68794.632 Oracle Coherence GE 3.5.1/461 <D5> (thread=Cluster, member=n/a): Service Cluster left the cluster
2011-11-23 13:24:03,632 ERROR [STDERR] 2011-11-23 13:24:03.632/68794.636 Oracle Coherence GE 3.5.1/461 <Error> (thread=ajp-0.0.0.0-8009-1, member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=0, BitSetCount=0
MemberId/ServiceVersion/ServiceJoined/ServiceLeaving
)

Answer 1

Please do a test with %Coherence_Home%\bin\multicast-test.cmd to check that multicast is OK....
and the do a test with %Coherence_Home%\bin\datagram-test.cmd to check your network performance....
Look at the documentation it is well documented...