Problem: Stopping cluster due to unhandled exception .. Unable to refresh

Question

Greetings
While testing a rolling upgrade of an application that uses Coherence 3.5.2 on 62-jvm cluster hosted on
11 physical machines, we encountered a situation where after the upgrade was completed, most jvms
in the system abruptly left the cluster. The physical hosts are running CentOS 5.4, the java used is
the 64-bit server version 1.6.0_16-b01. The test was run under a scenario that imposed "moderate"
load, with cpu usage on the physical machines never exceeding 60% busy, network bandwidth never
exceeding 5%, and with some free physical memory. Swapping did not occur during any time during
the test. I believe we are using the default tangosol-coherence.xml. We got the error below in
our coherence.log files on all of the systems, all at about the same time (within 10 milliseconds).
55 of the jvms left the cluster during the incident. There were 55 copies of the error message in
the various logs, all nearly identical except for the time and member id.
My questions include
- what does the error mean?
- what could cause it? (I investigated system logs, and found no evidence of the NIC cards
going off line at the time. Any suggestions about how to look for evidence of broadcast storm?)
- how can we keep it from happening again?
Many thanks for your help -
Mike Murphy
2011-04-26 17:34:14,629 Coherence Logger@9224544 3.5.2/463 ERROR 2011-04-26 17:34:14.629/1929.311 Oracle Coherence GE
3.5.2/463 <Error> (thread=PacketListenerN, member=30): Stopping cluster due to unhandled exception: com.tangosol.net.mes
saging.ConnectionException: Unable to refresh sockets: [UnicastUdpSocket{State=STATE_OPEN, address:port=10.48.88.116:809
1}, MulticastUdpSocket{State=STATE_OPEN, address:port=224.3.5.2:10013, InterfaceAddress=10.48.88.116, TimeToLive=4}, Tcp
SocketAccepter{State=STATE_OPEN, ServerSocket=10.48.88.116:8091}]; last failed socket: MulticastUdpSocket{State=STATE_OP
EN, address:port=224.3.5.2:10013, InterfaceAddress=10.48.88.116, TimeToLive=4}
at com.tangosol.coherence.component.net.Cluster$SocketManager.refreshSockets(Cluster.CDB:91)
at com.tangosol.coherence.component.net.Cluster$SocketManager$MulticastUdpSocket.onInterruptedIOException(Cluste
r.CDB:9)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:33)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketLis
tener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketTimeoutException: Receive timed out
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
at java.net.DatagramSocket.receive(DatagramSocket.java:712)
at com.tangosol.coherence.component.net.socket.UdpSocket.receive(UdpSocket.CDB:20)
at com.tangosol.coherence.component.net.UdpPacket.receive(UdpPacket.CDB:4)
at com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketListener.onNotify(PacketLis
tener.CDB:19)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)

Answer 1

Hi Mike,
It looks like you are having problems with multicast. You can run the multicast test described here
[http://download.oracle.com/docs/cd/E15357_01/coh.360/e15723/tune_multigramtest.htm]
which will help in diagnosing the problem