Cluster fragmentation

Question

What happens when part of my cluster cannot communicate with the other part?
For example, what if I have two machine in a cluster and at some point the two machines can longer communicate with each other but are otherwise connected to the network and running fine.
Will they both assume the other machine has left the cluster and form two independent clusters? This can lead to problems as an update made by one machine will not be reflected in the other. Basically each clusters will become out of sync with the db as the other cluster makes updates.
Or, will one machine somehow become the sole member of “the cluster�? and the other will not be in “the cluster�? at all. If this is the case, how does the machine that has left the cluster behave? Do all cache access methods throw exceptions until it is able to rejoin the cluster or what?

Answer 1

Hi Rohan,
The two islands will start looking for each otherthe
next timethe application code calls into the
Coherence API (e.g. CacheFactory.getCache(...)).When
they will see each other again depends on howflaky
your server/network/switch is.I have observed that when the split occurs, if the
one server that is effectively in an island by itself
does not make calls into the Coherence API, it
remains disconnected even though the other servers in
the cluster continue to make their own calls into the
API.Exactly, there is no need for this node to be in the cluster if it is not using the cache (i.e. accessing the API).
Also, I have seen this warning:
2005-06-23 09:18:47,270 INFO [STDOUT] 2005-06-23
09:18:47.270 Tangosol Coherence 2.5/290 <Warning>
(thread=Cluster, member=3): The member formerly known
as Member(Id=1, Timestamp=Thu Jun 23 09:18:46 UTC
2005, Address=A.B.C.139, Port=8088, MachineId=2955)
has been forcefully evicted from the cluster, but
continues to emit a cluster heartbeat; henceforth,
the member will be shunned and its messages will be
ignored.
Does this mean that the shunned Member 1 will never
be able to rejoin the cluster, since its messages
will be ignored?The member will attempt to rejoin the cluster once it has (1) shut down all its Coherence services and (2) the application calls into the Coherence API again.
Log messages on Member 1:
2005-06-23 09:18:47,390 INFO [STDOUT] 2005-06-23
09:18:47.389 Tangosol Coherence 2.5/290 <Error>
(thread=Cluster, member=1): This senior Member(Id=1,
Timestamp=Thu Jun 23 08:53:10 UTC 2005,
Address=A.B.C.139, Port=8088, MachineId=2955) appears
to have been disconnected from other nodes due to a
long period of inactivity
and the seniority has been assumed by the
Member(Id=2, Timestamp=Thu Jun 23 08:53:17 UTC 2005,
Address=A.B.C.140, Port=8088, MachineId=2956);
stopping cluster.
2005-06-23 09:21:47,908 INFO [STDOUT] 2005-06-23
09:21:47.908 Tangosol Coherence 2.5/290 <Info>
(thread=Thread-48, member=1): Restarting NamedCache:
ruleCache
2005-06-23 09:21:47.908 Tangosol Coherence 2.5/290
<Info> (thread=Thread-48, member=1): Restarting
Service: DistributedCache
2005-06-23 09:21:47.908 Tangosol Coherence 2.5/290
<Info> (thread=Thread-48, member=n/a): Restarting
cluster
Member 1 did appear to be operating as normal after
this ... do these log messages mean it really was
part of the cluster again or did it just think it
was, even though it was being shunned?Correct, it is now part of the cluster again. As I stated above a shunned member will attempt to rejoin the cluster.
This is the level of fault-tolerance and reliability that is built into Coherence from the start. However, I would still suggest fixing the flaky server/network/switch.
Later,
Rob Misek
Tangosol, Inc.
Message was edited by: rmisek