Fault tolerance of Distributed Locking Mechanism ?

Question

Is there a lock server for managing locks
     in a Coherence distributed cluster ?
     How fault tolerant is the locking mechanism ?
     What happens if a node which has acquired a lock dies ?
     If there is a lock server, what happens if the lock
     server dies ?
     Is there an upper limit on the number of nodes that
     a Coherence distributed cluster supporting distributed
     locking can have ? And how long does it take for
     the cluster to re-organize itself after a node has gone
     down ?
     Thanks!

Answer 1

A central lock server does not address any of the issues that you raise, and it also adds a single point of failure (SPOF) into the environment.
     The "link breaks" problem is known as the "split brain" scenario. There is no single correct answer for the split brain, and there cannot be (by principle) a single correct answer for it.
     Our plans for managing split brains involve the ability to define one site as a primary and one as a backup. If the primary becomes disconnected from the backup, it doesn't know if the backup is "down" or just disconnected, but it doesn't matter, because the primary is the primary so it needs to keep going.
     On the other hand, if the backup becomes disconnected from the primary, it may not know whether the primary went down or not, but it may be able to use a third party to find out. Consider points P, B, O1, and O2 where P is primary, B is backup, O1 is some other location and O2 is some other location. If B can reach O1 and O2, it knows that it still has connectivity. If B asks O1 and O2 if they can reach P, and neither can reach P, then it indicates that the problem is on P's side. Now the question is whether B should take over processing, which could be automatic, or it could be configured to be a manual switchover process.
     In either case, the answer could theoretically be incorrect, and like all unsolvable (literally unsolvable, not just unsolved) problems, you simply must choose which risks to mitigate, and how much expense you are willing to apply in order to reduce those risks.
     For example, Coherence 3.0 will allow you to run redundant networking across those sites, so losing a connection between two sites might actually require a minimum of three networks to fail completely. What's the cost of running three independent WANs over physically separated fiber? Especially if it is your own private cabling and networking and it's a 50km distance? It's pretty expensive, but that's how some financial services firms set up their DR sites.
     Like I said, there are no perfect answers, and there cannot be, but we provide the means for you to select from a variety of risk-mitigating solutions that in total can reduce the likelihood of any of these problems actually occurring into an infinitesimally small probability domain.
     Peace,
     Cameron Purdy
     Tangosol, Inc.