Dynamic and intelligent re-balancing of coherence partitions

Question

Can Coherence dynamically route more requests intelligently to better performing nodes on the same cluster? i.e. can it rebalance its partitions dynamically such that there are either more partitions or more frequently accessed partitions made to reside on the more powerful nodes of the cluster.
Edited by: 962776 on Oct 3, 2012 10:03 AM

Answer 1

962776 wrote:
Can Coherence dynamically route more requests intelligently to better performing nodes on the same cluster? i.e. can it rebalance its partitions dynamically such that there are either more partitions or more frequently accessed partitions made to reside on the more powerful nodes of the cluster.Hi,
this is kind of a loaded question, which is not as simple as it seems so let me step back a bit and put it into context.
Coherence maintains one copy of each partition which serves read and write behaviour.
It does not maintain multiple copies of the same partition even for read purposes (the redundant copies (backups) are just for high availability). This means that there is a single node which the client needs to communicate with, and does not need to maintain load information upon which it would need to decide where to route the request. You could possibly implement lagging-read behaviour to be serviced by the backup nodes yourself on top of Coherence, but it does not come out of the box, and you would need to transmit the load information to each other node which comes with an overhead. Lagging-read here meaning that you may read a not-up-to-date copy from backup nodes.
As for write operations, Coherence depends on there being only at most one active copy for write purposes, therefore you should not expect this to change anytime soon or not so soon...
As for your original question: can ownership be rebalanced based on the load?
It is not possible to do it out-of-the-box. Also, the functionality would come with its own gotchas:
- load information needs to be transmitted to the senior node (this may already be done behind the scenes)
- it may not be so simple to reconcile decisions based on load information with decisions based on balancing and dispersing partitions so that you get to a balanced and safe distribution while also optimizing the load
- Would you want to move partitions just based on the load, not only for availability/safety reasons? The current behaviour theoretically does something similar, if you consider the load cost being a constant one per partition. The operative word is constant.
Once a balanced and safe distribution is reached, Coherence does not want to move partitions around.
Why is this important? Because you cannot process operations while a partition is on the move.
If you wanted the load to be actually variable instead of constant 1, then it would possibly lead to much more frequent partition movements which would add additional load to the network and increase message latencies in general and in particular hinder responsiveness of the partitions on the move, which according to your request would likely be those which are already the most heavily loaded. That would possibly be counterproductive.
So yes, theoretically it is possible, you could implement it with 3.7.1+ by writing your own implementation of PartitionAssignmentStrategy. But is it practical? It really depends on your exact system.
An alternative approach which could tackle the problem may be changing your key partitioning algorithm so that it balances data in the partitions more evenly, maybe that will also translate to a more even distribution of the load. You would have to try and measure it. Also, you may want to look at BroadKeyPartitioningStrategy. Of course if you have a few very hot associated keys, that would make any single partition hot and that cannot really be helped.
Best regards,
Robert