Bucket vs Node Determinsm

Question

Hi Guys,
     We were wondering how a backup node is promoted to a primary in the event the primary fails. We appreciate that in a paritioned cache locating an object by key is deterministic in that the net result is knowing which node to go and fetch the data from.
     The question is though, if we have a large amount of data on a cache node and that cache node dies, must we expect the data to be transferred to another node before the primary continues. We sort of understand that the backup becomes the primary, but what happens exactly? We're wondering how this is achieved without having to stop the world while we move all the data around.
     Would it be fair to assume therefore that when you compute the location of the data you resolve the bucket but each node has a list of who owns each of the buckets? Thus allowing you to say, designate backup buckets and primary buckets and just to move things between lists? Therefore a rebalance can be a relativly fast operation with the new creation of the backup is an asynchrnous background task.
     Kind Regards,
     Max

Answer 1

Hi Max,
     Good to hear from you.
     > We were wondering how a backup node is promoted to a primary in the event the
     > primary fails.
     When the primary leaves the cluster, the direct result is a "member left" message within the cluster service on each member. The other services are likewise notified, and each conducts failover in its own specific manner. The partitioned cache service has a data structure (one that maintains information about ownership and backups, etc.) that includes references to the now-departed member, and as that departed member is removed from the data structure, the backups are automatically promoted. In other words, the chain of ascendency is set up well in advance, so the member leaving is actually a relatively small event in and of itself.
     For a particular member that was the first backup for some of the data managed by the primary, as it adjusts that ownership and backup data structure, it must also copy data from what is called a "backup map" into what is called a "backing map". (The "backing map" is what provides the actual data storage for a cache service, while a "backup map" is literally just a backup.) This operation can be more expensive, particularly if the backup is on disk, etc.
     > The question is though, if we have a large amount of data on a cache node and that
     > cache node dies, must we expect the data to be transferred to another node before the
     > primary continues.
     No. The new backups are made asynchronously, albeit as a higher priority operation.
     > We sort of understand that the backup becomes the primary, but what happens
     > exactly? We're wondering how this is achieved without having to stop the world while
     > we move all the data around.
     It should only slow down a bit (due to the backup operation occuring), not stop. Is that what you see?
     Peace,
     Cameron.