Master-Worker Implementation

Question

Hi all,
We would like to implement the Master-Worker pattern with coherence. To do so, we make our master load at startup all the computation descriptions from the database and put them into the distributed cache. Considering that the local storage parameter is set to false on the master’s configuration file, this action ensures that all our computations are fairly distributed over the workers. At runtime, all the incoming data of the entire platform go trough the master which will dispatch them to the corresponding worker(s). Thus the master must always know which member is responsible for a specific computation in order to minimize network traffic by sending only the relevant data to each concerned worker. To ensure that the master is always aware of the cache distribution, we keep our own Map<ComputationKey, Member> up-to-date by loading it at startup and by reloading it when a member joins or leaves the cluster. To load our map, we use an InvocationService to execute an agent which gets the content of the local BackingMap, on each worker. To send the data to a specific worker, we use another InvocationService to execute an agent which gives the data to the local data manager of the worker. The worker treats the given data then sends back the computed data to the master through a last InvocationService.
Is there a better way to do so? If not, after a MemberEvent has been thrown, is there a way to know if the redistribution is fully completed? Because currently, we launch too soon the agent which gives us the cache distribution, thus our map becomes invalid as it doesn't contain anymore the right amount of data.
Thanks for your Answer,
Nicolas

Answer 1

Hi Nicolas,
If a computation is bound to a single entry, the best approach would be to use the NamedCache.invoke() API, which provides the once-and-only-once execution guarantee. If a computation could be bound to multiple entires residing in a single partition (using a custom key association), that approach could still be used.
Moving forward, you should not assume that re-distribution can only be triggered by fail-over or fail-back events. We are currently working on dynamic weight-base distribution algorithms that would factor in the amount of actual data stored in a partition as well as CPU utilization related to that data. As a result, change in a data usage pattern could cause some re-distribution, though in either case (today or tomorrow), the distribution events are going to be very rare ones and in general the partition ownership could be counted on as being very stable.
Another point I'd like to make that checking the ownership information is a very efficient call, that does not require any network communications. In addition to currently exposed key-based ownership API, Coherence 3.4 introduces a member-based ownership PartitionedService request:
public PartitionSet getOwnedPartitions(Member member);If you choose to use the InvocationService to execute the computation agents, you could use the ownership check to quickly determine the stability of the distribution before and after a computational call (or just in cases when the computational call fails to return reliable data).
Regards,
Gene