Map/Reduce and Affinity

Question

First of all, let me describe use case.
Where are two types of objects (and accordingly two distributed caches): Child(cid, pid, cdata1, ....) and Parent(pid, pdata1, ...). Child's key have key association so, child objects stored in the same partition where its parent stored. So, I need to run the query against parents with several constraints applied. These set of constraints contains constraint which is based on data from corresponding child objects. How to do it efficiently without precalculation and putting precalculated value in Parent objects and applying constraint to it?
Example:
User(uid, firstname, lastname) and Address(aid, uid, zip) (user can have several addresses)
query is: select all users which firstname equal to 'John' and have address with zip equal to '65535'
The second case is pretty the same, but the query is slight different: select all addresses with zip equal to '65535' and which user's firstname equal to 'John'

Answer 1

Hi,
I was asking about immutability of fields used to establish the join relationship between the parent and the child and the rest of the attributes in one of the tables.
But it makes too many restrictions, I believe.
Ok, in the general case of map reduce you can just do an aggregation and idempotently put the reduced info to a cache in a DIFFERENT cache service from what you aggregating in an idempotent way (always the same data on the same key even for retries), which also means that the granularity of such data must be so that an entry of the reduced data must be reduced from entries only in a single partition of the mapped data if this is not the final reduce step.
Let's look at an example: you are doing a two step map/reduce.
First step maps entries like this:
Mapped data keys are:
P1K1
P1K2
P2K3
P3K4
where with the notation P2K3 I mean that Coherence assigned the key to partition 2. This assignment has to do with Coherence's key-partitioning strategy (which is pluggable, by the way), you cannot explicitly decide it on an entry-by-entry basis.
Let's assume that P1 and P2 are at the time both owned by node X, P3 is owned by node Y.
When you are doing an aggregation of this cache, it is possible (likely) that your aggregator on node X will get entries for keys P1K1, P1K2, P2K3.
Your aggregator must create a reduced result for (P1K1 + P1K2) which is a separate from the reduced result entry from P2K3. That result entry must only depend on the data in the entries from which it was reduced respectively (P1K1 + P1K2 or P2K3, but not both) and not on anything else. This is because Coherence may reexecute that
aggregation later on a node which has only P1 but not P2 if node X died, and the cache might also have one of the reduced results and you would end up either with corrupt results or you would have to write more complex code to cover this case.
You have to put these RP1K12, RP2K3, RP3K4 reduced entries to a cache separate from the mapped cache so that you are not facing the possibility of deadlocks or thread-starvation in the service of the aggregated cache.
This unfortunately will mean that the reduced result will likely not be on the same node as the mapped data, and that would mean lower performance compared to if you were able to do this in the same node in case your both caches you want to join together were on the same node.
Another approach, for affine data is that from the aggregator you locally enqueue the reduced data to a local queue from which its consumer fires off the second step of aggregations to the other cache to be mapped which aggregation holds the first reduced data in its state. You will have to care about duplicate checking of partial reduced data on the final reduce step so each step of your reduced data must carry its source partitions from all preceeding steps.
I will possibly show an example of this in a couple of days, but I will have to iron out a couple of things, first, like ensuring at-least once guarantees for starting the next step.
Alternatively you can build something similar with InvocationService which helps a bit by providing an asynchronous invocation possibility.
Best regards,
Robert