Detection of proxy node outage

Question

Question regarding detection of proxy node outage in extend client.
My impression is that once connection established, you won't notice that proxy node was gone until next time you try to access cluster (get, put, etc. etc.).
Or there is anything like MemberListener thing that client will be notified when proxynode is gone?
This cause us some issue since we will add a map listener within extend client, but we won't receive any notification if the proxy node is gone.
To address this issue, we are implement a heartbeat approach which client will have a background thread perform periodically cache get to ensure that the connection still valid.
Above approach worked fine for single proxy node approach, since if the heartbeat check failed, we know something wrong.
However, the heartbeat approach create new issue when there are multiple proxy nodes configured in client's remote cache scheme.
This is what we observed.
An extend client configured with a remote cache scheme has proxyA and proxyB as proxy list.
1. Extend client start with connection to proxyA.
2. Extend client subscribe to key ABC for cacheX
3. Kill proxyA process.
4. The background heartbeat check thread perform a heartbeat get to cacheX. Looks like during this cacheX.get() operation, Coherence will establish a new connection to proxyB automatically. Our application doesn't even notice there is a proxy node switch.
Now the issue is if an update to key ABC occurred between step 3 and 4. Our application won't receive any notification at all. And we don't even notice there is a new connection created to a different proxynode at all.
Our business requirement require us not to miss any entry update notification in extend client (unless if all proxy nodes are down) as long as there is at least one proxy node available.
Is there any better approach for this requirement?

Answer 1

J.K.
Turn out that member listener approach still have chance to lost update notification.
We found out that the update occurred between MemberLeft event notification and re-add map listener will be lost.
That window is small, however, our business requirement require that no lost of update event at all as long as the update operation succeeded (the update could be performed by another extend client connect to different proxy node).
Is it even possible to ensure that requirement using extend client?
Regards,
Chen