Disable PartitionListeners on service prior to cluster shutdown

Question

We're running a Coherence cluster of 70 storage nodes and a further 10 storage-disabled tcp-extend nodes, all running DefaultCacheServer.
I have a PartitionListener configured on a distributed cache service which sends an email to a distribution list in the event of a 'PARTITION_LOST' event. However, currently when the storage-enabled processes are stopped (by killing the processes), we get spurious emails with partition losses due to the cluster being shut down.
Is there a tidy, built in way to disable the partition listeners across a cluster before the storage nodes are stopped, and avoid sending these spurious mails? Note that the mails are not sent by the process that is currently stopping, but by one of the other processes when the PartitionEvent is detected.
I've tried using of Cluster.shutdown() but this does not stop the listeners from being called. I also tried using Cluster.isRunning() in the listener to determine whether to send the error email, but it appears the cluster gets restarted (and isRunning() returns true again) if CacheFactory.getCache() is called after Cluster.shutdown() has been called, so this is not a safe mechanism either.
My current solution is to register each partition listener as a JMX bean on the management service (Registry) as it's instantiated, and expose a disable() method. I then run a standalone java class which joins the cluster and iterates over those registered beans calling disable on each one, prior to stopping the storage processes.
The above solution works, but am I missing a built-in way of doing this? I'd rather use an out of the box solution if there's one that fits the bill.
Thanks in advance,
Giles Taylor

Answer 1

Hi Giles,
I don't think that there is a feature for this particular purpose, as for Coherence the death of a node is a usual event, it does not have the concept of "I am going to kill all nodes, now".
However, it is very easy to to implement it on your own.
Just have a special-purpose cache, on which you register an event listener. The event-listener listening for a certain change in the cache (e.g. you put a specific value to a specific key, to indicate that full shutdown will commence now) should then set a static flag which indicates that full cluster shutdown has started and therefore you do not need to send emails on partition loss from now on. Alternatively you can do the same thing with an InvocationService (that invocation service needs to be started on all nodes). Alternatively you can also put that special entry simply into a replicated cache (so checking it can be locally carried out).
Then in the partition listener you just check the static flag or the replicated cache, according to whichever approach from the previous ones you chose, and you send the mail only if you did not indicate that full shutdown is commencing.
Then if you want to do a full shutdown, you first carry out the action according to the approach which you chose, and only then start to kill nodes.
BR,
Robert