Other members not responding when a clustered member is not reachable

Question

We have an product that uses Coherence 3.3 we have cluster 5 members.
We are finding that if for whatever reason a cluster member is down or unresponsive (eg doing a garbage collection) the other members are then trying to do some work to compensate, which leaves them not responding to requests. The nodes have been configured in a unicast mode so each node has been specified in the configuration.
Examples we've seen:
1) member 3 does a stop the world gc all other members pause - we see the following in the logs on all other members
2009-12-31 00:20:18.291 Oracle Coherence EE 3.3/387 <Warning> (thread=PacketPublisher, member=6): Experienced a 1982 ms communication delay (probable remote GC) with Member(Id=1, Timestamp=2009-12-30 16:01:26.85, Address=172.29.4.16:8090, MachineId=46352, Location=process:27758@and-msg4); 30 packets rescheduled, PauseRate=0.0030, Threshold=901
In this instance we had a pause of 19 seconds.
Can we configure the nodes to not wait if communication to member is not possible. How?
2) Take a single node down during the day (off peak hours) The other members start logging a lot of coherence debugging which I've been told not to worry about, but during this period the applicatiokn is not processing requests and causes timeout to the calling servers.
When a member is removed from the cluster what are the other nodes doing (redistributing cache?)? How can we minimise the fallout from this so that we can handle this more gracefully.
I would be grateful for any help and any useful links to documentation. It may well be that our configuration is incorrect as we leave most things as the default values. I could do with some pointers on where to start looking and which parameters to tweak as there seem to be quite a few.
Regards
Fez
What can we do to minimise this?
Edited by: user12406699 on 04-Jan-2010 07:31

Answer 1

Hi there thank you for the suggestions.
1) I understand that the pause during the gc will be helped by ensuring they are as short as possible. I have tinkered with the CMS parameters and I think our old was becoming very fragmented and therefore hitting the point where a stop the world occurred. I have limited the number of threads and increased the heap space. After a couple of days we are seeing that the concurrent garbage collector is doing work more frequently but with no stop the worlds. So I think issue 1 has will be resolved by this.
2) This issue still exists. When we stop tomcat server for one of the nodes we see a pause and lots of debug messages from coherence during this time the other node are no longer processing requests as we can see the number of running requests increase and the time to respond increases as well across the node. Because of the amount of traffic we are recieving our clients soon timeout.
Going over the sugestions
a) graceful shutdown of the cluster node
How? Currently we stop traffic being driven to the node and then stop tomcat. Is there a way to also gracefully stop node being apart of the cluster??
b) Set <timeout-milliseconds>
I don't quite understand the documentation for this setting
"Default value is 60000.
Note: For production use, the recommended value is the greater of 60000 and two times the maximum expected full GC duration."
So we will be currently using the default. To me 60 seconds sounds very high for a production timeout which makes me think that this is not what I had in mind. Is there anything in the configuartion that may help?
Regards
Fez