"Partially lost connectivity" error message

Question

We're conducting some load testing and when we reach a certain threshold in the test, Coherence will report this error:
[2010-10-21 20:21:39,376] ERROR [Logger@9243969 3.5.3/465p7] (Log4j.CDB:3) - 2010-10-21 20:21:39.376/7166.760 Oracle Coherence GE 3.5.3/465p7 <Error> (thread=Cluster, member=6): This node appears to have partially lost connectivity: it receives responses from MemberSet(Size=2, BitSetCount=2, ids=[4, 7]) which communicate with Member(Id=5, Timestamp=2010-10-21 18:22:38.456, Address=10.13.9.35:13921, MachineId=3619, Location=site:int.qaload.lan,machine:bcla01,process:9977, Role=RNTCServerEmbedded, PublisherSuccessRate=0.9615, ReceiverSuccessRate=0.9879, PauseRate=0.0020, Threshold=1785, Paused=false, Deferring=false, OutstandingPackets=1, DeferredPackets=0, ReadyPackets=0, LastIn=91315ms, LastOut=3301ms, LastSlow=n/a), but is not responding directly to this member; that could mean that either requests are not coming out or responses are not coming in; stopping cluster service.
We've verified that we're not approaching the capacity of the NIC on the server or the network itself. The CPUs on the servers are also well under maximum levels.
Other than the obvious (i.e. a network connectivity problem), what would cause this error? In other words, under what circumstances is this error displayed?

Answer 1

Hi Jeff,
Not sure if you've seen the description of the message here ...
http://coherence.oracle.com/display/COH35UG/TCMP+Log+Messages
Cause : The communication link between this member and the member indicated by %s2 has been broken. However, the set of witnesses indicated by %s1 report no communication issues with %s2. It is therefore assumed that this node is in a state of partial failure, thus resulting in the shutdown of its cluster threads.
Action : Corrective action is not necessarily required, since the rest of the cluster presumably is continuing its operation and this node may recover and rejoin the cluster. On the other hand, it may warrant an investigation into root causes of the problem (especially if it is recurring with some frequency).
Thanks