Re:Question related to the Packet timeout error

Question

2010-03-08 09:11:05.273 Tangosol Coherence CE 3.2.2/371 <Warning> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-03-07 12:23:13.958, Address=xxxx, MachineId=5238) was unresponsive for 15369 ms, 93 packets have timed-out, PauseRate=0.0040, Paused=false, Deferring=true, OutstandingPackets=0, DeferredPackets=12, Threshold=512
2010-03-08 09:19:22.772 Tangosol Coherence CE 3.2.2/371 <Warning> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-03-07 12:23:13.958, Address=xxxx, MachineId=5238) was unresponsive for 16448 ms, 111 packets have timed-out, PauseRate=0.0042, Paused=false, Deferring=true, OutstandingPackets=0, DeferredPackets=10, Threshold=512
2010-03-08 09:30:52.651 Tangosol Coherence CE 3.2.2/371 <D5> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-03-07 12:23:13.958, Address=xxxx, MachineId=5238) was unresponsive for 28 ms, 20 packets have timed-out, PauseRate=0.0042, Paused=false, Deferring=true, OutstandingPackets=0, DeferredPackets=1, Threshold=512
2010-03-08 09:37:50.583 Tangosol Coherence CE 3.2.2/371 <Warning> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-03-07 12:23:13.958, Address=xxxx, MachineId=5238) was unresponsive for 10590 ms, 60 packets have timed-out, PauseRate=0.0043, Paused=false, Deferring=true, OutstandingPackets=0, DeferredPackets=5, Threshold=512
2010-03-08 09:50:00.635 Tangosol Coherence CE 3.2.2/371 <D5> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-03-07 12:23:13.958, Address=xxxx, MachineId=5238) was unresponsive for 132 ms, 20 packets have timed-out, PauseRate=0.0043, Paused=false, Deferring=true, OutstandingPackets=0, DeferredPackets=1, Threshold=512
2010-03-08 09:50:19.086 Tangosol Coherence CE 3.2.2/371 <Warning> (thread=PacketPublisher, member=2): Member(Id=3, Timestamp=2010-03-07 12:23:13.958, Address=xxxx, MachineId=5238) was unresponsive for 8954 ms, 73 packets have timed-out, PauseRate=0.0044, Paused=false, Deferring=true, OutstandingPackets=0, DeferredPackets=10, Threshold=512
1. Will the packet retry if they get this timeout error?
2. why is the logs timestamp and the packet timestamp is having very large gap nearly 1 day in some cases
3.How long does the packet wait before it timesout ,i see the unresponsive time varies from 80ms to 10000 ms
4.When do we get this warnings? is there any remedy?

Answer 1

Wow Coherence 3.2, I haven't seen that in years, it is a pre-Oracle version. I'd suggest considering an upgrade to 3.5, which should be doable without even recompiling your code, just restart the cluster with new jars.
But that isn't what you asked. This message was reworded in Coherence 3.3 and later to refer to these events not as packet timeouts, but as rescheduled packets do indicate that they are automatically retried.
Regarding the timestamp that is the timestamp of the cluster member to which the packet was sent. A members timestamp is the time at which it joined the cluster.
The default packet retry interval is 200ms, I.e. If we don't get an ACK to a packet in that time we will retransmit it. After 1m of this we will give up and remove either the sender or receiver from the cluster. The time intervals in the log messages are with respect to when the first ACK was expected, so a reported delay of 80ms means it took 280ms to delived a packet.
These messages highlight periods of failed communication, and usually are the result of garbage collections. Some of yours are quite bad (multiple second), indicating that you've likely overcommited your java heap. The effect of the outages should be poor overall cluster performance and if bad enogh should also cause heavily GCing nodes to be kicked out of the cluster.
Mark
Oracle (Tangosol) Coherence