Heartbeat time outs some times caused by clocks-out-of-sync?

Question

Heartbeat timeouts are usually caused by the java socket communication between the master and its slaves being slow (or dead).
This slowness in turn can be caused by a system being overloaded, something than can easily happen when we're running out of memory.
Memory problems can be caused by running against file system cache, which in turns eats up all memory, etc.
Some times the java socket connection between the master and remote hosts just is slow, also causing heartbeat timeouts.
Today I had a case where, in a multi-host situation, one of the remote hosts had his clock set five minutes early.
On the master this resulted in lots of 'slow getmessage' warnings telling you that it took longer than five seconds between the time the message was sent and that it was received. These timings use both the master and the remote host's current clock values.
In above case that always meant that every message was considered five minutes late.
Then, further on during the test the remote system's clock was reset and increased by five minutes.
That was it for the heartbeat timer, because it also uses the system's clock values.
So, if you run into heartbeat problems again, check your clocks.
In your 'xxxx-0.stdout.html' file you can verify, within reason, the clock settings, for instance:
11:52:19.013 11:47:17.605 task_run_all(): 395 tasks
11:57:39.004 11:57:39.000 Slow getMessage: hd2-0 1957 301404 REQUEST_SLAVE_STATISTICS
The first time value is the timestamp from the master at the time the message was received, the second is from the slave at the time that the message was sent, and as you can see, there is a five minute difference. The next message shows the now changed clock values.
The important message of this: make sure your clocks are synchronized in a multi-host environment.
Henk.

Answer 1

If you have more than one user account, these instructions must be carried out as an administrator.
Launch the Console application in any of the following ways:
☞ Enter the first few letters of its name into a Spotlight search. Select it in the results (it should be at the top.)
☞ In the Finder, select Go ▹ Utilities from the menu bar, or press the key combination shift-command-U. The application is in the folder that opens.
☞ Open LaunchPad. Click Utilities, then Console in the icon grid.
Make sure the title of the Console window is All Messages. If it isn't, select All Messages from the SYSTEM LOG QUERIES menu on the left. If you don't see that menu, select
View ▹ Show Log List
from the menu bar.
Enter the word "Starting" (without the quotes) in the String Matching text field. You should now see log messages with the words "Starting * backup," where * represents any of the words "automatic," "manual," or "standard." Note the timestamp of the last such message. Clear the text field and scroll back in the log to that time. Select the messages timestamped from then until the end of the backup, or the end of the log if that's not clear. Copy them (command-C) to the Clipboard. Paste (command-V) into a reply to this message.
If all you see are messages that contain the word "Starting," you didn't clear the search box.
If there are runs of repeated messages, post only one example of each. Don't post many repetitions of the same message.
When posting a log extract, be selective. Don't post more than is requested.
Please do not indiscriminately dump thousands of lines from the log into this discussion.
Some personal information, such as the names of your files, may be included — anonymize before posting.