Tons of "Slow getMessage: " errors on console

Question

Hi folks,
Since upgrading to 5.04.02, I've been seeing tons of "Slow getMessage: " notices show up on the console. There's a wide variety of these:
17:51:13.003 Slow getMessage: vdb-001-3    5795  3761 SLAVE_STATISTICS
17:51:29.622 Slow getMessage: vdb-004-0    5813  3786 SLAVE_WORK_COMPLETED
17:51:29.622 Slow getMessage: vdb-004-0    5814  3786 READY_FOR_MORE_WORK
17:51:47.574 Slow getMessage: vdb-001-0    5814  3759 HEARTBEAT_MESSAGE
The workload seems to running fine, and if I filter these out the latency seen by vdbench is only slightly higher than the latency seen by the storage system. The CPU load on the master is negligible. Master and slaves communicate over a 10GbE network with ping times less 0.5ms. System and user CPU load on the slaves is usually low, with most of the time spent in IO wait. Memory is also plentiful and each java process hasn't reached it's Xmx limit. There are times where each java process shoots up to 100% CPU on the processor it holds, but the high CPU for the java processes seems to happen only between runs where there's some communication happening between the master and slaves. Unfortunately the getMessage notices are showing up all the time. Only their contents seem to change.
Environment:
CentOS 7 VM's, 8 vCPU, 8GB RAM
1 master, 8 slaves
java version "1.7.0_71", OpenJDK Runtime Environment (rhel-2.5.3.1.el7_0-x86_64 u71-b14), OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)
vdbench java flags:  -d64    -Xmx896m  -Xms448m
So a couple of questions:
Are the messages indicative of any problems in the system?
What is likely to be causing these messages when everything seems to be otherwise fine?
How do I tell vdbench to not print these messages? Or do I need to resort to grep -v?
Any suggestions or pointers?
Thanks and take care,
Miroslav

Answer 1

Communication between the Vdbench master and its slaves is done using java socket.
Every interval=n seconds Vdbench asks its slaves for the latest interval's workload and performance statistics.
Throughout the years there have been intermittent problems with socket messages arriving far too late either on the Master or on the Slaves, causing reporting inconsistencies.
What you are seeing is an attempt to provide information about how slow the message traffic can get, this in the hope to some day truly figuring out what the heck is going on. If ANY message arrives more than two seconds late I now report how long it took from the 'send' to the receive'.
I agree, these messages do not belong on the console, and here is a fix: SlaveSocket.class
Place this in /vdbench50402/classes/Vdb
So what is causing the socket delays? Usually when a system is low on memory or free cpu cycles, or, in a multi-host environment maybe the socket traffic is competing with the data traffic of the workload generated by Vdbench. I have never found a real answer.
A second problem can occur when the socket traffic is delayed by more than three minutes, because that is when the Vdbench heartbeat monitor decides that everything is stuck and aborts. To change the heartbeat timeout from three to fifteen minutes add '-d27' as an execution parameter or 'debug=27' at the top of your parameter file.
Hope this helps.
Henk