Dr. Damien's Developmen​t - Debugging Race Conditions

Question

Awhile back, one of the forum posters suggested I share some of the techniques I have developed in my years as a LabVIEW programmer.  This is the first in an occassional series.  Race condition debugging is the topic of the day, since it was the topic which prompted the suggestion.
Anyone who has programmed LabVIEW for any period of time has probably run into a race condition.  You used a global variable when you shouldn't.  That local that used to be a simple scalar morphed into a cluster and then starting blapping your control booleans.  You copied a cluster and didn't reinitialize it correctly.  There are lots of ways to do it.  But how to do you find it?
Race conditions often rely on an exact timing and execution sequence.   A good clue that you have a race condition is that your code works correctly when you step through it with execution highlighting.  You need a way to see what is happening in real time.  I do this by sprinkling my code with the small VI attached below (LabVIEW 7.1 version, Windows only).  This VI writes strings to the debug device every time it is run.  These strings can be read by anything which will read the debug device, but the easiest way to watch them is using DebugView from the Microsoft SysInternals suite of applications.  DebugView gives you timestamps with processor tick count precision, so you can accurately determine exactly when the code runs (Note that Linux and Mac users can get much the same functionality by writing a string to stderr.  Since I have not actually done this, I cannot give details of use.).
Where do you put these function calls?  If you have no clue where the problem is, start with your top level loop.  Make sure you put identifying tags at each location.  Include the values of the variables you are trying to find race conditions in.  This should allow you to find the problem spots.  At this point, start placing more tags in a sort of binary search for the problem.  Experience has shown that you can usually find a race condition in under an hour using this technique, provided you can reproduce the race condition in the first place (which may be a real problem).
If you are using LabVIEW 8.0 or better, you can put a conditional compile structure around the error debugging code and leave it in your code so you don't have to remake it every time.
There are probably as many methods of debugging race conditions as there are LabVIEW programmers.  How do you do it?
This account is no longer active. Contact ShadesOfGray for current posts and information.
Attachments:
WinDebugLogging.vi ‏33 KB

Answer 1

Logging when things happen is a great way to start to track down race conditions. For logging purposes I use a queue-based logging system. There is a separate log viewer that's essentially a window with a string indicator with a built-in search mechanism, but there is also an option to dump it to file. As with the previous solution, it's cross-platform. 
Unfortunately, logging on its own is not enough to be able to track down race condition caused by local variable abuse since there is no data dependency.  There is no way to know whether the access of the local happened before or after the logging event. In this respect there is little that LabVIEW programmers can do other than to artifically create a data dependency which essentially removes the race condition that you were trying to track down in the first place. Given this, it would be great if LabVIEW had a built-in mechanism for logging read/write access to locals and globals, which in most cases are the primary causes of race conditions.
By the way, is "blapping" a technical term?  I will let the reader look up that term.