Tracking down a periodic delay in a cRIO

Question

Hi Folks,
I am having trouble with a mystery process delaying my main loop in a RT system. I am looking for advice on how to debug this problem. Thanks!
I have an industrial control project in LabVIEW 2009 SP1. It is running in a 9014 cRIO. The main time critical loop runs every 10 ms and both reads and writes using implicit Class 1 Ethernet/IP (Industrial Protocol, a standard from ODVA.) to an Allan Bradely Control Logix PLC.
The PLC has a watchdog counter. It writes a new value on the counter which is then written to the cRIO. The cRIO copies the counter value back to the PLC. The PLC counts how many "cycles" it takes to receive this value back from the cRIO. Each is cycle is 5 milliseconds, the time it takes to run one loop of the ladder logic. If the count of cycles gets to 15, an alarm is set and the control system is effectively shut down.
What I am seeing is that most of the time, the communications works just fine. The data gets through in 2 or 3 cycles, which is what you would expect. About every 6 seconds, the com fails and the watchdog counter spikes past 15. This appears to be something going on in the cRIO.
I did some tests in the cRIO. I set up a timer to tell me how long it was taking to get from the start of the main loop to various points in the loop. The loop is broken up by frames of a sequence structure. What I found was that the loop always ran in the same amount of time, even when the delay was clearly observed. In other words, the delay is happening between iterations of the main loop.
I also tried to use the Remote Execution Trace Toolkit to see what I could see. I set up some code to detect the delay and then trigger a send of the trace data. At best I can get 1.4 miliiseconds of data. Since the delay itself is 90 milliseconds, I am not able to see an event that is relevant to my search.
So how do I figure out what is going on?

Answer 1

Not knowing if you are a very experience LV RT programmer, it is difficult to know if the following are obvious to you. The cRIO developers guide is very good at explaining how to handle deterministic and non-deterministic processes, and how you need to decide how to distinguish and implement them and how LV RT prioritises their operation. Chapter 3 is really useful, here are a couple of quotes:
"Your application might consist of both deterministic and nondeterministic tasks. Deterministic processes must complete on time, every time, and therefore need dedicated processor resources to ensure timely completion. Separate the deterministic tasks from all other tasks to ensure that they receive enough processor resources. For example, if a control application acquires data at regular intervals and stores the data on disk, you might need to handle the timing and control of the data acquisition deterministically. However, storing the data on disk is inherently a nondeterministic task because file I/O operations have unpredictable response times that depend on the hardware and the availability of the hardware resource. Therefore, you want to use a Timed Loop for the data acquisition task and a While Loop for data logging. While Loops should be used for nondeterministic tasks or unbounded code (file I/O, network communication, and so on)."
And:
"When a time-critical loop is called, it is the only loop on the block diagram that executes." and "Timed Loops are single-threaded, so you can serialize your tasks using error wires to guarantee that the data nodes execute in the same order every time the loop is called. If you implement these tasks as two separated Timed Loops with equal priority, they still execute in the same thread, but you cannot guarantee the order of execution." and "You generally should use Timed Loops only when implementing a periodic task since any type of blocking introduces jitter."
So perhaps put that non-deterministic communication loop in a while loop rather than a timed structure, otherwise it could be block the other timed loops from running. I'm sure somebody far more experienced with LV RT would come up with a more rigorous recommendation - but hopefully this is a useful pointer.
Consultant Control Engineer
www-isc-ltd.com