NI-1744 Smart Cameras go offline using Vision Builder AI and LV2010?

Question

I've run into a perplexing and frustrating problem.
I have a set of 6 NI-1744 smart cameras used to monitor sample motion in an automated system
A central robot moves samples between 6 satellite chambers; there is a camera mounted above the entrance to each of the satellites; the sample carriers are drilled with up to 10 holes encoding their numbers in binary (0-1023). There is also an L-shaped registration mark drilled near the region for the number encoding. Matters are complicated somewhat in that the second generation of carriers has a slightly more complicated registration mark (which includes the simple L of the first generation).
What I would like to accomplish is this:
Most of the time, I just want to be able to see what's under each camera, illuminating from above using a ring light attached to the camera. Call that Inspection A.
I may need to capture an image without the ring light. Call that Inspection B
When a sample transfer is occurring, I need to run a more complicated inspection (Inspection C):
A light is turned on underneath the end of the sample transfer arm (the "fork"), backlighting the region where the hole pattern would be if a carrier is in fact present.
Acquire an image of the fork with the backlight on.
Check to see if the image shows the fork to be empty. If so, return that information and the inspection is complete.
If not, look for the (backlit) registration mark and establish a coordinate system for the holes. Look to see which holes are present, calculate the sample number, return the data and the inspection is complete
If not, look for the 2nd gen registration mark. If it is found, establish the coordinates as in step 4, calculate the number, return the data and complete.
If none of the registration patterns are found, turn on the ring light, capture an image and return it so the operator can intervene and enter the appropriate data.
I have successfully built the inspection, and it appears to work in the Vision Builder configuration interface, or if run from the Vision Builder inspection interface. If I attempt to *use* the inspection, accessing it via the VBAI interface in Labview, one or more of the cameras will hang after one or more inspections and stops responding to LabVIEW. It will take multiple reboots to get it back on line and visible to either VB or Labview.
Originally, I had thought to configure the camera to run inspections continously and select inspection A, B, or C based on a variable I could access using the Labview Shared Variable Engine.  Every time I tried that the camera in question would hang. My current sort-of-successful software uses the VBAI functions in LabVIEW to open Inspection A, B, or C on the camera, run that inspection a single time and return the results and inspection image. That still ends up with one or more cameras hanging, especially if I've added the case to handle the more complicated registration mark.
I think I may be running out of memory in the camera. I have occasionally received an out of memory error message when running the inspection in debug (step) mode in the VB configuration interface, at which point the camera will disconnect from the VB interface. When I look at the system monitor tab for a camera in MAX, I can see that it is showing me 11.5M free/124M total memory, and 72.3Mfree/118M total disk space. If I understand those numbers, that means that less than 10% of the memory is free when the inspection starts!
I'm only communicating with the cameras via the ethernet interface. Are there any software components I don't need to have installed on the 1744 (MAX lists about 12 different things installed!) in that case?
Alternatively, it appears there are VBAI functions available that might let me acquire an image and then process it via a local copy of Vision Builder running on the host PC, rather than in the camera. Is that so, and if so, would it be faster than running the inspection in the camera?
I've attached a zipfile with the inspection that appears to stall the camera, and samples of typical images captured of the empty fork and a sample holder.
Kevin Roche
Advisory Engineer/Scientist
Spintronics and Magnetoelectronics group
IBM Research Almaden
Attachments:
Carrier Read Problem.zip ‏657 KB

Answer 1

The good news: thanks to some offline assistance from Brad, my cameras are no longer crashing.
The bad news: my labview VI using VBAI functions still fails for one or more cameras after a while (typically >12 hours of monitoring). 
It's very odd: the cameras are still visible online.
I can ping them.
I can connect to them via VBAI, run inspections that way (either in configuration mode or inspection mode), and disconnect successfully.
My calling VI, however insists that it can't connect (usually with a -354700 error from VBAI Connect.vi, occasionally with a -354705).
I am using shift registers in the monitor loop of the main VI so that I can pass the VBAI session reference for each camera back to the subvis that actually load and run the desired inspection once a connection has been made. After an indeterminate time, one or more of those appears to go bad, but if I attempt to reset and connect to the camera again (I did include a control to let me close and reconnect if necessary from the monitor loop), I still get the above errors.
The only way to get them back online within LabVIEW this morning proved to be to stop the main VI and reset all the controls to default, reboot the cameras, and then restart the VI. At that point it was able to connect to all 6 again and has been running happily for over 8 hours.
I set up the persistent sessions using the shift registers because I have observed in the past continually opening and closing resources like that can lead to memory problems. It also dramatically reduced my cycle time when all I want to do is get the latest images from the cameras to under 2 seconds.
Any ideas? Is there some subtlety to disconnecting/reconnecting to the VB in the cameras via Labview that I'm missing?
I am working on a slightly smarter version of the inspection that can be allowed to run continously in a camera and simply return the appropriate result when asked, rather than having to start and stop different inspections as I do now, but that is not ready for deployment yet. It's the symptom of running just fine for X hours and then losing one or more cameras that is baffling me right now.
Kevin Roche
Advisory Engineer/Scientist
Spintronics and Magnetoelectronics group
IBM Research Almaden