RT Remote Front Panel issues.

Question

I'll start with my setup:
2 PXI-1042 chassis connected via fiber-optic using NI PXI-8336 (MXI-4) cards.
Chassis 1:  PXI-8108 embedded controller running LabVIEW Real-time OS (2010), PXI-6528 DIO, PXI-8336 MXI-4.
Chassis 2:  MXI-4 in slot 1, 4 PXI-8334 DMM, PXI-6528 DIO, PXI-6281 M-series DAQ.
The real-time executable monitors analog and digital inputs and DMM measurements, displays the values using numeric and boolean indicators, generates alarms and allows users to acknowledge alarms via a Remote Front Panel.  The code has a custom error handler that traps errors and stores them in a global array.  This allows the code to display them on a console and allows the user to acknowledge and clear errors, and scroll through multiple error messages.  We used property nodes sparingly.
The system is monitored by two laptops with the correct run-time engine installed (both laptops have the LabVIEW development suite).  Under normal conditions, one laptop keeps control of the Remote Front Panel at all times.
The problem:
The system was running flawlessly for 3 months of continuous use.  Then, for no apparent reason, the indicators that display measurements from "Chassis 2" (specifically the DAQ) stopped updating. 
If the user refreshed the browser, the Remote Front Panel client would reconnect to the RT Web Server and get updated values, but they were always static.  Essentially, refreshing the browser took a "snapshot" of the data, but the indicators did not update continuously as before.  At the same time, DIO inputs to "Chassis 1" responded appropriately to changes in state. 
Refreshing the browser on one laptop would update the values for that laptop only, and the other laptop could be refreshed independently and have different values displayed, but both laptop's indicators were static.  During this time, the DMM measured inputs were not changing and the "Chassis 2" DIO inputs did not change state (i.e., we could not perturb the inputs), so we could not determine if those indications were affected.
Also, the error console did not respond to user input until the browser was refreshed and the client reconnected to the target.
We know the RT application is running on the target because there is code that monitors the status of the loops by changing state every loop iteration and displays the status to the user.
Our short-term solution was to reboot the RT controller by powering down both chassis, starting "Chassis 2", then starting "Chassis 1".  It's been running for a few days now without incident.
Since the data displayed after refresh was accurate based on comparison to gages in the system, we do not suspect a measurement problem.
Possible culprits:
1.  The RT Web Server has malfunctioned in some way (memory problems, synchronization, etc.)
2.  The MXI-4 link.
3.  Cosmic rays?  Sunspots?  Nearby radar stations?  Neighboring HAM radio operators (with massive amplifiers)?
If anyone has any ideas, they are welcome.  We are concerned that this will be an ongoing problem, if only every few months or so.

Answer 1

Could you clarify a bit on how the PXI-6528 was acting while you were seeing these problems? Does this card have values display on the Remote Front Panel, and if so were they still updating properly? I believe you were saying that, but I wanted to be sure because that makes a big difference. 
If everything became static, including the DIO card, at the same time, then I'd say see if your browser updated, or something in your network settings was changed without you noticing. Perhaps an update ran and reset some of your firewall settings, or an Internet Explorer update changed something that required a reboot. 
If just the outputs of chassis 2 were the problem, and the DIO card values on your remote front panel still worked well, then it is most likely a problem with your MXI-4 cards or your fiber optic cable that cause the change in behavior. 
Miles G.
National Instruments
Applications Engineer