The first binary file write operation for a new file takes progressively longer.

Question

I have an application in which I am acquiring analog data from multiple
PXI-6031E DAQ boards and then writing that data to FireWire hard disks
over an extended time period (14 days).  I am using a PXI-8145RT
controller, a PXI-8252 FireWire interface board and compatible FireWire
hard drive enclosures.  When I start acquiring data to an empty
hard disk, creating files on the fly as well as the actual file I/O
operations are both very quick.  As the number of files on the
hard drive increases, it begins to take considerably longer to complete
the first write to a new binary file.  After the first write,
subsequent writes of the same data size to that same file are very
fast.  It is only the first write operation to a new file that
takes progressively longer.  To clarify, it currently takes 1 to 2
milliseconds to complete the first binary write of a new file when the
hard drive is almost empty.  After writing 32, 150 MByte files,
the first binary write to file 33 takes about 5 seconds!  This
behavior is repeatable and continues to get worse as the number of
files increases.  I am using the FAT32 file system, required for
the Real-Time controller, and 80GB laptop hard drives.   The
system works flawlessly until asked to create a new file and write the
first set of binary data to that file.  I am forced to buffer lots
of data from the DAQ boards while the system hangs at this point. 
The requirements for this data acquisition system do not allow for a
single data file so I can not simply write to one large file.  
Any help or suggestions as to why I am seeing this behavior would be
greatly appreciated.

Answer 1

I am experiencing the same problem. Our program periodically monitors data and eventually save it for post-processing. While it's searching for suitable data, it creates one file for every channel (32 in total) and starts streaming data to these files. If it finds data is not suitable, it deletes the files and creates new ones.
On our lab, we tested the program on windows and then on RT and we did not find any problems.
Unfortunately when it was time to install the PXI on field (an electromechanic shovel on a copper mine) and test it, we've come to find that saving was taking to long and the program screwed up. Specifically when creating files (I.E. "New File" function). It could take 5 or more seconds to create a single file.
As you can see, field startup failed and we will have to modify our programs to workaround this problem and return next week to try again, with the additional time and cost involved. Not to talk about the bad image we are giving to our costumer.
I really like labview, but I am particularly upset beacuse of this problem. LV RT is supposed to run as if it was LV win32, with the obvious and expected differences, but a developer can not expect things like this to happen. I remember a few months ago I had another problem: on RT Time/Date function gives a wrong value as your program runs, when using timed loops. Can you expect something like that when evaluating your development platform? Fortunately, we found the problem before giving the system to our costumer and there was a relatively easy workaround. Unfortunately, now we had to hit the wall to find the problem.
On this particular problem I also found that it gets worse when there are more files on the directory. Create a new dir every N hours? I really think that's not a solution. I would not expect this answer from NI.
I would really appreciate someone from NI to give us a technical explanation about why this problem happens and not just "trial and error" "solutions".
By the way, we are using a PXI RT controller with the solid-state drive option.
Thank you.
Daniel R.
Message Edited by Daniel_Chile on 06-29-2006 03:05 PM