Periodic spectra storing in TDMS

Question

I need to periodically store spectra from 4 voltages and 4
currents, each  spectra signal has 100 spectral lines. This means 800 spectral
lines is one record which is created each aggregation interval (200ms up to 10
min – user selectable). For long time I use binary data format for this and it
works fine and fast. Now I need to optionally store data into TDMS data format
for my customer who want to process them in DIAdem.
I found only one example for periodical storing of spectra
signal: Express
write data (time and frequency domain).vi. I tried to create data
file with approximatelly 500 records. The data file is about 2MB. But to show
this file in LabVIEW TDMS File Viewer takes tens of second having CPU
utilisation on 100%!!! If 
I load this 2MB file into DIAdem it makes also significantly longer
responses in comparison with example files!!!
I am
looking for efective way how to proceed for this type of data with TDMS format.
Consequently I will need to read data in LabVIEW, but the reader must be fast.
I have few typical tasks for reader:
To read time sequence of
selected harmonic on selected signal for the whole measurement
To read one record (all
harmonics) from one signal
Please
help.
Bilik

Answer 1

Hi Bilik,
One reason you are getting poor performance in the TDMS code you posted is that you are writing one data point at a time to each channel.  Since each buffer write in TDMS is prefaced with a binary buffer header describing its contents, you are creating a binary header for every data point you are storing.  The LabVIEW 2009 Beta, which is already live, contains TDMS VIs with an updated option to create a single binary header for the entire file (and forego the option of changing anything during the acquisition)-- with this option you would not encounter this problem.
But you can sidestep this problem with already shipping LabVIEW versions by invoking a special channel-level property set to automatically buffer channel writes up to a certain minimum buffer size, then flush that minimum buffer to disk automatically.  This avoids excessive binary header writes without changing your code inside the acquisition loop, though it does require you to add the channel property sets outside (left of) the acquisition loop.  Just assign the "NI_MinimumBufferSize" property to something like 500 or 1000 or 2000 values on each of the channels.  This also avoids the need to defrag the TDMS file afterwards, since you aren't creating it with serious fragmentation to start with.  However, I did notice after adding all this to your code that the write performance of your binary file is still much better than TDMS, and this I don't understand.
Perhaps Herbert can take a look at the edits I made and suggest further improvements.
Brad Turpin
DIAdem Product Support Engineer
National Instruments
Attachments:
TimeFreqDomain-TDMS.vi ‏33 KB
TimeFreqDomain-BIN.vi ‏27 KB