TDMS write buffer performance

Question

I am writing log data to a TDMS file.  We log one sample for each of 16 channels at the beginning and end of each step in a test profile and every minute in between.
The profile has 640 steps, most shorter than one minute.  This results in approximately 400,000 writes to the TDMS file with only one data point per channel.  As a result the file is very fragmented and takes a long time to open or defragment.
In this post Brad Turpin mentions a fix that works well but greatly diminished the TDMS write performance.
http://forums.ni.com/ni/board/message?board.id=170&message.id=403179&query.id=7209265#M403179
I also found that it takes about 40 seconds to set the NI_MinimumBufferSize attribute on 10,240 channels. (640 groups * 16 channels)
I did test this and it works very well but it took hours to generate a file of dummy data using this method.  Generating the dummy data file with the same number of writes but not using the buffer size attribute took seconds. 
In this post Brad also mentioned that LV 2009 contains TDMS VIs with an updated option to create a single binary header for the entire file.
I have not been able to find any more references to this nor have I found the attribute to set this functionality.
Does anybody know how to set this attribute or have any suggestions on how to better deal with my file structure?
Thanks,
Dave 

Answer 1

Are you writing one value per channel for all 16 channels with a single call to TDMS Write, or are you calling TDMS Write 16 times for that? Consolidating this into one call to TDMS Write should improve performamce and fragmentation quite a bit. In addition to that, you could set NI_MinimumBufferSize to the maximum number of values a channel in a step can have. After each step you could call TDMS Flush in order for these buffers to be flushed to disk. That should further reduce fragmentation.
The feature Brad mentioned is used in 2009 automatically, you don't need to enable it. Unfortunately, that won't do much for your application, because you create new channels for every step, which results in a new binary header for every step. The 2009 improvements would only kick in if you would use the same channels all the way through.
We are currently working on some improvements to the TDMS API that will help making your use case a lot more efficient. These are not yet available to customers though, so I'll describe some ways of working around the issue. It's not going to be pretty, but it'll work.
1) The TDM Streaming API is built for high performance, but even more than that, it is built so whatever you do with it, it will always create a valid TDMS file. These safety measures come at a cost, especially if you have a large number of channels and/or properties versus a rather small number of values per channel. In order to better address use cases like that for the time being, we have published a set of VIs a while ago, which will write TDMS files based on LabVIEW File I/O functions. This API does a lot less data processing in the background than the built-in API and therefore is a lot more efficient tackling use cases with thousands of channels.
2) Another way of improving performance during your test steps is to push some of the tasks that cost a lot of performance out into a post processing step. You can merge multiple TDMS files by concatenating them on a file system level. A possible workaround would be to write one TDMS file per step or maybe one every 10 or 100 steps and merge them after the test is done. An example VI for concatenating TDMS files can be found here. 
Hope that helps,
Herbert
Message Edited by Herbert Engels on 05-12-2010 11:33 AM