TDMS DAQmx decode

I have some DAQmx measurement data in TDMS format, and would like to analyze this data in Matlab or custom python. I have read and understand the TDMS File Format Internal Structure documentation, but the documentation claims that "This article does not describe how to decode DAQmx data. If you need to read a TDMS file with software that implements native support for TDMS (without using any components provided by National Instruments), you will not be able to interpret this data."
I have tried several online solutions: TDMS Reader and ConvertTDMS(v9) at Matlab File Exchange, both of which failed due to interleaved data formatting or presence of RawDAQmx data. I have also tried (and might still try to debug) a python version (pyTDMS), which might be failing due to poor file format documentation.
I am next going to try the NI solution: the DLL + Matlab. I haven't gotten into attempting an implementation yet, but from the looks of it, it's going to be a hack. Particularly because I now have to install multiple Windows SDKs.
I have two questions:
1) Am I missing something? I realize NI needs to make a profit from their product, but for the documentation to be missing key information and to have to install Microsoft SDKs is inexcusable. Here in the real world, NI, we have mixed shops and several vendors provide the tools that we need. Your website claims TDMS works in Matlab, but this is obviously not the case, at least not easily. And Excel compatibility is great, but not when you have the problems of large data sets that TDMS is supposedly designed for. NI, I've tended to avoid your products in the past because I dislike graphical programming, but you seem to be specifically preventing interoperability which might give me a new reason.
2) What references (if any) are available regarding this "decoding" of DAQmx data? What specifically is different if NI code isn't used?

I wanted to hopefully explain some of the background information to hopefully answer some of the specific questions that you have.
Please understand that NI is in no way intending to keep a closed system here. TDMS strives to be an open file format with interoperability with several environments. In that mission, NI provides several APIs to read back TDMS files: .Net API, Excel importer, C API, CVI API, and even a Matlab plugin. These provides APIs provide the best user experience as they are designed around TDMS (eg. functions refer to "groups", "channels", "properties", etc).
It sounds like you had an issue with the Excel add-in with large data sets; I knew that this was an issue in Excel 2003 and previous versions as Excel did not support large data sets. I thought with Excel 2007, we update the importer to work well with Excels new capabilities. If not, we should definitely look into that.
The link that the first reply mentions should yield the Matlab API that works with DAQmx scaled data. If you have further problems with it, please make sure to let us know.
I wasn't sure what your concerns surrounding Windows APIs are as I'm not aware of any of our products installing a Windows API. Could you please elaborate if its a particular concern so that we can look into that?
The reason why I mention all of these supported APIs is not to push them because they are NI-provided solutions. I mention them because I believe that they will make your application the most successful.
We document our raw binary layout, but this should only be used if there's not an easier way to get your job done.
All that being said, to answer your second question, the raw data in a DAQmx TDMS file simply consists of data in the same exact format that you get if you were to call DAQmx Read Raw. The output is a binary blob of interleaved channel data from the ADC without calibration or scaling. The process of deinterleaving that data and applying all of the necessary information with the exact numerical precision is fairly complex and error-prone if left to every user. In many ways, thi is why we created the built-in TDMS logging feature into DAQmx - so that users could stream with the best performance possible without having to implement difficult algorithms. DAQmx does not provide documentation for its raw data format.
I do not recommend trying to implement that code yourself if we have a solution that integrates well and does the heavy lifting for you. If you really want to look into some of the details, we can aid you some in this effort.
Sorry for the long post, but I wanted to make sure you understand that there was no ill will from NI in our approach here. I hope your application is successful, and please let us know if you have any follow-up questions.
Thanks,
Andy McRorie
NI R&D

Similar Messages

TDMS Timestamp Decoding - Help!!

I am in need of help with decoding the timestamp in TDMS files. Specifically in the properties section. The following (very vague documentation) provides limited details:
http://zone.ni.com/devzone/cda/tut/p/id/5696
Specifically looking for bit definition of the "tdstimestamp" type. As an example I extract the following hex values from a TDMS file:
44 00 00 00 EA D8 C5 AC 4F 9E 56 BC C2 2F F3 C3 00 00 00 00
I know that the first four bytes (44 00 00 00) signify the timestamp datatype, but how do I decode the remaining bytes. The approximate time of above is 11:55:41AM on March 4, 2008.
Thanks for any help!

Matt - Thanks, that works great!!
Dave, I see that the MATLAB pluggin was just updated to include MATLAB 2008b. We take data in our labratory using the TDMS format. When we distribute this to other researchers, they often use MATLAB (and of course Labview).
For about a year now, the Matlab pluggin has been out of date and would not work with the last 3 releases of MATLAB. Since it is a compiled C library, we're stuck and can not modify it. As the internal TDMS format is not changing and only low level matlab commands are required to read it, I believe that Matlab users have been looking for a simple Matlab function to just take a tdms file and make a similar .mat file (not a dll). This is what I have done (and thanks to Matt for the timestamp help).
The Excel TDMS add-in is very nice and is a good reason to use the TDMS format (in addition to its good support in VI libraries), but I often have to fend off requests to write large ASCII or flat binary files (in addition to our TDMS files) so that they can be read in MATLAB. I don't want to have the burden of making/distributing a LV executable that makes this interim file format and then creates a .mat file.

Parallel TDMS Logging

Good day all,
Just for a brief summary, I have a PCI-6280 M-series board using LV2010.
Currently, I have 4 Differential AI inputs created on one task, and a Linear Encoding Input using a counter on another task. As you can probably guess, I would like to know the different AI measuremetns at every position I am at (using the linear encoder). Since this is such a big task, I have decided to use the "DAQmx Configure Logging.vi" for logging into a TDMS file.
However, I noticed that both tasks could not be written or "streamed" at the same time. I read on this link that a way to do this is to concactenate the files together. However, I was wondering if this would affect how "synchronized" the position and measurements would be. Is there a better way to do this?
Thanks
Lester
Solved!
Go to Solution.

Hey Lester,
Since all the concatenation is done as post-processing it should not affect the synchronization of your data. Another forum you can check for information is here:http://forums.ni.com/t5/Multifunction-DAQ/Multiple-TDMS-DAQmx-streaming-to-the-same-TDMS-file/td-p/1.... Please let me know if you have any additional questions.
Regards,
Kevin
Product Support Engineer
National Instruments

DAQmxConfigureLogging

Hello,
We would like to use the DAQmxConfigureLogging library function to stream analog input data to disk. We will be using the DAQmx_Val_LogAndRead option to allow reading of acquired data.
The question we have is if it is possible to asynchronously write event markers into the TDMS files created after acquistion is started?
Thank you,

Hi jharris66,
Yes it is possible, but there are some some things to consider before you incorporate this into your code. The following forum post discusses that, http://forums.ni.com/t5/Multifunction-DAQ/Multiple-TDMS-DAQmx-streaming-to-the-same-TDMS-file/td-p/1... .
Bobby Breyer
Applications Engineer
National Instruments

I write digital port by 'DAQmx Configure Logging.vi' and receive TDMS file with 8 boolean channels. How can I write to 1 integer channel?

Hello!
I want to write 1 digital port from PXI-6536 with streaming to TDMS file.
I'm writing by 'DAQmx Configure Logging.vi' and become TDMS file with 8 boolean channels.
How can I write to 1integer channel?
Attachments:
1.JPG ‏27 KB

Hey Atrina,
The actual data stored on disk is just the raw data (that is, a byte per sample in your case). It's really just a matter of how that data is being represented in LabVIEW whenever you read back the TDMS file.
I'm not sure if there is a better way to do this, but here is a way to accomplish what you're wanting:
Read back the TDMS file as a digital waveform. Then there's a conversion function in LabVIEW called DWDT Digital to Binary. This function will convert that set of digital channels into the "port format" that you're wanting. I've attached an example of what I mean.
Note: When looking at this VI, there are a few things that the downgrade process did to the VI that I would not recommend for these TDMS files. It added a 1.0 constant on the TDMS Open function, and it set "disable buffering" on the TDMS Open function to false; you can get rid of both of those constants.
Message Edited by AndrewMc on 01-27-2010 11:21 AM
Thanks,
Andy McRorie
NI R&D
Attachments:
digitalconvert.vi ‏13 KB

DaqMX - Access TDMS dt and timestamp Before "Stop Task" is called

When using DaqMX to store sampled data to a TDMS file, how can I access the Start Time and dt? Is there a property Node, etc I can read from? The TDMS file does not include the datestamp and dt until AFTER the "Stop Task" is called (looks like another Labview bug), and I need to view it while the task is still running. See attached
Attachments:
DaqMX_TDMS_NoDatestamp_Example V2.vi ‏31 KB

I suppose I should clarify... So our application generates the TDMS file during functional test; the TDMS file is to be used for troubleshooting reasons only when a failure occurs. When we get a failure during functional test, the requirement is that the software pauses on failure. When this happens, we need to be able to open the TDMS file to look at various signals to see how they behaved during functional test to help us arrive at the Root Caue. The issue is that until the "DAQmx Close Task" VI is called, Labview does not store the To and DT information in the TDMS file (not sure why this is, appears to be a bug). Since we cannot terminate the functional test due to our Pause on Failure requirement, when we look at the TDMS file for debug purposes, it shows the timestamp starting in the year 1903, and it inferrs a DT of 1 second, rather than 10ms. This is very embarrassing to show to the customer.
It's obvious that Labview knows the DT and To, but only chooses to write it to the TDMS file when we Close the task. Any insight? TDMS files are made to be viewable while Labview is writing to them, but waiting until the end to store the Timing information is a Huge drawback to using them with DAQmx...

DaqMX & TDMS Logging - No Datestamp and dt until Close

In our application, we are continuously logging data to a TDMS file using the "DaqMX Configure Logging (TDMS).vi". However, there are times when we need to look at the TDMS data while the application is still running (i.e. when a failure is found). What I've noticed is that the initial timestamp and dt are missing from the TDMS file until the "Stop and Clear Tasks.vi" is called, which in our application only occurs when the application is being terminated.
In other words, when I go to look at the TDMS data while the app is running, the TDMS File Viewer shows the time starting in year 1903 and the wrong sample width until after the application exits, and then it shows the correct time and sample time. How can this be fixed so that the initial timestamp appears while the app is running?

Duplicate:
http://forums.ni.com/t5/LabVIEW/DaqMX-Access-TDMS-dt-and-timestamp-Before-quot-Stop-Task-quot-is/m-p...
Unofficial Forum Rules and Guidelines - Hooovahh - LabVIEW Overlord
If 10 out of 10 experts in any field say something is bad, you should probably take their opinion seriously.

DAQmx configure logging (TDMS) and counter

Hi,
I'm trying to stream analog data jointly with counter data to one TDMS file with use of DAQmx configure logging vi. This is quite new but powerful function but I can't manage with it successfully. I'm able to acquire analog data only and display no. of counts simultaneously (attached diagram). I have no idea how attach counter data to TDMS stream as separate channel group.
It is important for me to stream data with this VI bacause I'm sampling analog data with 95kHz per channel (PCI-6143 board).
Could you post any ideas how to resolve this problem?
Best regards
Mark
Attachments:
acq_analog_and_contr.PNG ‏44 KB

Eric, hear is short info about DAQmx configure logging http://zone.ni.com/devzone/cda/tut/p/id/9574.
Yes I thought about producer/consumer structure, It is first alternative way.
There is also another very simple way - connecting encoder to analog input and acquire together with analog signals. Of course It will be neccasary to process sigal from encoder after recording. I'm also concerning such solution
Ok I have to sleep with this problem
Best regards
Mark

Data collection was switched from an AI Config task writing to an hsdl file to synchronized DAQmx tasks logging to TDMS files. Why are different readings produced for the same test?

A software application was developed to collect and process readings from capacitance sensors and a tachometer in a running spin rig. The sensors were connected to an Aerogate Model HP-04 H1 Band Preamp connected to an NI PXI-6115. The sensors were read using AI Config and AI Start VIs. The data was saved to a file using hsdlConfig and hsdlFileWriter VIs. In order to add the capability of collecting synchronized data from two Eddy Current Position sensors in addition to the existing sensors, which will be connected to a BNC-2144 connected to an NI PXI-4495, the AI and HSDL VIs were replaced with DAQmx VIs logging to TDMS. When running identical tests, the new file format (TDMS) produces reads that are higher and inconsistent with the readings from the older file format (HSDL).
The main VIs are SpinLab 2.4 and SpinLab 3.8 in folders "SpinLab old format" and "Spinlab 3.8" respectfully. SpinLab 3.8 requires the Sound and Vibration suite to run correctly, but it is used after the part that is causing the problem. The problem is occuring during data collection in the Logger segment of code or during processing in the Reader/Converter segment of code. I could send the readings from the identical tests if they would be helpful, but the data takes up approximately 500 MB.
Attachments:
SpinLab 3.8.zip ‏1509 KB
SpinLab 2.4.zip ‏3753 KB
SpinLab Screenshots.doc ‏795 KB

First of all, how different is the data? You say that the reads are higher and inconsistent. How much higher? Is every point inconsistent, or is it just parts of your file? If it's just in parts of the file, does there seem to be a consistent pattern as to when the data is different?
Secondly, here are a couple things to try:
Currently, you are not calling DAQmx Stop Task outside of the loop; you're just calling DAQmx Clear Task. This means that if there were any errors that occured in the logging thread, you might not be getting them (as DAQmx Clear Task clears outstanding errors within the task). Add a DAQmx Stop Task before DAQmx Clear Task to make sure that you're not missing an error.
Try "Log and Read" mode. "Log and Read" is probably going to be fast enough for your application (as it's pretty fast), so you might just try it and see if you get any different result. All that you would need to do is change the enum to "Log and Read", then add a DAQmx Read in the loop (you can just use Raw format since you don't care about the output). I'd recommend that you read in even multiples of the sector size (normally 512) for optimal performance. For example, your rate is 1MHz, perhaps read in sizes of 122880 samples per channel (something like 1/8 of the buffer size rounded down to the nearest multiple of 4096). Note: This is a troubleshooting step to try and narrow down the problem.
Finally, how confident are you in the results from the previous HSDL test? Which readings make more sense? I look forward to hearing more detail about how the data is inconsistent (all data, how different, any patterns). As well, I'll be looking forward to hearing the result of test #2 above.
Thanks,
Andy McRorie
NI R&D

DAQmx TDMS streaming

When streaming data to a TDMS file in LabVIEW I have a DAQmx task that uses reference trigger. Is it possible to read the file during the acquisition? It appears that the code is not writing to the disk until the acquisition is complete, by then it's too late for the application I have in mind.
I tried to open the file, set the buffer size (smaller), and close the file, but that resulted in the file not even getting created (no data even after the acquisition had completed).
Any suggestions would be greatly appreciated.
David Dupont

Hi David,
Let's simplify the problem by ignoring TDMS for now. When you do a reference triggered acquisition with DAQmx, when does the data become available to be read from the DAQmx task? The answer to that question depends on the type of device.
I wrote a quick test VI (attached) to demonstrate the difference. It performs a reference triggered AI acquisition, acquiring 5 seconds of pretrigger data and 20 seconds of posttrigger data (at 100 S/s, so 500 pretrigger samples and 2000 posttrigger samples). A counter/timer is used to generate the trigger 7 seconds* after the start of the acquisition. During the acquisition, the "available samples per channel" and "total samples per channel acquired" properties are queried at regular intervals, as well as the "counter output pulse done" property indicating where the trigger was generated.
Here is the behavior of an M Series board: none of the samples are available to read until the entire acquisition is complete.
On M Series, until the acquisition is complete, AvailSampPerChan returns 0. This happens because the beginning of the posttrigger samples is determined by counting 2000 samples back from the last sample, and the beginning of the pretrigger samples is determined by counting 500 samples back from the beginning of the posttrigger samples. (As I mentioned
previously, a workaround is to use the RelativeTo and Offset properties to make the samples available sooner, but then you have a different problem: finding the trigger.)
X Series devices (as well as SC Express and NI 6535/6/7 devices, as far as I know) display a significant improvement: the pretrigger samples are available as soon as the trigger is detected, and the posttrigger samples are available as soon as they are acquired.
On X Series, AvailSampPerChan jumps from 0 to 500 when the trigger is detected, and then it increments continuously. Note that it is still impossible to read any pretrigger samples before the trigger is detected, because doing so would require predicting when the trigger will occur, before it occurs.
When you enable TDMS logging, DAQmx can't write samples to the TDMS file before they are available for reading, so reference triggering still limits when you can read back the data from the TDMS file.
What type of device are you using? Is it a USB device? USB also affects when data is available for reading, because data is transferred in larger chunks to reduce communication overhead.
Brad
* I didn't use a start trigger to synchronize the counter with the AI task, so the 7 second delay is not exact.
Brad Keryan
NI R&D
Attachments:
Ref Trig Data Arrival.vi ‏37 KB

DAQmx and TDMS not always working

Hello,
I am new to the DAQmx and TDMS vi's and am hoping someone will be able to help me. The attached code contains several case structures, however only one of them has the DAQmx and TDMS vi's running through them. My intent is to record the voltage output from two photodetectors as I scan the current of a diode laser controller. When I run this code it does not always record the values while I am scanning and when I quit the program the TDMS file viewer will show me no data. Sometimes, but rarely, it will show me some data. The stacked sequence within the case structure is the code that runs the laser controller and a couple of translational mounts. The DAQmx and TDMS vi's that should record the data are just next to that.
Thank you!
Alfredo
Solved!
Go to Solution.
Attachments:
LDC3714B_v9_test.vi ‏305 KB

I just ran the code about 5 times and it only recorded the data I was expecting once even though I changed nothing. I am attaching the code with the subvi's.
What I mean by "only one of them has the DAQmx and TDMS vi's running through them" is that the data is only recorded using these vi's once only one of the many case structures is activated. I'm simply pointing out that the vi's are not used anywhere else.
Also in the DAQmxRead vi, should I have Analog 2D DBL or Analog 1D Waveform?
I think the problem with the inconsistency in my code might be because I have a Stached Sequence Structure within the Case Structure where the vi's are.
Thank you.
Alfredo
Attachments:
Code.zip ‏252 KB

Extra empty groupName in TDMS file when using DAQmx Start New File & stop DAQmx while waiting for trigger?

I wrote a VI that listens to an external trigger coming from a PFI line and saves a user-defined number of pretrigger+posttrigger samples into disk, and then the DAQmx restarts to wait for the next trigger. I used the DAQmx Start New File.vi to change the filename of each of the next file. However, I found that for each of the resulting TDMS files, there is an additional group that is added in addition to the real data. For example, if the groupName is set to "task_voltage", then there is an additional groupName called "task_voltage #1".
I suspect that this is because the program preallocate diskspace using the old filename for the pretriggered samples, but when the trigger happens and the samples are ready to be written into disk, an actual new file is opened and saved. The reason I think this might be the case is because, say each file would be 50kB and I already have 1 file triggered and saved, before the second trigger comes in, I can see that the file size of the first file is twice as much as it should be (100kB), but the file size will return to noraml (50kB) after the second trigger happens and a second file is created.
Does anyone know if this is really the case? Is there a way to configure the task so that this doesn't happen?
Another question I have is that the program currently set up so that the VI will wait until a trigger happens to return the values. However, this means that if a trigger doesn't happen, the VI waits indefintely, and the "stop" button in the while loop is useless. I have to use the "Abort Execustion" button on the tool bar to stop the VI. Is there a way to interrupt and stop the DAQmx task even if the trigger doesn't happen?
Thanks in advance for any help!
Attachments:
pretrigger_loop_new_file.vi ‏29 KB

I don't believe there is a way to programmatically change/delete the group name in a TDMS file. What you can do however, is convert between TDMS and TDM using the VIs found in the Data Storage palette and then perform whatever modifications you need using the TDM utilities.
Applications/Systems/Test
National Instruments | AWR Group

TDMS Direct Integration in NI-DAQmx Logging but where is timestamp!

Hi
Wonder if anyone could advise please
I am quite impressed with Configure Logging.vi on Daqmx pallette streaming straigth to hardwire. I tried and it works great. However, data is missing time channel/ time stamps for individual points. Is there additional setting to achieve this?

Hi,
Thanks for answering!
Yeah, that`s a nice idea, but it is, like you said, not exactly what i`m searching for...
That`s my situation:
I have four analog channels (voltage-measurement) and i need to store the raw data in tdms format. So it would be nice to use the DAQmx to TDMS direct logging. But there is one Problem: I also have to log the sum of the four voltages and I want to log it to the same tdms-file as new channel.
If it is not possible to get the tdms-file reference out of the DAQ-Task, I have to use the traditional way (not using the direct logging) by reading the data and log it manually.
lg Reinhard

TDMS Direct Integration in NI-DAQmx Logging

Hi,
Is it possible to get the tdms file reference, when using the "TDMS Direct Integration in NI-DAQmx Logging" like that: http://www.ni.com/white-paper/9574/en/
I would need it to write manally data to an tdms-file while a DAQmx-task is logging the measured data to this file.
To open a second file reference is not possible, because it is blocked from the DAQ-task.
Best regards,
Reinhard

Hi,
Thanks for answering!
Yeah, that`s a nice idea, but it is, like you said, not exactly what i`m searching for...
That`s my situation:
I have four analog channels (voltage-measurement) and i need to store the raw data in tdms format. So it would be nice to use the DAQmx to TDMS direct logging. But there is one Problem: I also have to log the sum of the four voltages and I want to log it to the same tdms-file as new channel.
If it is not possible to get the tdms-file reference out of the DAQ-Task, I have to use the traditional way (not using the direct logging) by reading the data and log it manually.
lg Reinhard

TDMS Properties show polynom scale although DAQmx ist scaled linear

I aquire data with DAQmx and write the data to a TDMS-file . The input channel is scaled linear, as shown in the picture "Code Skalierung". In contrast to this linear scale the channel-Properties of the TDMS-file show a polynom scale with totally different values.
How can I scale my AI linear and the properties show the proper scale-values?
Solved!
Go to Solution.
Attachments:
Code Skalierung.png ‏117 KB
Dateimonitor.png ‏145 KB

In DAQmx, we have scaling optimizations where scales will get combined whenever possible. In your case, we have a polynomial scale for your given device (probably an M-Series or X-Series device) that will take the raw 2 bytes value for a sample and scale it to voltage. Then, your linear scale on top of that scale will actually get combined into that scale so that m gets multipled by the second coefficient and the b offsetting the "offset" coefficient. For example, in a x^2 + b x + c, the resulting optimized expression would be a x ^2 + m b x + c + b. The same kind of optimization happens with linear on linear scaling and in some other cases. For example, if there were 5 linear scales "in a row", these would resolve to a single linear scale representing all levels.
This has some key advantages: 1) It optimizes the performance of the operations as it only has to apply one scale, and 2) it helps some with precision in terms of floating point math, 3) in the case of TDMS logging, it saves some space in the file.
If you want to validate that this is what's happening:
1) Look at the data coming from DAQmx Read and compare it to the TDMS file.
2) Look at the channel property for the device scaling coefficients (I believe it's under something like Analog Input>>General Properties>>Data Transfer>>Device Scaling Coefficients in the Channel property node). Then do the above calculation.
3) Try changing your m and b to something like 2x and see the resulting coefficients accordingly modified.
Hope this helps.
Thanks,
Andy McRorie
NI R&D

TDMS DAQmx decode

Similar Messages

Maybe you are looking for