DAQmx TDMS streaming

When streaming data to a TDMS file in LabVIEW I have a DAQmx task that uses reference trigger.  Is it possible to read the file during the acquisition?  It appears that the code is not writing to the disk until the acquisition is complete, by then it's too late for the application I have in mind.
I tried to open the file, set the buffer size (smaller), and close the file, but that resulted in the file not even getting created (no data even after the acquisition had completed).
Any suggestions would be greatly appreciated.
David Dupont

Hi David,
Let's simplify the problem by ignoring TDMS for now. When you do a reference triggered acquisition with DAQmx, when does the data become available to be read from the DAQmx task? The answer to that question depends on the type of device.
I wrote a quick test VI (attached) to demonstrate the difference. It performs a reference triggered AI acquisition, acquiring 5 seconds of pretrigger data and 20 seconds of posttrigger data (at 100 S/s, so 500 pretrigger samples and 2000 posttrigger samples). A counter/timer is used to generate the trigger 7 seconds* after the start of the acquisition. During the acquisition, the "available samples per channel" and "total samples per channel acquired" properties are queried at regular intervals, as well as the "counter output pulse done" property indicating where the trigger was generated.
Here is the behavior of an M Series board: none of the samples are available to read until the entire acquisition is complete.
On M Series, until the acquisition is complete, AvailSampPerChan returns 0. This happens because the beginning of the posttrigger samples is determined by counting 2000 samples back from the last sample, and the beginning of the pretrigger samples is determined by counting 500 samples back from the beginning of the posttrigger samples. (As I mentioned
previously, a workaround is to use the RelativeTo and Offset properties to make the samples available sooner, but then you have a different problem: finding the trigger.)
X Series devices (as well as SC Express and NI 6535/6/7 devices, as far as I know) display a significant improvement: the pretrigger samples are available as soon as the trigger is detected, and the posttrigger samples are available as soon as they are acquired.
On X Series, AvailSampPerChan jumps from 0 to 500 when the trigger is detected, and then it increments continuously. Note that it is still impossible to read any pretrigger samples before the trigger is detected, because doing so would require predicting when the trigger will occur, before it occurs.
When you enable TDMS logging, DAQmx can't write samples to the TDMS file before they are available for reading, so reference triggering still limits when you can read back the data from the TDMS file.
What type of device are you using? Is it a USB device? USB also affects when data is available for reading, because data is transferred in larger chunks to reduce communication overhead.
Brad
* I didn't use a start trigger to synchronize the counter with the AI task, so the 7 second delay is not exact.
Brad Keryan
NI R&D
Attachments:
Ref Trig Data Arrival.vi ‏37 KB

Similar Messages

  • Design Suggestions for Multiple DaqMX Task Streaming App?

    I'm working on a LabVIEW application in which I'm streaming high-speed data to disk from multiple PXI devices simultaneously.  Each device has its own DaqMX task, and all tasks stream to the same file.  The PXI device configuration (which devices are in the chassis, which slots they're in, and which channels to read from each device) is determined at runtime.
    Does anyone have a suggestion for a design model for this?  To make matters worse, I'd like to be able to specify a channel to monitor its data during the streaming.  I'm thinking the Producer-Consumer model is the basic approach, and I'm at the point where I have an array of DaqMX tasks, one for each device.  I could probably extend that array to be an array of clusters containing:
       1) DAQ Command (e.g. Initialize, Start, Stop, Acquire, etc)
       1) Task ID
       2) Control reference to 2-D array (where each DaqMX read can be stored)
       3) Array of channel names (to allow selection of channel to be monitored
    This could be passed as notifier data, to a data collection subVI, but the part I'm struggling with is finding the best way to run X number of tasks in parallel, where X is not known until runtime.
    Any suggestions would be appreciated.

    Thanks for the input.  With regards to the file format, the decision has been made by my superiors not to use TDMS - unfortunate, but NI hasn't provided the information to write a MatLab file reader, which is a requirement.  So, I've created a custom file format tailored to the needs of my application, but generic enough to be used for other apps. With it, I've been able to stream 8 channels at 800KHz (4 channels each from 2 PXI-6120s) without breaking a sweat.  However, the performance varies greatly depending on which slots the cards are in (but that's a whole different discussion - see the PXI forum for that one).  Once NI solves that one, I'll feel a lot more comfortable.
    I have already made reentrant subVIs that can perform a specific DAQ task.  The problem with a for loop is that the VI sits and wait for a start trigger, then acquires the streaming data.  I can't start the next VI because I'm in the first one.  I thought about creating a data collection VI, and this VI would start up to 6 other VIs in parallel, based on how many and which cards were present.  It's a bit messy, because each slot can contain one of two devices, so I'd need to check which type it was before calling it.  I'm thinking I'll have to create the task list and the references to the data in the main GUI loop, and then pass this using a notifier or queue to the data collection loop. 

  • TDMS streaming problems

    Hi forum,
    i have problems streaming a lot of data to a file. I am using  write to measurment file with tdms in the while loop but my pc is to slow to do this streaming (100Ks/s for 10s). 
    For optimizing i was thinking to open a tdms file outside the while loop stream it to file in the recording state and then read it to another file in the logging state, clear it and i could stream again.
    But can i delate data from the tdms format, or is there something easier to do?
    Thanks and best
    Martin

    Where is the data coming from?
    You definately should be opening the file before your loop and closing it after your loop.  You can then write as much as you want inside of the loop.  Nothing extra should be needed.
    But if this data is coming from a DAQmx task, use the DAQmx Configure Logging and it will stream the data to disk for you.  You just have to tell it where to put the file.
    There are only two ways to tell somebody thanks: Kudos and Marked Solutions
    Unofficial Forum Rules and Guidelines

  • DaqMX & TDMS Logging - No Datestamp and dt until Close

    In our application, we are continuously logging data to a TDMS file using the "DaqMX Configure Logging (TDMS).vi". However, there are times when we need to look at the TDMS data while the application is still running (i.e. when a failure is found). What I've noticed is that the initial timestamp and dt are missing from the TDMS file until the "Stop and Clear Tasks.vi" is called, which in our application only occurs when the application is being terminated.
    In other words, when I go to look at the TDMS data while the app is running, the TDMS File Viewer shows the time starting in year 1903 and the wrong sample width until after the application exits, and then it shows the correct time and sample time. How can this be fixed so that the initial timestamp appears while the app is running?

    Duplicate:
    http://forums.ni.com/t5/LabVIEW/DaqMX-Access-TDMS-dt-and-timestamp-Before-quot-Stop-Task-quot-is/m-p...
    Unofficial Forum Rules and Guidelines - Hooovahh - LabVIEW Overlord
    If 10 out of 10 experts in any field say something is bad, you should probably take their opinion seriously.

  • DAQmx configure logging (TDMS) and counter

    Hi,
    I'm trying to stream analog data jointly with counter data to one TDMS file with use of DAQmx configure logging vi. This is quite new but powerful function but I can't manage with it successfully. I'm able to acquire analog data only and display no. of counts simultaneously (attached diagram). I have no idea how attach counter data to TDMS stream as separate channel group.
    It is important for me to stream data with this VI bacause I'm sampling analog data with 95kHz per channel (PCI-6143 board).
     Could you post any ideas how to resolve this problem?
    Best regards
    Mark
    Attachments:
    acq_analog_and_contr.PNG ‏44 KB

    Eric, hear is short info about DAQmx configure logging http://zone.ni.com/devzone/cda/tut/p/id/9574.
    Yes I thought about producer/consumer structure, It is first alternative way.
    There is also another very simple way -  connecting encoder to analog input and acquire together with analog signals. Of course It will be neccasary to process sigal from encoder after recording. I'm also concerning such solution
    Ok I have to sleep with this problem  
    Best regards
    Mark

  • TDMS DAQmx decode

    I have some DAQmx measurement data in TDMS format, and would like to analyze this data in Matlab or custom python.  I have read and understand the TDMS File Format Internal Structure documentation, but the documentation claims that "This article does not describe how to decode DAQmx data.  If you need to read a TDMS file with software that implements native support for TDMS (without using any components provided by National Instruments), you will not be able to interpret this data."
    I have tried several online solutions: TDMS Reader and ConvertTDMS(v9) at Matlab File Exchange, both of which failed due to interleaved data formatting or presence of RawDAQmx data.  I have also tried (and might still try to debug) a python version (pyTDMS), which might be failing due to poor file format documentation.
    I am next going to try the NI solution: the DLL + Matlab.  I haven't gotten into attempting an implementation yet, but from the looks of it, it's going to be a hack.  Particularly because I now have to install multiple Windows SDKs.
    I have two questions:
    1) Am I missing something?  I realize NI needs to make a profit from their product, but for the documentation to be missing key information and to have to install Microsoft SDKs is inexcusable.  Here in the real world, NI, we have mixed shops and several vendors provide the tools that we need.  Your website claims TDMS works in Matlab, but this is obviously not the case, at least not easily.  And Excel compatibility is great, but not when you have the problems of large data sets that TDMS is supposedly designed for.  NI, I've tended to avoid your products in the past because I dislike graphical programming, but you seem to be specifically preventing interoperability which might give me a new reason.
    2) What references (if any) are available regarding this "decoding" of DAQmx data?  What specifically is different if NI code isn't used?

    I wanted to hopefully explain some of the background information to hopefully answer some of the specific questions that you have.
    Please understand that NI is in no way intending to keep a closed system here. TDMS strives to be an open file format with interoperability with several environments. In that mission, NI provides several APIs to read back TDMS files: .Net API, Excel importer, C API, CVI API, and even a Matlab plugin. These provides APIs provide the best user experience as they are designed around TDMS (eg. functions refer to "groups", "channels", "properties", etc).
    It sounds like you had an issue with the Excel add-in with large data sets; I knew that this was an issue in Excel 2003 and previous versions as Excel did not support large data sets. I thought with Excel 2007, we update the importer to work well with Excels new capabilities. If not, we should definitely look into that.
    The link that the first reply mentions should yield the Matlab API that works with DAQmx scaled data. If you have further problems with it, please make sure to let us know.
    I wasn't sure what your concerns surrounding Windows APIs are as I'm not aware of any of our products installing a Windows API. Could you please elaborate if its a particular concern so that we can look into that?
    The reason why I mention all of these supported APIs is not to push them because they are NI-provided solutions. I mention them because I believe that they will make your application the most successful.
    We document our raw binary layout, but this should only be used if there's not an easier way to get your job done.
    All that being said, to answer your second question, the raw data in a DAQmx TDMS file simply consists of data in the same exact format that you get if you were to call DAQmx Read Raw. The output is a binary blob of interleaved channel data from the ADC without calibration or scaling. The process of deinterleaving that data and applying all of the necessary information with the exact numerical precision is fairly complex and error-prone if left to every user. In many ways, thi is why we created the built-in TDMS logging feature into DAQmx - so that users could stream with the best performance possible without having to implement difficult algorithms. DAQmx does not provide documentation for its raw data format.
    I do not recommend trying to implement that code yourself if we have a solution that integrates well and does the heavy lifting for you. If you really want to look into some of the details, we can aid you some in this effort.
    Sorry for the long post, but I wanted to make sure you understand that there was no ill will from NI in our approach here. I hope your application is successful, and please let us know if you have any follow-up questions.
    Thanks,
    Andy McRorie
    NI R&D

  • TDMS Direct Integration in NI-DAQmx Logging but where is timestamp!

    Hi
    Wonder if anyone could advise please
    I am quite impressed with Configure Logging.vi on Daqmx pallette streaming straigth to hardwire. I tried and it works great. However, data is missing time channel/ time stamps for individual points. Is there additional setting to achieve this?

    Hi,
    Thanks for answering!
    Yeah, that`s a nice idea, but it is, like you said, not exactly what i`m searching for...
    That`s my situation:
    I have four analog channels (voltage-measurement) and i need to store the raw data in tdms format. So it would be nice to use the DAQmx to TDMS direct logging. But there is one Problem: I also have to log the sum of the four voltages and I want to log it to the same tdms-file as new channel.
    If it is not possible to get the tdms-file reference out of the DAQ-Task, I have to use the traditional way (not using the direct logging) by reading the data and log it manually.
    lg Reinhard

  • TDMS digital data

    Hello,
    we are developing datalogging system for our customer and we need to save analog and digital data to TDMS file. There is probably no problem with analog data streaming. But on the other side writing digital data inside TDMS file looks like a problem... So we tried to use background streaming (DAQmx Logging) where size of file is acceptable (8xDI(1B) x 512samples + TDMS header). But we need to log analog and digital data with timestamp channel so we can not use DAQmx background streaming instead we must program our application. When we used TDMS write for digital waveform, digital data result in TDMS viewer and DIAdem is OK but file size is (8x1B for each digital channel). We also look at TDMS by DAQmx logging where internal file structure (attached file  tdms_tsdata.png) contains parameter NI_DigitalLine which is bit in data byte index (i think). Another parameter is NI_DataType = 5 (UInt8). From these two params I deduced some ideas and do some tests with unsatysfying results. Only one of tested ways was to write direct UInt8 of DI (entire port) to ensure file size ... 8xDI=1B x samples. However those TDMS can not be correctly read by TDMS Viewer as structure of digital inputs but only as number.
    Is there any chance to write manually and effectively digital data in TDMS file (like DAQmx logging)?
    Thank you in advance for any hint.
    Michal.
    Attachments:
    tdms_tsdata.png ‏40 KB

    Hi kel072,
    You are right, you can not write just as one  byte and expect Diadem, or TDMS file viewer to interpret directly as 8 lines boolean data.
    This is an expected behavior because there is no information about the number of lines.
    Therefore, I have two suggestions:
    1. You can build your custom TDMS file viewer (pack your data as u8 bit, transfer it into the TDMS file and later build your own VI that will read from TDMS, reconstruct the data in boolean 1D array and display the data). In this scenario, Diadem will still not be able to interpret them as 1 bit data.
    2. Write your data to a binary file in the format you specify. Then you can read it directly in LV or any other programming language that supports that format, or you can build a data plugin for Diadem that will allow it to interpret the data in the particular way you want ( and use it for your application). This method requires the most programming but gives you the most flexibility (packing data, display in Diadem, display in LabVIEW etc).
    Check the links bellow for more information on the data plugin:
    1. http://www.ni.com/white-paper/13803/en/
    2. http://www.ni.com/example/29822/en/

  • TDMS read error-getd​atatype

    Brief background on my project.  I have a PXI system setup to read signals from DSA devices, S-Series, and an Encoder.  I am using matlab to make function calls to the NI Libraries in DAQmx, and nilibddc.h.  I collect data and stream data to disk with the ability to view data during acquisition.  Acquisition occurs for nearly a minute over 37 channels at 100ksamp/sec.  So the .TDMS files are fairly large.  I synchronize my systems by a sort of hybrid master slave device configuration.  I use automatic synchronization across similar device types, and to synchronize different device types i use the sample clock and start trigger from a DSA device to sync my other devices.  Since I use automatic synchronization the .TDMS files can have anywhere between 1 channel or 20 channels in one file.  This makes for a very large file and Microsoft Excel Importer doesn't cut it due to the limitation on size and time it takes to load.  So I am using the nilibddc.dll to read the data from the files which seems to be pretty smooth so far. I have not attempted to read a file with a minute worth of data but worse case I can read chunks of the data file at a time so I am not terribly worried about that.  My problem occurs when I read data from my .tdms file containing my encoder data.
    The part where it gets hung up is my function calling for DDC_GetDataType.  Now when I use my code to read data from analog .tdms files it works smoothly and does everything I want it to.  The error I get for my counter signals is an unexpected error (-6204.)  Now the interesting thing is when I use labview 2009 on another computer, i can read the files with no problem.  I am guessing my labview 2009 uses different libraries though.  For example when I read the file in labview, i go to the functions pallete, select file i/o, tdms streaming, and use either read .tdms or tdms viewer function.  When I am on my primary computer, I am using a call to the DIAdem Connectivity Library I downloaded from NI. The microsoft excel importer seems to also be hung up at the same location (getdatatype).  I found this when inspecting the log file it creates when I try to read it.  Any help would be appreciated.  I have attached the error log I get from microsoft .tdms importer.  I also attached a sample .tdms file with encoder position data in it.  If trying to open outside of labview, don't be fooled if it tells you there is no data available.  It lies, using labview 2009 you can read all 500ksamps without a problem.  Using TDMS viewer or read tdms atleast.  Thanks.
    Solved!
    Go to Solution.
    Attachments:
    Testlog4.tdms.log ‏5 KB
    TDMS Encoder.zip ‏48 KB

    I went ahead and used the tdms_ebd.dll that you attached and tried it out with my code.  Unfortunately I am still getting hung up at the same location on TDMS files with CI data stored in them.  The getdatatype functions returns an unexpected error.  If I try to bypass this in my code by manually setting the datatype, it then runs into the issue where it acts as if there is no data in the file.  I believe the functions asking for the length of the data is where it returns a 0. Now I did download the TDMS C DLL from the link to NI's website.  This is the one I am currently using with the tdms_ebd.dll that you attached in your response. 
    I am trying to read the information into matlab.  I have code written up that is very similar to what most people have done, and I believe might even be what NI has released as a starting point for TDMS reading in matlab. One example that is closely related to my code is found here. Import into Matlab. My code is not as heavily formatted and I would have to go through the same process in order to release my code.  My code is basically the code though that he refers to earlier in his message found in the NI devzone. One thing I wanted to note is this though, depending on where the TDMS C DLL is downloaded, the C formatting for the functions differs slightly.  For example I have found that there are header files with a _m added to the file.  I assume these files were made specifically for usage with matlab, since they are a lot easier to work with then the actual header files.  I do have both types and have tried both types and still no luck. 
    Let me know if you need any more information, but as of right now, the update seems to not have fixed my problem.  Thanks.

  • Interleave​d data saving and the TDMS Viewer VI.

    Hello,
    I'm using LabVIEW 2009 Service Pack 1 ( ver 9.0.1 32-bit ).
    I was trying to take a piece of data and save it in interleaved mode.
    The save operation seems to be completed correctly, but when the TDMS Viewer VI is launched it crashes, and I'm unable to browse the data.
    I'm attaching the example I used.
    What I want to know is:
    1) am I using the write properly? What I'm trying to acomplish is to push down an array if I16 to a TDMS write block, recieve 30 channels of data this way.
    2) I managed to use this VI and it crashes, on two PC's, one is a win7 32bit, the second one was an XP. But let me know if this runs properly for you.
    Maciej
    Attachments:
    InterleavedDataErr.zip ‏16 KB

    I have a buffer that can accumulate up to 65536 I16 values.
    In that buffer I get for example 6x 10000 values.
    Every "chunk" of the 10000 values is my interleaved data. One point is one channel.
    My data is shaped like in the opening post:
    http://forums.ni.com/t5/LabVIEW/TDMS-Streaming-in-​interleaved-mode/m-p/1189201#M513943
    Since I'm running a windows machine, sometimes I'll get 5x 10000 values, sometimes it will be 3x , sometimes 6x during one buffer swipe.
    The data rate and the channel length is variable.
    But let's talk about one case:
    10k channels and 10 kHz rate of I16 vals.
    That gives me roughly 200 MB/s of data that I need to get out of the buffer.
    What I want to do on my PC is take that data, and save it interleaved to a TDMS file, while being able to safely stay above the 200 MB/s.
    I would be intersted in aiming for 300 MB/s so that if we push our card further down the line, I'm not stuck with the streaming bandwitch.
    I have noticed that unfortunately if you use an ammount of channles that is 10000 the writing performance drops dramatically.
    What I figured out is that I'm not conducting 1 write on 10000 channels on a small ammount of data. But rather than that a large accumulated amount of data (like 8000 x 10000 of I16)
    And I split the save of 10000 channels into 10x 1000 channels save.
    First channs from 0-999 , than 1000-1999 etc.
    All is done in interleaved mode.
    This is Very Very fast indeed. I got rates of 360 MB/s. That's smoking fast!
    Now the trick is... to handle the appropriate data shaping in the RAM, from my buffer, to the large shaped RAM chunk, that will be cut in to pieces for Streaming. LV is designed for data flow and copies data a lot, hence I get easlity in the out of memory problem, I'm going to have to use some ticks like storing the data structure via a VI Server call, or perhaps queues don't know yet. But I'm going to do some reading on that and it seems that this should be achieavable.
    Thank you both for your input,
    I will post if I get stuck.
    Maciej

  • TDMS file error occurs after several hours

    I have a compiled LabVIEW 2010 SP1 executable running on a Windows 7 PXI computer that is occasionally producing the following error message:
    "LabVIEW: LabVIEW could not load the TDMS file component.
    The top-level VI <my VI's name> was stopped at unknown on the block diagram of  <my VI's name>"
    This program is interfacing with equipment under test over a number of PXI cards and ethernet connections. It alters inputs to the system based on script commands, while receiving and displaying outputs. During testing, state changes are sent to a TDMS file. The system can run for hours but will usually produce the above error message... eventually.
    The error seems to indicate that a LabVIEW file is missing, but the TDMS files are being created, so I may not be understanding the meaning of the error. Does anyone know what the actual cause(s) of this error message are?

    The code writes to the TDMS file using the "TDMS Write" VI from the TDM Streaming Palette. These write VIs are located in 12 different sub VIs in a program loop within my main VI. Every 50ms they execute, service the I/O,  and write to the TDMS file if data in the particular sub VI has changed state. Some are writing floating point numbers, most are writing strings. There are about 750 total channels being written. I know that posting the code would be more usefull than describing it, but I think posting the source code would be frowned upon by my company. There are over a hundred custom VIs and external hardware connections that are required to effectively run this code anyway.
    Regarding timing, there doesn't seem to be a particular amount of time that has to pass before this issue occurs. I was given a screenshot that shows the error occured after 11 hours of running. I was also told that the same error had occured about 6+ hours into a test once. I can say that this error doesn't seem to occur quickly, as in within the first hour or so. It seems to require long duration tests.

  • Merge tdms file

    Hi,
    is there an easy way to merge a set of tdms-files (with same structure) to one file?
    For example:
    Measurement1.tdms
    Measurement2.tdms
    Measurement3.tdms
    to 
    Measurement.tdms

    Hi Andre,
    in what way do you want to "merge" the files?
    I think you could read the files out in Measurement Studio with the TDM Streaming Lib or the Diadem Connectivity Lib and save the data into one file.
    Let me know, if you have any questions.
    Best regards,
    Melanie

  • Tdm importer error when opening tdms file

    I am getting a error when trying to open a tdms file that had been opened by labview, written to, and then closed. The error is:
    USI encountered an exception:
    (175):Error while initializing interface
    I can sort of understand this error occuring when I have the file open in labview (actually I consider this to be a bug since there is no reason why another application should not be able to access a file in read only mode even if it is currently open by another application)
    but in my case I am definitely closing the tdms reference and it still will not import unless I stop the top level vi.

    It is a known bug that if you open a TDMS file with an option other
    than "read-only", no other process can get access to that file, even if
    it is just for reading. The issue is filed as CAR#  49794JWJ
    (status: fixed). This only applies if the file is still open, which you
    commented on, but which doesn't appear to be your actual problem.
    Apparently, you are using the Storage VIs to handle your TDMS files,
    otherwise you couldn't get a USI exception. Have you tried opening your
    file with the TDMS File Viewer on the TDM Streaming palette? If that
    doesn't work, please provide us with your TDMS file, and if possible,
    your VI. This would be a problem that we're not aware of at this point.
    Thank you,
    Herbert

  • How to group measurement data into different groups using TDMs

    Hi, frnds!
    I have designed a labview program for measuring the power current characteristics of a laser diode. I want to save the current and power arrays measured at different temperatures in different groups. I tried to implement this but I was not successful. The current and power measurements at different temperatures are being saved only in the first group and the rest are being empty! Please help me save the measurements in different temperature groups.
    I need this urgently!
    Thanks in advance!
    Kumar

    Perhaps a snippet of the code you are using would help. When you write to the TDMS file with the low level functions, you just need to specify the separate group names. Have you read over  the NI TDMS File Format, Introduction to LabVIEW TDM Streaming VIs, and Writing TDM and TDMS files articles? Have you tried any examples out of the Example Finder?
    As a basic example:
    Certified LabVIEW Developer

  • Error in open tdms file

    can some one help me why I get data type error when I want to open a tdms file by Read From Measurement File?I can open and view and analyse my file in diadem but I can't use them by labview!

    The Express VIs support only numeric channels, and some time stamp channels. The error you're seeing typically comes from files that contain string channels. You can double-check that by using the TDMS File Viewer from the TDM Streaming palette. The functions on the TDM Streaming palette (e.g TDMS Read) allow for easy access to particular channels, so you wouldn't have any issues with unsupported data types. You can check the "Write/Read Event Data" example VIs for more details on how to handle numeric and string channels in the same file.
    Hope that helps,
    Herbert

Maybe you are looking for