Read time of tdm file increases
Hello,
I recently recorded a lot of data in tdm format. I originally saved it in an excessively organised and complex way which resulted in large header files and was very slow. I changed it and most of the data is fine, except 6 files in the old format. Just trying to read them to get the data out and then convert it into an easier format is taking far too long. Each file has ~15000 channel groups, each with 3 channels (I won't do this again...). I have witten a small vi to test the reading speed. Excluding opening and closing of the file, if I just find a channel group by name, reading one channel from that group takes about 1 second. This would still take days to read it all but would be acceptable. The strange thing is, is that if I do this in a loop for successive channel groups which is clearly necessary, it takes longer and longer each iteration. This is behaviour that I have always seen when reading tdm files, but it hasn't caused a serious problem before. I can't find any information about why though, so I hope I'm just doing something stupid. I have attached the speed testing vi (Labview 2010). Running it for 5 iterations, the time taken to read the channels is
0.82304 s
1.58809 s
2.42514 s
3.56820 s
5.60632 s
The channels it is reading all have the same number of values, and it makes no difference if I start at a different channel group (by adding a number to i in the loop).
Does anyone have any explanation for this behaviour?
Thank you very much for your help, and I'm sorry if it's something tht has been asked before,
James
Solved!
Go to Solution.
Attachments:
ReadBadTDMSpeedTest.vi 44 KB
TDMS is an all binary format whereas TDM posses an XML header, otherwise they are very similar. Being a binary format you would expect it to read faster especially as you mentioned that the header was complex.
There are a number of un-closed references in your code and I wonder if this was partly the issue, although this wouldn't explain why converting to TDMS fixed the problem?
Beyond that, it would be nice to try this without using the express VIs to see if the problem still occurs.
Also, where you able to see if this was a memory issue? And if the read time continued to increase beyond what you posted?
Anyway, I am glad we found a solution to your problem.
Nick C.
Cardiff University
Similar Messages
-
How to read and create TDM Files in a Java Application?
Hi,
I want to read and write TDM-Files in a Java-Application.
I found the TDMHeaderWriter for Java, that can write TDM-Files, but the reading part is still missing.
Perhaps anyone knows an equivalent in Java to the TDM C DLL and the TDM C Header DLL?
Thanks VerenaCool
Can you please give me feedback on this project if you have finished it?
Matthias
Matthias Alleweldt
Project Engineer / Projektingenieur
Twigeater? -
How can I read in a TDMS file into Sound and Vibration Assistant
I have recorded a signal from 2 channels from a PXI 4496 DAQ card. I recorded this signal with the TDMS file format from a VI in Labview. I would like to import these signals into the Sound and Vibration Assistant for freq. analysis. In sound and vibration assistant there is an import logged tdms from signal express but I can not seem to import my original signal? How can I import this data?
Hello All,
I am trying to read TDMS file using LABVIEW. The TDMS file is created by me for reading data from one wire devices.
I have created simple vi. attached below which read data from TDMS file. I am able to read other tdms files, I have with this program but not which one created by me. I am not able to attach file with .tdms exentiont. So I have attached it with .xlsx extention. Can anyone please guide me where is the problem in TDMS file? Am I missing anything while creating it?
Thanks
Attachments:
EditTDMS (1).vi 14 KB
data.xlsx 17 KB -
Does anyone know how to read in a TDMS file into Matlab?
I am trying to read a TDMS file into Matlab. Are the two compatible? How can I import this file into matlab?
A search can turn up wonderful things, such as:
Reading TDM/TDMS Files with The MathWorks, Inc. MATLAB® Software -
Timing problems reading tdm files
Hello NI community,
my first post because of a very annoying problem with the Storage VIs from the File I/O VIs and Functions palette. I spent a long time to create a program for analyzing tdm files and I spent almost the same time trying to fix this problem now
My program opens a single tdm file, reads in the data and analysis it, displays the result (e. g. 10 DBL values) and closes the tdm file afterwards, then it opens the next tdm file... .
The Problem is, that the execution time increases permanently e. g. starting at 50 ms after 4000 tdm files read it takes about 1 s! So it takes days to read in more than 10000 tdm files. Also it takes many minutes to close the program after reading many (e. g. 4000) files!
Maybe the Storage VIs store the data in the background on a server or something else and do not release this data after closing the tdm file.
Has anyone an idea how to fix this problem? How to release all resources after closing the tdm file. Is there an alternative method to read in tdm files without using the buggy Storage VIs.
Converting tdm to tdms does not help, converting time increases the same way.
Wait time (0,5 s) after closing the tdm file does not help.
Settings for tdm VIs: open (read only)
LabView2011 SP1
Thanks in advance
DanielHello Norbert,
thank you for your reply!
Attached you´ll find a simplified example for testing.
It reads the same .tdm file multiple times. The behavior is the same than in my application.
Copy the files on your computer, select the correct folder for the .tdm file in the VI and press start. You´ll see the execution time for opening the .tdm file, for reading the data and also for closing the .tdm file rising.
For example "Read data" duration on my computer:
Loop #: Duration 2: Factor to duration at start: Stopping program:
1 19 ms < 5 s
1000 42 ms 2
2000 63 ms 3
4000 101 ms 5
8000 185 ms 10
10000 222 ms 12 > 3 min
The tdm-files I need to read in are with .bin data files and bigger. Reading lasts about 100 ms for the first tdm-file. The factor is almost the same like in this example. So it takes about 1 s after reading 6000 tdm-files.
Best Regards
Daniel
Attachments:
Read_tdmFileMultipleTimesTest.vi 876 KB
Read_tdmFileMultibleTimesTest.zip 1265 KB -
I am needing to look at select data of a TDMS file in an analyze portion of a stat machine. The VI is to test both AC and DC motors. I am getting 18 different reading from a power analyzer. I am writting these to a TDMS file for ease of passing through the rest of the states. In the "Analyze" state I am wanting to look at 13 of readings an compare them to see is they are within tollerence. I will then use this check to highlight the reading green or red.
My questions are:
Should I even have an 'Analyze' state or should I do the analysis in the state that the reading is taken?
If I do use the 'Analyze' state is pulling the data from the TDMS file the best way to do this?
Also, if you see any other possible errors in the vi, please let me know. This is my first experience with LabView and it is steep learing curve.
Thanks,
Attachments:
AC TEST PANEL.vi 524 KBSince you are using a State machine approach, I would suggest to keep that idea and add the Analyze state to keep the code organized. Plus, you can add a shift register to pass around the acquired data which will be more efficient than reading from the TDMS file.
For future reference, when using the state machine while you analyze, log, etc...you are forcing sequential execution and loosing time that can be used for hardware I/O. When this becomes a concern you can implement the following structure. Keep up the good work
Alejandro | Academic Program Engineer | National Instruments -
Hello!
I want to write 1 digital port from PXI-6536 with streaming to TDMS file.
I'm writing by 'DAQmx Configure Logging.vi' and become TDMS file with 8 boolean channels.
How can I write to 1integer channel?
Attachments:
1.JPG 27 KBHey Atrina,
The actual data stored on disk is just the raw data (that is, a byte per sample in your case). It's really just a matter of how that data is being represented in LabVIEW whenever you read back the TDMS file.
I'm not sure if there is a better way to do this, but here is a way to accomplish what you're wanting:
Read back the TDMS file as a digital waveform. Then there's a conversion function in LabVIEW called DWDT Digital to Binary. This function will convert that set of digital channels into the "port format" that you're wanting. I've attached an example of what I mean.
Note: When looking at this VI, there are a few things that the downgrade process did to the VI that I would not recommend for these TDMS files. It added a 1.0 constant on the TDMS Open function, and it set "disable buffering" on the TDMS Open function to false; you can get rid of both of those constants.
Message Edited by AndrewMc on 01-27-2010 11:21 AM
Thanks,
Andy McRorie
NI R&D
Attachments:
digitalconvert.vi 13 KB -
Time from .xls file is divided
Hello I,m new with LabVIEW 10 and i have a problem with reading time from .xls file with the read excel values.vi.
In excel the time format is 00:00:20 but in LabVIEW (with Strings) i get (i think) a divided result 0,000231481481481481 for 00:00:20 and so on.
00:00:15
00:00:20
00:00:17
00:00:20
00:00:08
00:00:20
00:00:09
I would like to get the same format in the .vi as it is in .xls file.
Picture added.
Could you please helpe me with this...
Attachments:
String problem.JPG 36 KBThat VI returns the value of the cell, not the formatted string. The issue with time values between Excel and LabVIEW has been brought up numerous times before. Excel does not use the same datetime starting point as LabVIEW. See, for example, this discussion: http://forums.ni.com/t5/LabVIEW/Read-custom-excel-time-data-in-LabView/m-p/775359
-
How to convert tdm files to lvm files?
I am trying to convert a number of TDM files to LVM files. I got error (2553) when I execute my VI. I dont know why.
1. Can anybody help me? I attached the VI.
Thank you.
Hyunseok
Attachments:
Conversion TDM to LVM.vi 121 KBHi Hyunseok
I changed your VI to a single file converter (see attched VI) and it works out well with LabVIEW 2009. Is there any special within your TDM files? Can you provide an example file which does not work?
Additionally I'm curious why you want to convert TDM to LVM since there are a bunch of tools to directly work with TDM files in other applications (Find Resources for Reading and Writing TDMS Files in Third-Party Products)
Stefan
Attachments:
Simple Conversion of TDM to LVM.vi 61 KB -
Hello,
how can I make a recording an analog input in first .tdms file (for example in 1 second) followed by a recording in another file (for example in 1 second)?The simplest would be to configure the DAQmx Streaming feature to write your analog reads to a TDMS file. You can use a DAQmx Read property node to set the number of samples to put into each TDMS file. A new file will automagically be created once that number of samples have been streamed.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines -
Storage VIs internal error, TDMS file
I use LabView 8.5.1, DAQmx 9.0.2. and a module NI-USB6216.
I configure on my application a task as follows: 15 channels activated with a sample rate of 1sample/10s (very slow). My acquisition has to run for a few days or more.
Sometimes, I obtain the error "Storage VIs internal" error (code -2500). then I cannot read my TDMS file (see attached the file) and I cannot treat my data.
Do you know the exact meaning of this error on the case of a TDMS file. Do yo have a solution to treat my data?
Best regards, Daniel.
Solved!
Go to Solution.
Attachments:
Data.zip 436 KBI tried with LabVIEW 2010 and didn't see the problem when reading your attached TDMS file.
Please download the latest TDMS installer from here:
http://zone.ni.com/devzone/cda/tut/p/id/9995
Please backup tdms.dll on your computer like in c:\program files\national instruments\shared\tdms\ and then run the installer. -
Error reading TDMS file with inconsistend dt
Hello all,
I would like to ask your help to understand how to read a TDMS file which I am opening and closing based on a user input. As a result the data in the TDMS file does not have a consistent time step, which may be causing my problem, but I really don't know. I keep getting the error 2501, that I have an invalid reference.
Or, if I continue and bypass the error, the extract portion VI gives me an error stating that I need to define an integer multiple of dt.
I am not sure how to solve the first point as I have the reference wired directly to the read TDMS vi.
I have attached a screen shot of the error and the portion of the VI with the offending error (it is the code connected to the read TDMS read vi).
My goad is to extract a portion of the data to do calculations on and display a result to the user.
thanks for any comments.
A. Lopez
Attachments:
error-2501.JPG 198 KBI think you open multiple times the same TDMS file, you should open it before the while loop. And I don't see a close tdms function...
Also check for errors on the open TDMS function.
Ton
Message Edited by TonP on 02-14-2007 07:45 AM
Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas
LabVIEW, programming like it should be! -
I am trying to read TDMS files directly into Matlab using the NI nilibddc.dll. I have read most of the posts on here but still have a problem...
Within Matlab I can open and read a file and get the data but I cant find anything that tells me what the sample interval (between samples) is for a particular file. I guess its in the header somewhere but how do I access it?
I cant find anything that tells what function to use to get it from the header.
In fact i cant find anyhting that tells me what the functions are in the DLL.
I tried opening the CHM help file, this has a list of all the functions but when you click on one in the contents it reports 'the address is not valid'. It seems the link is borken somehow in the help file (this is on XP)!
any help gratefully receieved!
mikehi mike,
if your data is stored in form of waveforms, this information can be retrieved from the channel:
Entity
Property
Data type
Description
tdm_channel
wf_xname
string
Name of the x-channel of the waveform
tdm_channel
wf_xunit_string
string
Unit text of the x-channel
tdm_channel
wf_start_time
time
Time stamp for the waveform
tdm_channel
wf_start_offset
double
Starting offset (to be added to the time stamp)
tdm_channel
wf_increment
double
Increment
tdm_channel
wf_samples
long
Number of samples in one shot (needed by LabVIEW when concatenating
more consecutive shots). This property is used by DIAdem and DataFinder to decide whether the tdm_channel is a waveform (wf_samples > 0) or not (wf_samples = 0, or not existing).
tdm_channel
wf_time_pref
string
(enum: "absolute" or "relative") Additional information for LabVIEW to
determine if the x-channel is treated as time (absolute) or as index
(relative). (not available in LVM) -
Time stamp to name a TDMS file
Hi:
I am using data from excel files to create a time stamp, after that,my VI creates TDMS files with this data in files with 5000 samples, also I am using time stamp to named the TDMS files, but my problem is that my first file takes the name 180000.000, How can I modify this? because I need my file´s name will be the first data from excel.
I attached my VI and two excel files, the idea is that the first file will be 84327.500 and the second will be 84740.950
Attachments:
test1.xlsx 150 KB
test2.xlsx 151 KB
testfile.vi 56 KBFirst, let me congratulate you on taking your first steps to learning LabVIEW. If you have not already done it, I highly recommend you go through the online tutorials. You have an actual project to work on, so application of what you learn will be fairly immediate.
For your actual program issue, you need to read the data from the Excel file and convert it to a timestamp before you set your filename. The order of operation of your program is to set the TDMS file name while simulataneously reading from the Excel file. After you have already created the TDMS file, you convert the data to a timestamp and write this to your timestamp front panel control. Note that the first point you write to the TDMS file may or may not be what you think it is. It could easily be the same 180000.000 value you are using to name the TDMS file. You have what is known as a race condition. The order of operation between the read and write of the timestamp control is not known and can be variable. It is one of the reasons that you should NEVER use local variables to pass data around. Use wires. In LabVIEW, wires are the data and take the place of variables in a text base language. Front panel controls should be used only for user interace reasons. In your loop, you should wire the timestamp from its current conversion (wired into the timestamp on the front panel) directly to the TDMS write (currently wired from a local variable of timestamp).
A couple of other things. You do not need to convert your data arrays to matrices to index them. This can be done directly with Index Array from the array palette. This single primitive will replace the Array Subset, convert to matrix, and index matrix primitives.
You do not need to go through a string conversion to produce a timestamp. The Date/Time to Seconds primitive in the timing palette will do it directly. You will need to populate the cluster input.
When you posted your code, you did not include your subVIs. In the future, include all your VIs, either separately or in a ZIP file.
Finally, to solve your intial problem, use Index Array to pull the first row of data from the Excel file, then convert this to a timestamp using Date/Time to Seconds. Use this to create your TDMS file.
You may have noticed, if you do all this, you do not need your timestamp front panel control, other than as a simple monitor.
Good luck! Welcome to LabVIEW.
This account is no longer active. Contact ShadesOfGray for current posts and information. -
How to append time stamp to TDMS file
hello all, i would like to append time stamp to a tdms file as the 1st column of the file... the pattern should look like this.. can anyone suggest how to create such a file?
Time
Data
0.001
2.5545
0.002
2.653
0.003
1.215
0.004
2.6586
Now on LabVIEW 10.0 on Win7I got the below oepratin to do.. i used a Write to Measurement File.vi and selected tdms format and it worked out... however, when i read the same file using read measurement file.vi and selected tdms option in ti, i am observing that on the x-axis graph, i get real-time stamp instead of decimals... i want x-axis as decimals instead of timestamp, and when i selected decimals in waveform graph properties --> display format--> x-axis, i could not select decimal instead of time-stamp... can anyone please let me know how to get what i intend to do?
Now on LabVIEW 10.0 on Win7
Attachments:
Untitled 2.vi 68 KB
Maybe you are looking for
-
FM for read cost center distribution infotype 1018
Hi Gurus, Can anybody provide FM for read OM Infotype 1018 - cost center distribution. Note : Requirement is get cost centers which are maintained in 1018 as POSITION based. Regards, Ranjith
-
Screensaver broken under Mountain Lion
Well, just upgraded to Mountain Lion, and the default screensaver doesn't work like it use to. Previously, I pointed it to use the directory that contained all of my photos, in which there are a bunch of sub-directories (photos/2010, photos/2011, pho
-
hi.. i want to load a swf into mdi window in flex.. here my swf size is 200x200.. when i m trying to load this swf into mdi window using swfloader it exceeds the mdi window.. it is not load with original swf size(200x200).. how to achive this..
-
Hi. I have a project to use LabVIEW as a server to do many tasks. What i want is: LabView receive remote data from internet about a ECG. LabView will also run an MatLab/Octave script to analysis the data. This data will be available from remote acces
-
I have an iPod photo 60gb and (2) computers - an iMac and an iBook. I've been syncing my photos and my iTunes music from the iBook, with no problems. Unfortunately, space on my 30gb iBook is running dreadfully low, and I've decided to have my iMac be