Appending tdms files problem

Hello all,
I've developed a VI that logs data from a 6 hours test. Sampling rate is 10k resulting to 3,600 TDMS files each of them saving 6 columns (AI channels) of 60,000 rows. These are raw data that are going to be post processed calculating i.e rms values, FFTs and so on.
I am in a stage where I should take a decision of whether I should use DIAdem for post processing or develop my own LabView code. By far I would prefer DIAdem as I am not very experienced Labview programmer. My questions are:
1) Drag&dropping 3,600 tdms files to the Data Portal results as expected memory overflow. Is there anyway of appending all these massive data for processing? I.e being able to have a chart of 6 hours and the corresponding rms values?
2) Is there any way of (if it is required) compressing my samples? I.e get an average of rms values every 10 or 50 cycles?
I have been through most of the webcasts on DIAdem and I really wanna take a swift decision of how I should orient my post processing work and your help and suggestions are most welcomed.
Thank you
Panos

Herbert,
does this mean you can stitch TDMS files together?
Like this:
Ton
Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas
LabVIEW, programming like it should be!

Similar Messages

How to append time stamp to TDMS file

hello all, i would like to append time stamp to a tdms file as the 1st column of the file... the pattern should look like this.. can anyone suggest how to create such a file?
Time
Data
0.001
2.5545
0.002
2.653
0.003
1.215
0.004
2.6586
Now on LabVIEW 10.0 on Win7

I got the below oepratin to do.. i used a Write to Measurement File.vi and selected tdms format and it worked out... however, when i read the same file using read measurement file.vi and selected tdms option in ti, i am observing that on the x-axis graph, i get real-time stamp instead of decimals... i want x-axis as decimals instead of timestamp, and when i selected decimals in waveform graph properties --> display format--> x-axis, i could not select decimal instead of time-stamp... can anyone please let me know how to get what i intend to do?
Now on LabVIEW 10.0 on Win7
Attachments:
Untitled 2.vi ‏68 KB

Problem viewing data after import TDMS file

Environment SignalExpress 3.0. Windows XP.
I have a number of logs created and saved in a SignalExpress 3.0 project. I'm trying to import a single log into a new project and view the data in a data view window.
So I do:
Open SignalExpress
File->Import->Logged Signal from SignalExpress TDMS files
Navigate to the log folder named - "25062009_135820_-_Nano_Carbon_Pyrograf_T1"
Double click the tdms file named - "Voltage.tdms"
I see the following tree structure in the Logs pane :
Logs
    - 25/06/2009 13:58:20
        - Nano Carbon Pyrograf T1
            - Voltage - Dev3_ai0.
If I then try to drag Voltage - Dev3_ai0 onto a data view it doesn't work.
I've inspected the original project which I used to create the log tried to import above. Comparing the tree structure it looks:
Logs
    - 25/06/2009 13:58:20 - Nano Carbon Pyrograf T1
        - Voltage
            - Dev3_ai0
In this project I can drag Dev3_ai0 onto a data view with no problem.
Notice the difference in tree structure in the Logs pane? I'm pretty sure SignalExpress is parsing the log wrongly when it imports it!
Mike

Hi Mike
I know that we have communicated over email but I just wanted to respond here so others can see the workaround.
I have investigated the issue and have been able to reproduce the issue, even in Signal Express 2009. The problem is caused by the name you specify for the log when you begin recording. If the character "-" is present at any point in the name of the log, Signal Express does not import the tdms file correctly. This is not an issue with the tdms file, as it opens fine with LabVIEW, but must be an issue with Signal Express. I have reported this as bug under reference #182846. For now, you should not use the "-" character in the name of your log.
Regards
Rich
NI | UK

Reading TDMS files in Matlab - problems with the example code

I want to use the TDMS files to store data and do offline analysis in Matlab, version 2009b (either on Mac OS10.6 or Windows XP).
To test the data import in Matlab I tried the example from http://digital.ni.com/public.nsf/allkb/0EEADA99DC7D00A4862572E30037C3A2 , but without success.
I tried either with and without the path definition (as explained on the page I referred to before) at the start, and got back the following errors:
with:
??? Error using ==> loadlibrary>deltempfiles at 559
nilibddc_m.i.
Error in ==> loadlibrary at 452
deltempfiles(delfiles);
Error in ==> Read_TDMData at 7loadlibrary([examplePath '\nilibddc'], hfile);
without:
??? Error using ==> loadlibrary>deltempfiles at 559
nilibddc_m.i.
Error in ==> loadlibrary at 452
deltempfiles(delfiles);
Error in ==> Read_TDMData at 36loadlibrary(NI_TDM_DLL_Path,NI_TDM_H_Path);
I found a few messages on the issue, dating from 2006 and 2007, that didn't lead to a satisfying solution and was hoping that, by now, there would be a solution to my problem.
Thanks a lot
Remco
Solved!
Go to Solution.

Hi Musk,
Thanks for your answer.
Unfortunately it didn't help as my problem is not discussed there. I have a problem with this line:
loadlibrary(NI_TDM_DLL_Path,NI_TDM_H_Path);
I said before that I was using 2009b, but I was mistaken, I have R2008a, but the 'loadlibrary' is opened in a 2008b editor (no idea why). On the Mac I have 2009b installed.
Thanks
Remco

Memory usage problem with TDMS files

Hi,
I have used TDMS write file to save my real time data . I need to leave my VI, running for a long time (even one day ) then I process the acquired data in a seperate vi.
My problems are
I olways get memory usage message during data acquization. Also even if I finish acquization ( for example make a 2GBytes TDMS file I get the same message when I want to try to read it.
To read the TDMS fisrt I open ,configure ,read and then close the TDMS ref.Then I use the data from the TDMS as input to my loop.
By using this method I olways get (Memory usage message ) and I need to stop the process loop. Would you please help me how can I avoid this problem

your problem is the attempt to attach all the data in one big goo in memory. A 2GB TDMS file needs probably a little more than 2GB in memory if you want to read all its contents at once. That is BAAAD! You should try to write your analysis in a way that can read the data in smaller chunks and process each chunk individually instead of trying to load the entire GIGA blob at once. Computers can have a lot of memory nowadays but that does not mean that you should attempt to operate on GigaByte data in memory in one go.
Rolf Kalbermatter
CIT Engineering Netherlands
a division of Test & Measurement Solutions

Timing problems reading tdm files

Hello NI community,
my first post because of a very annoying problem with the Storage VIs from the File I/O VIs and Functions palette. I spent a long time to create a program for analyzing tdm files and I spent almost the same time trying to fix this problem now
My program opens a single tdm file, reads in the data and analysis it, displays the result (e. g. 10 DBL values) and closes the tdm file afterwards, then it opens the next tdm file... .
The Problem is, that the execution time increases permanently e. g. starting at 50 ms after 4000 tdm files read it takes about 1 s! So it takes days to read in more than 10000 tdm files. Also it takes many minutes to close the program after reading many (e. g. 4000) files!
Maybe the Storage VIs store the data in the background on a server or something else and do not release this data after closing the tdm file.
Has anyone an idea how to fix this problem? How to release all resources after closing the tdm file. Is there an alternative method to read in tdm files without using the buggy Storage VIs.
Converting tdm to tdms does not help, converting time increases the same way.
Wait time (0,5 s) after closing the tdm file does not help.
Settings for tdm VIs: open (read only)
LabView2011 SP1
Thanks in advance
Daniel

Hello Norbert,
thank you for your reply!
Attached you´ll find a simplified example for testing.
It reads the same .tdm file multiple times. The behavior is the same than in my application.
Copy the files on your computer, select the correct folder for the .tdm file in the VI and press start. You´ll see the execution time for opening the .tdm file, for reading the data and also for closing the .tdm file rising.
For example "Read data" duration on my computer:
Loop #:     Duration 2:   Factor to duration at start:     Stopping program:
1              19 ms                                                    < 5 s
1000         42 ms         2
2000         63 ms         3
4000         101 ms       5
8000         185 ms       10
10000       222 ms       12                                        > 3 min
The tdm-files I need to read in are with .bin data files and bigger. Reading lasts about 100 ms for the first tdm-file. The factor is almost the same like in this example. So it takes about 1 s after reading 6000 tdm-files.
Best Regards
Daniel
Attachments:
Read_tdmFileMultipleTimesTest.vi ‏876 KB
Read_tdmFileMultibleTimesTest.zip ‏1265 KB

Problem opening multiple .tdms files in subroutine

Heya-
I'm using LabVIEW 8.2, professional development system.
I'm using the Open / Read / Close TDMS functions as part of a subroutine in a larger program. The subroutine is supposed to open one of several TDMS files (each of which contains bulk data saved previously), extract the data, then restructure into a 3D array, which is passed back to the main program. The particular file TDMS file to open is controlled by the main program. The TDMS Close function is used after the data is extracted.
When running solo, the subVI performs as expected. When running as a subVI, it functions fine on the first call, but subsequent calls to open different data sets fail to actually open the TDMS file (or at least, the Read TDMS function generates an empty array). However, returning to the orginal file that was opened, the subVI works fine - it opens the TDMS file. It doesn't matter which file was intially used - whichever one was opened first functions fine, any others do not. That would make me think that the first TDMS file isn't being closed, except that I am using the Close TDMS function after reading.
Attached are the subVI (TDMS_to_3D_array), a quick test program that uses the subVI, and a .zip of two data sets
Not sure if the global variables located in the SubVI will default, so they are: UpperMaxCol: 55, UpperMinCo: 0, UpperMaxPixel: 48, UpperMinPixel: 6
Thanks in advance for the help
Dan
Attachments:
Data.zip ‏1060 KB
TDMS_to_3D_array.vi ‏54 KB
TDMS_loadfault_tester.vi ‏39 KB

Hi Dan,
Perhaps you would try a few things.
Use the TDMS Flush VI in the subVI before the TDMS Close VI.
Try running the program with the subVI open.
Wire the TDMS File refnum from the subVI back to the parent and observe its value as the program switches between TDMS files.
Perform the TDMS read in the parent VI, wire the TDMS Data and the Array Size of the TDMS dataset to the subVI. Does this make any difference?
Try the subVI without the Channel Name string indicator.
Include error cluster wires throughout the entire subVI. Wire them through the TDMS VIs, then through the nested For structures, and then back to the parent VI.
Let us know what happens.

How to store data (in tdm files)

I'm looking for advice on how to store data in tdm/tdx files.
The main challenge is like this: At regular time intervals there is new data available (say every 10 minutes there are 10 minutes of new data available). All data is time based. So every ten minutes I may create a new tdm/tdx file containing this data segment. However, when i want to analyze the data i dont necessarily want to view 10 minutes of data from all channels, but maybe 20 hours of data from one particular channel.
The way I've achieved this so far is to manually load each 10 minute segment of this particular channel and then add these parts together. This is both time consuming and cumbersome. Is there a better way to do this that I've simply not discovered yet? Is there a better way to store data to simplify this process?
One solution is of course to save data from several 10-minute frames in one file, but seeing as there is a never ending supply of data i cant simply save all data in one giant file, it has to be split up at some point, and the problem will still remain.
One factor to have in mind here is that this is rather large amounts of data (maybe 10GB each day), so the option to simply load all the data into memory goes out the window rather quickly.
Feel free to ask if I've not made myself very clear

Hi salte,
If your test ever ended, and if you had LabVIEW 8.20 or higher, I would recommend using a TDMS file, which handles data appeands flawlessly. DIAdem 10.1 now also does data reduction and index windowing during file loading for TDM / TDMS / DataPlugin files, so that you could easily load out only the part of the file you wanted to look at. But since you describe your data acquisition as never stopping and amassing 10 GB per day, I agree that it would be impractical to use only 1 data file. So we are stuck with some number of files which each contain a part of your measurement. This approach can have advantages, since you can save operational properties for each "buffer" such as average value, dominant frequency, ambient room temperature, etc., and later on you can use the DataFinder to query out only the data buffers which meet specific conditions based on these properties. The problem remains to load and assemble data based on multiple files. This is an old problem in DIAdem, and one for which I have an efficient and I hope satisfactory workaround application. It does what you describe already doing in the minimum amount of time and with the minimum amount of user interaction, and it can be highly parametrized to suit your particular situation. The ideal way to start the application would be as part of a ResultsList custom menu. This would enable you to query out the buffers you wanted, highlight those rows in the ResultsList, right-click and choose your custom menu, and WHAM! the selected buffers for your queried channel(s) are automatically appended together in the DataPortal. Launching from a ResultsList custom menu would mean that you could skip the file dialog and just read out the data sources directly from the ResultsList selection.
Let me know what you think,
Brad Turpin
DIAdem Product Support Engineer
National Instruments
Attachments:
Importing Data from Multiple DataPlugin Files.zip ‏198 KB

Error reading TDMS file with inconsistend dt

Hello all,
I would like to ask your help to understand how to read a TDMS file which I am opening and closing based on a user input. As a result the data in the TDMS file does not have a consistent time step, which may be causing my problem, but I really don't know. I keep getting the error 2501, that I have an invalid reference.
Or, if I continue and bypass the error, the extract portion VI gives me an error stating that I need to define an integer multiple of dt.
I am not sure how to solve the first point as I have the reference wired directly to the read TDMS vi.
I have attached a screen shot of the error and the portion of the VI with the offending error (it is the code connected to the read TDMS read vi).
My goad is to extract a portion of the data to do calculations on and display a result to the user.
thanks for any comments.
A. Lopez
Attachments:
error-2501.JPG ‏198 KB

I think you open multiple times the same TDMS file, you should open it before the while loop. And I don't see a close tdms function...
Also check for errors on the open TDMS function.
Ton
Message Edited by TonP on 02-14-2007 07:45 AM
Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas
LabVIEW, programming like it should be!

Reducing TDMS file size?

Hello,
I'm streaming analogue data acquired at 32 kS/s to disk in 32768 data point writes using TDMS Write.vi .
A while back I feel that I saw an article about reducing the size of the TDMS files with some tricks. I think one of them might have been not writing unused high significance bits(?).
I can't find that article now and I"m wondering if anyone had suggestions for reducing file size? Right now its about 0.25 MB/s. This is manageable, but it would be nice to reduce this if it were easily possible.
Thanks in advance,
Frenk

Hello,
Hmmm... I've run into an unexpected problem. I'd like to scale the unscaled data from DAQmx for some calculations during the data collection, but it isn't scaling the way I would have expected.
My range is +/-10V and the largest unscaled numbers for data filling that range are very roughly about +/-125000. I would have expected them to approach 2^31 (one bit for sign)!! How can one scale this data?
Also, my saved TDMS file doesn't seem to have the information that "Cont Acq&Graph Voltage - Graph Unscaled Data from File (TDMS).vi" needs to scale it.
Regarding I16, my data is unusual in that it contains a signal of interest that is about 80 dB below the stimuli that are also in the measurement. For that reason I need the full I32 to avoid excessive quantization error in the signal of interest.
Thanks again!
Frenk

Reading TDMS files in Matlab

I am trying to read TDMS files directly into Matlab using the NI nilibddc.dll. I have read most of the posts on here but still have a problem...
Within Matlab I can open and read a file and get the data but I cant find anything that tells me what the sample interval (between samples) is for a particular file. I guess its in the header somewhere but how do I access it?
I cant find anything that tells what function to use to get it from the header.
In fact i cant find anyhting that tells me what the functions are in the DLL.
I tried opening the CHM help file, this has a list of all the functions but when you click on one in the contents it reports 'the address is not valid'. It seems the link is borken somehow in the help file (this is on XP)!
any help gratefully receieved!
mike

hi mike,
if your data is stored in form of waveforms, this information can be retrieved from the channel:
Entity
Property
Data type
Description
tdm_channel
wf_xname
string
Name of the x-channel of the waveform
tdm_channel
wf_xunit_string
string
Unit text of the x-channel
tdm_channel
wf_start_time
time
Time stamp for the waveform
tdm_channel
wf_start_offset
double
Starting offset (to be added to the time stamp)
tdm_channel
wf_increment
double
Increment
tdm_channel
wf_samples
long
Number of samples in one shot (needed by LabVIEW when concatenating
more consecutive shots). This property is used by DIAdem and DataFinder to decide whether the tdm_channel is a waveform (wf_samples > 0) or not (wf_samples = 0, or not existing).
tdm_channel
wf_time_pref
string
(enum: "absolute" or "relative") Additional information for LabVIEW to
determine if the x-channel is treated as time (absolute) or as index
(relative). (not available in LVM)

How to send TDMS file using TCP/IP

Hi,
I'm working with a cRIO-9025, chassis cRIO-9118 and NI 9239 module.
I've done an acquisition (voltage) where data is in 1D Wfm SGL. I've saved this data using a TDMS file, but now I want to send this file to an other computer (in this case mine) because the file is stored in the target. To do that I've implemented this example:
http://zone.ni.com/devzone/cda/epd/p/id/2842#0requirements
The problem is that the tdms file is created but it hasn't any data. I attach the image of the program. The program is done in other project (not which I use to make acq) to make it easy and then implement it when it works.
*The name file in the client is not used because I had a conflict with path.
What I'm looking for is make an acquisition of 10 minutes every 30 minutes and then send this data to an other computer (or others). I'm not sure if it's the best way, maybe is better to send data at the same time as the acquisition is being done, but my theacher told me that is easier if I send the file when acquisition has finished. To make acq I use FPGA and RT.
I would appreciate your help. Thanks!!
Solved!
Go to Solution.

I believe the FTP VIs are built into LabVIEW 2011SP1 - I have the VI present, but I could have installed the internet toolkit (don't know).
Look in the following location: Function Palette -> Connectivity -> Internet -> FTP
You're really looking for the VI named, "FTP Get File.vi". What this VI does is contacts an FTP server and requests a file from it. If your files are on the target, and you know the filename that will be created (such as files created in a sequence) you can request the file from the target and have them copied to the local machine (Windows). RT cannot "push" files to the Windows side unless you have an FTP server (like FileZilla Server) running on the Windows host.
It's a very simple one-VI solution. You specify host (the IP Address of the target you're pulling files from), the remote path for the file (for instance if the file is located in c:\ni-rt\data\myfile.zip then that's what you put as the remote path of the file), the local path for the file (where you want the file copied to, along with the name of the file, such as c:\mydir\myfile.zip), the username to log into the FTP server with (default is 'anonymous') and the password to use (default is an empty string). The importan thing to note is whether to use a binary transfer or not - if the file is a text file, no binary transfer needed; if the file is a binary file then you get better results using a binary transfer - I prefer to always use binary transfers, as non-binary transfers actually "interpret" and "convert" text information in the files.
-Danny

Time stamp to name a TDMS file

Hi:
I am using data from excel files to create a time stamp, after that,my VI creates TDMS files with this data in files with 5000 samples, also I am using time stamp to named the TDMS files, but my problem is that my first file takes the name 180000.000, How can I modify this? because I need my file´s name will be the first data from excel.
I attached my VI and two excel files, the idea is that the first file will be 84327.500 and the second will be 84740.950
Attachments:
test1.xlsx ‏150 KB
test2.xlsx ‏151 KB
testfile.vi ‏56 KB

First, let me congratulate you on taking your first steps to learning LabVIEW. If you have not already done it, I highly recommend you go through the online tutorials. You have an actual project to work on, so application of what you learn will be fairly immediate.
For your actual program issue, you need to read the data from the Excel file and convert it to a timestamp before you set your filename. The order of operation of your program is to set the TDMS file name while simulataneously reading from the Excel file. After you have already created the TDMS file, you convert the data to a timestamp and write this to your timestamp front panel control. Note that the first point you write to the TDMS file may or may not be what you think it is. It could easily be the same 180000.000 value you are using to name the TDMS file. You have what is known as a race condition. The order of operation between the read and write of the timestamp control is not known and can be variable. It is one of the reasons that you should NEVER use local variables to pass data around. Use wires. In LabVIEW, wires are the data and take the place of variables in a text base language. Front panel controls should be used only for user interace reasons. In your loop, you should wire the timestamp from its current conversion (wired into the timestamp on the front panel) directly to the TDMS write (currently wired from a local variable of timestamp).
A couple of other things. You do not need to convert your data arrays to matrices to index them. This can be done directly with Index Array from the array palette. This single primitive will replace the Array Subset, convert to matrix, and index matrix primitives.
You do not need to go through a string conversion to produce a timestamp. The Date/Time to Seconds primitive in the timing palette will do it directly. You will need to populate the cluster input.
When you posted your code, you did not include your subVIs. In the future, include all your VIs, either separately or in a ZIP file.
Finally, to solve your intial problem, use Index Array to pull the first row of data from the Excel file, then convert this to a timestamp using Date/Time to Seconds. Use this to create your TDMS file.
You may have noticed, if you do all this, you do not need your timestamp front panel control, other than as a simple monitor.
Good luck! Welcome to LabVIEW.
This account is no longer active. Contact ShadesOfGray for current posts and information.

Editing String in.TDM file

With Set Properties it is possible to add one or more strings to a .TDM file (or is it to another file that is linked to the .TDM file?). Similarly it is possible to read that string with Get Properties.
My current impossibilities are:
- If each measurement is appended to the preceding measurement how do I read a record that is not the last one saved?
- If one or more strings has to be edited how can the modification replace only the affected strings? This implies a Read VI that is also capable to Write (replace).
I am not finding much information on this file format and would appreciate knowledgeable insight.
Regards,

TDM = 2 Files
The Storage VIs create a .TDM file (XML-based, for descriptive properties) and a .TDX file (binary, contains waveform / array data).
One or many channel groups
Each measurement can be appended to a file as a new channel group, or the values can be appended to channels that were stored earlier (resulting in one channel group per file). Use the "Always create new channel group/channel" switch to decide which way of appending data you want to use. You might want to use the "Data viewer" example shipping with LabVIEW to have a look at the hierarchy of channel groups and channels.
Note that when appending values to a channel, "Write Data" uses the channel name to find the right channel to append data to.
Get and set properties on particular channel groups
To modify data on a particular channel group, you need to obtain a "storage refnum (channel group)" to this channel group. There are two ways of doing that:
- Use "Read Data", configured to object type "channel group". The "channel groups" output will give you the array of refnums you need (the attached VI shows how to obtain and use these refnums).
- Wire the "storage refnum (file)" that comes out of "Open Storage" to "Get Properties". The "channel groups" property will be an array of all channel group refnums in your file (the "Write channel groups" and "Read channel groups" examples shipping with LabVIEW show how to do that).
"Read Data" should be used if you want to apply a query condition or if you are e.g. looking up channels across channel groups. "Get Properties" can give you all channel groups in the order they were stored in, or e.g. all channels of a particular group.
"Read Data" and "Set Properties"
"Set Properties" will replace all properties that you have selected and that show up on the block diagram as terminals. All other properties remain unchanged. The properties will be written only to the one file, channel group or channel that the incoming refnum refers to. "Read Data" doesn't write anything by itself, but you can use the output refnums of "Read data" to apply "Set Properties" to whatever you have read.
Attachments:
GetAndSetPropertiesInTdmFile.vi ‏115 KB

How can I append excel file rather than replace it?

Hi,
I have the attached VI set to write data to Excel, One case structure sends the data to cell B6, the other sends the data to cell B7. The problem is, the data is deleted from one cell when new data is added to another.
I think I want to be able to append the data without deleting other cells. Any ideas?
Thank you
Todd
Attachments:
Flow meter write to Excel question.vi ‏180 KB

Toddzilla,
There are a couple different ways to append to excel files. A couple links that might get you going are:
Can I Add New Data to an Existing Excel File Without Erasing Old Data?
Append Data to an Excel Document Every Iteration Using Report Generation Toolkit
Are you trying to write over cells B6 or B7, or are you trying to append within the cell itself?
Hopefully this can get you on the right track. When I append to files I tend to insert rows or append to the end of the file.
Scott A
SSP Product Manager
National Instruments

Appending tdms files problem

Similar Messages

Maybe you are looking for