Split a TDMS file

Hi,
I would like to record 32 signals in TDMS file.
But I want to acquire for example 50 000 points per files. I made a VI which do this but I have a problem because for one file, I can have 50 000 points for one channel et the double , 100 000 for the other channel, I don t know how i t s possible because I limited to 50 000 points.
Maybe I made a mistake in my code, could you help me please
I put my VI in attachment file.
Thank you very much
Romaric GIBERT
Attachments:
final4.vi ‏282 KB
Log_Channels_(SubVI).vi ‏15 KB

The solution is to not make the TDMS files so large. Perhaps modify the code so it starts a new file every 1/2 hour or something. Could you post a screen shot of the part of the code that is doing the saving?
Something that just occurred to me is that the TDMS file format is optimized for writing - not reading so stuff is just sort of streamed into the file with an index keeping straight what goes with what. The results is a file where the data from individual channels can be very fragmented, like parts of a file on a hard drive. I think there is a routine for defraging a TDMS file. Alternately, you could save the data in a temporary file (not necessarily TDMS) and resave it to the final TDMS file after all the data is collected.
Mike...
Certified Professional Instructor
Certified LabVIEW Architect
LabVIEW Champion
"... after all, He's not a tame lion..."
Be thinking ahead and mark your dance card for NI Week 2015 now: TS 6139 - Object Oriented First Steps

Similar Messages

Out of memory error while opening 1GB TDMS file, no splitting possible

Hi all,
the file in question contains about 1,4 Million channels, each has 295 integer points. The channel name is the time stamp where its measurement has been taken.
Diadem cannot load them completely and only displays the first 65.000 channels.
Labview (TDMS read) cannot even read small chunks of data. I've tried separating the file using Split TDMS File Memory Independent LV2009.vi (https://decibel.ni.com/content/docs/DOC-20054) but to no avail - the error I get is Code 2 (Memory is full).
I use Win7 and Labview 2012 SP1 32Bit.
Thanks in advance for any help.
Best regards, Markus
Solved!
Go to Solution.

Markus,
the term "channel" refers commonly to data coming from a single transducer. So if you have e.g. 50 transducers in your system, you should have 50 channels.
Acquiring data continuously generates chunks of data per time unit (e.g. 1kHz acquisition rate will generate 1000 samples per second). The chunks should be saved in the file to its appropriate channel. TDMS Write gives you the option to save data repeatedly to the same channels (again, group/channel name the important parameter!).
Nevertheless, the overall amount of data will not be less, even if you distribute them over channels in a better way. Still, loading ALL data at once will not be possible in a 32bit process.
BUT: If channels have all data as recommended, loading a singel channel at once will give you all data from the transducer for the complete measurement time. Then you can display/modify the data as requested, maybe store back to another file (or the same as new channel).
Having less channels could also improve loading times, but maybe there is no difference at all.... i don't have specific benchmarks on this.
Depending on the algorithms you are using, data reduction can improve overall times.
Norbert
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.

How to store data (in tdm files)

I'm looking for advice on how to store data in tdm/tdx files.
The main challenge is like this: At regular time intervals there is new data available (say every 10 minutes there are 10 minutes of new data available). All data is time based. So every ten minutes I may create a new tdm/tdx file containing this data segment. However, when i want to analyze the data i dont necessarily want to view 10 minutes of data from all channels, but maybe 20 hours of data from one particular channel.
The way I've achieved this so far is to manually load each 10 minute segment of this particular channel and then add these parts together. This is both time consuming and cumbersome. Is there a better way to do this that I've simply not discovered yet? Is there a better way to store data to simplify this process?
One solution is of course to save data from several 10-minute frames in one file, but seeing as there is a never ending supply of data i cant simply save all data in one giant file, it has to be split up at some point, and the problem will still remain.
One factor to have in mind here is that this is rather large amounts of data (maybe 10GB each day), so the option to simply load all the data into memory goes out the window rather quickly.
Feel free to ask if I've not made myself very clear

Hi salte,
If your test ever ended, and if you had LabVIEW 8.20 or higher, I would recommend using a TDMS file, which handles data appeands flawlessly. DIAdem 10.1 now also does data reduction and index windowing during file loading for TDM / TDMS / DataPlugin files, so that you could easily load out only the part of the file you wanted to look at. But since you describe your data acquisition as never stopping and amassing 10 GB per day, I agree that it would be impractical to use only 1 data file. So we are stuck with some number of files which each contain a part of your measurement. This approach can have advantages, since you can save operational properties for each "buffer" such as average value, dominant frequency, ambient room temperature, etc., and later on you can use the DataFinder to query out only the data buffers which meet specific conditions based on these properties. The problem remains to load and assemble data based on multiple files. This is an old problem in DIAdem, and one for which I have an efficient and I hope satisfactory workaround application. It does what you describe already doing in the minimum amount of time and with the minimum amount of user interaction, and it can be highly parametrized to suit your particular situation. The ideal way to start the application would be as part of a ResultsList custom menu. This would enable you to query out the buffers you wanted, highlight those rows in the ResultsList, right-click and choose your custom menu, and WHAM! the selected buffers for your queried channel(s) are automatically appended together in the DataPortal. Launching from a ResultsList custom menu would mean that you could skip the file dialog and just read out the data sources directly from the ResultsList selection.
Let me know what you think,
Brad Turpin
DIAdem Product Support Engineer
National Instruments
Attachments:
Importing Data from Multiple DataPlugin Files.zip ‏198 KB

Opening large tdms files in excel takes forever, can anything be done?

After recording some daq channels for 4 hours my file is 120mb large, with about 20columns of data and 150k rows. Opening this in excel takes at least 5min, several not responding screen fades, and growing fear that all my data is impossible to get to.
What can I do about this? Is there a split tdms file option I can use or a way to speed up excel maybe?
Solved!
Go to Solution.

The solution is to not make the TDMS files so large. Perhaps modify the code so it starts a new file every 1/2 hour or something. Could you post a screen shot of the part of the code that is doing the saving?
Something that just occurred to me is that the TDMS file format is optimized for writing - not reading so stuff is just sort of streamed into the file with an index keeping straight what goes with what. The results is a file where the data from individual channels can be very fragmented, like parts of a file on a hard drive. I think there is a routine for defraging a TDMS file. Alternately, you could save the data in a temporary file (not necessarily TDMS) and resave it to the final TDMS file after all the data is collected.
Mike...
Certified Professional Instructor
Certified LabVIEW Architect
LabVIEW Champion
"... after all, He's not a tame lion..."
Be thinking ahead and mark your dance card for NI Week 2015 now: TS 6139 - Object Oriented First Steps

TDM file with multirate interleaved data?

I am developing a datalogging application that samples several channels at different rates and I need a way to graph the data quickly. After figuring out that reading my own custom file format byte-by-byte takes a long time in LabVIEW, I started looking at TDM files.
I found the TDM Header Writer VIs, and it looks like I can just create a header that points to my binary data. However, I have several channels which can be sampled at different rates. For example:
Channel 1 data
Channel 2 data
Channel 3 data
Channel 1 data
Channel 1 data
Channel 2 data
Channel 1 data
Channel 1 data
Channel 2 data
Channel 3 data
Is it possible to create a header for data stored like that? Or would I need to include data for all of the channels at the highest frequency? ...
Channel 1 data
Channel 2 data
Channel 3 data
Channel 1 data
Channel 2 data (dummy)
Channel 3 data (dummy)
Channel 1 data
Channel 2 data
Channel 3 data (dummy)
Channel 1 data
Channel 2 data (dummy)
Channel 3 data (dummy)
Channel 1 data
Channel 2 data
Channel 3 data
I am thinking about creating 2 files: one for low speed data (0-100S/s) and one for high speed (100-1000S/s). I would simply insert the dummy data for channels that aren't the highest speed.
Am I going about this the right way? Is there an easy way to store the data the way I _really_ want to (all the channels interleaved without dummy data)? Any ideas or suggestions would be greatly appreciated.
Thanks,
John

John,
Have you considered splitting the data as it is acquired and writing each channel to it's own binary file and then using a TDM header file to relate the acquired channels?
That would be how I would approach this problem but as I haven't tried this before I can't tell you how successful it will be (yet).
I'll try looking into this some more and post again when I have some better information.
In the meantime here are some resources that you may or may not have seen:
Introduction to the LabVIEW TDM Header Writer VIs
LabVIEW TDM Header Writer VIs
Introduction to LabVIEW Data Storage VIs
~
Simon H
Applications Engineer
National Instruments

Tdms files

Hi,
I have a PCI 6132.
I´m using a program to acquire datas in continuos mode, and save it in my pc.
I'm using TDMS files to record it. But this files are too long. I need record it in multiple files to make easy to read after.
Using just one file, it has more than 10Gb, so, I need make multiples files with 1Gb each one, for example.
Thanks,
Marcela

Hi,
I think the better idea is to write data into multiple TDMS files when streaming data, instead of split the file into multiple files when you finish writing.
There are at least several ways to split, it depends on your application. For example, if you have multiple groups or channels, you can read from the big one TDMS file group by group and write to new TDMS file for each group. And, you can read samples by samples, for example, if you have 1000 samples in the big one file and you can split the 1000 samples into 10 files and each new file contain 100 samples. Another way is based on TDMS file format, TDMS file is consist of "segments", you can divide the big one file into new files and for each new file, it contains at least one segment (it would be a little complicated for the implementation, you need to know where the segment starts from, etc).
Please refer to www.ni.com/tdms for more reference.

Break large TDMS file into small files

hello all
My TDMS file is around 3g, and needs changed into around 10M size files.
I ran McRorie's splitFiles.vi ‏15 KB in this page. and set the number of samples per file as 5000000, however, I cannot get the results I need. every small file is only 1KB, and no data inside. What is the possible problem in this?
Also, I tried to write a vi based on sample vi(Read TDMS File) by adding one "write to measurement file.vi". However, when I set the small file size as 10M Byte inside the"write to measurement file.vi", the first file could be around 20M, and the next few files may be correct as 10M, and then it just stop splitting, edding with a file even much larger than original file. I uploaded my vi here, maybe someone can help to find some mistake for me.
Thanks very much!
Wuwei Mao
Solved!
Go to Solution.
Attachments:
Read TDMS File.vi ‏54 KB

Hi Wuwei,
After giving the correct data type to TDMS Read node in splitFiles.vi, it works as expected. ( See the attached two VIs: createFile.vi and the modified splitFiles.vi)
Because I don't know how you created your TDMS file, I write a new 3G bytes TDMS file, which has one group and one channel. The data type of samples is unsigned 16-bit Integer. The total number of samples is 1610612736. Then I set the number of samples per file to 5000000 as you did. So after splitting, each file size is 5000000*(16/8) bytes (around 10M bytes).
Please make sure the followed steps have been done before you run the splitFiles.vi:
1. The TDMS file will be split has been put on the proper path;
2. The correct group and channel names have been given;
3. The correct data type to TDMS Read node has been given.
Because your second option using "write to measurement file.vi" to split TDMS file will lose some information, such as group and channel names. So I suggest using the method used by splitFiles.vi to accomplish your goal.
Jie Zheng
NI R&D

Error reading TDMS file with inconsistend dt

Hello all,
I would like to ask your help to understand how to read a TDMS file which I am opening and closing based on a user input. As a result the data in the TDMS file does not have a consistent time step, which may be causing my problem, but I really don't know. I keep getting the error 2501, that I have an invalid reference.
Or, if I continue and bypass the error, the extract portion VI gives me an error stating that I need to define an integer multiple of dt.
I am not sure how to solve the first point as I have the reference wired directly to the read TDMS vi.
I have attached a screen shot of the error and the portion of the VI with the offending error (it is the code connected to the read TDMS read vi).
My goad is to extract a portion of the data to do calculations on and display a result to the user.
thanks for any comments.
A. Lopez
Attachments:
error-2501.JPG ‏198 KB

I think you open multiple times the same TDMS file, you should open it before the while loop. And I don't see a close tdms function...
Also check for errors on the open TDMS function.
Ton
Message Edited by TonP on 02-14-2007 07:45 AM
Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas
LabVIEW, programming like it should be!

How can I open a tdms file and replace a subset of data then save that change without re-writing the entire file again?

Hi all,
Is it possible to open a tdms file and make a small change an an array subset then save the file without having to save the whole dataset as a different file with a new name? That is to say, is there something similar to "Save" in MS Word rather than "Save As"... I only want to change a 1D array of four data points in a file of 7M data points.
I am not sure if this make sense? Any help is apreciated.
Thanks,
Jack

You can use either one, but for your application, I would use the synchronous. It requires far less setup. When you open the file, set both enable asynchronous and disable buffering to FALSE to enable you to use synchronous with arbitrary data sizes.
Attached code is LabVIEW 2011.
This account is no longer active. Contact ShadesOfGray for current posts and information.
Attachments:
UpdateTDMS.zip ‏20 KB

How to get the last data point from a TDM file in LabVIEW?

Hello,
I am using LabVIEW to analyze some rather large TDM files, and I need a way to get only the last data point. So far, the only way I have been able to accomplish this is by reading the entire file. Is there a property in the TDM file or a function in LabVIEW that will allow me to get the index of the last item in a channel?
Thanks!
Christina

Do you want to avoid reading whole file and want to be able to reach or get the index of last value of channel? is there any specific reason? I am not sure you could do it without loading the whole file. But the easiest way would be just to use array functions "array size" would give you the index of last element.
-Nilesh
Kudos are (always) welcome for the good post. :-)

Newly written tdm/tdms file not found

Hi,
I am trying to save some simple thermocouple data to a tdm/tdms file. I run my vi and it seems to work just fine. I am using the NI 9213 getting started example as a basis and have added tdms open, tdms set properties, tdms write, and tdms close. I can see the data streaming on the waveform chart but when I hit stop and go look for the saved file I can't seem to find it. What am I doing wrong?
Thank you
Solved!
Go to Solution.

SRamirez wrote:
I set it to create. When I run it once for a new folder it runs just fine but if I try again I get error 6.
You are specifying a folder? You should be giving a full file path (like c:\temp\blah.tdms). What you might want to do is build a file name based on the data/time. This way you won't have to worry about overwriting files and what not.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
Attachments:
Build File Name.png ‏9 KB

How to compare signals from two different .tdms files?

i'm developing a lie detection system in labview.so inorder for that i need to compare the threshold physiological signals from the .tdms file with the signals i acquire continuelsy from the subject for each question asked.i use respiration monitor and heart beat monitor together with the sensorDAQ to acuire the physiological signals.so which are the function i should make use of?i have the following toolkits
1. advanced signal processing toolkit
2. adaptive filter toolkit
3. digital filter design toolkit
4. biomedical toolkit
please give me advise on this.
thank you.
Solved!
Go to Solution.

Lie detector systems (polygraphs) generally work by looking at the physiological responses to the own subjects baseline, and do this in real-time. You could use a file that represents the stored "baseline" for the subject, but the overall environment, circumstances, general state of the subject, etc. may not be similar enough to easily see subtle changes.
From the stored file, you might determine a set of parameters or thresholds that indicate "normal, relaxed" for each signal. You could read the file into your LabVIEW program, process each channel, and save these parameters. Then when you have the new data, you can read the new data file in and process it in a similar way and then determine whether the parameters are outside the limits that you established for a "lie".
You may get more responses to this question if posted in the Biomedical User Group
Visit the NI Biomedical User Group at:
www.ni.com/biomedusers

How do I select more then one channel to view in tdms file viewer graph panel

I have a TDMS file that I can view with the labview 2013 TDMS file viewer and I can select one signal out of 15 signals to view on the graph panel. however I would like be able to look at 2 or 3 signals
on the same graph but can't seem to be able to select more then one signal at a time. Is there a way to select more then one to view?

I've modified the TDMS viewer in the past to allow selecting multple channels. It's a little bit of a pain because the tree control in the viewer only allows you to select one item. You have to edit the tree to allow multiple items. Now your selection is an array instead of a scalar. This means the rest of the code has to be updated, and the references to the tree in subVIs need to be updated. Not impossible but it takes some time. Alternativly open it in Excel with the free add-in then you can select the two columns you want and insert a graph. Not as simple for the user but using Excel, which users should be familiar with might mean users need less training on data files.
Unofficial Forum Rules and Guidelines - Hooovahh - LabVIEW Overlord
If 10 out of 10 experts in any field say something is bad, you should probably take their opinion seriously.

How does DIADEM import TDMS files? How gets every channel his number and groupindex? How can I determine which channel has which groupindex and number?

I store different channels in a TDMS file.
I like to have a time channel at the first position with group index 1 and number 1.
When I read the TDMS file with DIADEM the time channel (Float64) is on a differernt position, and the channels are not sorted alphabetically.
Here are my questions:
How does DIADEM import TDMS files?
How gets every channel his number and groupindex?
How can I determine which channel has which groupindex and number?
Best regards
Joerg

Hi Jörg,
i suppose that you´re programme whose create the *.tdms file is writing on false position. Try to create datas with timechannel on first indes in diadem, then save it and then open it again. you see that all is correct. So please tell me what programm in what version do you use and please attache it here.
Did you use the library for creating *.tdms files like in the link ?
http://zone.ni.com/devzone/cda/tut/p/id/6471
Here you find the gtdms_8.x.zip - when you extract it and opened the *.llb you find vi´s for all functions e.g. writing 2d array of strings to *.tdms file
when you open the subvi´s then you see how created and writing datas/structure to *.tdms files. Because *.tdms is binary you can´t see structure with open it in editor.
When you don´t have Labview you can use the 30 days test of current version 8.5 under following link
german version download link
https://lumen.ni.com/nicif/d/lveval/content.xhtml
english version download link
https://lumen.ni.com/nicif/us/lveval/content.xhtml
Hope it helps
Best Regards

.tdm file storage vi's slow editing vi's

I am developing software using Labview 8.5 and have recently added some file storage vi's for the .tdm format for use in DIAdem 10.2. Upon using the open storage, set file properties, write channel group, write channel, and close storage vi's, the editing has become extremely slow - for any operation I try to perform there is a 10-15seconds delay before it occurs. If I remove the .tdm vi's from my diagram then the editing goes back to normal and changes/operations occur almost instantaneously as expected. My project is not large as far as I am concerned (Front panel - 121k, block diagram=1538k, code=0.2k, data=957k, total=2617k, total vi size on disk=389k)
I saw a similar post to the .tdm storage vi's causing this to occur back in 2005 and was supposed to have been on the fix list but it does not appear that it has 3 years later. I am sure many others are using these vi's so not sure what I may have done differently to cause the editing to slow down so much.
Anyone with ideas and fix would be greatly appreciated as I cannot afford to sit around 10-15 seconds for every step of my developement.

Yes, sometimes Storage VIs are slow. But They are Express VIs, supplying the easy configuration for users. If you feel they are too slow, I suggest that you can use TDMS files instead.

Split a TDMS file

Similar Messages

Maybe you are looking for