ASCII vs. TDMS

I have this big packet that claims TDMS files are smaller than ASCII files.
I have a 4x36790 array that i save to an ascii file and tdms file. The ascii file (862 kb) is significantly smaller than the tdms file (1.12 mb).
Can anyone offer an explanation why this NI chart is not supporting my findings?
Message Edited by elset191 on 05-08-2009 03:42 PM
Tim Elsey
LabVIEW 2010, 2012
Certified LabVIEW Architect
Solved!
Go to Solution.
Attachments:
test.JPG ‏11 KB
File size test.vi ‏13 KB
chart.JPG ‏92 KB

What is the magnitude of the numbers you are storing?
If your dataset is relatively small, the amount of overhead the TDMS file format adds for channel names and grouping is greater than the compression you get by storing the data as a binary file. Your data set isn't huge, but isn't all that small either.
However, you are saving doubles which would cost you 8 bytes per array element. With ASCII, you are storing the elements with about 6 or so characters per number since the default setting is 3 decimal places. You have a tab separator, 3 decimal places, the decimal point, and 1 character for each digit on the integer side of the decimal point.
Unfortunately, the VI you posted was missing some subVI's and typedef's and didn't reflect the image you posted. But cleaning it up and running it with a small 4x4 array with just a 3 in the lower right corner (all rest zeroes), I got 107 bytes for ASCII and 616 for TDMS. And the TDMS index file added another 456 bytes. So my test run showed both possibilities, very little data overall so that the extra info in the TDMS format was a greater percentage of the file. And relatively small numbers that took only 6 bytes to show and separate in ASCII as opposed to 8 bytes to store as a double in binary.
By looking at your file sizes, I say each value you are storing is costing about 2 more bytes with binary compared to an ASCII string with 3 decimal places. (147,160 array elements, ~258,000 difference in file size)
One disadvantage of ASCII is that the formatting to text will limit your precision to some amount that you define by the format string, but if you remain in binary, the value will be stored with all the precision/resolution that was present in its native datatype.

Similar Messages

Access files (ascii and TDMS) by blocks, step by step

Whener I perform a data acquisition and save the data I save to files (either ascii or TDMS, depending on acquisiton length, sampling rate, etc).
Afterwards, when I want to access the data, the only way I have managed to is by reading it all at a time. And so, I have to turn to other softwares such as MatLab for postprocessing. And also, if files are bige enough, I sometimes have memory related troubles when reading complete files.
I would like to know how to, because I am sure it can be done, how to do a loop for reading files by blocks, step by step; conrolling file position, block length, etc.
Regards,
usuario

You may want to read Managing Large Data Sets in LabVIEW. It walks through many of the issues you are facing here, although it is a bit dated. After you read that, also read the LabVIEW help on the In Place Element Structure and the Data Value Reference. Approach both with caution, since both break data flow (i.e. can be a potential race condition), but they help when dealing with copies of large data.
Text files can be read incrementally by either line or character. Since the ideal block read size for a Windows file system is about 65,000 bytes, I like to read in my text files 65,000 bytes at a time and doing any parsing from the memory buffer. When the buffer runs out, I get another 65,000 bytes until the file is done. <shamelessAdvertising>This method will be the subject of my next post on object-oriented file I/O, which will, unfortunately, be about two to three weeks from now, due to my "real job" getting in the way .</shamelessAdvertising>. If you are in a hurry and enjoy complex code, the Read From Measurement File Express VI in LVM mode also uses this method. If you open the front panel of one and dive into the code, you can see it in action.
Good luck! Let us know if you need more help.
This account is no longer active. Contact ShadesOfGray for current posts and information.

ASCII to tdms file converter vi?

I have a data file that is in ascii format. I need to convert it to TDMS format to read into another vi.
Thanks.
D. Gray

Hi DGray,
you can read your data with "Read From Spreadsheet File.vi" and write it with the tdms functions.
Mike

How to convert tdms to ascii files?

Hello,
I need to convert *.tdms files to an ascii file. I have 16 channels and scalling coefficient information in the tdms file, With the TDMS-Image Viewer i can see the 1-D array waveform perfectly, but I cant write it to an ascii file. When I use a signalexpress.vi or the spreetsheat.vi for writing an LVM/ascii file, information gets lost. (Red symbol at INPUT!!!!)
Please answer me, because I dont have lots of experience with labview and an ascii file is more easier to handle for me! I found only export to matlab and exel, but i just want it in an ascii file. Reading the tdms and opening it is no problem, but connecting the right blocks to the 1D array waveform without dataloss doesnt work.
thank you for reading
danny

Thank you for reply,
i have the new version 8.5. With signalexpress.vis i can do asciis but the red symbol (at input) is there so i dont want to get information lost.
The tdms file has header information for scalling factors and unscalled data for 16 channels.
so because tdms files are faster for writing i want to convert them to ascii with a Convertion Vi, but it seems that with an express vi he stops because it cant handle handle with the headerinformation.
now i use the signalexess vi to write tdms data, before i did it manully by setting channel and group information.
Please tell me if the red symbol AT input of a Signalexpress VI does matter? its a 1D array wavefrom that goes in to the input.
thank you for this discussion, i hope that i dont need to use the matlab import option for tdms!!
greetings danny

How to store a file path in a TDMS Property

I need to store some Windows paths to other files as TDMS properties, but (according to another forum post I now can't find), the following characters are all illegal in TDMS properties: . : , ; ' \ @ < > # [ ] % ( ) { } | * ? = ! "
Many of those are used in paths. I thought of converting each of them to its ASCII character code like an HTTP URL, but the '%' character is in the list, too. Can anyone recommend an approach to handling this situation?
Solved!
Go to Solution.

I actually did something I almost never do. I searched DIAdems online help. I found a page on name conventions for properties on TDMS files which states the following:
"If you use one of the following characters in a custom property name, DIAdem replaces this character with an underscore: .,:, ,, ;, ', \, @, <, >, #, [, ], %, (, ), {, }, |, *, ?, =, !, ", ^, $, &, +, -, /, and a space. DIAdem also replaces a leading number in the property name with an underscore."
It however does not state anywhere that I've found that you can't use these symbols in a text as part of the property data.
Best Regards
David

Insert time stamp in ASCII file

Hi all!
I am trying to log data values from several channels (so far 4) into one ASCII file including a time stamp in the first column.
The goal is one file that contains one time stamp per row (in the first column) and one column per channel. It may looks like this:
TIME Ch1 Ch2 Ch3 Ch4
t1        a1     b1     c1    d1
t2        a2     b2     c2    d2
I want to save one row (time stamp and 4 channels) each minute as above into the same file. Nice would be to creat a new file each day.
Problems:
1. Using the "Save to ASCII/LVM" step: How can I get SignalExpress to insert a time stamp in the first column of each row?
    I have no problem inserting the 4 signals that generate me one column per signal. And using "Overwrite once, then append to file" I get one file for all signals.
    Any idea how to insert a time stamp this way? Or how to creat a new file every 24 hours?
2. Another way I tried is to use the "Recording Option". But how can I get all the four signals saved in one file?
    After the DAQmx acquisition step I process the 4 channels individually (one scaling step and one threshold step per channel - because of different factors per channel).
    Activating the Recording Option (which would generate a new file every 24 hours, but not starting at 12pm each day...) I get 4 single TDMS-files (and their corresponding tdms_index and meta.txt files),
    one file per signal I am recording. I assume this is because they are 4 individual signals I'm recording and not a group of signals. Can I group individual signals together?
Thanks for your help
SSC
PS: I work with SignalExpress 2.5 and have a NI USB-6210 connected.

In the recording options tab you can set a stop condition to be 24 hours of data. And then select the "Restart Log Automatically" option to start logging for another 24 hours. You can also configure the logging to auto create an ASCII file once logging stops. With this configuration an ASCII file will be created every 24 hours. To enable ACSII file generation, select Tools>>Options.. And then set "Automatically export log to ASCII file to true".
I believe that you will get one ASCII file for each TDMS file. If you are logging 4 channels from the same device you should be able to log the entire group which would allow you to get only 1 TDMS / ASCII file.

Recording time on the Save to ASCII%2FLVM step

To Ni application engineers:
I have Signal Express 3.0, and I would like to find out if there are ways to put real time stamp (or relative time based on DAQ rate).   We would like the time stamp to be in the 1st column in Excel along with the mean statistical data on the right side of it.   (See attached project file for detail)
Time |   Load  | Stroke  | Strain 1 | Strain 2 |     .....and so on
Our clients been constantly asking us about this, and we just not able to give them a good answer due to the fact that signal express 3.0 doesn't give such option.  This is quite urgent, and I hope you guys can help.
If the newer version of signal express can do such task, we would be interested in an upgrade.   I would assume the newer version will run on Windows7 Professional 64bit.
Thanks in advance for this matter.
Kin Chu
Integrated Technologies, Inc
Attachments:
sample.seproj ‏1204 KB

Hi Kin,
Unfortunately SignalExpress does not have the capability to export the real time to Excel. Using Xo and DeltaX to calculate the absolute timestamp in Excel is likely the best option.
Additionally, you can create a 'Save to ASCII/LVM' step that will log the data to a TDMS file and you can save the time as relative or absolute. However, the absolute time is given in the number of seconds elapsed since January 1st, 1904 (as explained here: http://digital.ni.com/public.nsf/allkb/69CABEE9F0BA9A6586256B41006B0BBB?OpenDocument).
-Jake B.

DIAdem 2012 displaying all channels as "Noname" for .tdm files

Hi
I'm experiencing a problem with DIAdem 2012 when opening .tdm files.
I've reinstalled all NI software components, and still the problem remains.
All channels are displayed as "Noname", "Noname_1" etc.
Even worse, all values are displayed as either "0" or "NOVALUE".
See attached image for example.
The attached image is displaying the "Example.tdm" file which is opened by default when starting DIAdem.
When opening the "Example.csv" file the problem is not there (channels have names such as "Time", "Velocity" and values).
So what could be the problem related to reading/loading .tdm files? (Note, as I said, I just reinstalled DIAdem from scratch)
Attachments:
DIAdem_noname.jpg ‏270 KB

Hi salte,
"TDM_DataPlugin" is the default name that the ASCII DataPlugin Wizard would give to a DataPlugin you or someone else created to open files with a "*.tdm" file extension. I'm betting that's where it came from-- the behavior you see in the Data Portal is also consistent with loading a TDM file as if it were an ASCII file. DIAdem does not install this DataPlugin, and I highly recommend that you remove it.
To remove the DataPlugin, go to the DataPlugin dialog, select the the "TDM_DataPlugin" in the list, and hit the red "X" icon or the <Delete> key on your keyboard. To get to this dialog first start in either the NAVIGATOR or SCRIPT panel, then depending on your DIAdem version select one of the following two menu paths:
"Settings >> Options >> Extensions >> DataPlugins..."
"Settings >> Extensions >> DataPlugins..."
Once you've removed the "TDM_DataPlugin", you will again be able to drag TDM files from the NAVIGATOR tree view into the Data Portal.
Brad Turpin
DIAdem Product Support Engineer
National Instruments

Buffered TDMS file still very large

Hello, I am trying to to stream to a TDMS file. I am writing approximately 10 channels of data at 20 Hz. I was creating a new file evey 5 mins or so and defragging the file once I was finished with it, which was giving me a reasonable size of file. I decided that I would like to make bigger files (over the period of hours) and the the defragging would be too computationally costly to do on the fly. I have been trying to set a buffer for the TDMS files but the resultant files are very large. You can tell the file isn't being writen until its closed (from watching its size in windows explorer) which suggests the file is being buffered. However its about 6 times bigger than an ASCII file of the same data, with and index almost as big again.
Does anyone have any ideas why it would appear to buffer but not actually reduce the data size. I'm running Windows XP, LabVIEW 10.0.
When I create each new file I run this to set the buffers. (I have also hard wired in a buffersize of 100000 on a one minute file to no avail.
Niall
Thanks
Niall

I has this marked as the solution but it turns out it only helped one of my programs. I am still getting very large (fragmented) TDMS and index files for another program. The problem is that if I just defrag the (supposedly buffered) file, it interupts the data logging because it takes so long. I'm 99% sure that it *is* buffereing as it doesn't write till it closes the file, and if I use the read properties function it reads back a set buffer size. Here is the VI that actually writes the data. There bit at the top is for writing an optional acii file so you can ignore that.
Its maybe a bit hard to see whats going on in the next one, but this is where the file is created before being passed to the setbuffer VI which I posted earlier. It also closes that last file
Its really hacking me off now and holding me up from going no to other stuff. It would be great if someone had some ideas.
Thanks
Niall

TDMS Timestamp Decoding - Help!!

I am in need of help with decoding the timestamp in TDMS files. Specifically in the properties section. The following (very vague documentation) provides limited details:
http://zone.ni.com/devzone/cda/tut/p/id/5696
Specifically looking for bit definition of the "tdstimestamp" type. As an example I extract the following hex values from a TDMS file:
44 00 00 00 EA D8 C5 AC 4F 9E 56 BC C2 2F F3 C3 00 00 00 00
I know that the first four bytes (44 00 00 00) signify the timestamp datatype, but how do I decode the remaining bytes. The approximate time of above is 11:55:41AM on March 4, 2008.
Thanks for any help!

Matt - Thanks, that works great!!
Dave, I see that the MATLAB pluggin was just updated to include MATLAB 2008b. We take data in our labratory using the TDMS format. When we distribute this to other researchers, they often use MATLAB (and of course Labview).
For about a year now, the Matlab pluggin has been out of date and would not work with the last 3 releases of MATLAB. Since it is a compiled C library, we're stuck and can not modify it. As the internal TDMS format is not changing and only low level matlab commands are required to read it, I believe that Matlab users have been looking for a simple Matlab function to just take a tdms file and make a similar .mat file (not a dll). This is what I have done (and thanks to Matt for the timestamp help).
The Excel TDMS add-in is very nice and is a good reason to use the TDMS format (in addition to its good support in VI libraries), but I often have to fend off requests to write large ASCII or flat binary files (in addition to our TDMS files) so that they can be read in MATLAB. I don't want to have the burden of making/distributing a LV executable that makes this interim file format and then creates a .mat file.

Convert tdms to binary

How to convert large size ( > 500 MB) TDMS files to .binary or ascii..
i found so many examples to do this using labview.. but my requirement is if there is no labview installed on my system then how can i convert and analyze the data from TDMS files..

Make an executable and an installer out of the one that suits you! Then you can use it without LabVIEW, unless I'm missing something obvious!
Also run a defrag on your file it helps.
Beginner? Try LabVIEW Basics
Sharing bits of code? Try Snippets or LAVA Code Capture Tool
Have you tried Quick Drop?, Visit QD Community.

Multiple TDMS Questions

I haev a few questions on TDMS.
1. Is it true that TDMS file is a LabVIEW specific binary, and you can't read it without using LabVIEW?
2. I was able to read the TDMS file in excel. Howcome I can do that? Is excel supposed to be able to read TDMS?
3. Is it true that TDMS is smaller than a regular ASCII file, given the same data? It seems so.
4. If 3 is true, I don't understand why, since I was able to look at the raw data of the TDMS directly in excel, i assume that the data is stored as ASCII and not binary inside the TDMS file.
5. It seeems like TDMS comes in pair, a tdms file and a tdms_index file. Can the tdms be accessed with only the tdms file without the tdms_index and vice versa?
Thanks!
Kudos and Accepted as Solution are welcome!

And I suppose it should be mentioned that there are a few CAR's pertaining to some of the behavior of if you use the Advanced TDMS functions (which you only really need to use if you need to overwrite or replace data in an existing TDMS file).
And if you acquire data one point at a time, make sure you don't put the points in one at the time in the TDMS file as you will create a huge file consisting mostly of duplicated header information (which you can then use the TDMS defrag to fix). It can be better to collect the points in a software/memory array and only push them to the TDMS file once every xx samples.
I find that I enjoy working with TDMS files for my project, but of course my data is awesome for TDMS too, I get my data in "traces" so each logical unit (channel) comes pre-packaged as an array of thusands of points, so my TDMS files tend to be very "efficient".
Here is a copy paste of one of the Advanced issues I ran into:
I have done some more research on the first issue you reported. (Error:
-2529 in [Adv] [TDMS Set Channel Information])
Issue/Feature A) Flush required after channel property writes:
Create some file level properties and write and over-write them.
Create a group and add and overwrite some properties Create a channel in the
group, add a property then try to do a synchronous write. You will get an
error message. Either do a flush between the property write OR write the
channel data first then do the channel property writes.
As it turns out, there is already a Corrective Action Report (CAR) filed
with the LabVIEW Development Team. Here is the CAR#310023 You can look for
this number in the future releases of LabVIEW.
QFang
CLD LabVIEW 7.1 to 2013

Tdms vs cluster where when speed doesn't matter

hello,
The question is in the title: in case speed doesn't matter wich could be advantage using one or other of methode.
I 'm working on calibration bench at low speed ( T° of water , humidity, barometer) so hdd access is not a parameter.
i have to store , author, date ~20 data information like that an then test data ( ~10 * 2Darray(~10*10) ).
I'm new to this kind of programming . I think both are possibles , but i would like to start knowing what can be the
best before involving to much in one method before realizing the other was better adapted.
What are avantage disavantages like easy of coding, scalability, ...
Best regards
Tinnitus
CLAD / Labview 2011, Win Xp
Mission d'une semaine- à plusieurs mois laissez moi un MP...
RP et Midi-pyrénées .Km+++ si possibilité de télétravail
Kudos always accepted / Les petits clicks jaunes sont toujours appréciés
Don't forget to valid a good answer / pensez à valider une réponse correcte

For scalability and future expansion, I would strongly recommend you save your data as discrete units, not clusters. Be sure to include a file format version number. This will allow you to change the format in the future and still read old files. More importantly, it will allow old code to fail gracefully (if necessary) when attempting to read newer, different format, files. Make sure you take this into account in your design. Some options in LabVIEW:
TDMS - would be perfect for you, except it does not support 2D arrays. In your case, the workaround (storing multiple 1D arrays) is not very onerous. This is probably your first choice. It can be easily converted to ASCII and/or read by Excel if needed.
Config File - this API set would also work fairly well for you. It is text based, so be careful of your formatting for any numbers you save. It is easy to lose resolution and get into problems. Once, again, arrays are not natively handled. You can work around this easily enough by converting the arrays to strings before writing. Again, you will probably want to create a key for each line of the array, since extra CR/LFs will cause problems when reading.
HDF5 - this is a far more difficult method, but avoids the issues of the previous two. You can get the API here (a Google search will give you more LabVIEW APIs, this one is rather old). HDF5 is difficult to learn but highly flexible and can be read by most commercial analysis packages.
Good luck. Let us know if you need more information.
This account is no longer active. Contact ShadesOfGray for current posts and information.

Fehlermeldung beim Laden eigener TDM-Dateien

Beim Laden selbst abgespeicherter TDM-Dateien bekomme ich folgende Fehlermeldung: Das Laden der Datei " ***** " mit dem Ladeverfahren "TDM" ist fehlgeschlagen. Inhalt der LOG-Datei: C:\Daten\Terex RH70\RH70_Beschl\0609XX_RH70_Beschl_2.tdm(338) : fatal error xercesc:AXParseException 15 : Invalid character (Unicode: 0x1C)
fatal error (42) :Errors while processing the model
Speichere ich meine Dateien als DAT-Dateien ab, kann ich sie öffnen.
Ich benutze DIADem 9.1

<meta http-equiv="CONTENT-TYPE" content="text/html; charset=utf-8"><meta name="GENERATOR" content="OpenOffice.org 2.0 (Linux)"><meta name="AUTHOR" content="Andreas Krantz"><meta name="CREATED" content="20061122;10474500"><meta name="CHANGED" content="16010101;0">

Es handelt sich hier um einen Effekt
der auftritt weil in einem Namen oder einem Attribut ein Control
Zeichen Auftritt. In DIAdem 10 ist dieses Verhalten abgestellt und
kann nicht mehr auftreten. Die Datei ist aber nicht verloren. Man
kann sie einfach in einem Texteditor öffnen. In die angegeben
Zeile springen und das Zeichen (Es sollte relativ offensichtlich sein
welches) durch ein beliebiges anders (z.B. ?) zu ersetzen. Anschliessend kann die Datei wieder
gelesen werden.
Hier konkret:
Datei: 0609XX_RH70_Beschl_2.tdm
Zeile:338
Technisches zu XML:
Als XML definiert wurde, hat das
Standardisierungskonsortium einige Zeichen Ausgeschlossen. Das waren
die, die in der frühen Computergeschichte als Steuerzeichen
(Control-Characters) bekannt waren. Ein recht berühmtes ist wohl
BELL (0x07) oder ESC (0x1B). Diese Zeichen wurden nicht benutzt um
Texte oder Inhalte darzustellen sondern um das Terminal zu steuern.
Es handelt sich hier um die Zeichen die
in einer ASCII Tabelle die ersten 30 bilden. also von 0x00 bis 0x1F.
Allerdings gibt es bei den Steuerzeichen Ausnahmen die zum
formatieren von Text benötigt werden. Der XML Standard definiert
3 davon.
Tab Stop: 0x9
Wagenrücklauf(CR) 0xd
Zeilenvorschub(LF) 0xa
Wer bei solchen Worten an die gute alte
Schreibmaschine denkt liegt nicht ganz falsch. So kam es, dass die ersten 30 Zeichen
der guten alten ASCII tabelle in XML nicht darstellbar sind. Sie
lassen sich auch nicht in anderer Form schreiben. Ein XML Parser ist
dazu angehalten solche Dateien die besagte Zeichen enthalten als
ungültig zu markieren.

TDMS problem with spikes

Hello guys,
At first, I'm a bloody beginner using SignalExpress and DIAdem :-/
I recorded some data (signals of two acceleration sensors) using SignalExpress 2010.
Now I want do open them with DIAdem for post processing ... reading the TDMS files works but the data seems to be corrupted somehow.
There are spikes (>1E+300) in the signal sequence which shouldn't be there (I don't see them when I open the file in SE2010).
Please see the attatched screenshot (it explains the problem better)
Greetings
Marius
Attachments:
20120310_Signal_Sequence.jpg ‏74 KB

Hello YongqingYe,
OS language is German (Windows XP), SE2010 language, too.
I only used standard ASCII characters as far as i know.
At the moment I dont't have the chance to start a new measurement.
I'm looking for the device name (portable NI device, I noted it down somewere).
I attached the meta.txt file thats created while using SE.
Greetings
Marius
Attachments:
Beschleunigung_meta.txt ‏1 KB

ASCII vs. TDMS

Similar Messages

Maybe you are looking for