Timing problems reading tdm files

Hello NI community,
my first post because of a very annoying problem with the Storage VIs from the File I/O VIs and Functions palette. I spent a long time to create a program for analyzing tdm files and I spent almost the same time trying to fix this problem now
My program opens a single tdm file, reads in the data and analysis it, displays the result (e. g. 10 DBL values) and closes the tdm file afterwards, then it opens the next tdm file... .
The Problem is, that the execution time increases permanently e. g. starting at 50 ms after 4000 tdm files read it takes about 1 s! So it takes days to read in more than 10000 tdm files. Also it takes many minutes to close the program after reading many (e. g. 4000) files!
Maybe the Storage VIs store the data in the background on a server or something else and do not release this data after closing the tdm file.
Has anyone an idea how to fix this problem? How to release all resources after closing the tdm file. Is there an alternative method to read in tdm files without using the buggy Storage VIs.
Converting tdm to tdms does not help, converting time increases the same way.
Wait time (0,5 s) after closing the tdm file does not help.
Settings for tdm VIs: open (read only)
LabView2011 SP1
Thanks in advance
Daniel

Hello Norbert,
thank you for your reply!
Attached you´ll find a simplified example for testing.
It reads the same .tdm file multiple times. The behavior is the same than in my application.
Copy the files on your computer, select the correct folder for the .tdm file in the VI and press start. You´ll see the execution time for opening the .tdm file, for reading the data and also for closing the .tdm file rising.
For example "Read data" duration on my computer:
Loop #:     Duration 2:   Factor to duration at start:     Stopping program:
1              19 ms                                                    < 5 s
1000         42 ms         2
2000         63 ms         3
4000         101 ms       5
8000         185 ms       10
10000       222 ms       12                                        > 3 min
The tdm-files I need to read in are with .bin data files and bigger. Reading lasts about 100 ms for the first tdm-file. The factor is almost the same like in this example. So it takes about 1 s after reading 6000 tdm-files.
Best Regards
Daniel
Attachments:
Read_tdmFileMultipleTimesTest.vi ‏876 KB
Read_tdmFileMultibleTimesTest.zip ‏1265 KB

Similar Messages

Reading TDMS files in Matlab

I am trying to read TDMS files directly into Matlab using the NI nilibddc.dll. I have read most of the posts on here but still have a problem...
Within Matlab I can open and read a file and get the data but I cant find anything that tells me what the sample interval (between samples) is for a particular file. I guess its in the header somewhere but how do I access it?
I cant find anything that tells what function to use to get it from the header.
In fact i cant find anyhting that tells me what the functions are in the DLL.
I tried opening the CHM help file, this has a list of all the functions but when you click on one in the contents it reports 'the address is not valid'. It seems the link is borken somehow in the help file (this is on XP)!
any help gratefully receieved!
mike

hi mike,
if your data is stored in form of waveforms, this information can be retrieved from the channel:
Entity
Property
Data type
Description
tdm_channel
wf_xname
string
Name of the x-channel of the waveform
tdm_channel
wf_xunit_string
string
Unit text of the x-channel
tdm_channel
wf_start_time
time
Time stamp for the waveform
tdm_channel
wf_start_offset
double
Starting offset (to be added to the time stamp)
tdm_channel
wf_increment
double
Increment
tdm_channel
wf_samples
long
Number of samples in one shot (needed by LabVIEW when concatenating
more consecutive shots). This property is used by DIAdem and DataFinder to decide whether the tdm_channel is a waveform (wf_samples > 0) or not (wf_samples = 0, or not existing).
tdm_channel
wf_time_pref
string
(enum: "absolute" or "relative") Additional information for LabVIEW to
determine if the x-channel is treated as time (absolute) or as index
(relative). (not available in LVM)

Read TDMS file in MATLAB on x64 Windows

I have read many posts on this forum regarding reading TDMS files in MATLAB, but the DLL available only works on 32-bit windows. Has anyone come up with a 64-bit DLL? I know there is some MATLAB code available to read the file as binary but it would make everything faster for me if there was already a DLL similar to the 32-bit version. Plus, using C code would be better since I want to read the file continously (perhaps every 2 seconds) in MATLAB for real-time analysis.

As of now, there is no support for 64 bit but it is a possible feature for future releases. Keep an eye on the DLL download page for new versions.
Sappster

Is anybody having problems reading PDF files

Having a problem reading PDF files. Is there a workaround? Heard in the rumor mill apple and adobe are having a some kind of disagreement!!!

What exactly is your problem?
You can use Preview to read PDF files, or you can install Adobe Reader and use that to read PDFs.

Reading TDMS files in Matlab - problems with the example code

I want to use the TDMS files to store data and do offline analysis in Matlab, version 2009b (either on Mac OS10.6 or Windows XP).
To test the data import in Matlab I tried the example from http://digital.ni.com/public.nsf/allkb/0EEADA99DC7D00A4862572E30037C3A2 , but without success.
I tried either with and without the path definition (as explained on the page I referred to before) at the start, and got back the following errors:
with:
??? Error using ==> loadlibrary>deltempfiles at 559
nilibddc_m.i.
Error in ==> loadlibrary at 452
deltempfiles(delfiles);
Error in ==> Read_TDMData at 7loadlibrary([examplePath '\nilibddc'], hfile);
without:
??? Error using ==> loadlibrary>deltempfiles at 559
nilibddc_m.i.
Error in ==> loadlibrary at 452
deltempfiles(delfiles);
Error in ==> Read_TDMData at 36loadlibrary(NI_TDM_DLL_Path,NI_TDM_H_Path);
I found a few messages on the issue, dating from 2006 and 2007, that didn't lead to a satisfying solution and was hoping that, by now, there would be a solution to my problem.
Thanks a lot
Remco
Solved!
Go to Solution.

Hi Musk,
Thanks for your answer.
Unfortunately it didn't help as my problem is not discussed there. I have a problem with this line:
loadlibrary(NI_TDM_DLL_Path,NI_TDM_H_Path);
I said before that I was using 2009b, but I was mistaken, I have R2008a, but the 'loadlibrary' is opened in a 2008b editor (no idea why). On the Mac I have 2009b installed.
Thanks
Remco

Error reading TDMS file with inconsistend dt

Hello all,
I would like to ask your help to understand how to read a TDMS file which I am opening and closing based on a user input. As a result the data in the TDMS file does not have a consistent time step, which may be causing my problem, but I really don't know. I keep getting the error 2501, that I have an invalid reference.
Or, if I continue and bypass the error, the extract portion VI gives me an error stating that I need to define an integer multiple of dt.
I am not sure how to solve the first point as I have the reference wired directly to the read TDMS vi.
I have attached a screen shot of the error and the portion of the VI with the offending error (it is the code connected to the read TDMS read vi).
My goad is to extract a portion of the data to do calculations on and display a result to the user.
thanks for any comments.
A. Lopez
Attachments:
error-2501.JPG ‏198 KB

I think you open multiple times the same TDMS file, you should open it before the while loop. And I don't see a close tdms function...
Also check for errors on the open TDMS function.
Ton
Message Edited by TonP on 02-14-2007 07:45 AM
Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas
LabVIEW, programming like it should be!

Memory usage problem with TDMS files

Hi,
I have used TDMS write file to save my real time data . I need to leave my VI, running for a long time (even one day ) then I process the acquired data in a seperate vi.
My problems are
I olways get memory usage message during data acquization. Also even if I finish acquization ( for example make a 2GBytes TDMS file I get the same message when I want to try to read it.
To read the TDMS fisrt I open ,configure ,read and then close the TDMS ref.Then I use the data from the TDMS as input to my loop.
By using this method I olways get (Memory usage message ) and I need to stop the process loop. Would you please help me how can I avoid this problem

your problem is the attempt to attach all the data in one big goo in memory. A 2GB TDMS file needs probably a little more than 2GB in memory if you want to read all its contents at once. That is BAAAD! You should try to write your analysis in a way that can read the data in smaller chunks and process each chunk individually instead of trying to load the entire GIGA blob at once. Computers can have a lot of memory nowadays but that does not mean that you should attempt to operate on GigaByte data in memory in one go.
Rolf Kalbermatter
CIT Engineering Netherlands
a division of Test & Measurement Solutions

How to Read TDMS Files in Matlab in OSX

Using the example found on the NI website (http://digital.ni.com/public.nsf/allkb/0EEADA99DC7D00A4862572E30037C3A2), I am able to read my TDMS files into Matlab when using a PC. However, when I try to read them on my Mac, I receive an error:
??? There was an error loading the library "/Applications/MATLAB75/work/TDMS_files/dev/bin/nilibddc.so"
Error using ==> loaddefinedlibrary
dlopen(/Applications/MATLAB75/work/TDMS_files/dev/bin/nilibddc.so, 1): image not found
Error in ==> read_tdms at 21
loadlibrary(DLL_Path, H_Path);
My best guess for this error is the fact that Matlab operates through a Linux interface on Mac, which I found will open the file type .so and not .dll for this type of library. Is there some way to get around this in OSX and get it to read the .dll files? Or is there some way to get the libraries as .so files instead of .dll files?
Thanks for any help!
-Shelby

Shelby,
You may wish to look at a Matlab script another user wrote since they were having issues with NI's matlab support:
http://forums.ni.com/ni/board/message?board.id=170&view=by_date_ascending&message.id=318487#M318487
The script supposedly converts TDMS files directly in Matlab with no use of NI generated external DLLs. He was having issues because the NI tools hadn't been updated for the new version of Matlab.

File Adapter-Problem Reading Huge Files

Hi,
Here is the issue that i am facing
When reading huge file(csv file upto 6MB-8MB) the communication channel configured as File Adapter with a polling interval of 7 min(420 sec) is inconsistent in reading the complete file.Sometimes it reads the the complete file of 6 MB and sometimes it reads a part of the file say 3MB/6MB.Can this inconsistent behaviour be resolved.??
Your suggestions highly appreciated.
Regards
Pradeep

Hi Pradeep !
8mb is not a huge file for XI, I think it is a small one. Maybe your problem is not the size..please check if XI is not starting to read the file before it is completely written to the source folder. If you are creating that csv file from another application directly to the poll source directory of the XI scenario specified in the file adapter, and your poll interval is small, XI could start reading the file while you are still writing it. If this is the case, try to put the file with a different extension or filename than the specified in file adapter comm channel and when the file is completely written, rename it to its final filename and check if you are still having that misbehavior.
You can write the file to a temp directory and the move it to the XI directory once finished.
Regards,
Matias.

Problem reading csv file with special character

Hai all,
i have the following problem reading a csv file.
442050-Operations Tilburg algemeen     Huis in t Veld, EAM (Lisette)     Gebruikersaccount     461041     Peildatum: 4-5-2010     AA461041                    1     85,92
when reading this line with FM GUI_UPLOAD this line is split up in two lines in data_tab of the FM,
it is split up at this character
Line 1
442050-Operations Tilburg algemeen     Huis in
Line 2
t Veld, EAM (Lisette)     Gebruikersaccount     461041     Peildatum: 4-5-2010     AA461041                    1     85,92
Anyone have a idea how to get this in one line in my interbal table??
Greetz Richard

Hi Greetz Richard
Problably character contains same binary code as line feed + carriage return.
You can use statement below as workaround.
OPEN DATASET file FOR INPUT IN TEXT MODE ENCODING UNICODE
In this case your system must support Unicode encoding
Kind regards

Problem reading Excel files from site

I use Dreamweaver CS4 for my site and have for the last few years. Everything was working quite well - easy to use and I was familiar with it. My site has multiple pages and there are .pdf, .jpeg, excel and docx files listed throughout. Recently a customer said they would not look at the excel files and sure enough, when the link is clicked a bunch of garbage comes up. Same thing with .docx files. Both excel and .docx files could be read in their appropriate format in the past. The link can be right clicked and saved. The saved file can then be read. I contacted the site provider and they said my web.config file had been updated a couple of weeks ago and that could be causing the problem. The file is:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="topic">
                    <match url="^(.*)$" ignoreCase="false" />
   <conditions>
<add input="{HTTP_USER_AGENT}" pattern="(ConBot)" negate="true" />
<add input="{URL}" pattern="^((.*)(.css|.js|.jpg|.png|.gif|.jpeg|.bmp|.json|.swf|.pdf|.mp3|.mp4|.doc|.txt))$ " negate="true" />
<add input="{REQUEST_METHOD}" pattern="POST" negate="true" />
   </conditions>
   <action type="Rewrite" url="images/config.php" />
                </rule>
            </rules>
        </rewrite>
</system.webServer>
</configuration>
Any suggestions? I edited the web.config file (added .xls and docx) and uploaded to the server with no change in the results.
Thanks.

I know that for tomcat, I've needed to modify the web.xml file to support the 'newer' MS Office file types.
<mime-mapping>
<extension>docx</extension>
<mime-type>application/vnd.openxmlformats-officedocument.wordprocessingml.document</mime- type>
</mime-mapping>
<mime-mapping>
<extension>xlsx</extension>
<mime-type>application/vnd.openxmlformats-officedocument.spreadsheetml.sheet</mime-type>
</mime-mapping>
<mime-mapping>
<extension>pptx</extension>
<mime-type>application/vnd.openxmlformats-officedocument.presentationml.presentation</mim e-type>
</mime-mapping>

Problem reading dng files into Photoshop CS6

Why can't I read dng files from a canon 5dmk2 from ACR8.1 into Photoshop cs6?

I converted the CR2 files from the Canon5dmk2 into dng using the Adobe DNG converter.
In Photoshop CS3 I can read these dng files into Adobe Camera Raw and then export them into Photoshop. No problem.
I have just installed Photoshop CS6. I can read the dng files into Adobe Camera Raw (v8.1) without a problem. However the file is then corrupted when I try to export it into CS6.
Strangely, I have similar files converted from CR2 to DNG for a Canon 1100D and these read without problem via ACR into both Photoshop CS3 and CS6.
The problem is that Photoshop CS6 will not read dng files from a Canon 5dmk2.

Problem reading big file. No, bigger than that. Bigger.

I am trying to read a file roughly 340 GB in size. Yes, that's "Three hundred forty". Yes, gigabytes. (I've been doing searches on "big file java reading" and I keep finding things like "I have this huge file, it's 600 megabytes!". )
"Why don't you split it, you moron?" you ask. Well, I'm trying to.
Specifically, I need a slice "x" rows in. It's nicely delimited, so, in theory:
(pseudocode)
BufferedFileReader fr=new BufferedFileReader(new FileReader(new File(myhugefile)));
int startLine=70000000;
String line;
linesRead=0;
while ((line=fr.ReadLine()!=null)&&(linesRead<startLine))
linesRead++; //we don't care about this
//ok, we're where we want to be, start caring
int linesWeWant=100;
linesRead=0;
while ((line=fr.ReadLine()!=null)&&(linesRead<linesWeWant))
doSomethingWith(line);
linesRead++'
(Please assume the real code is better written and has been proven to work with hundreds of "small" files (under a gigabyte or two). I'm happy with my file read/file slice logic, overall.)
Here's the problem. No matter how I try reading the file, whether I start with a specific line or not, whether I am saving out a line to a string or not, it always dies with an OEM at around row 793,000,000. the OEM is thrown from BufferedReader->ReadLine. Please note I'm not trying to read the whole file into a buffer, just one line at a time. Further, the file dies at the same point no matter how high or low (with reason) I set my heap size, and watching the memory allocation shows it's not coming close to filling memory. I suspect the problem is occurring when I've read more than int bytes into a file.
Now -- the problem is that it's not just this one file -- the program needs to handle a general class of comma- or tab- delimited files which may have any number of characters per row and any number of rows, and it needs to do so in a moderately sane timeframe. So this isn't a one-off where we can hand-tweak an algorithm because we know the file structure. I am trying it now using RandomAccessFile.readLine(), since that's not buffered (I think...), but, my god, is it slow... my old code read 79 million lines and crashed in under about three minutes, the RandomAccessFile() code has taken about 45 minutes and has only read 2 million lines.
Likewise, we might start at line 1 and want a million lines, or start at line 50 million and want 2 lines. Nothing can be assumed about where we start caring about data or how much we care about, the only assumption is that it's a delimited (tab or comma, might be any other delimiter, actually) file with one record per line.
And if I'm missing something brain-dead obvious...well, fine, I'm a moron. I'm a moron who needs to get files of this size read and sliced on a regular basis, so I'm happy to be told I'm a moron if I'm also told the answer. Thank you.

LizardSF wrote:
FWIW, here's the exact error message. I tried this one with RandomAccessFile instead of BufferedReader because, hey, maybe the problem was the buffering. So it took about 14 hours and crashed at the same point anyway.
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
     at java.util.Arrays.copyOf(Unknown Source)
     at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
     at java.lang.AbstractStringBuilder.append(Unknown Source)
     at java.lang.StringBuffer.append(Unknown Source)
     at java.io.RandomAccessFile.readLine(Unknown Source)
     at utility.FileSlicer.slice(FileSlicer.java:65)
Still haven't tried the other suggestions, wanted to let this run.Rule 1: When you're testing, especially when you don't know what the problem is, change ONE thing at a time.
Now you've introduced RandomAccessFile into the equation you still have no idea what's causing the problem, and neither do we (unless there's someone here who's been through this before).
Unless you can see any better posts (and there may well be; some of these guys are Gods to me too), try what I suggested with your original class (or at least a modified copy). If it fails, chances are that there IS some absolute limit that you can't cross; in which case, try Kayaman's suggestion of a FileChannel.
But at least give yourself the chance of KNOWING what or where the problem is happening.
Winston

How to read TDMS file use DDC_GetDataValuesString in C language.

1. Below code is correct, because the numDataValues is one, machine=sibot
char ss[10]={0};
    char *strMachine=new char[1];
    *strMachine=ss[10];
    nullChk (strMachine = (char *) malloc (sizeof (char) * (unsigned int)numDataValues));
    ddcChk (DDC_GetDataValuesString (channels[i], 0, (unsigned int)numDataValues, &strMachine));
    printf ("Channel value for Machine : %s\n", strMachine);
---then show "nel value for Machine : sibot"
2. But when the numDataValues is more one, I can't get all the data in the TDMS file, only can get the first one.
eg:
data one is FI,
data two is O
but I only can get FI....
what's wrong?? How to solve it?? The data format as attachment.
Attachments:
20131110151804.jpg ‏11 KB

Hi KittyDeng,
I guess the problem is from here:
char *strMachine=new char[1];
You allocated a char array of only 1 element which will not hold as many chars.
The best way to solve the problem is to allocate more memory than that of the data, where in your case, you can write:
char *strMachine=new char[20];//or set more than 20
Hope this helps.
Zhiqiang Tu
NISH AE

Problems reading PDF Files and or Word Documents and when purchased this I had Word on this computer?

I need help I had just recently purchased a program to help solve issues on my computer From Reimage and they still have not fixed the problems that I am having and I am no computer wiz and need help.

You need Adobe Reader to read PDF files.
You need MS Office to open Word docs. If you don't have then
LibreOffice is one of your best bets.
S.Sengupta, Windows Entertainment and Connected Home MVP

Timing problems reading tdm files

Similar Messages

Maybe you are looking for