How to concatenate TDM files in DiaDem

I've seen the post on how to concatenate CSV files on DiaDem ... but how do you concatenate TDM files together?
I'm surprised this functionality isn't embedded into the software as it is a fairly common task, and the script is likely beyond most user's ability to maintain.
Solved!
Go to Solution.

Hi Brad,
Maybe I didnt explain myself that well, my fault. The case is:
As a result of acquired data post analysis with LabVIEW, I am logging into TDMS files 1.7GB of data. It's a file of 15 columns (properties) each of 1,080,000 rows ( properties values).
If I decide saving all the data into one file ( resulting into a 1.7GB file), DIAdem cannot load it into Data Portal. The program gets frozen even after waiting for 30mins.
If I save the data into multiple files (i.e like I tried last time in 3 files of around 500MB and one around 270MB) the 3 first files are loaded into data portal but when the loading of the 270MB is under way, DIAdem gives me an indexing error (sorry cant recall right now what is says exactly).
In the case of saving into multiple files, apparently I want to concatenate the TDMS file channels so I can i.e graph plots for the whole duration of my test. A reappending VBScipt you kindly provided me, works fine with any file I used that was not as massive as the ones I am dealing now. When I am using it with the above mentioned files, in the end I just get a time channel and none of the channels in my TDMS files.
Using a VBScipt that requires that I load my data into Data Portal before running it, it's simply impossible as DIAdem stalls as I mentioned above.
Sorry, I havent been using scipting before so I might be missing something here. As it goes for those changes you suggested. i will give it a try but could you please confirm which scipt we are talking about now exactly? As for concatenate files, the code Herbert provided in that thread is great but it does nor address exactly my problem.
Thanks,
Panos
Message Edited by Saloutios on 06-14-2010 04:27 AM

Similar Messages

How to call tdms file in DIAdem programatically using LabVIEW

Hi all,
I'm working on a project which has to acquire and stamp data in every 5 ms and store the data from the test cycle start up to almost a months' time. We have decided to store the data in the TDMS file format. We are going to use DIAdem for report generation. I'm new to NI diadem. So I need to know whether it is possible to acquire at the rate of 50Ks/s of around 15 channels and store data to the TDMS file and how we can call a TDMS file into DIAdem and generate reports in the HTML format.

Hi Jothiraja,
Yes, I think this is an excellent choice. In DIAdem you can just drag and drop the TDMS file from the NAVIGATOR tree view on the left into the Data Portal on the right in order to load the TDMS file. The REPORT panel in DIAdem has a "File >> HTML Export..." menu you can use to export your configured report in HTML format. Both the data loading and the HTML export can also be run programmatically in DIAdem, if you need to automate the reporting process.
On the LabVIEW side, how large is your acquisition buffer going to be? 1000 values on each of the 15 channels? 5000 values on each of the channels? If your acquisition buffers are going to be smallish, then you may want to consider creating a "NI_MinimumBufferSize" property on each of the 15 channels, assigning a value to this property that is the number of channel values you want to flush to disk at a time. The question here is how much of the resulting TDMS file will be stored data vs. binary buffer header information. If your resulting TDMS_Index file is tiny compared with its TDMS file, then you can ignore this tip. Alternatively, you could run the "TDMS Defragment.vi" at the end of the data acquisition, or equivalently load the acquired TDMS file into DIAdem and then re-save it over the original. In both of these cases you will get just one binary buffer at the top of the TDMS file, and all the rest of it will be stored data. Again, you can compare the size of the TDMS file with its TDMS_Index file to show you what the effect is.
TDMS is plenty fast enough to stream the 15 channels at 50kS/s that you propose, and I believe you will be very happy with the choice of TDMS as your file format and DIAdem as your reporting engine. I'll also be happy to help you through whatever obstacles you may run into along the way.
Brad Turpin
DIAdem Product Support Engineer
National Instruments

Getting "... file has and invalid format. (Error no. 11)" when importing .tdms file into DIAdem 11

I developed a script for processing .tdms files in DIAdem 11.1, now I'm deploying the script on the very PXI system that created the .tdms files. This system had DIAdem 11.0, but when I try to open (DataFileLoadSel) the tdms file in my script I get the error message:
"D:\MyFileName.tdms" has an invalid fiel format. (Error no. 11)
The tdms files are created by a LabView application.
If I delete the correspondin .tdms index file, I get a little further, it loads the first two of four groups, using DataFileLoadSel on the first group, and DataFileLoadRed on the 2nd group, on third group, another DataFileLoadRed, I get the following error:
Error in <MyScript> (Line: x, Column: y):
Cannot load the file "D:\MyFileName.tdms" with the loader "TDMS"
Further information:
Cannot open the file "D:\MyFileName.tdms".
Is there a difference between DIAdem 11.0 and 11.1 that affects the import of .tdms files, if so, can I get around it?
Thanks,
Eric
Solved!
Go to Solution.

Hello eberg,
Before we go into more detailed error trapping, could you please try to install the TDMS 2.0 format on the DIAdem 11.0 computer (DIAdem 11.1 and LabVIEW 2009 come with the TDMS 2.0 stuff already installed).
Please get the download here: http://zone.ni.com/devzone/cda/tut/p/id/9995
Once installed, please try running the Scripts again (in DIAdem 11.0) and let us know if that fixed the issue. It might not help to install this, but it's a quick thing to try before we dig deeper into the issue.
Best regards,
Otmar
Otmar D. Foehner
Business Development Manager
DIAdem and Test Data Management
National Instruments
Austin, TX - USA
"For an optimist the glass is half full, for a pessimist it's half empty, and for an engineer is twice bigger than necessary."

How to save tdm file from GPI extension DLL

Hi,
Is there any way to save data channels to tdm file format from GPI extension dll.
I've tried lDataSave function, but I get DAT file format.
I've also tried using Diadem.TOCommand interface, It'works with CmdExecuteASync method, but not with CmdExecuteSync (I've thought using a new thread with CmdExecuteASync, but what about synchronization ?).
As I need to run a process like, do something, save tdm file, then do another thing, I'm a little lost.
Thanks for any suggestion.
JMA.

Hello!
First: I have no solution!
Second: Some remarks to using a OLE interface to DIAdem in GPI.
My experience is that there are only few situations and circumstances where it might be the only way to solve a problem. There are some critical things to know. If you create a ToCommand interface it isn't guaranteed that you get an interface to the proccess you are running in. In normal cases it will be but you can't be sure! CmdExecuteSync doesn't work because usually your command is initiated out of a script. While running a script DIAdem will not execute another command.
The problem with CmdExecuteASync is that the command will be put onto the DIAdem command stack. It will be executed after the script is finished. You can not be sure that the channel data is the same as it was at the time you called CmdExecuteASync. Multiple Calls to CmdExecuteASync will be executed after another.
Sorry that I do not have a solution but perhaps I can avoid some frustrating DIAdem-OLE-Test-Sessions.
Matthias
Matthias Alleweldt
Project Engineer / Projektingenieur
Twigeater?

How to read TDMS file use DDC_GetDataValuesString in C language.

1. Below code is correct, because the numDataValues is one, machine=sibot
char ss[10]={0};
    char *strMachine=new char[1];
    *strMachine=ss[10];
    nullChk (strMachine = (char *) malloc (sizeof (char) * (unsigned int)numDataValues));
    ddcChk (DDC_GetDataValuesString (channels[i], 0, (unsigned int)numDataValues, &strMachine));
    printf ("Channel value for Machine : %s\n", strMachine);
---then show "nel value for Machine : sibot"
2. But when the numDataValues is more one, I can't get all the data in the TDMS file, only can get the first one.
eg:
data one is FI,
data two is O
but I only can get FI....
what's wrong?? How to solve it?? The data format as attachment.
Attachments:
20131110151804.jpg ‏11 KB

Hi KittyDeng,
I guess the problem is from here:
char *strMachine=new char[1];
You allocated a char array of only 1 element which will not hold as many chars.
The best way to solve the problem is to allocate more memory than that of the data, where in your case, you can write:
char *strMachine=new char[20];//or set more than 20
Hope this helps.
Zhiqiang Tu
NISH AE

How to Read TDMS Files in Matlab in OSX

Using the example found on the NI website (http://digital.ni.com/public.nsf/allkb/0EEADA99DC7D00A4862572E30037C3A2), I am able to read my TDMS files into Matlab when using a PC. However, when I try to read them on my Mac, I receive an error:
??? There was an error loading the library "/Applications/MATLAB75/work/TDMS_files/dev/bin/nilibddc.so"
Error using ==> loaddefinedlibrary
dlopen(/Applications/MATLAB75/work/TDMS_files/dev/bin/nilibddc.so, 1): image not found
Error in ==> read_tdms at 21
loadlibrary(DLL_Path, H_Path);
My best guess for this error is the fact that Matlab operates through a Linux interface on Mac, which I found will open the file type .so and not .dll for this type of library. Is there some way to get around this in OSX and get it to read the .dll files? Or is there some way to get the libraries as .so files instead of .dll files?
Thanks for any help!
-Shelby

Shelby,
You may wish to look at a Matlab script another user wrote since they were having issues with NI's matlab support:
http://forums.ni.com/ni/board/message?board.id=170&view=by_date_ascending&message.id=318487#M318487
The script supposedly converts TDMS files directly in Matlab with no use of NI generated external DLLs. He was having issues because the NI tools hadn't been updated for the new version of Matlab.

How to send TDMS file using TCP/IP

Hi,
I'm working with a cRIO-9025, chassis cRIO-9118 and NI 9239 module.
I've done an acquisition (voltage) where data is in 1D Wfm SGL. I've saved this data using a TDMS file, but now I want to send this file to an other computer (in this case mine) because the file is stored in the target. To do that I've implemented this example:
http://zone.ni.com/devzone/cda/epd/p/id/2842#0requirements
The problem is that the tdms file is created but it hasn't any data. I attach the image of the program. The program is done in other project (not which I use to make acq) to make it easy and then implement it when it works.
*The name file in the client is not used because I had a conflict with path.
What I'm looking for is make an acquisition of 10 minutes every 30 minutes and then send this data to an other computer (or others). I'm not sure if it's the best way, maybe is better to send data at the same time as the acquisition is being done, but my theacher told me that is easier if I send the file when acquisition has finished. To make acq I use FPGA and RT.
I would appreciate your help. Thanks!!
Solved!
Go to Solution.

I believe the FTP VIs are built into LabVIEW 2011SP1 - I have the VI present, but I could have installed the internet toolkit (don't know).
Look in the following location: Function Palette -> Connectivity -> Internet -> FTP
You're really looking for the VI named, "FTP Get File.vi". What this VI does is contacts an FTP server and requests a file from it. If your files are on the target, and you know the filename that will be created (such as files created in a sequence) you can request the file from the target and have them copied to the local machine (Windows). RT cannot "push" files to the Windows side unless you have an FTP server (like FileZilla Server) running on the Windows host.
It's a very simple one-VI solution. You specify host (the IP Address of the target you're pulling files from), the remote path for the file (for instance if the file is located in c:\ni-rt\data\myfile.zip then that's what you put as the remote path of the file), the local path for the file (where you want the file copied to, along with the name of the file, such as c:\mydir\myfile.zip), the username to log into the FTP server with (default is 'anonymous') and the password to use (default is an empty string). The importan thing to note is whether to use a binary transfer or not - if the file is a text file, no binary transfer needed; if the file is a binary file then you get better results using a binary transfer - I prefer to always use binary transfers, as non-binary transfers actually "interpret" and "convert" text information in the files.
-Danny

How to read .wav files with DIADEM

I have done a acoustic analysis with a microphone, and I have save the different Files with .wav format.
I need to read these waveform files with DIADEM.
Diadem recognize the different files however it returns the message "Diadem d'ont load the files ---.wav"

Hello Jean-Julien!
Get and install the Audio Data PlugIn. You can get ist here: http://zone.ni.com/devzone/cda/epd/p/id/44
Matthias
Matthias Alleweldt
Project Engineer / Projektingenieur
Twigeater?

How to import .tdms file to Matlab running in Unix system?

I was using DAQmx to acquire 16 channels of sampled data to .tdms format and later on will need to import the data to a Matlab programme. My question is, the Matlab we have is installed in a Unix system. But the tdms to Matlab example given in NI website, as in the link below, has problem when i run it in Unix system.
http://digital.ni.com/public.nsf/websearch/0EEADA99DC7D00A4862572E30037C3A2?opendocument&Submitted&&...
Errors occurred in the "loadlibrary" function. I was told that in Unix system, a different file instead of .dll file shall be used?
Can anyone make an example to read tdms data, that will work in a Matlab installed in Unix system?

I have plenty of experience working in LabVIEW and with TDMS however very little in using it in other software. I have downloaded the example code on this end but since I don't have your software installed on my machine, there is not a ton I can do in walking you through it. However, I understand this is a very helpful newsgroup for that environment.
Since you said you modified the header, to make sure everything is in line prior to entering other program, this is a good resource on the TDMS file structure.
Logan H
National Instruments
Applications Engineer

How to convert tdm files to lvm files?

I am trying to convert a number of TDM files to LVM files. I got error (2553) when I execute my VI. I dont know why.
1. Can anybody help me? I attached the VI.
Thank you.
Hyunseok
Attachments:
Conversion TDM to LVM.vi ‏121 KB

Hi Hyunseok
I changed your VI to a single file converter (see attched VI) and it works out well with LabVIEW 2009. Is there any special within your TDM files? Can you provide an example file which does not work?
Additionally I'm curious why you want to convert TDM to LVM since there are a bunch of tools to directly work with TDM files in other applications (Find Resources for Reading and Writing TDMS Files in Third-Party Products)
Stefan
Attachments:
Simple Conversion of TDM to LVM.vi ‏61 KB

Double clicking TDM file opens DIAdem but does not load file?

Every other Windows program automatically loads the file when application opens. Is there a setting for this?
NOTE: Using DIAdem 10.2 on Windows Vista Ultimate 64 Bit and Windows XP Pro 32 Bit.

CaliDevGuy,
You can try this file. It will update your registry files. I tried it on my computer and it works. You can load it into registry by double-clicking it.
Regards,
Song Du
Application Engineer
National Instrument
Regards,
Song Du
Systems Software
National Instruments R&D
Attachments:
CB_diadem.zip ‏1 KB

How does DIADEM import TDMS files? How gets every channel his number and groupindex? How can I determine which channel has which groupindex and number?

I store different channels in a TDMS file.
I like to have a time channel at the first position with group index 1 and number 1.
When I read the TDMS file with DIADEM the time channel (Float64) is on a differernt position, and the channels are not sorted alphabetically.
Here are my questions:
How does DIADEM import TDMS files?
How gets every channel his number and groupindex?
How can I determine which channel has which groupindex and number?
Best regards
Joerg

Hi Jörg,
i suppose that you´re programme whose create the *.tdms file is writing on false position. Try to create datas with timechannel on first indes in diadem, then save it and then open it again. you see that all is correct. So please tell me what programm in what version do you use and please attache it here.
Did you use the library for creating *.tdms files like in the link ?
http://zone.ni.com/devzone/cda/tut/p/id/6471
Here you find the gtdms_8.x.zip - when you extract it and opened the *.llb you find vi´s for all functions e.g. writing 2d array of strings to *.tdms file
when you open the subvi´s then you see how created and writing datas/structure to *.tdms files. Because *.tdms is binary you can´t see structure with open it in editor.
When you don´t have Labview you can use the 30 days test of current version 8.5 under following link
german version download link
https://lumen.ni.com/nicif/d/lveval/content.xhtml
english version download link
https://lumen.ni.com/nicif/us/lveval/content.xhtml
Hope it helps
Best Regards

DIAdem taking a long time to load 700MB TDMS file to data portal

I'm trying to load a 700MB TDMS file into DIAdem ,4 channels 10Hz sampling rate over 4 days . When i drag the file into the data portal , diadem freezes up , its on a 2GB machine, file located on the desktop, not running any other programs and it went 50min without being able to load before i made it quit. seems to work fine on smaller files though . Was wondering if this sounds normal and if anyone knows a way around it.
yin
Solved!
Go to Solution.

Thanks for all your input.
YongqinYe and Brad : I've tried defragmenting but this also seems to be taking a long time (in excess of ) on 2 different machines. The index file is 300MB, so about 75% of the *.tdms,
Otmar: i also tried registry loading the data but this still took a long time on this particular file although it worked great on smaller files (5MB).
Brad : the file does not open in the labVIEW TDMS viewer although i'm able to view it in chunks of 65000 using the TDMS importer for excel.
Stefan: the file does have explicit time stamp channel .
Any extra help would be appreciated but the way I'm thinking of doing it now is to use much longer buffer size (from only 6 to a few hundred) when collecting the data to decrease the level of fragmentation.

Concatenate two files in 1 using UNIX command

Hi All,
I need a help in Concatenating the 2 files in 1 file in unix from sap.
I have coded someting like below:
DATA: command LIKE rs37a-line.
DATA: BEGIN OF tabl OCCURS 0,
          line(2000),
      END OF tabl.
command = 'cat /tmp/file1.txt /tmp/file2.txt /tmp/file3.txt > /tmp/file4.txt'.
CALL 'SYSTEM' ID 'COMMAND' FIELD command
              ID 'TAB'     FIELD tabl-sys.
But i am able to see the new 1 file created in unix but not able to get the Content of 2 files in this new file created (its blank).
Please suggest me how to concatenate 2 file with content to new 1 file. am i doing anything wrong in my code>
please help me
Mahesh
Edited by: mahesh s on Apr 8, 2011 3:33 PM

Hi Mahesh,
Concatenating files in UNIX works like this:
cat chapter01 chapter02 chapter03 > book
So from what you posted, I can't see any mistake.
Aside from that:
- Rae is right, check whether you have a typo concerning case-sensitivity
- Check whether you have the right (chmod) to read the tempfiles
- Are you allowed to issue the "cat" Command?
- Is there any log information in PROT?
Whenever I want to issue shell commands on the AS, I usually write a shell script, send it to my basis team and have them set a link to the sap-directory. I then execute the script via CALL FUNCTION 'SXPG_STEP_XPG_START' which also return a logtable to you, so you can see directly whether your commands are wrong, or the chmod is wrong or whatever. I highly reccomend you doing it like this for more transparency.
best regards,
Lukas

Concatenating TDMS Files

Hello,
I'm writing a script in DIAdem 2011 which combines data from multiple sources into a single TDMS-File. Exporting into multiple TDMS files works fine( I can open them in DIAdem) but I'm having trouble combining all files to one TDMS-File. Speed is important, so I tried to avoid loading all data into the dataportal and saving again (this works though).
If I understood correctly it is possibly to directly concatenate the binary contents of the files into a single file. Is this correct? Because using the MS-Dos copy command or using a VBS-Script in DIAdem with Stream-Objects both results in TDMS-Files that I can't open anymore.
The error message says:
"TDS Exception in Intialize: Tds Error: TdsErrNotSupported(53):"
It seems only one of the files produces these problems. When I load this one file and save it again as TDMS I can sucessfully combine them and load the resulting TDMS without errors.
Perhaps the file is fragmented and this leads to the error (the offending file is measurement data)? Is this expected behaviour for fragmented files?
How can i efficiently combine these files? I have to be able to combine multiple hundreds of files ( in pairs. e.g. combine 400 TDMS in pairs to 200 TDMS files) fast and i guess opening 200 files, saving again and combining is rather slow.
Is there an efficient way to solve this task?
As the offending file is actually TDMS-Data from a BLOB-Datafield in a SQL-database another solution might be loading the TDMS directly from the database into the dataportal and saving from DIAdem, instead of dumping the BLOB-data into a TDMS-File on the HDD( i guess this results in a defragmented file as opening and saving resolves the problems).
But I'm not sure how to achieve this. I'm using ADO to interface with the SQL-DB. The code looks approximately like this:
Set recordSet = CreateObject("ADODB.Recordset")
recordSet.Source = 'QUERY
recordSet.open
Set fso = CreateObject("Scripting.FileSystemObject")
'Create Stream object
Set BinaryStream = CreateObject("ADODB.Stream")
do
chunkSize=100
lngOffset=0
lngFileSize=recordSet.Fields.Item("Field").ActualSize
'Open the stream And write binary data To the object
BinaryStream.Open
do while lngOffset<lngFileSize
varChunk=RecordSet.Fields.Item("TDMS_DataInBLOB").GetChunk(chunkSize)
Call BinaryStream.Write(varChunk)
lngOffset=lngOffset+chunkSize
loop
'Save binary data To disk
Call BinaryStream.SaveToFile("path", adSaveCreateOverWrite)
BinaryStream.Close
Call RecordSet.MoveNext
loop until RecordSet.EOF
How can I change this code to write the data into the dataportal?
Thanks in adavance.
Best regards,
grmume

Hi Brad Turpin,
I have already tried concatenating the files with the copy command. I don't have a hex-editor at work but by looking at the filesizes and checking the contents with a texteditor the script seems to work but I can't open the created file.
The script I used looks like this:
private Sub mergeFiles(filePath1,filePath2,outFilePath,bDeleteInput)
if bDeleteInput then
Call ExtProgram("cmd.exe","/Q /c copy /b """&filePath1&""" + """&filePath2&""" /b """&outFilePath&""" &del """&filePath1&"""&del """&filePath2&"""")
else
Call ExtProgram("cmd.exe","/Q /c copy /b """&filePath1&""" + """&filePath2&""" /b """&outFilePath&"""")
end if
End Sub
To test your script I defined the following variables:
NewFileName="C:\Documents and Settings\username\Desktop\new folder\combined.TDMS"
ChanNames(0)="file1"
ChanNames(1)="file2"
Folder="C:\Documents and Settings\username\Desktop\new folder"
DosDrv="C:\Documents and Settings\username\Desktop\BatchFile"
It completed sucessfully but the result is the same. I can open the two seperate files(file1.tdms and file2.tdms) but I cannot open the combined.tdms.
Sadly I can't share the original files and I can't use DIAdem to fill the files with dummydata because opening in DIAdem and saving resolves the problems.
Do you have an idea what causes these problems? The possibility to combine tdms-files like that was a big bonus for me...
Can I somehow fill the files with dummydata without opening them in DIAdem? Or tell DIAdem to keep the original file structure but change the values?
Best regards,
grmume

How to concatenate TDM files in DiaDem

Similar Messages

Maybe you are looking for