Query multiple TDMS Files

Hi,
I would like to evaluate the option of using TDMS files instead of an SQL database.
I will need to simply extract/query results that follow some conditions from multiple TDMS files.
I found many articles but could not find one example of how to extract data from multiple tdms files.
Should I have Data Finder or DIADEM ? or can I simply use LabVIEW ?
I am attaching a simple VI that generates 50 TDMS files (at the same folder as the VI), with "random" properties ('Title' and 'Type') and "random" data colums ('Temperature', 'Channel', 'Power').
Save the VI in a new folder and Run it. it will generate 50 TDMS files (see block diagram).
Now, considering those files, (for example) how can one extract only the the data where Test='Test2' and Type='Type1' and (Temperature=25 or Power>0.4) ?
Amitai.
Attachments:
TDMS_Write_Random_Files.vi ‏15 KB

Hooovahh wrote:
The DataFinder toolkit is awesome when dealing with TDMS report indexing. Lots of good stuff can be done with report look ups. Find me a report that was made after this date, before this date, made on this PC, running this version of software, for a UUT ran in this nest location, with this operator name, that failed. Queries like that can be made very easily once the indexing is complete.
As for pricing. It is included with the developer suite, but I'm pretty sure it isn't included with any professional, or full package. If you go to a page like the PID Fuzzy Logic Toolkit you'll see in bold at the top it says it is included with Full and Professional.
http://sine.ni.com/nips/cds/view/p/lang/en/nid/209054
The DataFinder page doesn't have this.
http://sine.ni.com/nips/cds/view/p/lang/en/nid/209062
Oh and there are several examples found in the Example Finder. Download a trial and see if it is worth it.
According to here, the DataFinder Toolkit is an add-on that is included in the Developer Suite. So it looks like Hooovahh is correct.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines

Similar Messages

Read multiple tdms files

Two questions:
1) how can i rename the different signals in the tdms file? Now they are names "untitled x". I am looking for an elegant solution, because i record about 200 different Signals.
2) Is there an "easy way" to select, read and concatenate multiple TDMS files to show them in a graph?

I had a similar goal with grouping plots and rearranging them on the X-axis, so I put together this pretty quickly to search a range of TDMS files for a specific channel, should be able to adapt it pretty easy to suit your needs, once you manage to rename your channels.
Attachments:
Record Grouping.vi ‏48 KB

View multiple TDMS files at once?

We have to store data from 2 channels in 2 seperate TDMS files, but I need to view/analyze them togther. How can I open two or more TDMS files at the same time?
Solved!
Go to Solution.

Which solution do you mean? Diadem or rolling your own reader?
You could code up a very nice system to read TDMS files but keep in mind that TDMS is kind of a database in itself. If you want to store to Citadel it might be best to just write your data as a BLOB of variants. You'd still have to code something to pull the raw data out, decode it back into the original data and dislay it though.
Using LabVIEW: 7.1.1, 8.5.1 & 2013

Problem opening multiple .tdms files in subroutine

Heya-
I'm using LabVIEW 8.2, professional development system.
I'm using the Open / Read / Close TDMS functions as part of a subroutine in a larger program. The subroutine is supposed to open one of several TDMS files (each of which contains bulk data saved previously), extract the data, then restructure into a 3D array, which is passed back to the main program. The particular file TDMS file to open is controlled by the main program. The TDMS Close function is used after the data is extracted.
When running solo, the subVI performs as expected. When running as a subVI, it functions fine on the first call, but subsequent calls to open different data sets fail to actually open the TDMS file (or at least, the Read TDMS function generates an empty array). However, returning to the orginal file that was opened, the subVI works fine - it opens the TDMS file. It doesn't matter which file was intially used - whichever one was opened first functions fine, any others do not. That would make me think that the first TDMS file isn't being closed, except that I am using the Close TDMS function after reading.
Attached are the subVI (TDMS_to_3D_array), a quick test program that uses the subVI, and a .zip of two data sets
Not sure if the global variables located in the SubVI will default, so they are: UpperMaxCol: 55, UpperMinCo: 0, UpperMaxPixel: 48, UpperMinPixel: 6
Thanks in advance for the help
Dan
Attachments:
Data.zip ‏1060 KB
TDMS_to_3D_array.vi ‏54 KB
TDMS_loadfault_tester.vi ‏39 KB

Hi Dan,
Perhaps you would try a few things.
Use the TDMS Flush VI in the subVI before the TDMS Close VI.
Try running the program with the subVI open.
Wire the TDMS File refnum from the subVI back to the parent and observe its value as the program switches between TDMS files.
Perform the TDMS read in the parent VI, wire the TDMS Data and the Array Size of the TDMS dataset to the subVI. Does this make any difference?
Try the subVI without the Channel Name string indicator.
Include error cluster wires throughout the entire subVI. Wire them through the TDMS VIs, then through the nested For structures, and then back to the parent VI.
Let us know what happens.

Query multiple XML files

Hi,
Is there another way of making queries in multiple different XML files other than using XQuery?
I use CremeVM for a CDC but it doesn't have desiredAssertionStatus() method in its Class.class so XQuery doesn't work.
Another option to solve it will be to execute multiple XPath expressions but not a good solution I think.
Any suggestions?

Thanks Curt,
just for others.... My Function for reading XML:-
let XmlImportedFile = (FilePath, FileName) =>
let
Source = Folder.Files(FilePath),
File = Source{[#"Folder Path"=FilePath,Name=FileName]}[Content],
ImportedXML = Xml.Tables(File)
in
ImportedXML
in
XmlImportedFile
then it's use to combine into a custom Column:-
let
Source = Folder.Files("C:\Users\Tim\Data\Education\Collect\XML files January 2014"),
InsertedCustom = Table.AddColumn(Source, "XmlImportedFiles", each XmlImportedFile([Folder Path],[Name])),
RemovedColumns = Table.RemoveColumns(InsertedCustom,{"Content", "Name", "Extension", "Date accessed", "Date modified", "Date created", "Attributes", "Folder Path"}),
#"Expand XmlImportedFiles" = Table.ExpandTableColumn(RemovedColumns, "XmlImportedFiles", {"Name", "Header", "School", "Pupils"}, {"XmlImportedFiles.Name", "XmlImportedFiles.Header", "XmlImportedFiles.School", "XmlImportedFiles.Pupils"})
in
#"Expand XmlImportedFiles"
thanks again and great blog post :-)
Thank you for you time folks!

Concatenating TDMS Files

Hello,
I'm writing a script in DIAdem 2011 which combines data from multiple sources into a single TDMS-File. Exporting into multiple TDMS files works fine( I can open them in DIAdem) but I'm having trouble combining all files to one TDMS-File. Speed is important, so I tried to avoid loading all data into the dataportal and saving again (this works though).
If I understood correctly it is possibly to directly concatenate the binary contents of the files into a single file. Is this correct? Because using the MS-Dos copy command or using a VBS-Script in DIAdem with Stream-Objects both results in TDMS-Files that I can't open anymore.
The error message says:
"TDS Exception in Intialize: Tds Error: TdsErrNotSupported(53):"
It seems only one of the files produces these problems. When I load this one file and save it again as TDMS I can sucessfully combine them and load the resulting TDMS without errors.
Perhaps the file is fragmented and this leads to the error (the offending file is measurement data)? Is this expected behaviour for fragmented files?
How can i efficiently combine these files? I have to be able to combine multiple hundreds of files ( in pairs. e.g. combine 400 TDMS in pairs to 200 TDMS files) fast and i guess opening 200 files, saving again and combining is rather slow.
Is there an efficient way to solve this task?
As the offending file is actually TDMS-Data from a BLOB-Datafield in a SQL-database another solution might be loading the TDMS directly from the database into the dataportal and saving from DIAdem, instead of dumping the BLOB-data into a TDMS-File on the HDD( i guess this results in a defragmented file as opening and saving resolves the problems).
But I'm not sure how to achieve this. I'm using ADO to interface with the SQL-DB. The code looks approximately like this:
Set recordSet = CreateObject("ADODB.Recordset")
recordSet.Source = 'QUERY
recordSet.open
Set fso = CreateObject("Scripting.FileSystemObject")
'Create Stream object
Set BinaryStream = CreateObject("ADODB.Stream")
do
chunkSize=100
lngOffset=0
lngFileSize=recordSet.Fields.Item("Field").ActualSize
'Open the stream And write binary data To the object
BinaryStream.Open
do while lngOffset<lngFileSize
varChunk=RecordSet.Fields.Item("TDMS_DataInBLOB").GetChunk(chunkSize)
Call BinaryStream.Write(varChunk)
lngOffset=lngOffset+chunkSize
loop
'Save binary data To disk
Call BinaryStream.SaveToFile("path", adSaveCreateOverWrite)
BinaryStream.Close
Call RecordSet.MoveNext
loop until RecordSet.EOF
How can I change this code to write the data into the dataportal?
Thanks in adavance.
Best regards,
grmume

Hi Brad Turpin,
I have already tried concatenating the files with the copy command. I don't have a hex-editor at work but by looking at the filesizes and checking the contents with a texteditor the script seems to work but I can't open the created file.
The script I used looks like this:
private Sub mergeFiles(filePath1,filePath2,outFilePath,bDeleteInput)
if bDeleteInput then
Call ExtProgram("cmd.exe","/Q /c copy /b """&filePath1&""" + """&filePath2&""" /b """&outFilePath&""" &del """&filePath1&"""&del """&filePath2&"""")
else
Call ExtProgram("cmd.exe","/Q /c copy /b """&filePath1&""" + """&filePath2&""" /b """&outFilePath&"""")
end if
End Sub
To test your script I defined the following variables:
NewFileName="C:\Documents and Settings\username\Desktop\new folder\combined.TDMS"
ChanNames(0)="file1"
ChanNames(1)="file2"
Folder="C:\Documents and Settings\username\Desktop\new folder"
DosDrv="C:\Documents and Settings\username\Desktop\BatchFile"
It completed sucessfully but the result is the same. I can open the two seperate files(file1.tdms and file2.tdms) but I cannot open the combined.tdms.
Sadly I can't share the original files and I can't use DIAdem to fill the files with dummydata because opening in DIAdem and saving resolves the problems.
Do you have an idea what causes these problems? The possibility to combine tdms-files like that was a big bonus for me...
Can I somehow fill the files with dummydata without opening them in DIAdem? Or tell DIAdem to keep the original file structure but change the values?
Best regards,
grmume

Tdms files

Hi,
I have a PCI 6132.
I´m using a program to acquire datas in continuos mode, and save it in my pc.
I'm using TDMS files to record it. But this files are too long. I need record it in multiple files to make easy to read after.
Using just one file, it has more than 10Gb, so, I need make multiples files with 1Gb each one, for example.
Thanks,
Marcela

Hi,
I think the better idea is to write data into multiple TDMS files when streaming data, instead of split the file into multiple files when you finish writing.
There are at least several ways to split, it depends on your application. For example, if you have multiple groups or channels, you can read from the big one TDMS file group by group and write to new TDMS file for each group. And, you can read samples by samples, for example, if you have 1000 samples in the big one file and you can split the 1000 samples into 10 files and each new file contain 100 samples. Another way is based on TDMS file format, TDMS file is consist of "segments", you can divide the big one file into new files and for each new file, it contains at least one segment (it would be a little complicated for the implementation, you need to know where the segment starts from, etc).
Please refer to www.ni.com/tdms for more reference.

How can I make a query by date with several TDMS files?

Hi,
I have a project that can write and read TDMS files everyday (a file for each day with date and time). There, I can import those files to excel (I choose a file and load it). But, I have a question: How can I make a query by date with those TDMS files? I'd like make a time stamp for start date and stop date, where I could compare the TDMS file dates and sum the values that are in the channels where these files are similar. For example, I save a file with "02/01/2013" name, in other day "03/01/2013", in other day "04/01/2013"... so, i'd like put in time stamp for start date "02/01/2013" file and to stop date "04/01/2013" file, than, sum all values that range with my TDMS files existing. How can I make that?
Thanks all,
Val

Hello Val_Auto.
You're Brazilian, no? Me too. = ^ - ^ =
I converted VI to version of your LabVIEW (8.5). Is attached in this reply.
This VI search all your TDMS in a range of dates and join them in a single TDMS. I hope this is what you wanted.
Query TDMS is the main VI. The TDMS VI Search changes the date format that out from calendar control (which is DD / MM / YYYY) to DD-MM-YYYY. This is because you can't name files using "/". I chose "-" but, if necessary, you should change to keep the same format of your TDMS files.
If you have any doubt as to its operation or how to make changes to adapt VI for your application, keep at your disposal.
Thank you for your contact; I hope have helped you and succeed in your application.
Wesley Rocha
Application Engineer
National Instruments Brazil
Visite a nossa comunidade em PORTUGUÊS!!!
Attachments:
Query TDMS.vi ‏62 KB
tdms search.vi ‏24 KB

TDMS File Creates Multiple Tabs of Data. Want to create only one.

Hello NI Community,
I am currently using the DAQ Assistant with my string pots to record displacement over a long interval of time (see attached VI). I am then using the "Write To Measurement File" vi to save the data into a TDMS file (see the "Write to File Settings" picture attached for all settings chosen for file write). The file is being written, however when I open the TDMS file, the document creates a tab in Excel for every second of data recorded (please see the attached "). It looks like the TDMS file is grouping all the data from each second together and then creating a new group of data.
How do I get the file to write all the data onto a single tab instead of creating multiple? I do not want to have to concatenate hundreds of tabs into a single excel sheet.
Thanks,
Chris
Solved!
Go to Solution.
Attachments:
Write to File Settings.PNG ‏40 KB
TDMS File Multiple Tabs.PNG ‏57 KB
TDMSWrite.vi ‏108 KB

You should not be using the comment. I just removed that and it all worked for me.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines

Question about creating multiple output files from same query

I have a query like this:
select * from emp;
ename empno deptno
Scott 10001 10
Tiger 10002 10
Hanson 10003 20
Jason 10004 30
I need to create multiple output files in xml format for each dept
example:
emp_dept_10.xml
emp_dept_20.xml
emp_dept_30.xml
each file will have the information for employees in different departmemts.
The reason I need to do this is to avoid executing the same query 200 times for generating the same output for different departments. Please let me know if it is practically possible to do this.
Any input is greatly appreciated.
Thanks a lot!!

You can write a shell script to generate the multiple spools files for the same output. Below script may helps you.
#====================
#!/bin/bash
n=0
while [ $n -le 20 ]
do
n=`expr $n + 1`
sqlplus -s system/manager <<EOF
spool emp_dept_$n.xml
select count(1) from tab;
spool off
EOF
done
#====================

Save multiple sample rate data to TDM file

Hello, LV connoisseurs
I use 2 Multifunction boards and LV 7.1 to gather slow and fast data simultaneously, 'slow' being 10 temperatures at 150 Hz, 'fast' being 12 pressures at 15 kHz, rate factor between slow and fast is constant 100. My acquisition is set to 'continuous' with blocks of 15 and 1500 resp., so each sample set takes 0.1 sec.
Currently, I use 2 loops, one for each board. Slow data are written to .lvm, fast to .tdm, and this works fine.
But I wonder if in this configuration it might be possible to
- use one loop only (yes, trivial) with the main target being to
- write data into two channel groups of one .tdm-file, one for the 'fast' the other for the 'slow' data?
If at all possible, would this require the consumer/supplier scheme such as to allow the interspersing of data or can I do this directly?
Thank You for your input.
Michael

Just to make the distinction, you should probably be using TDMS (the S stands for streaming) instead of TDM if you are continuously writing data. TDM is more for writing a snap-shot and doesn't work so well for continuous data (big memory leaks last time I used it back in '06).
Also, as no time data is stored you probably also want separate timestamp channels then for your fast and slow data.
As mentioned, it is no problem having multiple writing loops using the same TDMS reference.
nrp
CLA

Question about creating multiple XML files from same query

I have a query like this:
select * from emp;
ename empno deptno
Scott 10001 10
Tiger 10002 10
Hanson 10003 20
Jason 10004 30
I need to create multiple output files in xml format for each dept
example:
emp_dept_10.xml
emp_dept_20.xml
emp_dept_30.xml
each file will have the information for employees in different departmemts.
We are using DBMS_XMLGEN package to generate XML.
The reason I need to do this is to avoid executing the same query 200 times for generating the same output for different departments. Please let me know if it is practically possible to do this.
Any input is greatly appreciated.
Thanks a lot!!

one solution i can think of is to use SQLX operator instead of dbms_xmlgen.
here is a sample example.
declare
l_xmltype xmltype;
l_deptno emp.deptno%type;
begin
for i in (select * from emp order by deptno)
loop
    select xmlconcat(
                xmlelement("ename", i.ename)
               ,xmlelement("sal", i.sal)
               ,xmlelement("detpno", i.deptno))
      into l_xmltype from dual;
    dbms_output.put_line(l_xmltype.GetClobVal());
end loop;
end;
/Now here you can open the query once, keep writing to the file till the deptno
is same, when the deptno changes, close the file and open a new file with new
deptno and start writing.
Note : in this way you will have to add the xmlprolog manually to each of the file which should not be an issue. after opening the file add the prolog string manually.
Hope this helps.

Script to auto zero channels in a TDM file with multiple channel groups

Hi friends,
I am trying to process some data but am having trouble modifying a script to autozero channels across multiple channel groups. The following script works to zero channels across one group. I use this script to process data in a TDM file with only one channel group. I need to be able to zero channels across multiple groups (5 groups). But the amount of groups I will have may change so I might need an option to enter the amount of groups or have the script determine the amount of groups in my TDM file.
Here is the code I use to zero the data. Thanks in advance for your help!
Sub UserDlg_Select
Call USERDLGINIT()
USERDLGCAPTION="Auto Zero Channel Based on Specified Points "
USERDLGTXT(5)="Number of points to Average"
USERDLGCOMMAND(5)="t1"
Call USERDLGOPEN()
End Sub
Call UserDlg_Select()
T2="1-"&T1
For I= 1 To ChnNoMax 'For all data channels except time
T4=CHNNAME(I) 'Create string for Channel name (I)
T3=ChnDim(T4) 'Create text variable for channel dimension
STATSEL(6)="Yes" 'Arith mean
STATCLIPCOPY="yes"
STATCLIPVALUE="No"
Call StatBlockCalc("Channel",T2,T4) 'Calculate Stats for Channel (I)
'Subtract Mean of first (T1) points for Channel (I)
Call FormulaCalc("Ch('temp1') := CH('"&T4&"') - statarithmean")
CHNCOMMENT("temp1")=("Channel zeroed over first "&T1)+"pts" 'Add Comment
ChnDim("temp1")=T3 'Add Dimension
CHNNAME("temp1")=T4+"-Z" 'Save updated data with -Z
'Delete Stat Channels
Call CHNDEL("ArithmeticMean")
Call CHNDEL("Sum")
Next 'End For loop

Hi rvillalta319,
Sounds like you'll need to adapt your code to use both a for loop and some implementation of the GroupCount variable (or ChannelGroups.Count) as seen here:
http://zone.ni.com/reference/en-XX/help/370858K-01/varoff/groupcount/

How to store data (in tdm files)

I'm looking for advice on how to store data in tdm/tdx files.
The main challenge is like this: At regular time intervals there is new data available (say every 10 minutes there are 10 minutes of new data available). All data is time based. So every ten minutes I may create a new tdm/tdx file containing this data segment. However, when i want to analyze the data i dont necessarily want to view 10 minutes of data from all channels, but maybe 20 hours of data from one particular channel.
The way I've achieved this so far is to manually load each 10 minute segment of this particular channel and then add these parts together. This is both time consuming and cumbersome. Is there a better way to do this that I've simply not discovered yet? Is there a better way to store data to simplify this process?
One solution is of course to save data from several 10-minute frames in one file, but seeing as there is a never ending supply of data i cant simply save all data in one giant file, it has to be split up at some point, and the problem will still remain.
One factor to have in mind here is that this is rather large amounts of data (maybe 10GB each day), so the option to simply load all the data into memory goes out the window rather quickly.
Feel free to ask if I've not made myself very clear

Hi salte,
If your test ever ended, and if you had LabVIEW 8.20 or higher, I would recommend using a TDMS file, which handles data appeands flawlessly. DIAdem 10.1 now also does data reduction and index windowing during file loading for TDM / TDMS / DataPlugin files, so that you could easily load out only the part of the file you wanted to look at. But since you describe your data acquisition as never stopping and amassing 10 GB per day, I agree that it would be impractical to use only 1 data file. So we are stuck with some number of files which each contain a part of your measurement. This approach can have advantages, since you can save operational properties for each "buffer" such as average value, dominant frequency, ambient room temperature, etc., and later on you can use the DataFinder to query out only the data buffers which meet specific conditions based on these properties. The problem remains to load and assemble data based on multiple files. This is an old problem in DIAdem, and one for which I have an efficient and I hope satisfactory workaround application. It does what you describe already doing in the minimum amount of time and with the minimum amount of user interaction, and it can be highly parametrized to suit your particular situation. The ideal way to start the application would be as part of a ResultsList custom menu. This would enable you to query out the buffers you wanted, highlight those rows in the ResultsList, right-click and choose your custom menu, and WHAM! the selected buffers for your queried channel(s) are automatically appended together in the DataPortal. Launching from a ResultsList custom menu would mean that you could skip the file dialog and just read out the data sources directly from the ResultsList selection.
Let me know what you think,
Brad Turpin
DIAdem Product Support Engineer
National Instruments
Attachments:
Importing Data from Multiple DataPlugin Files.zip ‏198 KB

Schema (*.xsd) for TDM file format?

Our application (written in C#) has to...
- parse the TDM file for specific information
- edit the content of specific tags on the TDM file
We use the normal MS XML engines to process the xml information in the TDM.
Question: Is there an xml schema (*.xsd) available for the xml tags inside the TDM that we can use to...
- validate the TDM
- know which values are allowed inside the tags?
So far I found nothing that would help me during my two day cruise thru internet, zone.ni.com, Online-Help.
Thanx, Oliver Thost.
red-ant measurement technologies and services e.k.

Hello Mr. Romainczyk,
Thanx for your reply to our post at zone.ni.com.
My Email: [email protected]
- We are NI customer in Germany.
- We bought DIAdem SSP contract to our new DIAdem 9 license.
- We also have bought LabView Developer Suite 1 year subscription.
- We program a LV application for PXI 8186 controller.
I also started a SSP technical query via telephone concerning the same topic but until now there is no answer.
In our project we need technical information concerning the TDM file format since we will develop an C# application that works with TDM files.
From the beginning:
1) Our LV application logs vibration measurement data (multiple channel) into multiple WAV-files that are referenced channels for the TDM.
2) So DI
Adem is able to directly read and process the WAV-data via TDM.
3) The third application (written in C#) has to read the TDM information and�
- delete one channel or
- change axis names or
- change samplerate for one channel or, or, or�
That�s why we are searching for a complete technical description of the TDM xml-schema:
- Description of all available tags
- Summary of all allowed data inside the tags
and so on
We already have the description for the DAT format.
Best regards, Oliver Thost
red-ant measurement technologies and services e.k.

Query multiple TDMS Files

Similar Messages

Maybe you are looking for