Tdms Store Millisecond

My requirement looks simple. I need to acquire data from 8 analog channels and store in a flat file. I need to acquire data every 5 ms and store the same. This has to go for 10 to 30 days. This is huge data. Latter, I need to retrieve the data based on the date and time specified by the user. The user wants the report with data shown for every 5 ms interval in the specified date / time range. I tried creating a tdms file, i converted the timestamp to string and stored as a column. But though the conversion and the array (using probe) shows the time stamp to the millisecond, the data is stored on intervals of 15 ms. Though there is no data loss in the storage, the TS is not actual. It is happening in the following way
TS                                      Value
10:15:01:100    1
10:15:01:100    2
10:15:01:100 3
10:15:01:116 15 so on.
Where can the problem be? Can we not store realtime time (to the milisecond) in tdms.

First:
Your title mention TDMS, your post mentions flat file, what is it you need?
In case of TDMS it's quite easy, read the data from the DAQ device in a waveform at 200 Hz, store the data into the TDMS file with a duration of 1 hour or something.
You can get the Starttime and samplerate from each channel in a TDMS file quite easy so that shouldn't be a problem.
There is no need to jump through loops (or strings) to store time info in a TDMS file.
The fact you see a 15 ms jump is OS based, the timestamp (as returned by 'Get Current Time') is retrieved from the OS clock which runs at approx. 60 Hz (or 16 ms.)
Ton
Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas
LabVIEW, programming like it should be!

Similar Messages

Efficiency of using activeX for Excel

I've been using data logging for our test fixtures in .tsv files, but our engineers want to automate it further by organizing some new data into several sheets, so I'm looking into manipulating Excel directly using ActiveX.
My question is how efficient is it to write this in ActiveX compared to just making a text file? I'm concerned that, since we're going to be writing some 50000 cells or something, that it'll suck up a lot of memory to do it through ActiveX.
Is this a legitimate concern, or am I just being paranoid?
Anything worth doing is worth over-doing.
LabVIEW Junkie

Thanks for all the help, guys!
A couple of things -- first off, I'd love to look at that code if you get a chance. But what I'm concerned about is if I create an array of 50000 values or something like that and write it all in one go at the end of the program, two things are going to happen: 1) it's going to take a really long time to finish the program, and 2) if the program crashes in that final step, I'll lose all the data. That second reason is why I currently write a line every cycle to the .tsv file.
What I'm thinking of doing, is meeting you halfway -- rather than write each cell individually, write each line individually within the spreadsheet. This should reduce the memory used (hopefully) and increase performance.
I haven't really played around with TDMS yet, I'll look into it. A couple questions on that -- does it write in excel format? The reason I ask is the second sheet that I'm supposed to add is not going to contain any changing data -- it's just to store some constants that are set on the front panel. If TDMS stores data in excel format, I can write it to TDMS, transfer to Excel, and then use ActiveX to add the second worksheet with the constants at the end of the program. How's that sound?
Thanks again!
Anything worth doing is worth over-doing.
LabVIEW Junkie

How to store a file path in a TDMS Property

I need to store some Windows paths to other files as TDMS properties, but (according to another forum post I now can't find), the following characters are all illegal in TDMS properties: . : , ; ' \ @ < > # [ ] % ( ) { } | * ? = ! "
Many of those are used in paths. I thought of converting each of them to its ASCII character code like an HTTP URL, but the '%' character is in the list, too. Can anyone recommend an approach to handling this situation?
Solved!
Go to Solution.

I actually did something I almost never do. I searched DIAdems online help. I found a page on name conventions for properties on TDMS files which states the following:
"If you use one of the following characters in a custom property name, DIAdem replaces this character with an underscore: .,:, ,, ;, ', \, @, <, >, #, [, ], %, (, ), {, }, |, *, ?, =, !, ", ^, $, &, +, -, /, and a space. DIAdem also replaces a leading number in the property name with an underscore."
It however does not state anywhere that I've found that you can't use these symbols in a text as part of the property data.
Best Regards
David

How to store data (in tdm files)

I'm looking for advice on how to store data in tdm/tdx files.
The main challenge is like this: At regular time intervals there is new data available (say every 10 minutes there are 10 minutes of new data available). All data is time based. So every ten minutes I may create a new tdm/tdx file containing this data segment. However, when i want to analyze the data i dont necessarily want to view 10 minutes of data from all channels, but maybe 20 hours of data from one particular channel.
The way I've achieved this so far is to manually load each 10 minute segment of this particular channel and then add these parts together. This is both time consuming and cumbersome. Is there a better way to do this that I've simply not discovered yet? Is there a better way to store data to simplify this process?
One solution is of course to save data from several 10-minute frames in one file, but seeing as there is a never ending supply of data i cant simply save all data in one giant file, it has to be split up at some point, and the problem will still remain.
One factor to have in mind here is that this is rather large amounts of data (maybe 10GB each day), so the option to simply load all the data into memory goes out the window rather quickly.
Feel free to ask if I've not made myself very clear

Hi salte,
If your test ever ended, and if you had LabVIEW 8.20 or higher, I would recommend using a TDMS file, which handles data appeands flawlessly. DIAdem 10.1 now also does data reduction and index windowing during file loading for TDM / TDMS / DataPlugin files, so that you could easily load out only the part of the file you wanted to look at. But since you describe your data acquisition as never stopping and amassing 10 GB per day, I agree that it would be impractical to use only 1 data file. So we are stuck with some number of files which each contain a part of your measurement. This approach can have advantages, since you can save operational properties for each "buffer" such as average value, dominant frequency, ambient room temperature, etc., and later on you can use the DataFinder to query out only the data buffers which meet specific conditions based on these properties. The problem remains to load and assemble data based on multiple files. This is an old problem in DIAdem, and one for which I have an efficient and I hope satisfactory workaround application. It does what you describe already doing in the minimum amount of time and with the minimum amount of user interaction, and it can be highly parametrized to suit your particular situation. The ideal way to start the application would be as part of a ResultsList custom menu. This would enable you to query out the buffers you wanted, highlight those rows in the ResultsList, right-click and choose your custom menu, and WHAM! the selected buffers for your queried channel(s) are automatically appended together in the DataPortal. Launching from a ResultsList custom menu would mean that you could skip the file dialog and just read out the data sources directly from the ResultsList selection.
Let me know what you think,
Brad Turpin
DIAdem Product Support Engineer
National Instruments
Attachments:
Importing Data from Multiple DataPlugin Files.zip ‏198 KB

TDMS for RETAIL/STORE

Hello Expert Zone,
Finally I started working on the CRM extraction, and here is my scenario.
Source system: 300 warehouses worth of member data.
Target: Blank/empty.
Objective: Copy /extract member/membership data from source by warehouse type ( 320 - 340 only ), to bring in 20 warehouse worth of data, and nothing else..
What template should be used in this case? appreciate if someone can share your inputs with step by step procedure.
Thanks,
V.

Hello Vikram,
I see this query related to the discussion we had in your previous thread:
http://scn.sap.com/message/15025928#15025928
If i understand correctly, you have CRM loyalty management coupled with SAP POS.
What i understand is that there is SAP Retail system and there is SAP CRM.
For CRM data reduction, TDMS provides date criteria. TDMS CRM does not reduce CRM loyalty management data, which means that the data in these tables will be fully copied across from sender to receiver system.
We do have TDMS Retail solution where we do reduction based on below.
1) Store/Site(In retail terminology)
2) Customer number
3) Date
Let me know if i have answered your question.
Best Regards,
Madhavi

How to store configuration data from clusters to TDMS files

It is a common use case to write measured data to a TDMS file, while
the configuration information for your test is stored in a cluster (and
that cluster might contain additional clusters, arrays etc). TDMS does not have
native support for clusters and some other data types.
This LAVA discussion thread
contains a VI library that breaks clusters of arbitrary complexity down
to their basic elements and writes their contents to TDMS properties
either at file, group or channel level. It also contains code that
reads your data back from TDMS and recreates the original cluster type.
Please review the whole thread in order to understand what to expect and to make sure you have the latest version.
Herbert

Change
while (rs.next()) {
     System.out.println(rs.getString("piece_id") + "," + rs.getString("piece_num")+ "," + rs.getString("piece_ver"));
     FileOutputStream fileobj = new FileOutputStream(f);
     DataOutputStream writeInsideFileObj = new DataOutputStream(fileobj);
     writeInsideFileObj.writeBytes(rs.getString("piece_id"));
     writeInsideFileObj.writeBytes(",");
     writeInsideFileObj.writeBytes(rs.getString("piece_num"));
     writeInsideFileObj.writeBytes(",");
     writeInsideFileObj.writeBytes(rs.getString("piece_ver"));
     fileobj.close();     //here my data is replacing and only last row is storing..Kindly help me it is very urgent
}to
FileOutputStream fileobj = new FileOutputStream(f);
DataOutputStream writeInsideFileObj = new DataOutputStream(fileobj);
while (rs.next()) {
     System.out.println(rs.getString("piece_id") + "," + rs.getString("piece_num")+ "," + rs.getString("piece_ver"));
     writeInsideFileObj.writeBytes(rs.getString("piece_id"));
     writeInsideFileObj.writeBytes(",");
     writeInsideFileObj.writeBytes(rs.getString("piece_num"));
     writeInsideFileObj.writeBytes(",");
     writeInsideFileObj.writeBytes(rs.getString("piece_ver"));
fileobj.close();- Roy

USB 6009 and TDMS write

Dear all!
I'm using USB-6009 device to adquire data from a physical channel. In the following vi, the number of samples to be adquired is set to 1000, with a 100Hz rate. Therefore, as I understand, the number of samples that should be read inside the while loop where DAQmx read is placed should be 10 for each iteration. However, I've tried to store this information in a TDMS file and I'm getting less number of samples as expected. I wouldn't like to lose any data.
I've read overthere that the USB-6009 is a software-timed device, so, can this be the reason of this anomaly? Moreover, if I analyze the adquired samples using DIAdem software, I see some samples are being repeated. In this case, am I overwriting some data and reading past values?
If I need to replace my data adquisition device with a hardware-timed one, which one should I get?
Thank you very much in advance,
Miren
Attachments:
USB6009_TDMS.vi ‏25 KB

When you run the sample clock in "Continuous Samples" mode, the # of samples only determines the buffer size. This is outlined in the help for that function.
As a result, you have nothing pacing the internal loop: If there are samples available when it reaches the DAQmx Read, it passes out as many as are available in the buffer. You might get 1, 2, 100; it depends on how long the other operations in the loop take.
In general, DAQmx loops like this can be paced either by time or by # of samples. Looks like you want sample pacing: wire the number of samples you want to collect to the "# of samples per channel" input of the DAQmx read. Then the loop will wait at the daqmx read until that many samples are ready, and then it will proceed.
-Barrett
CLD

How to store measurement data in a single database row

I have to store time-data series into a database and do some analysis using Matlab later. Therefore the question might be more a database question rather than Diadem itself. Nevertheless I'm interested if anyone has some best practices for this issue.
I have a database which holds lifecycle records for certain components of same nature and different variants. Depending on the variant I have test setups which record certain data. There is a common set of component properties and a varying number of environmental measurements to be included. Also the duration of data acquisition varies across the variants.
Therefore having tables appears to be non-optimal solution for storing the data because the needed number of columns is unknown. Additionally I cannot create individual tables for each sample of a variant. This would produce to many tables.
So there are different approaches I have thought of:
Saving the TDM and TDX files as text respectively as BLOB
This makes it necessary to use intermediate files.
Saving the data as XML text
I don’t know yet if I can process XML data in Matlab
Has anybody an advice on that problem?
Regards
Chris

Chris
Sorry for the lateness in replying to your post.
I have done quite a bit of using a Database to store test results. (In my case this was Oracle DB, using a system called ASAM-ODS)
My 2 Cents:
Three functions were needed by users for me. 1) To search and find the tests, and 2) To take the list of Tests and process the data into a report/output summary. 2) If the file size is large, then being able to import the data quickly into analysis tool speeds up processing.
1) Searching for test results.  This all depends on what parameters are of value for searching. In practice this is a smaller list of values(usually under 20), but I have had great difficulty getting users to agree on what these parameters are. They tend to want to search for anything.   The organization of the searching parameters has direct relationship to how you can search. The NI Datafinder does a nice job of searching for parameters, so long as the parameter data are stored in properties of Channel Groups or Channels. It does not search or index if the values are in channel values.
Another note: Given these are searchable parameters, it helps greatly if these parameters have a controlled entry, so that the parameters are consistent over all tests, and not dependent on free form entry by each operator. Searching becomes impossible if the operators enter dates/ names in wildly different formats.
2) Similar issue exists if put values into databases. (I will use the database terms of Table and column(Parameter) and Row (instance of Data that would be one test record.)
The sql select statement, can do fast finds, if store the searchable parameters in rows of a table. Where would have one row for each Test Record.   The files I worked with have more than 2000 parameters.   Making a table that would store all of these, and be able to search for all of these, makes a very large number of rows. I did not like this approach, as it has substantial maintenance time, and when changes are made, things get ugly quick.
3)This is where having the file format be something that the analysis tool can quickly load up is beneficial.   Especially if the data files are large. In DIAdem's case, it reads TDM,TDMS files very quickly into the dataportal.   It can also read in the MDF or HDF files, but these are hierarchical structures that require custom code to traverse the information, and get the information into dataportal to use in Analysis /reporting. (Takes more time to read data, but have much more flexibility in the data structure than the two level tdm/tdms format.)
My personal preferences
I would not want to put the test data into a Table row. Each of the columns would be fixed and the table would be very wide in columns.
>
I personally like to put the test Data into a file, like TDMS, MDF, or HDF and then the database would have a entry for the reference to the attachment. The values that are in the database is just the parameters that are used for test Searching, either in datafinder or in sql commands in the user interface.
Hopefully these comments help your tasks some.
Respectfully,
Paul
tdmsmith.com

How does DIADEM import TDMS files? How gets every channel his number and groupindex? How can I determine which channel has which groupindex and number?

I store different channels in a TDMS file.
I like to have a time channel at the first position with group index 1 and number 1.
When I read the TDMS file with DIADEM the time channel (Float64) is on a differernt position, and the channels are not sorted alphabetically.
Here are my questions:
How does DIADEM import TDMS files?
How gets every channel his number and groupindex?
How can I determine which channel has which groupindex and number?
Best regards
Joerg

Hi Jörg,
i suppose that you´re programme whose create the *.tdms file is writing on false position. Try to create datas with timechannel on first indes in diadem, then save it and then open it again. you see that all is correct. So please tell me what programm in what version do you use and please attache it here.
Did you use the library for creating *.tdms files like in the link ?
http://zone.ni.com/devzone/cda/tut/p/id/6471
Here you find the gtdms_8.x.zip - when you extract it and opened the *.llb you find vi´s for all functions e.g. writing 2d array of strings to *.tdms file
when you open the subvi´s then you see how created and writing datas/structure to *.tdms files. Because *.tdms is binary you can´t see structure with open it in editor.
When you don´t have Labview you can use the 30 days test of current version 8.5 under following link
german version download link
https://lumen.ni.com/nicif/d/lveval/content.xhtml
english version download link
https://lumen.ni.com/nicif/us/lveval/content.xhtml
Hope it helps
Best Regards

Converting tdm to lvm/ working with large amount of data

I use a PCI 6251 card for data aquisition, and labview version 8; to log 5 channels at 100 Khz for approximately 4-5 million samples on each channel (the more the better). I use the express VI for reading and writing data which is strored in .tdm format (file size of the .tdx file is around 150 MB). I did not store it in lvm format to reduce the time taken to aquire data.
1. how do i convert this binary file to a .mat file ?
2. In another approach, I converted the tdm file into lvm format, this works as long as the file size is small (say 50 MB) bigger than that labview memory gets full and will not save the new file. what is an efficient method to write data (say into lvm format) for big size files without causing labview memory to overflow? I tried saving to multiple files, saving one channel at a time, increased the computer's virtual memory (upto 4880 MB) but i still have problems with 'labview memory full' error.
3. Another problem i noticed with labview is that once it is used to aquire data, it occupies a lot of the computer's memory, even after the VI stops running, is ther a way to refresh the memory and is this mainly due to bad programming?
any suggestions?

I assume from your first question that you are attempting to get your data into Matlab. If that is the case, you have three options:
You can treat the tdx file as a binary file and read directly from Matlab. Each channel is a contiguous block of the data type you stored it in (DBL, I32, etc.), with the channels in the order you stored them. You probably know how many points are in each channel. If not, you can get this information from the XML in the tdm file. This is probably your best option, since you won't have to convert anything.
Early versions of TDM storage (those shipping with LV7.1 or earlier) automatically read the entire file into memory when you load it. If you have LV7.1, you can upgrade to a version which allows you to read portions of the file by downloading and installing the demo version of LV8. This will upgrade the shared USI component. You can then read a portion of your large data set into memory and stream it back out to LVM.
Do option 2, but use NI-HWS (available on your driver CD under the computer based instruments tab) instead of LVM. HWS is a hierarchical binary format based on HDF5, so Matlab can read the files directly through its HDF5 interface. You just need to know the file structure. You can figure that out using HDFView. If you take this route and have questions, reply to this post and I will try to answer them. Note that you may wish to use HWS for your future storage, since its performance is much better than TDM and you can read it from Matlab. HWS/HDF5 also supports compression, and at your data rates, you can probably pull this off while streaming to disk, if you have a reasonably fast computer.
Handling large data sets in LabVIEW is an art, like most programming languages. Check out the tutorial Managing Large Data Sets in LabVIEW for some helpful pointers and code.
LabVIEW does not release memory until a VI exits memory, even if the VI is not running. This is an optimization to prevent a repeatedly called VI from requesting the same memory every time it is called. You can reduce this problem considerably by writing empty arrays to all your front panel objects before you exit your top level VI. Graphs are a particulary common problem.
This account is no longer active. Contact ShadesOfGray for current posts and information.

IMac (27-inch, Late 2013) with iMac (27-inch, Late 2012) in TDM issue

With the late-2013 iMac 27" (3.5 GHz, 32 GB RAM, 3 TB Fusion, Mavericks 10.9.4) as the primary and the late-2012 iMac 27" (3.4 GHz, 16 GB RAM, 3 TB Fusion, Yosemite Beta 10.10) in TDM, the screen on the target display blinks off and on (goes black and comes right back after a split second). It happens intermittently, regardless of which Thunderbolt port is used to connect the two computers. The other Thunderbolt port on the primary is used to connect two external FW800 drives via a Thunderbolt-to-FW800 adapter.
Through exhaustive testing, and in constant consultation with AppleCare senior advisors, I have seemingly ruled out all obvious potential hardware or software culprits:
An AVR is feeding both computers consistent power.
Each computer's RAM and video cards have been tested with TechTool Pro 7: everything passed.
The Thunderbolt cable has been replaced.
Reset the NVRAM and SMC on both computers (though with all the testing I've done since, it might not hurt to do this again).
Both computers have been running the latest version of OS X (the target display is currently running Yosemite Beta).
Made sure to enable TDM after the primary iMac has completed its full startup process.
Turned down the display brightness on both computers.
Trashed the com.apple.windowserver.plist file on the late-2013 iMac.
None of these steps have resolved the problem and I am having NO other issues with either computer. I'm stumped, as are the AppleCare senior advisors I've been working with. This issue has been present since I first connected these two computers this way. Design flaw not revealed in testing? Problem part not revealed in testing?
Anyone here seen anything like this or have any ideas of what else I can try? Help would be greatly appreciated!
Thanks,
Dana

Purchase a Windows PC?? Gasp, indeed! No, after months of software and hardware troubleshooting with an AppleCare senior advisor, and me refusing to believe my brand new 2013 iMac could have a hardware defect, I finally took my entire configuration (both iMacs, voltage regulator, back-UPS) in to my local Apple Store. They were able to replicate the issue, but then misdiagnosed the problem and replaced an internal display cable. Within minutes of reconnecting everything at home, the problem returned. So AppleCare sent an authorized service tech to my home to replace the logic board. That seems to have finally solved the problem. Inconveniently, the tech didn't have what he needed to flash the new logic board with my serial number right then, so I had to wait two weeks to get that done. It did finally get done though, and now all is well.
Good luck with your issue!

How to store lead-time and price per MPN for a single Item

We are probably one of the only CEM's that use Oracle Applications in a one-to-one MPN-IPN (Oracle Item Part Number) relationship. We do not currently store multiple Manufacturers under a single Item. Rather, we use the Customer Item Cross Reference to rank the customer's AML, with each IPN tied to a single MPN.
We are looking at moving into a one-to-many "world" in Oracle but are not sure how we can handle different pricing, lead-time, preference, status (prototype, production, obsolete, do not use), etc. for each Manufacturer's Part Number under a single Item. How are other CM's handling this???

Hi Greg,
Unfortunately, DIAdem Datafinder doesn't really support much in the way
of Date/Time search outside of the File Level Creation Date property.
In order to make the creation time of the Channels searchable you would
want to create some custom properties (or use the existing RegisterInt1
- RegisterInt6) to define your own searchable properties.
For example, since you know that you have one TDM file per day, you can
start your search by find the desired day on the File Property Level,
searching the "Creation Date" property. When you store your TDM
properties, you could store the hour creation as an integer (and
thereby searchable) in one of the RegisterInt properties of the
channel, or create your own custom "Hour" channel. You could similarly
save the minute and second property in their own custom properties.
Then, since you're saving integer values, you can then search those
values to determine the time that the channel was created.
I realize that this isn't ideal or elegant, as in total it requires 4
searches (date, hour, minute, second). But that is the most
straightforward way of searching your channels by creation date/time.
Hope this helps Greg, let me know if you have any other questions.
Dan Weiland

Best way to store data from a plot on hard drive

I have a DAQ set-up that will be on 24/7 for several days, and the whole time it's streaming data to an XY-Graph with a limited buffer. My question is, what's the best way to store my plot data in such a way that I can empty out my plot every few hours but if I want to pull up data from a few hours ago I would just have an option on my initial VI to view any old saved plot data.
Basically, I'm trying to not overload my RAM with all my plot data so I'd like to save it to the hard drive while my VI is running, and only display, for example the last 8 hours worth of data... I've been looking at TDMS, but I'm not sure if that's the right solution.
much thanks!

Hi,
You could save your data using TDMS as well as ASCII to simply load it in another application later.
Attached VI shows you how to save your data to a File after a certain amount of iterations, you can combine this then with your code and a timestamp.
Christian
Attachments:
Save to new file every N samples.vi ‏20 KB

How do you save arrays in tdm files?

Hi,
As part of a university project I am trying to save some program data to a .tdm file. On each channel I wanted a number of different types of data including an analogue wave form and a couple of 1-D DBL arrays. I have found an NI presentation that says that you can store arrays in them but dont seem to be having any luck. Any ideas what Im doing wrong? I am using labview 8.5.
Regards,
James

Hi James,
in a channel, you can save only one array as data. If you want to save more arrays then use several channels. You can organize them in groups. If the arrays must be in the same channel, then convert them to a spreadsheet string (in string palette) and save the string as a property.
Greetings,
shb

How to store date/time information in a channel?

Hello,
I'm logging manufacturing test data into TDM files from LabWindows CVI 8.0.
So far, I've organzied the data like:
one Channelgroup for each Measurement type
one Channel per DUT into eah Channelgroup
There is a new tdm file creatd each day.
For a later analysis, we want to able to easily search for all DUT's manufactured between Date/Time1 and Date/Time2 within all files.
Question:
What would be the most recommeded way to store the manufacturing date/time of the DUT with the channel?
I have assigned the channel name with a date/time string, - but it is kind of awkward to search for a range with string types.
Is there a better way?
Thanks,
Greg

Hi Greg,
Unfortunately, DIAdem Datafinder doesn't really support much in the way
of Date/Time search outside of the File Level Creation Date property.
In order to make the creation time of the Channels searchable you would
want to create some custom properties (or use the existing RegisterInt1
- RegisterInt6) to define your own searchable properties.
For example, since you know that you have one TDM file per day, you can
start your search by find the desired day on the File Property Level,
searching the "Creation Date" property. When you store your TDM
properties, you could store the hour creation as an integer (and
thereby searchable) in one of the RegisterInt properties of the
channel, or create your own custom "Hour" channel. You could similarly
save the minute and second property in their own custom properties.
Then, since you're saving integer values, you can then search those
values to determine the time that the channel was created.
I realize that this isn't ideal or elegant, as in total it requires 4
searches (date, hour, minute, second). But that is the most
straightforward way of searching your channels by creation date/time.
Hope this helps Greg, let me know if you have any other questions.
Dan Weiland

Tdms Store Millisecon​d

Similar Messages

Maybe you are looking for

Tdms Store Millisecond