Analyzing large TDMS files

I am developing an application that will measure an AC voltage at 100KHz for 4 hours. I need to calculate the voltage and frequency drift during 30 second intervals. My plan is to parse the data into small chunks (typically 1 second), calculate the RMS voltage and frequency, and determine the magnitude of drift that may be occurring. I have been able to collect data during short time intervals and analyzed it with the Tone Measurement VI and  RMS VI, but am not sure how to break up large data files into smaller chunks for analysis and store the results.

When you read a TDMS file, you can specify how many samples to read and where to start reading from.  All you need to do is read the data in chunks and process each chunk individually.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines

Similar Messages

  • Analyzing large TDMS files in time invervals

    We have TMDS files that are several hours long (sampled at 1024 S/s).
    We would like to measure the number of zero crossings and peaks in blocks one second long over the entire recording.  I can see that FFT has a Time Interval function, but I don't see any function of this nature for statistics. 
    As an additional "nice" to have we would like to have these blocks overlapping 75%.
    Carsten Thomsen

    Hi Carsten,
    Are you wanting to convert a Tachometer pulse to a Revolutions channel?  If so, I have a VBScript I can send you.  If not, what do you do with the number of zero crossings and peaks?  Are you counting just negative peaks/zero crossings, just positive peaks/zero crossings, or both?  How do you define a peak?
    Brad Turpin
    DIAdem Product Support Engineer
    National Instruments

  • Signal Express Large TDMS File Recording Error

    Hello,
    I have the following application and I am looking for some tips on the best way to approach the problem with Signal Express:
    I am attempting to using Signal Express 2009 (Sound and Vibration Assistant) to collect random vibration data on three channels over an extended period of time -- about 20 hours total.  My sample rate is 2kHz.  Sampling at that rate over that period of time invovles the creation of a very large TDMS file, which is intended for various types of analysis in signal express later or some other application later on.  One of the analysis functions to be done is a PSD (Power Spectral Density) plot to determine the vibration levels distributed over a band of frequencies during the log. 
    My original solution was to collect a single large TDMS file.  I did this with Signal Express recording options configured to save and restart "in current log" after 1 hour worth of data is collected.  I configured it this way because if there is a crash/sudden loss of power during data collection, I wanted to ensure that only up to an hours worth of data would be lost.  I tested this option and the integrity of the file after a crash by killing the SignalExpress process in the middle of recording the large TDMS file (after a few save log file conditions had been met).  Unfortunately, when I restart signal express and try to load the log file data in playback mode an error indicating "TDMS Data Corrupt" (or similiar) is displayed.  My TDMS file is large, so it obviously contains some data; however, Signal Express does not index its time and I can not view the data within the file.  The .tdms_index file is also present but the meta data.txt file is not generated.  Is there any way to insure that I will have at least partially valid data that can be processed from a single TDMS file in the event of a crash during mid-logging?   I don't have too much experience dealing with random vibration data, so are there any tips for generating vibration level PSD curves for large files over such a long time length?
    My solution to this problem thusfar has been to log the data to seperate .TDMS files, about an hour in length each.  This should result in about 20 files in my final application.  Since I want to take a PSD, which ends up being a statistical average over the whole time period. I plan on generating a curve for each of these files and averaging all 20 of them together to get the overall vibration PSD curve for the 20 hour time period.

    JMat,
    Based on the description of your application, I would recommend writing the data to a "new log" every hour (or more often). Based on some of my testing, if you use "current log" and S&V Assistant crashes, the entire TDMS file will be corrupted. This seems consistent with what you're seeing.
    It would be good if you could clarify why you're hoping to use "current log" instead of "new log". I'll assume an answer so I can provide a few more details in this response. I assume it's because you want to be able to perform the PSD over the entire logged file (all 20 hours). And the easiest way to do that is if all 20 hours are recorded in a continuous file. If this is the case, then we can still help you accomplish the desired outcome, but also ensure that you don't lose data if the system crashes at some point during the monitoring.
    If you use "new log" for your logging configuration, you'll end up having 20 TDMS files when the run is complete. If the system crashes, any files that are already done writing will not be corrupted (I tested this). All you need to do is concatenate the files to make a single one. If this would work for you, we can talk about various solutions we can provide to accomplish this task. Let me know.
    Now there is one thing I want to bring to your attention about logging multiple files from SignalExpress, whether you use "current log' or "new log". The Windows OS is not deterministic. Meaning that it cannot guarantee how long it takes for an operation to complete. For your particular application, this basically means that between log files there will be some short gap in time that the data is not being saved to disk. Based on my testing, it looks like this time could be between 1-3 seconds. This time depends heavily on how many other applications Windows has running at the same time.
    So when you concatenate the signals, you can choose to concatenate them "absolutely", meaning there will be a 1-3 second gap between the different waveforms recorded. Or you can concatenate them to assume there is no time gap between logs, resulting in a pseudo-continuous waveform (it looks continuous to you and the analysis routine).
    If neither of these options are suitable, let me know.
    Thanks, Jared 

  • Opening large tdms files in excel takes forever, can anything be done?

    After recording some daq channels for 4 hours my file is 120mb large, with about 20columns of data and 150k rows. Opening this in excel takes at least 5min, several not responding screen fades, and growing fear that all my data is impossible to get to.
    What can I do about this? Is there a split tdms file option I can use or a way to speed up excel maybe?
    Solved!
    Go to Solution.

    The solution is to not make the TDMS files so large. Perhaps modify the code so it starts a new file every 1/2 hour or something. Could you post a screen shot of the part of the code that is doing the saving?
    Something that just occurred to me is that the TDMS file format is optimized for writing - not reading so stuff is just sort of streamed into the file with an index keeping straight what goes with what. The results is a file where the data from individual channels can be very fragmented, like parts of a file on a hard drive. I think there is a routine for defraging a TDMS file. Alternately, you could save the data in a temporary file (not necessarily TDMS) and resave it to the final TDMS file after all the data is collected.
    Mike...
    Certified Professional Instructor
    Certified LabVIEW Architect
    LabVIEW Champion
    "... after all, He's not a tame lion..."
    Be thinking ahead and mark your dance card for NI Week 2015 now: TS 6139 - Object Oriented First Steps

  • Break large TDMS file into small files

    hello all
    My TDMS file is around 3g, and needs changed into around 10M size files. 
    I ran McRorie's splitFiles.vi ‏15 KB in this page.  and set the number of samples per file as 5000000, however, I cannot get the results I need. every small file is only 1KB, and no data inside. What is the possible problem in this?
    Also, I tried to write a vi based on sample vi(Read TDMS File) by adding one "write to measurement file.vi". However, when I set the small file size as 10M Byte inside the"write to measurement file.vi", the first file could be around 20M, and the next few files may be correct as 10M, and then it just stop splitting, edding with a file even much larger than original file. I uploaded my vi here, maybe someone  can help to find some mistake for me. 
    Thanks very much!
    Wuwei Mao
    Solved!
    Go to Solution.
    Attachments:
    Read TDMS File.vi ‏54 KB

    Hi Wuwei,
    After giving the correct data type to TDMS Read node in splitFiles.vi, it works as expected. ( See the attached two VIs: createFile.vi and the modified splitFiles.vi)
    Because I don't know how you created your TDMS file, I write a new 3G bytes TDMS file, which has one group and one channel. The data type of samples is unsigned 16-bit Integer. The total number of samples is 1610612736. Then I set the number of samples per file to 5000000 as you did. So after splitting, each file size is 5000000*(16/8) bytes (around 10M bytes).
    Please make sure the followed steps have been done before you run the splitFiles.vi:
    1. The TDMS file will be split has been put on the proper path;
    2. The correct group and channel names have been given;
    3. The correct data type to TDMS Read node has been given.
    Because your second option using "write to measurement file.vi" to split TDMS file will lose some information, such as group and channel names. So I suggest using the method used by splitFiles.vi to accomplish your goal.
    Jie Zheng
    NI R&D

  • How to get the last data point from a TDM file in LabVIEW?

    Hello,
    I am using LabVIEW to analyze some rather large TDM files, and I need a way to get only the last data point.  So far, the only way I have been able to accomplish this is by reading the entire file.  Is there a property in the TDM file or a function in LabVIEW that will allow me to get the index of the last item in a channel?  
    Thanks!
    Christina

    Do you want to avoid reading whole file and want to be able to reach or get the index of last value of channel? is there any specific reason? I am not sure you could do it without loading the whole file. But the easiest way would be just to use array functions "array size" would give you the index of last element. 
    -Nilesh
    Kudos are (always) welcome for the good post. :-)

  • DIAdem not saving all samples from TDMS file

    Hello to all,
    I have several large TDMS files (1.8Gb each) containing approx. 120 million sampels. When I try to open these fiels in Diadem it previews the correct amount of samples (120 Million), but when I try to save tha data in Matlab format it saves me only half the samples, 60 Million. Also, I have a Matlab script that opens the TDMS files and does exactly the same thing, it is not saving all samples. 
    Can you please advise me on this issue? Is there a way to automatically fragment large TDMS files into smaller ones?
    Best regards,
    Ion 

    Hello Jean Baptiste,
    It seems your TDMS file is fine since you have the correct number of data in DIAdem.
    It may comes from the conversion to the MatLab format. Maybe MatLab can't accept more than 60 millions data. You should ask that to the MatLab expert on their forum.
    However, if you want to fragment your TDMS file, you have to do it manually in DIAdem, when you load the file into the Data Portal, or you can automate the function by using VBScript. (Last panel in DIAdem)
    An other solution would be to fragment your TDMS file directly where you write it. (In LabVIEW maybe?)
    Regards,
    Benoit S. - Field Sales Engineer
    Certified LabVIEW Developer
    Certified TestStand Developer
    National Instruments France
    #adMrkt{text-align: center;font-size:11px; font-weight: bold;} #adMrkt a {text-decoration: none;} #adMrkt a:hover{font-size: 9px;} #adMrkt a span{display: none;} #adMrkt a:hover span{display: block;}
    Été de LabVIEW 2014
    12 présentations en ligne, du 30 juin au 18 juillet

  • Cannot load large CSV files in SignalExpress("Not enough memory to complete this operation" error)

    Hi guys,
    I'm new here and just  have browsed
    some of the related topics here regarding my problem but could not seem
    to find anything to help me fix this problem so I decided to post this.
    I currently have a saved waveform from an oscilloscope that is quite
    big in size(around 700MB, CSV file format) and I want to view this on
    my PC using SignalExpress. Unfortunately when I try to load the file
    using "Load/Save Signals -> Load From ASCII", I always get the "Not
    enough memory to complete this operation" error. How can we view and
    analyze large waveform files in SignalExpress? Is there a workaround on
    this? 
    Thanks,
    Louie
    P.S.>I'm very new to Signal Express and haven't modified any settings on it. 

    Hi Louie,
    Are you encountering a read-only message when you tried to save the boot.ini file? If so, you can try this method: right-click on My Computer >> Select "Properties", and go to the "Advanced" tab. Select "Settings", and on the next screen, there is a button called Edit. If you click on Edit you should be able to modify the "/3GB" tag in boot.ini. Are you able to change it in this manner? After reboot, you can reload the file to see if it helps.
    To open a file in SignalExpress, a contiguous chunk of memory is required. If SignalExpress cannot find a contiguous memory chunk that is large enough to hold this file, then an error will be generated. This can happen when fragmentation occurs in memory, and fragmentation (memory management) is managed by Windows, so unfortunately this is a limitation that we have.
    As an alternative, have you looked at NI DIAdem before? It is a software tool that allows users to manage and analyze large volumes of data, and has some unique memory management method that lets it open and work on large amounts of data. There is an evaluation version which is available for download; you can try it out and see if it is suitable for your application. 
    Best regards,
    Victor
    NI ASEAN
    Attachments:
    Clipboard01.jpg ‏181 KB

  • Thread death and xml files when analyzing large datasets

    i analyze large data sets. for example:
    4000 persons that each have a time series with 8,000 data points. i do this linearly, so (i think) i am only running the
    *public static void main(String[] args) { .... }*
    for(Person person : PersonSet) {  // there are 4,000 people
      List<Data> data = DataSink.getData();  // each List has a size of 8,000
      analyzer.analyze(person, data); // i do not start any threads in this method
    }i've run into, and over come, java.lang.OutOfMemoryError issues.
    now, my program runs fine for over two minutes, then it crashes and shows this error:
    C:\Documents and Settings\David\My Documents\NetBeansProjects\Test\nbproject\build-impl.xml:419: The following error occurred while executing this line:
    C:\Documents and Settings\David\My Documents\NetBeansProjects\Test\nbproject\build-impl.xml:286: java.lang.ThreadDeath
    this is how i expand memory:
    -Xms1250m -Xmx1250m
    i cannot understand how run-time memory management relates to xml files.?
    my IDE is NetBeans 6.1

    i've never dealt with ANT scripts, so this is like my first "Hello World" programming with ANT.
    yet,
    i can't understand why the jvm would say that the origin of a crash in a program thats be running for over 2-minutes
    would be in an ANT file?
    #more build-impl.xml
        <target name="-init-macrodef-java">
            <macrodef name="java" uri="http://www.netbeans.org/ns/j2se-project/1">
                <attribute default="${main.class}" name="classname"/>
                <element name="customize" optional="true"/>
                <sequential>
    // line #286                <java classname="@{classname}" dir="${work.dir}" fork="true">   // <--- line #286
                        <jvmarg line="${run.jvmargs}"/>
                        <classpath>
                            <path path="${run.classpath}"/>
                        </classpath>
                        <syspropertyset>
                            <propertyref prefix="run-sys-prop."/>
                            <mapper from="run-sys-prop.*" to="*" type="glob"/>
                        </syspropertyset>
                        <customize/>
                    </java>
                </sequential>
            </macrodef>
        </target>
        <!--
                    =================
                    EXECUTION SECTION
                    =================
                -->
        <target depends="init,compile" description="Run a main class." name="run">
    // line #419        <j2seproject1:java>     // <-- line #419
                <customize>
                    <arg line="${application.args}"/>
                </customize>
            </j2seproject1:java>
        </target>
        <target name="-do-not-recompile">
            <property name="javac.includes.binary" value=""/>
        </target>
        <target depends="init,-do-not-recompile,compile-single" name="run-single">
            <fail unless="run.class">Must select one file in the IDE or set run.class</fail>
            <j2seproject1:java classname="${run.class}"/>
        </target>
    ............................thanks

  • Buffered TDMS file still very large

    Hello, I am trying to to stream to a TDMS file. I am writing approximately 10 channels of data at 20 Hz. I was creating a new file evey 5 mins or so and defragging the file once I was finished with it, which was giving me a reasonable size of file. I decided that I would like to make bigger files (over the period of hours) and the the defragging would be too computationally costly to do on the fly. I have been trying to set a buffer for the TDMS files but the resultant files are very large. You can tell the file isn't being writen until its closed (from watching its size in windows explorer) which suggests the file is being buffered. However its about 6 times bigger than an ASCII file of the same data, with and index almost as big again.
    Does anyone have any ideas why it would appear to buffer but not actually  reduce the data size. I'm running Windows XP, LabVIEW 10.0.
    When I create each new file I run this to set the buffers. (I have also hard wired in a buffersize of 100000 on a one minute file to no avail.
    Niall
    Thanks
    Niall

    I has this marked as the solution but it turns out it only helped one of my programs. I am still getting very large (fragmented) TDMS and index files for another program. The problem is that if I just defrag the (supposedly buffered) file, it interupts the data logging because it takes so long. I'm 99% sure that it *is* buffereing as it doesn't write till it closes the file, and if I use the read properties function it reads back a set buffer size. Here is the VI that actually writes the data. There bit at the top is for writing an optional acii file so you can ignore that.
    Its maybe a bit hard to see whats going on in the next one, but this is where the file is created before being passed to the setbuffer VI which I posted earlier. It also closes that last file
    Its really hacking me off now and holding me up from going no to other stuff. It would be great if someone had some ideas.
    Thanks
    Niall

  • How to store data (in tdm files)

    I'm looking for advice on how to store data in tdm/tdx files.
    The main challenge is like this: At regular time intervals there is new data available (say every 10 minutes there are 10 minutes of new data available). All data is time based. So every ten minutes I may create a new tdm/tdx file containing this data segment. However, when i want to analyze the data i dont necessarily want to view 10 minutes of data from all channels, but maybe 20 hours of data from one particular channel.
    The way I've achieved this so far is to manually load each 10 minute segment of this particular channel and then add these parts together. This is both time consuming and cumbersome. Is there a better way to do this that I've simply not discovered yet? Is there a better way to store data to simplify this process?
    One solution is of course to save data from several 10-minute frames in one file, but seeing as there is a never ending supply of data i cant simply save all data in one giant file, it has to be split up at some point, and the problem will still remain.
    One factor to have in mind here is that this is rather large amounts of data (maybe 10GB each day), so the option to simply load all the data into memory goes out the window rather quickly.
    Feel free to ask if I've not made myself very clear

    Hi salte,
    If your test ever ended, and if you had LabVIEW 8.20 or higher, I would recommend using a TDMS file, which handles data appeands flawlessly.  DIAdem 10.1 now also does data reduction and index windowing during file loading for TDM / TDMS / DataPlugin files, so that you could easily load out only the part of the file you wanted to look at.  But since you describe your data acquisition as never stopping and amassing 10 GB per day, I agree that it would be impractical to use only 1 data file.  So we are stuck with some number of files which each contain a part of your measurement.  This approach can have advantages, since you can save operational properties for each "buffer" such as average value, dominant frequency, ambient room temperature, etc., and later on you can use the DataFinder to query out only the data buffers which meet specific conditions based on these properties.  The problem remains to load and assemble data based on multiple files.  This is an old problem in DIAdem, and one for which I have an efficient and I hope satisfactory workaround application.  It does what you describe already doing in the minimum amount of time and with the minimum amount of user interaction, and it can be highly parametrized to suit your particular situation.  The ideal way to start the application would be as part of a ResultsList custom menu.  This would enable you to query out the buffers you wanted, highlight those rows in the ResultsList, right-click and choose your custom menu, and WHAM! the selected buffers for your queried channel(s) are automatically appended together in the DataPortal.  Launching from a ResultsList custom menu would mean that you could skip the file dialog and just read out the data sources directly from the ResultsList selection.
    Let me know what you think,
    Brad Turpin
    DIAdem Product Support Engineer
    National Instruments
    Attachments:
    Importing Data from Multiple DataPlugin Files.zip ‏198 KB

  • .tdm file storage vi's slow editing vi's

    I am developing software using Labview 8.5 and have recently added some file storage vi's for the .tdm format for use in DIAdem 10.2.  Upon using the open storage, set file properties, write channel group, write channel, and close storage vi's, the editing has become extremely slow - for any operation I try to perform there is a 10-15seconds delay before it occurs.  If I remove the .tdm vi's from my diagram then the editing goes back to normal and changes/operations occur almost instantaneously as expected.  My project is not large as far as I am concerned (Front panel - 121k, block diagram=1538k, code=0.2k, data=957k, total=2617k, total vi size on disk=389k)
    I saw a similar post to the .tdm storage vi's causing this to occur back in 2005 and was supposed to have  been on the fix list but it does not appear that it has 3 years later.  I am sure many others are using these vi's so not sure what I may have done differently to cause the editing to slow down so much.  
    Anyone with ideas and fix would be greatly appreciated as I cannot afford to sit around 10-15 seconds for every step of my developement.

    Yes, sometimes Storage VIs are slow. But They are Express VIs, supplying the easy configuration for users. If you feel they are too slow, I suggest that you can use TDMS files instead.

  • How to split large video files into multiple individual clips by date

    Hello. I'm looking for a freeware/shareware program (if one exists, or willing to pay for one if it does this well) that will:
    1) Analyze a large AVI file identifying the start/start points of each segment of video and the dates the video segments were shot.
    2) Then allow me to organize (for example, join 2 or more segments together if it's a video with a lot of start/stop points during the same shot) the segments identified in step 1 into logical clips that could be split apart from the 1 large AVI file.
    3) Lastly, it would split my one large AVI file into the multiple clips or files that I identified in step 2.
    A piece of software called FootTrack does a great job analyzing AVI files as I describe in steps 1 & 2, but it doesn't allow you to split or export the one avi file into multiple avi files.
    I really don't want to have to do this one clip at a time, I've got to believe there's a program out there that will automate this process for me.
    Thanks

    I don't know of a way to automate it. I just did about 15 years worth. It wasn't too hard. I use an application called mpegstreamclip. Its very easy and free to download either from the apple site or direct from its site.
    http://www.squared5.com/
    Bascically you set the in and out points with the i and o keys and export.

  • Can I save a TDM file without its TDX data?

    I have hundreds of data files that have been sorted (by date).  In certain cases, I want to analyze subsets of this data in particular groupings.  I would like to save these groupings in a TDM file but I don't want to another copy of the bulk data.  However, I notice that if I save a TDM, it automatically saves the TDX, and I'm not sure it's possible to save/open a TDM without its same-name TDX file.  Does anyone else have a similar need and/or know of a data management method that can suit my need?
    Thanks.
    Solved!
    Go to Solution.

    Hmm...  But it seems that the TDMS file simply puts all the information in a single file and, in the end, is almost as big as the TDX file.  What I'm trying to do is only save the meta data but not the bulk data, since I already have a copy of the bulk data in another set of TDM/TDX files. 
    Maybe I'm asking the wrong question.  Here is a simplified example of what I'm trying to do:
    I have several (let's say, 6) original TDM files.  Each of these original TDM files have 20 or more channels.  I've created a new TDM file, pulling out only 5 of the channels from each of the original TDM files but organizing them into 6 groups corresponding to the original 6 TDM files.  From this new TDM file, I've created a TDR file where the plots are based on the 6 channel groupings.  Due to this TDR to new TDM association, I think I need to save the new TDM.  However, that means I now have a new giant file that contains duplicate data from the original TDM files.  I don't want this duplicate data.  Being very new to DIAdem, maybe I'm just going about this the wrong way because I haven't figured out the "right way" to accomplish my needs.  Can anyone advise?

  • Read time of tdm file increases

    Hello,
    I recently recorded a lot of data in tdm format. I originally saved it in an excessively organised and complex way which resulted in large header files and was very slow. I changed it and most of the data is fine, except 6 files in the old format. Just trying to read them to get the data out and then convert it into an easier format is taking far too long. Each file has ~15000 channel groups, each with 3 channels (I won't do this again...). I have witten a small vi to test the reading speed. Excluding opening and closing of the file, if I just find a channel group by name, reading one channel from that group takes about 1 second. This would still take days to read it all but would be acceptable. The strange thing is, is that if I do this in a loop for successive channel groups which is clearly necessary, it takes longer and longer each iteration. This is behaviour that I have always seen when reading tdm files, but it hasn't caused a serious problem before. I can't find any information about why though, so I hope I'm just doing something stupid. I have attached the speed testing vi (Labview 2010). Running it for 5 iterations, the time taken to read the channels is
    0.82304 s
    1.58809 s
    2.42514 s
    3.56820 s
    5.60632 s
    The channels it is reading all have the same number of values, and it makes no difference if I start at a different channel group (by adding a number to i in the loop).
    Does anyone have any explanation for this behaviour?
    Thank you very much for your help, and I'm sorry if it's something tht has been asked before,
    James 
    Solved!
    Go to Solution.
    Attachments:
    ReadBadTDMSpeedTest.vi ‏44 KB

    TDMS is an all binary format whereas TDM posses an XML header, otherwise they are very similar. Being a binary format you would expect it to read faster especially as you mentioned that the header was complex.
    There are a number of un-closed references in your code and I wonder if this was partly the issue, although this wouldn't explain why converting to TDMS fixed the problem?
    Beyond that, it would be nice to try this without using the express VIs to see if the problem still occurs. 
    Also, where you able to see if this was a memory issue? And if the read time continued to increase beyond what you posted?
    Anyway, I am glad we found a solution to your problem.
    Nick C.
    Cardiff University

Maybe you are looking for