Best way to record 50 kS/s data to file

I am trying to read data from a DAQ at a sampling rate of 50 kS/s and record it to a file. My application needs to run for at least a few hours. Initially, I tried a producer-consumer loop pattern with a Write to Measurement File Express VI, but it wrote the data too slowly, and my queue filled up. Now I am trying to use the Write to Spreadsheet File VI with or without a producer-consumer loop pattern (see attached VIs). Both seem to write data to file, but neither writes the number of datapoints I would expect (both have far fewer datapoints).
What is the best way to write this data to file? This seems like a basic question, so if it has already been discussed at length in another forum, or if there are examples of it which someone could direct me to, that would also be appreciated.
Thank you!
Solved!
Go to Solution.
Attachments:
Record Data 8-7-13 with Producer Consumer.vi ‏54 KB
Record Data 8-7-13 without Producer Consumer.vi ‏48 KB

I agree that TDMS is probably the best way to go for fast data streaming.
I also want to point out that you can read TDMS files with some spreadsheet software. This makes it easier to manage and provides a method for human readability.
Add in tool for Open Office Calc
http://www.ni.com/white-paper/6849/en
Add in tool for MS excel
http://www.ni.com/white-paper/4906/en
Jeremy P.
Applications Engineer
National Instruments

Similar Messages

  • Best way to stream lots of data to file and post process it

    Hello,
    I am trying to do something that seems like it should be quite simple but am having some difficulty figuring out how to do it.  I am running a test that has over 100 channels of mixed sensor data.  The test will run for several days or longer at a time and I need to log/stream data at about 4Hz while the test is running.  The data I need to log is a mixture of different data types that include a time stamp, several integer values (both 32 and 64 bit), and a lot of floating point values.  I would like to write the data to file in a very compressed format because the test is scheduled to run for over a year (stopping every few days) and the data files can get quite large.  I currently have a solution that simply bundles all the date into a cluster then writes/streams the cluster to a binary file as the test runs.  This approach works fine but involves some post processing to convert the data into a format, typically a text file, that can be worked with in programs like Excel or DIAdem.   After the files are converted into a text file they are, no surprise, a lot larger than (about 3 times) the original binary file size.
    I am considering several options to improve my current process.  The first option is writing the data directly to a tdms file which would allow me to quicly import the data into DIAdem (or Excel with a plugin) for processing/visualization.   The challenge I am having (note, this is my first experience working with tdms files and I have a lot to learn) is that I can not find a simple way to write/stream all the different data types into one tdms file and keep each scan of data (containing different data types) tied to one time stamp.  Each time I write data to file, I would like the write to contain a time stamp in column 1, integer values in columns 2 through 5, and floating point values in the remaining columns (about 90 of them).  Yes, I know there are no columns in binary files but this is how I would like the data to appear when I import it into DIAdem or Excel.  
    The other option I am considering is just writing a custom data plugin for DIAdem that would allow me to import the binary files that I am currently creating directly into DIAdem.  If someone could provide me with some suggestions as to what option would be the best I would appreciate it.  Or, if there is a better option that I have not mentioned feel free to recommend it.  Thanks in advance for your help.

    Hello,
    Here is a simple example, of course here I only create one value per iteration in the while loop for simplicity. You can also set properties of the file which can be useful, and set up different channels.
    Beside, you can use multiple groups to have more flexibility in data storage. You can think of channels like columns, and groups as sheets in Excel, so you see this way your data when you import the tdms file into Excel.
    I hope it helps, of course there are much more advanced features with TDMS files, read the help docs!

  • What is the best way to kill/stop a data load?

    Hi.
    What is the best way to kill/stop a data load?
    I have a data load from my QA R/3 system that is extracting 115.000.000+ records. The problem is that the selection in the function module used in the data source does not work, and the problem was not detected because of the nature of the data on the development system.
    I could kill processes owned by my background user (on both R/3 and BW) but I risk killing other loads, and sometimes the job seems to restart if I just try to kill processes. If I remove transactional RFCs in SM58 the load does not terminate; I only skip one or more datapackages. I have also tried to change the QM-status in the monitor to red, but that does not stop the load either...
    So isn't there a nice fool-proof way of stopping a dataload?
    Best regards,
    Christian Frier

    Hi,
    There r 2 ways to kill the job.
    One is using transation RSMO locate the job and display the status tab double click on the yellow light that is shown on the line total, a pop will come 'set overall status ' is displayed select the desired status that is red and save it. Then return to the monitor page and select the header tab double ckick on the data target right click and then goto 'manage',there should be request sitting there probably with yellow lights , highlight the line with the faulty request click the delete button then click refresh button.
    Second is goto SM37 and click on the active selection and enter the jobname and then click excute the particulr job should appear highlight the jobname then click on the stop iconthat appears on the taskbar( 3 rd from left)
    hope it is clear.
    Regards-
    Siddhu

  • Best way to  back up your data

    Which is the fastest and best way to back up your data in case of any problem ? Still to transfer to an external HD ?
    Thanks

    Hi Ferro;
    Best and simplest way to back up is Time Machine to an external drive.
    I think that any backup plan should alway be to an external drive. If you backup to an internal drive and the Mac fails, what good is your backup then?
    Allan

  • What is the best way to record a video(not big) for free on a mac w/isight?

    What is the best way to record a video(not big) for free on a mac w/isight? I need to just make a short (less than 5 minute, no special effects) video, but I can't figure out any programs that will make this possible for me.
    Thanks.

    Hi Carolyn,
    you can use iMovie to record a video for free on a Mac with iSight.
    1. Open iMovie HD
    2. Choose “Create a new project”
    3. Name the project for example “iSight Movie” (first field). In the second field, let the default location “Sequences”, in the third one, video format, choose “iSight”. Push “Create”
    4. iMovie window opens, next to the scissors, choose the camera (Camera mode), then iSight. Now you appear on the screen, right?
    5. Press “Record with iSight” in the main picture window.
    6. Press the same button to stop. Your new clip appears now in the clip panel at the right.
    7. You are done!
    iMac G5 PPC 2,1 Ghz   Mac OS X (10.4.8)  

  • What is the best way to record a project to an external recorded via firewire. Also, who makes a recorder that works with a mac?

    What is the best way to record a project from the timeline to an external recorder via firewire? Also, who makes a recorder that works with a mac and recordes in realtime? This is possible right?

    While theoretically possible, sometimes the camera people disable recording back to tape from the computer due to DRM (digital rights managment) issues. They will allow tape to tape transfers however.
    Test your process first is all I can advise.
    x

  • What's the best way to record different articulations in 1 phrase?

    In my first violins I am using forte legato, pianissimo pizz, forte tremelo and forte marcato in 1 phrase within 4 bars. What now would be the best way to record this? I am using Peter Siedlaczek's Complete Classical Collection. Do I make a seperate track for every effect and record 1 bar on every track? Or do I use the Key Switch for every effect withing 1 track? What is the most easiet way of working and especiallyu, what used not all my processor power? I only have a PowerPc with w GB ram.

    Seems like you've answered your own question as those are the two ways to do what you are trying to do. Using the key-switch will be less CPU intensive than using multiple instances of the plugin.

  • Best way to transfer internal HD data in Mavericks to Mac with Yosemite ?

    Hello,
    I'm using a 2007 iMac with Mavericks and will be getting a new one which will presumably come with Yosemite installed. What's the best way to transfer all the data from my internal HD on the old system, to the new one ?
    I use SuperDuper to make backups to external HD's, so if I make a bootable copy of the mac HD to an ext HD using SuperDuper, will everything function fine despite the different OS's ?
    Thanks,
    Matrose.

    You can make a bootable copy of your system now, but you won't be able to boot from it with the new computer. They are not usually backwards compatible with the OS. But when you first boot into the new system, use SetUp Assistant to migrate the data from the cloned copy. That will work just fine.

  • What is the best way of recording bass

    Hi Peeps
    What is the best way to record a bass guitar in Logic. I'm going through a Fast Track Pro Audio Interface. The songs we are recording a disco cover, like get down on it and freak out, so need quite a warm rounded tone to them with a solid level.
    Thanks
    Matt

    Hi Peeps
    What is the best way to record a bass guitar in Logic. I'm going through a Fast Track Pro Audio Interface. The songs we are recording a disco cover, like get down on it and freak out, so need quite a warm rounded tone to them with a solid level.
    Thanks
    Matt

  • BEST WAY TO RECORD FROM TV

    Whats the best way to record from TV and then be able to edit the footage, without losing quality?
    Is it with a dvd recorder, how to you get the files off the recorder???
    Thanks
    Simon

    If you already have a digital video camera, you can probably do it without buying anything extra. I have done this by connecting the video output from my able box to an input on my video camera, then connected the camera to the computer using the FireWire cable. I opened iMovie and imported the video from there. Of course, it's a bit of a pain, so I haven't done it often, and the quality is only as good as your camera is capable of. You may need to buy the cable to connect the cable box to the camera, though mine came with such a cable.

  • What is the best way to record video games on my mac

    what is the best way to record video games on my mac?

    I'm unclear about what you're asking - could you be a little clearer? I know you say that you want to "record video games" but I'm not sure if you're asking if you could, for instance, create a movie of video games as they're being played or what.
    Regards,
    Clinton

  • Best way to record electric guitar

    Hi all,
    after months of playing with GarageBand, a fantastic tool, I decided to try to record something with my electric guitar, just to see if I could be able to compose a song
    Well, I recently got a Dual G5 PowerMac, I saw it has a line in and a digital audio connector. I even got an iMic, a leftover from the "old" Dual G4 I had.
    So far, which is the best way to record my electric guitar? I already have the 1/4" -> 1/8" adaptor, should I just plug it in the iMic or the line in port on the rear of the Mac?
    Or , should I use a pre-processor before sending the signal to the Mac (I have an effect simulator from yamaha)?
    And, in case neither of the 2 scenarios are really worth it, what do you suggest?
    I already tried using the search feature, but I really couldn't find 1 clear topic that summed up my questions... sorry 'bout that.
    Thanks in advance,
    Regards

    You can certainly try the Line-In, you won't hurt anything. Most people find that it doesn't work very well because guitar is not "Line-Level".
    Others question the quality of the A/D converts built in to the Mac. Personally, I doubt the iMic's are any better, but the level problem usually forces one away from the direct Line In without some form of pre-amp.
    I've used the iMic (though with a mixing console) and found it acceptable. A higher quality interface (read much more money) yields much better results ... it's always a trade off.

  • What's the best way to record a phone conversation to GB?

    i am doing a small podcast with my best friend "the joseph podcast" and we have a few celebrities calling in to our show, we can't believe it either, Angela Kinsey from "the office" and Charlie Day from "it's always sunny in philadelphia" and we were wondering what the best way to record a phone conversation would be. we need help and we are pretty much poor so keep it cheap. thanks so much!

    If I were trying to record something very important, likely "get one chance only" type of thing, I would not use GB to do the recording. I would use a tape recorder and then later transfer the recording to GB.
    I'd use something like this BELL-SONECOR TR-70 Tele-Recorder Adapter for $13.
    --HangTime [Will Compute for Food] B-|>
    Note: I am an Amazon Associate, if you purchase this item via my link I will get a small commission)

  • Best way to record CDs and vinyl to iPad

    I am looking for the best way to record pre-recorded music, (CDs, vinyl) to my iPad. I have an M-audio interface I used with my MacBook.  I am looking for the best app to use, hopefully with an analog style VU meter with a needle. thank you.

    Hi Paul
    Thanks for the response. My HTC can record my video, but it is my old phone so is no longer connected to network or even works as a phone. This is why I need to either beam/bluetooth the file, or use a cable if that's the way to do it.
    Unless I'm missing some other obvious way. I suppose if I have to, I can record video, either on my HTC or digicam then convert and drop into itunes, synch and then its on my iphone to upload to YouTube (if this format works for YouTube)
    Long story short - I'm looking for a quck way to record video and upload using the iphone, on the go (ie. not needing my home PC)
    Thanks again!

  • Best way to spool DYNAMIC SQL query to file from PL/SQL

    Best way to spool DYNAMIC SQL query to file from PL/SQL [Package], not SqlPlus
    I'm looking for suggestions on how to create an output file (fixed width and comma delimited) from a SELECT that is dynamically built. Basically, I've got some tables that are used to define the SELECT and to describe the output format. For instance, one table has the SELECT while another is used to defined the column "formats" (e.g., Column Order, Justification, FormatMask, Default value, min length, ...). The user has an app that they can use to customize the output...which leaving the gathering of the data untouched. I'm trying to keep this formatting and/or default logic out of the actual query. This lead me into a problem.
    Example query :
    SELECT CONTRACT_ID,PV_ID,START_DATE
    FROM CONTRACT
    WHERE CONTRACT_ID = <<value>>Customization Table:
    CONTRACT_ID : 2,Numeric,Right
    PV_ID : 1,Numeric,Mask(0000)
    START_DATE : 3,Date,Mask(mm/dd/yyyy)The first value is the kicker (ColumnOrder) as well as the fact that the number of columns is dynamic. Technically, if I could use SqlPlus...then I could just use SPOOL. However, I'm not.
    So basically, I'm trying to build a generic routine that can take a SQL string execute the SELECT and map the output using data from another table to a file.
    Any suggestions?
    Thanks,
    Jason

    You could build the select statement within PL/SQL and open it using a cursor variable. You could write it to a file using the package 'UTL_FILE'. If you want to display the output using SQL*Plus, you could have an out parameter as a ref cursor.

Maybe you are looking for