Allocating enough memory to open a large data file

Question

Hello,
I am currently running an experiment which requires me to collect data at a very high rate, creating enormous data files. At first, I ran into the problem of LabVIEW not having enough memory to save all of the data to the hard-disk at once (it seeme to have no problem with a huge array during data collection, only when it went to save it to a file), so I programmed LabVIEW to only save 100000 samples at a time. This seemed to work fine, and I was able to collect a 550MB data file in this way. However, now that I would like to analyze that data, LabVIEW can't read the data from the file into an array, giving me the same insufficient memory errors as before. My system has 3GB of memory, so in theory LabVIEW should be able to get enough to open it, however Windows doesn't seem to want to allocate what it needs. Is there any way that I can override windows and allocate enough memory to LabVIEW for it to be able to open this file and work on the data?

Answer 1

BrockB wrote:
The data is all in a 2xN tab-delimited array, and I'm using the 1d array output of the "read from spreadsheet" vi, as I only need the first of the two columns. What I meant to say earlier was that I would still have to read all of the data from the file first if I wanted to break it up into pieces to use later. Labview seems to get stuck on reading the data from the file, not actually on opening it. It also seems like breaking the data up would be a much bigger hassle than just allowing labview to use more of the 3GB that I have available (most of which is sitting unused anyway).
First of all, any ASCII formatted file is an extreme waste of space and comparatively expensive to write and read. It is only recommeded for small files intended to be later read by humans.
"Read from Spreadsheet file" must first internally read the entire file as a string, then chop it up looking for tab and linefeed characters, then scan the parts into DBL. If you want the first column (instead of the first row), it then also needs to transpose the entire 2D array, requiering another copy in memory. As you can see, you'll easily have quite a few data copies plugging up your RAM. ("Read/Write from/to spreadsheet file" are plain VIs. You can open their diagram to see what's in there. )
For datasets this size, you MUST use binary files. Try it and your problems will go away.
Message Edited by altenbach on 11-11-2008 12:06 PM
LabVIEW Champion . Do more with less code and in less time .