How to extract a column out of a large ASCII file?

Question

Hi all.
After searching the board and applying several solution approaches my problem still remains. Maybe you can help me.
The data source i've to deal with are large ASCII files (~540 MB) with 14 columns (delimiter: TAB). Each column represents one channel. The number of characters in each "field" is variable. I have to read user defined columns (=channels) out of each data set. Needless to say that reading the whole file runs into memory problems.
If anyone has an idea i would be happy
Thanks in advance.
Greets
Kane

Answer 1

I hate to defocus you, but there is a more efficient way to do this.  My apologies that I do not have the time to write code, but here is the pseudo code.
Create an array for your output greater than or equal to what you think you will need.
Read a 65,000 character chunk from the file (or the rest of the file, whichever is smaller).
Use the string search functions functions to find successive line ends and the appropriate tab character delimiters for your column.
Convert and replace the element in your output array.
When done, trim your output array to the right size.
If you drop an LVM read, convert it to a regular VI, and dive in, you will see an example of this type of process.  The idea is to keep disk reads, which are very inefficient, to a minimum.  It also minimizes your memory allocations, because you do not need to resize your input buffer for every line.  Problems you will need to deal with (which are handled by the LVM read) are such things as:
Your line crosses a chunk boundary.
The end-of-file creates a smaller chunk than 65,000 characters (the optimum chunk size for Win32 systems).
The end-of-line character is not well defined (in your case, this is probably not an issue)
Searching for a character can produce memory allocations
You may want to try reading the data as a U8 array instead of a string and doing your searches on that instead of the string.
I have always wanted to write the piece of code, but never had the time or reason to do so.  Good luck.  I will try to help if I can.
This account is no longer active. Contact ShadesOfGray for current posts and information.