Add data to csv input file

Question

Im using a csv file as my data input for a test. The test data is simply recorded in an array. How do I add this array data back to the original csv file, as an additional column, once the stop button is clicked?
Solved!
Go to Solution.

Answer 1

Actually, you do not have to read the whole file at once, although that does make the programming easier.  You can read and write data in chunks to keep memory usage down and file write efficiency up (65,000 byte file chunks give best read and write performance on Windows systems).  You will end up with a steadily increasing memory buffer as you read a chunk from disk, add to it, then write what you can to disk before reading another chunk.  At the end, you will have a buffer the same size as your added data that you can then stream to disk at the end to finish the file.  The algorithm would go something like this:
Read N bytes from disk
Modify as needed by adding data at the end of each row.  Your data is now of size N + M
Write N bytes to disk, keeping the remaining M in a memory buffer
Read the next N bytes from disk
Modify and append to the current M bytes from the previous iteration.
Write N bytes to disk, keeping the remaining in a memory buffer
Repeat steps  4 - 6 until the entire file has been read.  You now have a buffer of left over data.
Stream leftover data to end of file in 65,000 byte chunks.
In theory, this would give better performance, if done well, since you do not need to convert the entire file to numbers, then back again, but only need to convert the numbers going into the file into text.  In practice, if you are not very careful, all the appending and reshaping of strings and arrays can cause memory issues if you don't reuse and manage the buffers intelligently, causing slowdowns.  This can be made disk speed limited, if done correctly.
This account is no longer active. Contact ShadesOfGray for current posts and information.