Determining the max allowable buffer size

Question

I am working on a program which works with very large amounts of data. Each job that is processed requires the program to read through as many as several hundred files, each of which has tens of thousands of values. The result of the operation is to take the data values in these files, rearrange them, and output the new data to a single file. The problem is with the new order of the data. The first value in each of the input files (all several hundred of them) must be output first, then the 2nd value from each file, and so on. At first I tried doing this all in memory, but for large datasets, I get OutOfMemoryErrors. I then rewrote the program to first output to a temporary binary file in the correct order. However, this is a lot slower. As it processes one input file, it must skip all around in the output file, and do this several hundred times in a 60+ MB file.
I could tell it to increase the heap size, but there is a limit to how much it will let me give it. I'd like to design it in a way so that I could allocate as big of a memory buffer as I can, read through all the files and put as much data in as it can into the buffer, output the block of data to the output file, and then run through everything again for the next block.
So my question is, how do I determine the biggest possible byte buffer I can allocate, yet still have some memory left over for the other small allocations that will need to be done?

Answer 1

The program doesn't append, it has to write data to the binary file out of order because the order that it reads data from the input files is not in the order that it will need to be output. The first value from all of the input files have to be output first, followed by the second value from every input file, and so on. Unless it is going to have several hundred input files open at once, the only way to output it in the correct order is to jump around the output file. For example, say that there are 300 input files with double values. The program opens the first input file, gets the first value and outputs at position 0. Then it reads the second value and must write it to position 300 (or byte position 8*300 considering the size of a double). The third value must go to position 600, and so on. That is the problem, it must output the values in a different order than it reads them.
Without giving any VM parameters to increase the heap size, Java will let me create a byte buffer of 32MB in size, but not much more. If the data to be output is more than 32MB, I will need to work on the data in 32MB chunks. However, if the user passes VM parameters to increase the heap size, the program could create a larger buffer to work on bigger chunks and get better performance. So somehow I need to find out what a reasonable buffer size is to get the best performance.