Very interesting! Who faster?

Question

Suppose you hava a direct bytebuffer(size 50M) to read file content by filechannel.
once the buffer is full,just clear the buffer and continue to read the remaining part of the file.
which one is faster,reading a 300M's file or reading ten 30M's files?
I did my test and found: one 300M file: 30M/second
ten 30M files: 40M/second
if the file is 600M,the reading rate continues to be slower: 21M/second.
why this happen?

Answer 1

The limit to the size of file that you can map into memory is not related to the amount of physical memory
you actually have. However there is a fixed limit of about 1.2 to 1.8 gig on most 32 bit platforms. If you are
lucky enough to be using a 64 bit JVM then this is not a problem. Therefore I now have to recomend that
you don't use this approach (there are issues with closing and then creating new mappings that will
hinder you)
Opening a file has quite a high cost associated with it. But the cost is fixed and when you are reading
5Gig the cost is ignorable. It is probably too small, when comapred to the time taken to read the file, to be taken
into account.
The low performance for larger files could be because the OS is allocating more memory to the disk
cache and therefore forcing your JVM's memory to be paged to the swap file (on the hard disk) this is a
very slow process.
File fragmentation could also account for the slower access time for larger files. What filesystem are you using?
some are better then others for large files. What is the block size on the disk? larger block sizes waste
more space for small files but allow more efficient access to large files.
Above all else, how much memory are you dedicating to your VM (the -Xmx option) please ensure that the
value here is less then the amount of pysical memory your computer has (by a few hundred Meg) otherwise
performance will be appaling.
I would not bother too much with trying to improve the performance at this stage as it is likely that the final
performance will be limited by the processing that you perform on the file you are reading (well you won't
actually know this until you profile the working code) then you will have a basis from which to consdier
optimising the program. Also there is little point fine tuning your application for your computer when it
is unlikely that that will be its final destination. Different computers have different hardware and are
capable of very different performances for a similar operation.
As you know that there is likely to be a optimisation issue in this part of the code perhaps you should
abstract it (write an interface) then you can swap implementaions easilly at a later stage (and compare
them). Perhaps using a different implementaion for different hardware configurations.
matfud