Working with big vectors

Question

My program works with a large vector of objects.
The problem is that the vector is to big to be stored in memory.
Is there any utility that will store the whole vector in a file
and seamlessly cache parts of it to memory ?
Thanks.

Answer 1

It seems I will have to build my own file mapping.
Since all my objects have the same size (e.g. all
Integers or String),
what is the best and fast way to read a specific
object (e.g. vector[200]) ? Also, what is the fast way
to read a bulk of objects (e.g. vector[200-300]) ?If your are dealing with integers or Strings I don't think you should go for a Vector filled with objects solution. You can minimize the memory requirements quite a bit if you use arrays.
Integers. Why not use an int array? Each int takes up 4 bytes in memory but an Integer Object adds an overhead of maybe 16 byte. If you use int's your memory need shrinks to a fifth.
Strings. A char takes 2 bytes. Maybe you can store a String as a byte array instead, thus saving 1 byte per char. The memory need shrinks to a half. It would be even more efficient if you just dumped all strings in one large byte array (with a null byte terminating each string). In addition you would need a second array with the start positions to the strings.
These are the densest in memory data structures you can get. If you still have too much data you will have to go for files. The in memory solutions I've suggested can be directly used with files (just substitute array with file).