Working with a lot of elements in collections

Question

Hi,
I'm facing a problem: i need to store many items (something about 1,000,000) in a structure similar to Collection or Hashtable, but these are all stored in memory. I want to know if is there any structure like these that stores the items in some secundary storage (such as HD), because i'm getting an OutOfMemoryException when trying to use Hashtable to store this data.
Thanks,
Saulo.

Answer 1

I don't think there's anything in the standard libraries that can handle this, but it's something you can put together fairly quickly.
The key idea is that instead of maintaining the values directly, you wrap all of the values with a java.lang.ref.SoftReferences. A SoftReference will discard the value when memory starts running out. You then assign all of your SoftReferences to a java.lang.ref.ReferenceQueue which notifies you that a value is about to evicted so that you can write it to disk.
Here's how it looks in memory for a hash table:
key -- soft reference -- value
Here's what you need to write:
A put method that wraps the values in a reference before entering it into the table.
A get method that first checks if a value is in the table, and if not, if the value has been evicted to disk.
A thread that repeatedly calls remove() on the reference queue to catch values as they are evicted.
The class to look at is sun.misc.SoftCache. It demonstrates everything you need for implementation except for writing the value to disk. If you used SoftCache, you would just need to change processQueue () to write your objects to disk and fill () to read the object back from disk.