Problem in iterating through huge number of entries ... memory issue

Question

I am having large number of entries in lacs.
I need to check each entry with previous ones , so I need to iterate through the each and every entry previous to current entry.
Here I have put all the entries in arraylist and I am iterating though this arraylist and validating current entry.
As I am using the arraylist for storing all the entries , the system is taking lot of memory because I am putting all in the memory.
Is there anyway to resolve this memory issue?

Answer 1

If you have a hundred thousand entries, to verify all the entries you would need to compare the later entries with the earlier ones such that:
Entry 1 = 100,000 accesses
Entry 2 = 99,999 accesses
Entry 5 = 99,995 accesses
Entry 10 = 99,990 accesses
Entry 99,990 = 10 accesses
Entry 99,995 = 5 accesses
Entry 99,999 = 1 access (only to be compared with the 100,000th entry)
It would then make sense to keep a certain (fixed) amount of entries in memory, those which get accessed the most, then write to file those which get accessed the least frequently. Have a method that can decide, based on the index of the entry, whether to get it from the ArrayList or to read it from the file. That will give you good performance.
BufferedReader would be a good candidate to read the file, because it can skip more bytes at once should you need a line very late in the file. Also consider using RandomAccessFile and seek(entryNumber * ENTRY_SIZE) if your data can be represented with a fixed size in bytes.
s
Edited by Looce at 2009-03-25 20:33:11 GMT [Added BufferedReader advice]