File Related problem--- Urgent

Question

Hi Friend's
I have a Problem that and i am Looking for the solution. please Help me.
There is a big file that content 5 lack record or may be more then this. my task is that i have to read this file and if there is any duplicate record then i have to remove this. for that i take two file.
1. out.csv
2. Reject.csv
out.csv content unique record and that record witch are duplicate i have to keep this on reject.csv
Logic...................
1. First i read my main file and keep this record in a vector. now i again read a record from main file and check that this record is in vector or not. if yes-- then keep this in reject and if not then keep this in out file. but this logic fail because of that when record reached till 2.5 lack then exception come out of memory. vector size is too much big.
2. in second for i read a record from main file and then write this in out file. next time when i read this file
then at the same time i read out.csv and match main file reaord with the record of out file. if match then keep this in Reject.csv else write this in out.csv. this will also fail becouse it is taking more then 10 hours as i feel.
so i want help in the problem so that i can form my task in very short time period.......................................
it's very urgent for me. my project is dependent on this thing. Help me.
Thanks

Answer 1

Well 2.5 lakh records is a huge number as far as applying logic is concerned and keeping that in memory like a vector or something is certainly a bad idea.
If the Stated was a give case to me.I'd try apply data pagination here is a sample logic which i can think off.
1).I'd first define what is so called a Records as an entity or a javabean which overides equality method of two objects depending on stated logic
2).I'd first fix a constant pagesize depending applying some probablity logic.
3).Now i'd choose three set of files on which i delete the second file once the merging process is completed.
4).Now on the first go i'd like to parse the the intial file and extract records and save them as JavaBean instances and save in a SortedSet / TreeSet interface.
5).Consequtively, i'd be running a different thread which works on a Random Access file and inserts records in that temp random access file.
6).Now,if the size of Set attains the fixed page size cache the results in Random Access file using the other thread and the empty the set once
7).Would repeat the process uptil end of the parent file.
8).Now,from the second file apply your initial logic only change is you'd using a Set of Entites.
9).That should reduce the your memory problem to a larger extent.
and The reason why i'm going for Set doesn't save duplicate records.
or as said adviced by my fellow posters you might think of using a database in place and solve the issue with simple database queries and expressing the entities accordingly.
REGARDS,
RaHuL