Fastest Way to Load Text Files Containing Thousands of Words

Question

Hi, can anyone give me suggestions for fast loading of files containing many bytes in the form of a single word on a different line. The average file size is 45 kb. I'm currently splitting the words into separate files each containing 4000 words a piece. I use a different thread to read each file and then add all of the words into a single vector, formed by joining the vectors of each reader thread. Any suggestions for speeding up the process? (Currently, it takes about 30 seconds to load 52,000 words). My OS is Windows2000

Answer 1

Something seems wrong. There is no way that what you describe should take anywhere close to that amount of time, unless the computer is a constraint. I created a file of 56,000 lines, each line consisted of the byte string "abcdefgh" plus a \r\n,
for a total of 10 bytes per line - a total file size of 560,000 bytes (547 kb)
The following program read the file, created an ArrayList, and added the 56,000 lines to the list in less than 1 second, about what I expected. You might want to run it and see what you get.
import java.io.*;
import java.util.*;
class Zxin
    public static void main(String[] args)
        throws IOException
        FileInputStream fis = new FileInputStream(
            "C:/Documents and Settings/Chuck/My Documents/Java/junk/word.txt");
        InputStreamReader isr = new InputStreamReader(fis);
        BufferedReader br = new BufferedReader(isr);
        List<String> list = new ArrayList<String>();
        String data;
        while ((data = br.readLine()) != null)
            list.add(data);
        System.out.println("The list contains " + list.size() + " entries");
        System.out.println("The first entry in the list is: " + list.get(0));
        br.close();
}It printed these 2 lines:
The list contains 56000 entries
The first entry in the list is: abcdefgh