Parse large (2GB) text file

Question

Hi all,
I would like your expert tips on efficient ways (speed and memory considerations) for parsing large text files (~2GB) in Java.
Specifically, the text files in hand contain mobility traces that I need to process. Traces have a predefined format, and each trace is given on a new line in the text file.
To obtain the attribues of each trace i use java.util.regex.Pattern and java.util.regex.Matcher.
Thanks in advance,
Nick

Answer 1

Memory mapped files are faster when you need random access and you don't need to load all the data, however here it just add complexity you don't need IMHO.
I suspect most of the time is taken by the parser so if you customise your parser it could be faster. Here is a simple custom parser
public static void main(String... args) throws IOException {
    String template = "tr at %.1f \"$obj(1) pos 123.20 270.98 0.0 2.4\"%n";
    File file = new File("/tmp/deleteme.txt");
//        if(!file.exists()) {
        System.out.println(new Date()+": Writing to "+file);
        PrintWriter pw = new PrintWriter(file);
        for(int i=0;i<Integer.MAX_VALUE/template.length();i++)
            pw.printf(template, i/10.0);
        pw.close();
        System.out.println(new Date()+": ... finished writing to " + file + " length= " + file.length() / 1024 / 1024 + " MB.");
    long start = System.nanoTime();
    final BufferedReader br = new BufferedReader(new FileReader(file), 64 * 1024);
    for(String line;(line = br.readLine()) != null;) {
        int pos = 6;
        int end = line.indexOf(' ', pos);
        double time = Double.parseDouble(line.substring(pos, end));
        pos = line.indexOf('s', end+12)+2;
        end = line.indexOf(' ', pos+1);
        double x = Double.parseDouble(line.substring(pos, end));
        pos = end+1;
        end = line.indexOf(' ', pos+1);
        double y = Double.parseDouble(line.substring(pos, end));
        pos = end+1;
        end = line.indexOf(' ', pos+1);
        double z = Double.parseDouble(line.substring(pos, end));
        pos = end+1;
        end = line.indexOf('"', pos+1);
        double velocity = Double.parseDouble(line.substring(pos, end));
    br.close();
    long time = System.nanoTime() - start;
    System.out.printf(new Date()+": Took %,f sec to read %s%n", time / 1e9, file.toString());
{code}
prints
{code}
Sun May 08 09:38:02 BST 2011: Writing to /tmp/deleteme.txt
Sun May 08 09:42:15 BST 2011: ... finished writing to /tmp/deleteme.txt length= 2208 MB.
Sun May 08 09:43:21 BST 2011: Took 66.610883 sec to read /tmp/deleteme.txt
{code}