Fastest Text Parsing Technique

Question

Hi all,
I have a set of huge text files that I need to quickly parse and remove all the asteriks characters from. At the moment my simple program looks like this:
import java.io.*;
public class QuickParse
     public static void main(String[] args) throws Exception
          BufferedReader in = new BufferedReader(new FileReader(args[0]));
          PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(args[1])));
          String str = "";
          while ((str = in.readLine()) != null)
               out.write(str.replaceAll("\\*","") + "\r\n");
          in.close();
          out.close();
}Is there a faster way of doing this? These files are well over a gig each. I'm aware that at the moment this program may well be leaving lots of String objects in memory, but I'm not sure what to do about it, or if this would slow the process down.
Thanks for any help or ideas.

Answer 1

I created a test file with 100k lines and total file size of about 10.6M. Each line looked like this:
0*1000000000*2000000000*3000000000*4000000000*5000000000*6000000000*7000000000*8000000000*9000000000*
I never know exactly how to test IO since the second time you read a file it is usally faster because part of the file is in memory. Anyway, I ran my test program 6 times changing the order of method invocation within each run. Here are my results:
c:\java\temp>java QuickParse 1
Replace : 2297
Tokenize: 1859
Movebyte: 1938
c:\java\temp>java QuickParse 2
Tokenize: 1000
Movebyte: 1937
Replace : 2219
c:\java\temp>java QuickParse 3
Movebyte: 1937
Replace : 2281
Tokenize: 1563
c:\java\temp>java QuickParse 1
Replace : 2297
Tokenize: 1469
Movebyte: 1922
c:\java\temp>java QuickParse 2
Tokenize: 1015
Movebyte: 1906
Replace : 2219
c:\java\temp>java QuickParse 3
Movebyte: 1906
Replace : 2281
Tokenize: 969
Order of method execution didn't seem to matter. The results (from best to worst) was always:
a) Tokenize
b) Movebyte
c) Replace
Here is the code I executed:
import java.io.*;
import java.util.*;
public class QuickParse
     public static void main(String[] args) throws Exception
//          String input = "C:\\Java\\j2sdk1.4.2\\src\\javax\\swing\\JComponent.java";
          String input = "data100000";
          if (args[0].equals("1"))
               replace(input, "output.replace");
               tokenize(input, "output.tokenize");
               movebyte(input, "output.movebyte");
          else if (args[0].equals("2"))
               tokenize(input, "output.tokenize");
               movebyte(input, "output.movebyte");
               replace(input, "output.replace");
          else if (args[0].equals("3"))
               movebyte(input, "output.movebyte");
               replace(input, "output.replace");
               tokenize(input, "output.tokenize");
     public static void replace(String input, String output) throws Exception
          long start = System.currentTimeMillis();
          BufferedReader in = new BufferedReader( new FileReader( input ) );
          BufferedWriter out = new BufferedWriter( new FileWriter( output ) );
          String str = "";
          while ((str = in.readLine()) != null)
               out.write(str.replaceAll("\\*","") + "\r\n");
          in.close();
          out.close();
          System.out.println("Replace : " + (System.currentTimeMillis() - start) );
     public static void tokenize(String input, String output) throws Exception
          long start = System.currentTimeMillis();
          BufferedReader in = new BufferedReader( new FileReader( input ) );
          BufferedWriter out = new BufferedWriter( new FileWriter( output ) );
          String str = "";
          while ((str = in.readLine()) != null)
               StringTokenizer st = new StringTokenizer(str, "*");
               while (st.hasMoreTokens())
                    out.write(st.nextToken());
               out.write("\r\n");
          in.close();
          out.close();
          System.out.println("Tokenize: " + (System.currentTimeMillis() - start) );
     public static void movebyte(String input, String output) throws Exception
          long start = System.currentTimeMillis();
          BufferedInputStream in = new BufferedInputStream( new FileInputStream( input ) );
          BufferedOutputStream out = new BufferedOutputStream( new FileOutputStream( output ) );
          byte[] buffer = new byte[4 * 1024];
          int bytes;
          while ( (bytes = in.read( buffer )) != -1 )
               for (int i = 0; i < bytes; i++)
                    if (buffer[i] != (byte)'*')
                         out.write( buffer, i, 1 );
          in.close();
          out.close();
          System.out.println("Movebyte: " + (System.currentTimeMillis() - start) );
}