Fastest way to count the number of occurences of string in file

Question

I have an application that will process a number of records in a plain text file, and the processing takes a long time. Therefore, I'd like to first calculate the number of records in the file so that I can display a progress dialog to the user (e.g. " 1234 out of 5678 records processed"). The records are separated by the string "//" followed by a newline, so all I need to do to get the number of records is to count the number of times that '//' occurs in the file. What's the quickest way to do this? On a test file of ~1.5 Gb with ~500 000 records, grep manages under 5 seconds, whereas a naive Java approach:
BufferedReader bout = new BufferedReader (new FileReader (sourcefile));
               String ffline = null;
               int lcnt = 0;
               int searchCount = 0;
               while ((ffline = bout.readLine()) != null) {
                    lcnt++;
                    for(int searchIndex=0;searchIndex<ffline.length();) {
                         int index=ffline.indexOf(searchFor,searchIndex);
                         if(index!=-1) {
                              //System.out.println("Line number " + lcnt);
                              searchCount++;
                              searchIndex+=index+searchLength;
                         } else {
                              break;
               }takes about 10 times as long:
martin@martin-laptop:~$ time grep -c '//' Desktop/moresequences.gb
544064
real     0m4.449s
user     0m3.880s
sys     0m0.544s
martin@martin-laptop:~$ time java WordCounter Desktop/moresequences.gb
SearchCount = 544064
real     0m42.719s
user     0m40.843s
sys     0m1.232sI suspect that dealing with the file as a whole, rather than line-by-line, might be quicker, based on previous experience with Perl.

Answer 1

Reading lines is very slow. If your file has single byte character encoding then use something like the KMP algorithm on an BufferedInputStream to find the byte sequence of "//\n".getBytes(). If the file has a multi-byte encoding then use the KMP algorithm on a BufferedReader to find the chars "//\n".getCharacters() .
The basis for this can be found in reply #12 of http://forum.java.sun.com/thread.jspa?threadID=769325&messageID=4386201 .
Edited by: sabre150 on May 2, 2008 2:10 PM