How to get file line count.

Question

Hey guys,
How to get file line count very fast? I am using BufferedReader to readLine() and count. But when dealing with big file, say several GB size, this process will be very time consuming.
Is there any other methods?
Thanks in advace!

Answer 1

What I'd do is you create an infofetcher, register a listener, implement gotMore() and have that scan for '\n'
Some might suggest getting rid of the listener/sender pattern or use multiple threads to make ii faster. This might help a little, but only if your I/O is super-duper speedy.
you are welcome to use and modify this code, but please don't change the package or take credit for it as your own work.
InfoFetcher.java
============
package tjacobs.io;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Iterator;
* InfoFetcher is a generic way to read data from an input stream (file, socket, etc)
* InfoFetcher can be set up with a thread so that it reads from an input stream
* and report to registered listeners as it gets
* more information. This vastly simplifies the process of always re-writing
* the same code for reading from an input stream.
* <p>
* I use this all over
     public class InfoFetcher implements Runnable {
          public byte[] buf;
          public InputStream in;
          public int waitTime;
          private ArrayList mListeners;
          public int got = 0;
          protected boolean mClearBufferFlag = false;
          public InfoFetcher(InputStream in, byte[] buf, int waitTime) {
               this.buf = buf;
               this.in = in;
               this.waitTime = waitTime;
          public void addInputStreamListener(InputStreamListener fll) {
               if (mListeners == null) {
                    mListeners = new ArrayList(2);
               if (!mListeners.contains(fll)) {
                    mListeners.add(fll);
          public void removeInputStreamListener(InputStreamListener fll) {
               if (mListeners == null) {
                    return;
               mListeners.remove(fll);
          public byte[] readCompletely() {
               run();
               return buf;
          public int got() {
               return got;
          public void run() {
               if (waitTime > 0) {
                    TimeOut to = new TimeOut(waitTime);
                    Thread t = new Thread(to);
                    t.start();
               int b;
               try {
                    while ((b = in.read()) != -1) {
                         if (got + 1 > buf.length) {
                              buf = IOUtils.expandBuf(buf);
                         int start = got;
                         buf[got++] = (byte) b;
                         int available = in.available();
                         //System.out.println("got = " + got + " available = " + available + " buf.length = " + buf.length);
                         if (got + available > buf.length) {
                              buf = IOUtils.expandBuf(buf, Math.max(got + available, buf.length * 2));
                         got += in.read(buf, got, available);
                         signalListeners(false, start);
                         if (mClearBufferFlag) {
                              mClearBufferFlag = false;
                              got = 0;
               } catch (IOException iox) {
                    throw new PartialReadException(got, buf.length);
               } finally {
                    buf = IOUtils.trimBuf(buf, got);
                    signalListeners(true);
          private void setClearBufferFlag(boolean status) {
               mClearBufferFlag = status;
          public void clearBuffer() {
               setClearBufferFlag(true);
          private void signalListeners(boolean over) {
               signalListeners (over, 0);
          private void signalListeners(boolean over, int start) {
               if (mListeners != null) {
                    Iterator i = mListeners.iterator();
                    InputStreamEvent ev = new InputStreamEvent(got, buf, start);
                    //System.out.println("got: " + got + " buf = " + new String(buf, 0, 20));
                    while (i.hasNext()) {
                         InputStreamListener fll = (InputStreamListener) i.next();
                         if (over) {
                              fll.gotAll(ev);
                         } else {
                              fll.gotMore(ev);
InputStreamListener.java
====================
package tjacobs.io;
     public interface InputStreamListener {
           * the new data retrieved is in the byte array from <i>start</i> to <i>totalBytesRetrieved</i> in the buffer
          public void gotMore(InputStreamEvent ev);
           * reading has finished. The entire contents read from the stream in
           * in the buffer
          public void gotAll(InputStreamEvent ev);
InputStreamEvent
===============
package tjacobs.io;
* The InputStreamEvent fired from the InfoFetcher
* the new data retrieved is from <i>start</i> to <i>totalBytesRetrieved</i> in the buffer
public class InputStreamEvent {
     public int totalBytesRetrieved;
     public int start;
     public byte buffer[];
     public InputStreamEvent (int bytes, byte buf[]) {
          this(bytes, buf, 0);
     public InputStreamEvent (int bytes, byte buf[], int start) {
          totalBytesRetrieved = bytes;
          buffer = buf;
          this.start = start;
     public int getBytesRetrieved() {
          return totalBytesRetrieved;
     public int getStart() {
          return start;
     public byte[] getBytes() {
          return buffer;
ParialReadException
=================
package tjacobs.io;
public class PartialReadException extends RuntimeException {
     public PartialReadException(int got, int total) {
          super("Got " + got + " of " + total + " bytes");
}