Processing large file in multiple threads

Question

I have a large plain text file (~1GB) that contains lots (~500,000) of chunks of text, separated by the record separator: "\n//\n". In my application, I have a class that reads the file and turns the records into objects, which is fairly time-consuming (the bottleneck is definitely in the processing, not the IO). The app is running on a muti-core machine, so I want to split the processing up over several threads.
My plan is to write a "Dispatcher" class, which reads the file from disk and maintains a queue of records, and a "Processor" class, which requests records from the Dispatcher object, turns the records into objects, and add()s them to an array. Is this the correct way to do it? Can anyone point me in the direction of a tutorial or example of this type of multi-threaded file processing?

Answer 1

public class TaskExecutionr {
    private static final int NTHREADS = 10; // Probably want to do System.getNumberOfProcessors+1 here
    private static final Executor exec
            = Executors.newFixedThreadPool(NTHREADS);
    private static Collection result = new ConcurrentLinkedQueue();
    public static void main(String[] args) throws IOException {
        FileReader reader =
        while (true) {
            final String text = reader.read(500,000); // read 500.000 lines to a string.
            Runnable task = new Runnable() { // create a new runnable that deals with the text
                public void run() {
                    handle(text);
            exec.execute(task); // Place the task in the Executor
    private static void handle(Socket connection) {
        // make your objects here
        result.add(<your new object>);
}http://www.javaconcurrencyinpractice.com/listings/TaskExecutionWebServer.java
Edited by: Fireblaze-II on Jun 19, 2008 5:48 AM