Fast file system search not possible in java?

Question

hi,
I'd like to find all xml files on a disk as quickly as possible. The simple implementation takes far too long (e.g. over 30 seconds for a disk with 10,000 files and 1000 folders).
This is a network drive, but the reason I think 30 seconds is slow is that the equivalent C code does it in much less time (over the same network conditions)
I've seen it run on a drive with 200000 files and 30000 folders, and this takes between 10-15 minutes, while C code takes around 1-2 minutes.
I'm not sure what the C code does exactly, and it may have access to much better OS calls than java can assume.
The only three things I can think of are
1) try to use fewer file objects (can't see a way of doing this)
2) try threading (i've done small tests before and not got any gain)
3) try using FileFilters (i've done small tests here too and not got any gain - I think they just implement what you'd do without them anyway..)
the other option is JNI but this is very bad since I need to run any many platforms..
profiling shows that most time is spent in the native code
17.0% 0 + 287 java.io.WinNTFileSystem.list
10.7% 0 + 181 java.io.WinNTFileSystem.getBooleanAttributes
but I've heard that switching between java/native is very expensive? so there may be benefits from doing it all in one go somehow?
any help would be really appreciated,
thanks,
asjf
ps. caching the file list is not really an option since it needs to run fast on new targets

Answer 1

heres some code that is much faster when I/O blocking is the main problem (seems to work much better on network drives..)
it uses a slightly bizarre (?) thread pool of workers that process a list of Invocations.
(this is one big file Pool.java)
asjf
ps. I don't normally write code like this :)
import java.util.*;
import java.lang.reflect.*;
import java.io.*;
public class Pool {
List invocations;
public void invoke(Invocation a){
invocations.add(a);
synchronized(invocations) {
invocations.notify();
//System.out.println("Added an invokation");
ThreadGroup threads;
Set pool;
Pool(int noThreads){
pool = new HashSet();
threads = new ThreadGroup("Pool of "+noThreads);
invocations = Collections.synchronizedList(new LinkedList());
for(int i=0; i<noThreads; i++){
Worker w = new Worker(i+" of "+noThreads);
Thread t = new Thread(threads, w);
pool.add(w);
t.start();
new Thread(new Runnable() {
public void run(){
try{
while(poolActive) {
int active = 0;
for(Iterator i = pool.iterator(); i.hasNext(); ){
Worker w = (Worker) i.next();
if(w.working) active++;
Thread.sleep(3000);
System.out.println(invocations.size() + " : " + active);
}catch(Exception e){
e.printStackTrace();
}).start();
volatile boolean poolActive=true;
public void join() {
try {
boolean alldone = false;
while(!alldone) {
alldone=true;
for(Iterator i = pool.iterator(); i.hasNext(); ){
Worker w = (Worker) i.next();
if(w.working)
alldone=false;
if(invocations.size()>0)
alldone=false;
Thread.sleep(2000);
}catch(Exception e){
e.printStackTrace();
//System.out.println("--------------------------> all done");
poolActive=false;
synchronized(invocations) {
invocations.notifyAll();
public void status() {
threads.list();
public static void main(String [] arg) throws Exception {
long time=-System.currentTimeMillis();
Pool pool = new Pool(200);
Enumerater e = new Enumerater(pool);
Invocation in = new Invocation(null,Enumerater.getXML, new Object [] {new File(arg[0]),e.result});
pool.invoke(in);
//pool.status();
pool.join();
time+=System.currentTimeMillis();
System.out.println("Time taken "+time+"\nFound "+e.result.size());
class Worker implements Runnable {
String name;
volatile boolean working;
public Worker(String name) {
this.name = name;
public void run() {
try {
working=false;
while(poolActive) {
Invocation i = null;
synchronized(invocations) {
while(poolActive && invocations.size()==0){
invocations.wait();
//System.out.println("Hello from "+name);
if(invocations.size()!=0) {
working=true;
i = (Invocation) invocations.remove(0);
//System.out.println(name+" processed: "+i);
if(i!=null)
i.result=i.method.invoke(i.o,i.arg);
working=false;
}catch(Exception e){
e.printStackTrace();
class Invocation {
Object o;
Method method;
Object [] arg;
Object result;
public String toString(){
return o+" "+method+" "+arg;
public Invocation(Object o, Method method, Object [] arg) {
this.o=o;
this.method=method;
this.arg=arg;
this.result=null;
class Enumerater {
Set result;
static Pool pool;
static Method getXML;
static {
try {
getXML = Enumerater.class.getDeclaredMethod("getXML",new Class [] {File.class,Set.class});
}catch(Exception e){
e.printStackTrace();
Enumerater(Pool p){
pool = p;
result = Collections.synchronizedSet(new HashSet());
public static void getXML(File file, Set result) {
File [] children = file.listFiles();
if(children==null)
System.out.println(file.getAbsolutePath()+" has null children");
else
for(int j=0; j<children.length; j++)
if(children[j].isDirectory()) {
Invocation in = new Invocation(null,getXML, new Object [] {children[j],result});
pool.invoke(in);
else
if(children[j].getName().toUpperCase().endsWith(".XML"))
result.add(children[j]);