Looping through HUGE directory - How to do so most efficiently?

Question

Hello,
I have a HUGE directory (200k files or so) - currently I'm using the below method to read the file names and place the results into a map. However, this is on a windows box and to run the class that uses this, I have to increase the memory usage via args to get it to run. Is there a better/more efficient way to interate through a huge directory? So for example, if I want to compare each file name in the directory to another filename in a List, how can I get a glob of files (say 100 or so), check that list, and return to the position I left off at?
    private Map listAllFiles() {
        Map files = new HashMap();
        File dir = new File(DIR_PATH);
        if (dir.exists() && dir.isDirectory()) {
            try {
                dir = dir.getCanonicalFile();
                File[] f = dir.listFiles();
                for (int i = 0; i < f.length; i++) {
                    files.put(f.getName(), f[i].getPath());
} catch (Exception e) {
e.printStackTrace();
return files;

Answer 1

SuperBadLuck wrote:
for each file run a preparedstatementOh dear. Running 200 thousand SQL statements doesn't sound fast, even if you're using PreparedStatements.
I am gathering that this is your situation: You have a directory, and a list of files in a database. You want to delete any file that is in the directory but not in the database. Apparently selecting the whole list of files in the database overruns your heap space, so you want to break it down.
Here's what I'd do:
For a chunk of files length M:
cleanChunk(SortedSet files) {
   String lower = files.first();
   String upper = files.last();
   //execute query SELECT db_files.path FROM db_files WHERE db_files.path BETWEEN lower and upper ORDER BY db_files.path;
   for each row in result set {
     files.remove(row);
   for each path in files{
      delete_file(path);
}That leaves the implementation of how you call cleanChunk up to the caller (whether you use an iterator over the directory and add each file to a set, calling cleanChunk every M files and then clearing the set, or some other method). Then you just need to optimize M.
Edited by: endasil on 31-Mar-2008 3:30 PM