Help getting accurate metadata for files on network drives

Question

I need to read 1000's of very small files that are addressable through the file system API (File class). The location of the files may be a local drive or via a Windows shared drive. When the files are on a remote system, the files may actually be changed by a process running on that remote system. Much of the time, I can actually cache the information needed from the files, but if the file changes or is deleted then I have to detect the change and reload or delete from my in memory cache. Everything works great when on a local drive, but the File API doesn't work when on a Windows shared network drive.
Basically, File.lastModified(), File.exists(), File.canRead(), File.isFile() etc, all return inaccurate results when the base path is on a mapped network drive, the files are changed by a remote process, and then are immediately accessed by my Java process.
For example,
<li> if the File existed
<li> my program loaded and cached it
<li> then a remote process changes the file
<li> when I call file.lastModified() immediately following the change, it will report the old lastModified date.
So, the net result is that I can't detect the change and end up serving up old data. This is even worse when the file is deleted, but Java File.exists() reports true. If I put in an artificial wait using Thread.sleep() in a while loop, then eventually the correct lastModified and exists will be reported. However, this doesn't help.
I've tried a lot of different ways to try to get Java to report the correct info. So far, the only reliable way is to try to open the file and read the first byte, catching an exception if necessary. This seems to then force the file data to be updated and Java will then correctly report the lastModified date (or throw an exception if it no longer exists). However, attempting to open the IO to the file and reading a single byte pretty much invalidates any reason to cache the data in the files because most of the overhead is in opening and closing the streams.
Is there any way to force Java or Windows to update the information about a file? It seems to me that this probably has to do with Windows caching shared drive file information and giving Java bad data, but it would be really nice to be able to force a refresh a file info.
Example:
//Assume that this is already behind code that did a file.exists() or file.canRead();
        long instanceFileModifyTime = fileObject.lastModified();
        MyObject cachedResult = INSTANCE_CACHE.get(cacheKey);
        if ((null != cachedResult) && (instanceFileModifyTime == cachedResult.getLoadTime())) {
            result = cachedResult;
        } else {
            //Open IO and load the data
        }

Answer 1

I need to read 1000's of very small files that are addressable through the file system API (File class). The location of the files may be a local drive or via a Windows shared drive.This is a very bad idea.
When the files are on a remote system, the files may actually be changed by a process running on that remote system. Much of the time, I can actually cache the information needed from the files, but if the file changes or is deleted then I have to detect the change and reload or delete from my in memory cache.That's also a very bad idea. Application-side caching is fraught with problems and this is one of them.
Everything works great when on a local drive, but the File API doesn't work when on a Windows shared network drive.
Basically, File.lastModified(), File.exists(), File.canRead(), File.isFile() etc, all return inaccurate results when the base path is on a mapped network drive, the files are changed by a remote process, and then are immediately accessed by my Java process. That will be the network file system. Nothing Java can do about it. Java neither knows nor cares. You are at the mercy of what the networked file system does. And that's why it's a very bad design. Networked file systems are for users, not for applications.
So, the net result is that I can't detect the change and end up serving up old data.Only because you cached it, which is why that's a bad idea. If you hadn't cached it you would have had to read the file again and you would have got the current data. And it would have been slow, because networked file systems are slow, which is why that's a bad idea.
Basically you are trying to use a remote file system as though it was a transactional database: it isn't, so you are trying to write more stuff in front of it to make it better; instead you are making it worse.
What you should be doing is using a transactional database as a transactional database.