Do integrity checks by file comparison really work?

Question

Hello,
I use a program (Beyond Compare) which has a Folder Compare module, with a function for doing a binary comparison between the files inside two folders. I use it to check the integrity of my backups, by comparing my main hard disk files, with the ones on
the backup hard disks. If two files get recognized as being different, then one of them should be corrupt (if they didn't get modified "normally").
My question is: will this method work to detect real file corruptions? Will it detect any kind of corruption, both copy corruptions, and "bit rot" corruptions? Will there be no problem regarding read cache? I mean: if the first file to compare gets read
from disk, and is kept in a read cache (by Windows or by the hard disk, or by something else), will the second file get read from the same cache (if Windows or the hard disk think they are identical files)? Do Windows and hard disks have some kind of procedure
to detect if a file to read from disk is already available in cache, even if it is in a different folder than the "original" one? Perhaps some kind of file-checksum-system which decides that the files are same? (And this system would not notice if the file
to compare is corrupt). If this would be true, then integrity checks by file comparison would not work, because in practice the same file would be read twice (first from disk, and then from cache), instead of reading both the two files to be compared from
disk.
I have already done tests by manually "corrupting" files (changing slightly the contents, while keeping size and timestamp the same), and it works (the files get recognized as different). But I'm not sure if it will work also with "real" corrupt files.
I'm interested mostly about Windows 8 Pro 64bit and NTFS (but would like to know also in general).
Thanks.

Answer 1

I also have Beyond Compare and have used it to check backup data.
Yes, Windows does RAM caching.  If you have a comparison program read enough file data that you
can be sure the RAM cache is flushed, then you can be sure that reading the file data back and comparing it with the original data is a valid way to ensure you have uncorrupted backups.
Note that NTFS now does online recovery, so just the act of reading back the data can trigger processes inside the file system implementation that will recover the data if it should start to experience "bit rot" (weakening or failure of the
storage medium).  But this is not strictly necessary as other operations, such as defragmentation etc., will occasionally access the disk data as well.
Some time back I wrote a small utility that calculates a CRC-32 on all the data from all the files in a set of folders.  This is another way that re-reading all the data can be triggered, as well as producing a summary number that can be easily
compared to determine that all is well - though one doesn't need my software to do it...  There are hash programs available that can accomplish the same things.  Search for SHA-1 programs, but beware there can be malware associated with free programs
and download sites.
It's good that people think about data integrity.  There's all too little of that nowadays.
-Noel
Detailed how-to in my eBooks:  
Configure The Windows 7 "To Work" Options
Configure The Windows 8 "To Work" Options