How to find file duplicates only matching size, not md5

Question

Hey guys, first post here.
I thought I'd share this, since it was quite useful for me. It may not be the most elegant solution, but it worked for me .
So, I have a terabyte (or so) of data, spanning 51,000+ files. Some are greater than 4GiB, many/most are ~4MiB. I have a (relatively) slow 1.6GHz Athlon, which doesn't really like md5-ing huge files , and I didn't need md5, just a size comparison. Unfortunately, all of the duplicate-file-finding tools I tried (including fslint, fdupes, dmerge, and dupseek) didn't have an option to not md5/sha1/hash in some form or another, or I did not find it .
So I decided to use fdupes (it's nice and simple for my needs), and trick it out .
I renamed /usr/bin/md5sum as md5sum.old, and created /usr/bin/md5sum with the following content:
#!/bin/bash
du $1|awk '{print $1}'|md5sum.old
Yes, my bash skills suck, and there are probably better ways of doing it, but again, it worked in this situation . So it's basically taking the du output of the file passed to md5sum (hopefully, again, it worked), shortening that to just the size, and then md5'ing that, which is much faster, and based on the size.
So now, you just run (e.g.):
fdupes -r /path/to/files > /path/to/duplicatefilelist
The -r is for recursive, and I'm redirecting it to a file because I know there will be a lot of dupes. If you don't have many, you could skip that.
Then, fdupes scans the number of files, and has a nice progress percentage so you can see how much is done. After this is done, you'd want to remove /usr/bin/md5sum, and restore the old md5sum .
I hope this is useful to someone else, and please share if there's a better way of doing this.
Thanks!

Answer 1

SahibBommelig wrote:
You also might want to try rmlint (self-advertisement...) or rdfind, which both are able to find twins much faster than fdupes.
If you really want only to compare the size you'd better go with karol's method.
Ah, I find that md5's aren't that slow after all (yes, I realize that rmlint is smarter than simply md5'ing every file ). rmlint is very nice, thank you!
One thing though...can I specify a "preferred" dir (when passing multiple dirs to rmlint), so that it has the "originals," and the "duplicates" are from the other directories?
Thanks.