Apparent kompare bug(s)

Question

I apologise in advance for the length of this 1st-Arch-post-in-years. I value context more than brevity, I guess. Hopefully this is the longest post I'll throw at Arch for a long, long time.
I looked around (quite) a bit before filing this post.  I've used Arch off-n-on since 0.4, know the Arch site, the basics about Arch, and most of the "gotcha's" farly well - but I'm no sysop, either.
I have a project I've been needing and meaning to work on for years (and which continues to compound itself while I don't); a huge set of backups on CD/DVD of all my data (my *life*, really) for well over a decade - letters, graphics, email, the first 40 pages of my book, you name it. Over the years this 'little' library has grown to over 10 GB compressed (30,000 files, *most* of them ARCHIVES). My guess is that 2-4 GB (compressed) of the data is important, some very much so.
My archive collection consists of many recursed archives themselves, and they're in about 8 or 10 different formats, to boot (even *.ice, LMAO). Many are further embedded several levels into the main archives (yes, I know that's a bad idea, but I didn't know that when I created them...). I BADLY need to unpack all this stuff, find and toss the duplicates, and truly organise (and efficiently backup) the remainder. This effort will obviously require a lot of automation (or else my heirs will either do it, or not).
I'd found a couple of useful tools in the Windows world years ago, but I don't buy software and the key piece of my suite was shareware that disabled itself after 30 days - not nearly enough time to plow through this boatload of bits.
A few months back I built a system with an Athlon XP 3500+ on an Asus A8N-E mobo w/512MB and 120 GB and 250 GB HD's. I'm hoping to go the RAID route soon - Linux software RAID, in fact. The 250 GB HD is running off an add-in PCI ATA card; both HD's have a UDMA channel all to themselves. The machine's got 4 extra fans, a nicely oversized 485W P/S, and the on-board sensors consistently show it running cool. I'm quite sure this isn't a H/W problem (I'm having no others; I built it myself...). This machine has been absolutely rock-solid in every other regard. I LOVE the system; now if I can simply use it for this one basically administrative project.
I finally have both the speed and the space required to tackle this now-monster project. When I "came home" to Arch a few months ago, I wanted to try several WMs/DMs; GNOME, KDE, E17, fluxbox and ratpoison (at least) are all loaded on my system, and I've booted most at least once. I'd assumed I'd have a LOT of options as to the tools to use.
I've been man-paging and googling and pacman -Q'ing (followed by more googling) for a couple of weeks through the course of my testing thus far, while trying various wrappers in various WMs. My concluson: Houston, we have a problem! <S>Maybe</S> Several.
In the past I've been partial to KDE (that's quickly changing), and it's what I started with this time around. As such, kompare seemed the obvious route to go, and had MOST (NOT all) of the functionality I needed (I also very much like it's GUI). Of course, kompare is simply a wrapper for diff (several diff variants, in fact, potentially). It's possible that diff is the problem. Hopefully further testing (and/or comments here) will make that determination more clearly.
Subsequent googling has revealed that the diff version shipped with Arch likely hasn't been completely functional at all levels in almost *5 YEARS* now. Diff's codefile timestamp is Sept. 3, *2002*! Version 2.8.1 (3.4 is current - just like *kompare*'s current version is 3.4). Additionally, directly below diff in that listing is diff3. Diff's codefile size is about 60K; diff3's is about 20K. *However*, they both have the same timestamp AND version # !!!
A *completely cosmetic* bug (http://bugs.kde.org/show_bug.cgi?id=138347) is fairly clearly the same bug as (https://bugs.launchpad.net/ubuntu/+source/kdesdk/+bug/45466 - 3-1/2 *YEARS* earlier!), and seemingly a trivial patch for a competent maintainer to create, at that. I've verified (w/o even trying) that this bug is resident in my system as we speak. Of course it is - the version of diff Arch is currently shipping was compiled almost a YEAR before the FIRST of the 2 bug reports above were filed! My Arch desktop O/S was installed less than a month ago using the 0.7.2 base ISO (kompare's timestamp is Jan. 20, 2007; version 3.4 - and it's a wrapper for a piece of code last compiled in 2002? What, the 2.0 kernel?). I've aggressively maintained my system current since my very recent install. See how well kompare/diff work on YOUR system...
Indeed, from the kompare bug page, it's hard to believe it's EVER been stable - or even used (see http://bugs.kde.org/simple_search.cgi?i … &offset=20 and just keep on clicking "View More Results" - or get yourself an energiser bunny to do it for you, maybe).
I copied the backups from CD's/DVD's to a clean, freshly-showered 64 GB (62.8, actually) ext3 partition (4K blocksize, defaults otherwise, journal included). I set up a top-level directory in this partition with 15 directories beneath it, one for each of my backup discs (whose contents range in size from 50 MB to 4.36 GB). I copied one archive to each directory, then unpacked them one at a time from the command line, removing the archive copy as I went. I ended up with about 15 GB of active data (25%).
I first configured kompare to use diff (w/no C/L options) running under KDE. As kdiff3 is kompare's preference, however, why is kdiff3 not already in the repos?
I'd not used it before, so I wasn't familiar with kompare's UI. I started out working with a few smaller subsets of the data I knew were near-perfect copies of each other and that wouldn't overload the filesystem. It seemed to act strangely; while I expected it to be well-optimised, both HD activity and CPU utilisation seemed extremely low for extended periods (and note my setup doesn't allow for a HDD LED for the HD on the controller card, where ALL the disk activity is going on).
After 3 runs in a row w/o a single errant file being found, I made a 2-byte mod to a small, non-critical text file that was part of the archive I was currently komparing, rebooting to make DAMNED sure cache was cleared, and reran kompare; identical results. The change STILL wasn't detected (the changed file still showed as being an exact copy of the original). The output I was getting was quite weird: usually, the 'kompare' would run at what seemed a rational speed; after a while a response screen would then open, *completely* devoid of data or messages. Immediately after that an odd dialogue box popped up; the title bar said "Error!", but the *message* was, basically, "No Duplicates Encountered." (or some such), or, more commonly, the same one I encountered repeatedly while searching the internet: "Could not parse diff output." (presumably because kompare still wasn't passing the proper parameters to diff). The program crashed about 1/2 the time (actually, give it a big enough block of data, and it'll crash every time). I was stunned 4 days later when ONE run (an exact duplicate of one I'd already attempted several times) resulted in "proper" output - *PERFECT*, with a complete and correct analysis - and completely unreproduceable, even after rebooting. It's ("It's" = proper analysis/output has) happened 3 times since, twice while working with data sets I would term 'trivial'.
So, in 2 weeks I got 3 runs w/o errors. I assure you I was trying at least 2 runs most days; a 1/2-dozen some days. By then, I also knew that kompare under KDE wasn't my only option - and I knew a bit more.
I use the "usual" repositories. When I finally checked diff's compile date, my jaw dropped: Sept. 3, *2002* !!!  It's version 2.8.1; the current stable release is 3.4, and 3.5 is currently available for D/L (albeit not released). V. 2.8.1 works with the 2.6 kernel about as well as the original GWBasic compiler ('interpreter') does in WinXP...and may have been this way for a long time.
More research (difficult, given the age of what little kompare documentation is available on the web). The primary maintainer apparently got removed from maintaining the package in 2003 (and, I presume, other packages, as well) for what appeared to be 'apathy' (MY words). He 'forgot' to include an update after saying he would do so (a patch to pass the appropriate parameters to diff, hopefully fixing the "Could not parse diff output." message, and maybe even straighten out my anomolies, as well).
Kompare hasn't been updated since. Diff3 IS currently being maintained. Actually, even though diff3 has apparently replaced diff, *kompare* prefers kdiff3. For that matter, from my reading thus far, diff, diff3, vimdiff and cmp ALL may possibly be useable (the latter would require some scripting). Further, ALL the following may possibly be made able to work as wrappers (in decreasing order of likelihood, IMNSHO): kompare, krusader, meld, and emelFM2.
So, diff is broken, and it doesn't appear that it's going to be fixed...I personally was a bit consternated when I realised just how off-the-rails kompare was. I see a function such as that provided by kompare as both *critical* AND as a 'black box' function - it HAS to work. Just like the ALU MUST compute 2+2 = 4, or the 'computer' becomes a boat anchor...
So, I D/L'ed meld (for GNOME; it runs under KDE but no help is avalable), then emelFM2 (written for E17, I believe). I've spent the past week trying various combinations of (a) kompare, krusader, meld and emelFM2 as wrappers for diff under (b) KDE, GNOME, and E17. The results have been SO inconsistent it's scary. This includes both giving apparently proper results with a big job on rare occasion (but not consistently reproduceable), and VERY bizarrely, on one occasion, wiping a brand new, 64 *GB* ext3 partition CLEAN. I assure you the remainder of my testing will be on a dedicated partition! This HD is almost braad-new and has given me ZERO other trouble.
Meld was the most consistently performing diff setup I used (on any WM; it worked on them all).
I plan to spend the next 1-2 weeks doing some detailed and structured testing of all possible combos of wrapper/driver/WM, rather than just diff. I also plan to see if it's possible to pass command-line parameters to diff from kompare (if I have the time to spend ANY more w/kompare; kompare clearly isn't what I need - even though it had the . Yes, I *do* know how many combos of 4 wrappers, 4 drivers, and 3 WM's there are (48); my college major was statistics, LOL. And 48 test runs is adequate to test each combo, ONCE...
To me, this task is like big sort jobs were on mainframes a (human, not computer!) generation ago: potentially very large, critical tasks, the results of which aren't always easy to manually double-check. Surely many, if not most, users have a manner of doing this. Diff has apparently been broken for maybe 5 years and THAT wouldn't have gone unnoticed - am I one of the VERY few to even try to use kompare? What do the rest of you use?
Note that after locating and reading as much kompare documentation as possible (I read more BUG reports than anything), I made several important discoveries:
(a) kompare isn't really set up for binary files; it's intended for use with text files. You can specify an option to force a text-mode search, but you apparently canNOT do the reverse.
(b) It err'ed out comparing two M$ 'Word' files, stating they "were binary"...
(c) Worst of all, for me: These files have been archived to CD or DVD, usually repeatedly, and then copied back to disk (I've lost 3 good-sized archives this way - which does serve to somewhat justify my apparently obsessive need to backup files). Kompare does do individual file comparisons (ie, if you specify both files on the command line) byte-by-byte (block-by-block, whatever), but if you request that it compare *directories*, filename and size are all it considers (it appears from the sparse documentation I've seen that this is corrected in the *current* version, . The duplicate-file tool suite I used in Windows did an MD5 hash on *every* file, no matter the size; I *knew* when a bit error had creeped in, and usually knew fairly quickly. Not finding out you have a bad backup until it's the only copy of your data you have left is no good!
Anyone have any suggestions on doing this task efficiently and reliably in Arch?
Again, I believe this to be an upstream problem in a way, but my guess is that if we were using the *current* diffutils to match the KDE version (and a maintainer had done his job in a timely fashion) this bug would NOT exist. My advice, at least until Arch can get diff3 (or, preferably, kdiff3) packaged, STAY AWAY FROM kompare !!!  At least with anything critical, at least until this issue is better understood.
Kompare (and diffutils, as well, IMO) needs to be removed from the 'public' repo's until it's close to fixed, at least. Just pkgbuild'ing the current diffutils version (or kdiff3!) is likely the easiest way out. Who knows, MAYBE I'll take a stab at that myself. No promises.
For ANYONE still reading, WOW! TIA. Really. Sorry for the tome.
Blue Skies...g
"It's a lot easier to throw grenades than it is to catch them!"  -- Tim Russert

Answer 1

[intro, backups, bla...]I BADLY need to unpack all this stuff, find and toss the duplicates, and truly organise (and efficiently backup) the remainder.
Ok, that's what it's all about.
I'd found a couple of useful tools in the Windows world years ago, but I don't buy software and the key piece of my suite was shareware that disabled itself after 30 days - not nearly enough time to plow through this boatload of bits.
Total Commander rocks.
[killed half a dozen paragrahps or so] It's possible that diff is the problem.
Contratulations. You managed to post 3K of text but still failed to mention what the problem is.
Subsequent googling has revealed that the diff version shipped with Arch likely hasn't been completely functional at all levels in almost *5 YEARS* now. Diff's codefile timestamp is Sept. 3, *2002*! Version 2.8.1 (3.4 is current - just like *kompare*'s current version is 3.4).
http://www.archlinux.org/packages/search/?q=diffutils
You will notice that package isn't flagged out-of-date. That's because the (still current) upstream version is (tadaa!) 2.8.1. Please tell us where you found that 3.4 version.
Additionally, directly below diff in that listing is diff3. Diff's codefile size is about 60K; diff3's is about 20K. *However*, they both have the same timestamp AND version # !!!
WTF??? What listing? You know that diff and diff3 are different programs but still part of the same upstream sources, don't you?
[non-relevant nonsense...] See how well kompare/diff work on YOUR system...
They work fine.
[kompare bashing]
Take a look at the URL, bugs.kde.org, not bugs.archlinux.org. Got it?
I copied the backups from CD's/DVD's to a clean, freshly-showered 64 GB (62.8, actually) ext3 partition (4K blocksize, defaults otherwise, journal included). I set up a top-level directory in this partition with 15 directories beneath it, one for each of my backup discs (whose contents range in size from 50 MB to 4.36 GB). I copied one archive to each directory, then unpacked them one at a time from the command line, removing the archive copy as I went. I ended up with about 15 GB of active data (25%).
I think I could rest my case already at this point.
I first configured kompare to use diff (w/no C/L options) running under KDE. As kdiff3 is kompare's preference, however, why is kdiff3 not already in the repos?
It is: community/kdiff3 0.9.91-1
(I will skip some paragraphs about the actual kompare usage and come back to that at the very end.)
I use the "usual" repositories. When I finally checked diff's compile date, my jaw dropped: Sept. 3, *2002* !!!  It's version 2.8.1; the current stable release is 3.4, and 3.5 is currently available for D/L (albeit not released). V. 2.8.1 works with the 2.6 kernel about as well as the original GWBasic compiler ('interpreter') does in WinXP...and may have been this way for a long time.
Please quit that bullshit and show us version 3.4 of the GNU diffutils.
Kompare hasn't been updated since. Diff3 IS currently being maintained. Actually, even though diff3 has apparently replaced diff, *kompare* prefers kdiff3. For that matter, from my reading thus far, diff, diff3, vimdiff and cmp ALL may possibly be useable (the latter would require some scripting). Further, ALL the following may possibly be made able to work as wrappers (in decreasing order of likelihood, IMNSHO): kompare, krusader, meld, and emelFM2.
Check http://websvn.kde.org/ to follow the development progress of kompare, it's part of kdesdk. diff3 is from diffutils 2.8.1 and I wouldn't call that "maintained", but you already know that. It is obviously not a replacement for diff, did you ever read the manpages? And wow, a KDE program prefers to use another KDE program!
[killed quite a bit]Diff has apparently been broken for maybe 5 years and THAT wouldn't have gone unnoticed - am I one of the VERY few to even try to use kompare? What do the rest of you use?
Again, diff is fine and I'm using kompare (or kdiff3?) almost daily, in krusader.
(a) kompare isn't really set up for binary files; it's intended for use with text files. You can specify an option to force a text-mode search, but you apparently canNOT do the reverse.
(b) It err'ed out comparing two M$ 'Word' files, stating they "were binary"...
Genius. Comparing binary files is useless. If you used MS Office you're hosed, that simple. More useful advice at the end.
Anyone have any suggestions on doing this task efficiently and reliably in Arch?
Yep.
Again, I believe this to be an upstream problem in a way, but my guess is that if we were using the *current* diffutils to match the KDE version (and a maintainer had done his job in a timely fashion) this bug would NOT exist. My advice, at least until Arch can get diff3 (or, preferably, kdiff3) packaged, STAY AWAY FROM kompare !!!  At least with anything critical, at least until this issue is better understood.
Upstream, right... diffutils have nothing to do with KDE at all... Arch has diff3 and kdiff3 packaged ... it's no 'issue' at all.
Okay, I apologize for that 'tone' of mine but I get pissed easily when wasting so much time on something as this.
I know your problem very well as I'm constantly backing up stuff, comparing directories and hunting duplicates.
After unpacking everything the first step to take is to run a comprehensive check for binary duplicates. My tool of choice for that task is Total Commander (Win32 shareware, no timeout, just a minor nag screen on start, works fine with Wine). Just a few random keyboard shortcuts you might find handy (see the help file for explanations): ctrl-b, ctrl-f1(-f6), ctrl-c,y, alt-f7
After killing them, go on to compare ("synchronize") similar directories, either with Total Commander, Krusader, rsync, diff -R, whatever.
If you encounter similar named files, check the dates and kill the older ones, or compare them if they're not binary.
And so on... you know best what kind of data you got and how to split it up.
Throwing plain diff on a set of thousands of files, possibly gigabytes is a huge waste of time and computing/power resources. Adding a frontend will make it crash most of the time, especially with binary files. I think using kdiff3 to compare the Arch Linux 0.8 base beta1 iso to the beta2 one alone should suffice. A few more gigabytes of RAM might help though.