A Tale of a Filevault Disaster and Recovery...

Question

Recently, one of my customers had a major disaster with their MacBook running 10.5.8:
It all started when they decided they needed to free up some disk space on the internal drive, and thus they began to trash a large number of files to accomplish this.
Apparently in doing so, they must have trashed some key files because they managed to permanently lock themselves out of their Filevault-encrypted Home folder; the filevault password no longer worked when they attempted to log back in after a restart, and after numerous unsuccessful login attempts, they received the message "your filevault-protected home folder did not open and needs to be repaired. Click OK to repair the folder.." Clicking OK did nothing, and at that point they requested my services.
Turns out also, they had no recent backup; the last backup was over 2 years old- they had misplaced the power cable for their Time Machine backup drive- a WD MyBook- and had never replaced it... (oye vay)
When I got the machine, I used the installer cd to create a root user and look at the internal drive's status- 53G of 233G used. Over the course of several days, I took the following actions:
1. Restarted and logged in as root, I attempted to open the user's sparsebundle with no success.
2. Restarted to the installer CD and ran Disk Utility's Repair Disk which returned no errors. Attempted to Repair Permissions but got no result appearing to hang..
3. Figuring "Ok, I can't open the sparsebundle yet, so I'll at least start improving the overall health of the file system"- I started up the MB in Target Disk Mode connected to my G5 Quad running 10.5.7. From my G5, I used DU and Diskwarrior against the MB's single volume internal drive. DU returned no errors but Diskwarrior returned problems in red on a directory rebuild, so I repeated with Diskwarrior until all was green (3x).
4. After this, tried again to open the user's sparsebundle with same negative result of the password not working.
5. At this point, running out of solutions but anticipating eventual success (against all odds) I copied the sparsebundle to a spare drive as a backup.
6. Did a Google search and eventually found an archived Apple discussions thread which seemed to indicate a possible solution:
http://discussions.apple.com/thread.jspa?messageID=5881960
To make a long story a bit shorter, what eventually totally saved my bacon was this info in a post by GuyEWhite in the archived thread:
<<< So Guy, if you're out there- A MILLION Thanks!! >>>
7. Type: sudo hdiutil attach /Users/username/username.sparseimage -nomount
8. A bunch of funky things will happen, but it eventually should list a number of volumes, one with an HFS attached to it, note if it says disk1s1 or disk2s2 or whatever.
9. Type: sudo fsck_hfs -f /dev/rdisk2s2 (rdisk2s2 being where the HFS was from step 8, adding an "r" at the front)
If fsck_hfs bails because of too many errors, just repeat until it says the FS is fine. That irks the bejeezus out of me as well, but has always worked for me so far.
7. Still in TDM, and working on the backup copy, I used the above commands to first "see" the sparsebundle volume, and then to run fsck against it. Sure enough, my results were similar to the above: major errors returned the first 3 or 4 times I ran it, then eventually it said the FS was OK, and then VOILA! My first success- the sparsebundle volume actually mounted in Disk Utility and Diskwarrior, both of which were running in the background.
8. The Filevault password now worked, and I ran Diskwarrior's "Rebuild Directory" multiple times against the mounted sparsebundle volume. Numerous errors were evident and Diskwarrior created a "Rescued Items" folder.
Obviously, whatever the client had done did a major FUBAR to her home folder. Eventually the number of errors in red was reduced to a point where it would not reduce further. About 1G of data appeared to be in the Filevaulted Home folder. This didn't square with the supposed size of the sparsebundle, but I left that issue hanging for the moment.
9. I repeated the command line procedure against the MB's internal drive- successfully mounting and decrypting the Filevaulted Home folder. Similar results were obtained where it took multiple passes of fsck to mount the disk, then a number of passes of Diskwarrior to repair the directory as much as possible.
10. Knowing that I had rescued at least some of the client's files, and that I had a "repaired" backup of the sparsebundle- I decided at this point to wipe the MB's interal drive and do a complete re-install of the system and all software updates- knowing I would be eventually transferring her user files back over.
11. Back to the discrepancy of visible files in the mounted sparsebundle:
I decided to get out the howitzers and use Data Rescue.
A "deleted files" scan found another 28G of actual user files!
12. The task of organizing and transferring the user files back over to the MacBook was pretty labor-intensive but ultimately pretty successful for the most part; there were however a significant number of corrupted files which could not be salvaged. These mostly included user prefs, some Pages documents, and some images. Most all of her music and movies were still intact.
At first I tried repairing some of the corrupt image files by using Graphic Converter to re-save images that Photoshop otherwise couldn't open. This proved to be incredibly tedious and instead, I simply imported all the images en masse into iPhoto. iPhoto very nicely refused import of all the corrupted images (something like a couple hundred out of multiple thousands).
The end result was quite satisfactory: I got the client a new power cable for their MyBook and a did a complete Time Machine backup to it. They were quite happy since I had recovered just about all their documents, music, movies, and photos...
But without finding that post and method by GuyEWhite, the outcome probably would have been far different, so again- thanks Guy!
<Edited by Host>

Answer 1

This is the single most useful entry for FileVault problems I ever encountered! Thanks a lot!
Just a few notes to add:
(1) Use this to access your FileVault image:
sudo hdiutil attach -nomount -readwrite -noautofsck -noverify /Users/(username)/(username).sparsebundle
Reminder: "(username)" is your short username. This will prevent Snow Leopard from traing to repair while accessing the FileVault bundle.
If your computer crashed your sparsebundle will be at "/Users/.(username)/(username).sparsebundle", not at "/Users/(username)/(username).sparsebundle", the difference is the "." in the folder name!
(2) After repairing with fsck_hfs you should detach the sparse bundle again:
hdiutil detach /dev/diskX
where "X" can be anything from 0 to whatever, matching the output after the "hdiutil attach" command.
Beswt,
Apple*