The System.keychain Conundrum: THE REAL SOLUTION.

Question

Symptom: After applying an update or some kind of instance where a shutdown occurred, upon rebooting, Mac OS X will demand a password for System.keychain. No password will ever work, not even root.
I had this happen to me and no one anywhere seemed to know how to fix it except for temporary workarounds that really didn't work.
First off, let me tell you what WON'T work as a solution:
1. Deleting offending AirPort network passwords in System.keychain - All you're doing is making OS X forget the passwords of those base stations in your Preferred Networks list (Network Preferences). If you attempt to put the passwords back and apply the changes, OS X will again demand the password for System.keychain.
2. Keychain First Aid - It will think that nothing is wrong with System.keychain, so as long as it is there.
3. Repair Permissions - Everyone believes that repairing permissions is the cure-all method for any problem with Mac OS X. It won't work because it already has the correct permissions (root:admin 755).
4. Deleting System.keychain itself - You won't ever get it back, and any base station added to your list of trusted networks will not have its password saved. Sure, it may save for YOUR login.keychain, but what if you have multiple users?
5. Replacing System.keychain with another from a known good system - This doesn't work either, and I'll explain why a little later.
6. Deleting /Library/Preferences/SystemConfiguration/com.apple.airport.preferences.plist - That has nothing to do with the problem.
7. Duplicating your login.keychain and renaming it as System.keychain - Sure, you'll be able to unlock it with your admin password, but that doesn't solve the problem.
No one - not even from Apple themselves (phone reps and store Genius
Bars) - knew how to fix this issue. Google or search the discussion boards all you want; you'll be wasting your time as no one really seems to know what the problem is and have employed one of those seven methods.
NOW FOR THE SOLUTION...
WARNING: This assumes that you are competent with the command line AND you have a working copy of OS X somewhere else nearby, preferably on an external disk partition. I'm not going to explain the basics of using the Terminal or how to access both of your systems at the same time. If you are afraid of screwing up, simply reinstall Mac OS X and say a few prayers that it will fix itself (assuming that another Software Update won't mess it up again).
The problem lies with a file deep in the bowels of OS X. It's /var/db/SystemKey. What it does is that it tells Mac OS X how to unlock the system keychain. It only knows the System.keychain specific to the computer, so if you import another System.keychain as a replacement, SystemKey won't know how to unlock it and you'll keep getting the annoying dialogs prompting for System.keychain's password.
So without further ado, this is how to stop the annoying dialogs once and for all:
1. AS THE ROOT USER, you will need to copy over /var/db/SystemKey from a known good system to your problematic system. Make sure that you preserve the permissions (0400). It is advisable that you are NOT booted from your problematic system.
2. If you have a good System.keychain, copy that over to /Library/Keychains on the problem system. If you accidentally deleted System.keychain, you can execute the following (again AS ROOT):
/usr/sbin/systemkeychain -C -f
This creates a new, working, empty System.keychain and effectively overwrites the old keychain.
3. Reboot to your system. You will be presented with different, more familiar (Change, Change All) dialogs. If you had any passwords saved in your list of preferred networks, just put them back in.
Best of luck.

Answer 1

A method that worked for me (and many others):
Delete the wireless router entry from the System keychain (do not just delete the password). The entry will automatically get re-created from the similar entry in the login (user) keychain. This solution was posted here within a few days of the release of 10.4.3.
bd