Error "kdc: Server not found in database" on attempted connections using Network User Credentials

Question

I am rebuilding my system after a recent debacle with Time Machine, which resulted in a complete wiping of my Open Directory contents. At this point, users can log into various computers on the network, when the hosts have been reconnected to the newly formed Open Directory and the trust certificate has been authorized.  However, when users attempt to connect to any file share, the Network User Account credentials fail to gain access.
I am running Mac OS X 10.9.4 on all systems. Two mac-mini's are running OS X Server 3.1.2.  One of these servers (mavericks1.pediatricheartcenter.org) is the Open Directory.  While testing the system, I am using the console on "Mavericks1," so the following discussion involves communication between the two server hosts only.
From Mavericks1, I open the console and attempt to connect to my file server, named fileserver.pediatricheartcenter.org.  I clear the console just prior to sending a "registered user" request to "FileServer" to gain access.  Careful examination of the console records shows the following:
1. The Network User is authorized with a message "ENC-TS pre-authentication succeeded".
2. Mavericks1 lists a console message that reads "kdc: Server not found in database: krbtgt/[email protected]:no such entry found in hdb"
3. Mavericks1 lists a console message that reads "kdc: Server not found in database: cifs/[email protected]: no such entry found in hdb"
4. The process registers what appears to be a final failure before trying again with "kdc: Failed building TGS-REP to 127.0.0.1:64390"
FileStorage.local does not exist in the DNS, nor does it exist on FileStorage.pediatricheartcenter.org. That (local) host name was removed when the domain host name for filestorage.pediatricheartcenter.org was created.
1. Why does the kerberos process reference a host name that does not exist?
2. What might be causing the failed authentication exchange?
3. What can be done to remedy the issue?

Answer 1

I spent some time on the phone with Apple Support on Friday.  Thank you to Linc Davis for providing some insights into the issues.
As a result of the conversation with Apple Support we learned the following, which I will report here for those who might find this page again:
First, OpenDirectories are extremely fragile.  Once you have turned on your OpenDirectory, do not do any of the following:
Do NOT change the host name.
Do NOT change the IP address.
If you are going to attempt either of these things, you should make a clone of your drive (not just a TimeMachine backup, a fully bootable clone, just in case).
Performing these activities (particularly the changing of the host name) will "break" your open directory, and the only way to rebuild the open directory is first to fully destroy the original.  Several services are also destroyed when OpenDirectory is broken, the most notable is Profile Manager.
DESTROYING OPEN DIRECTORY
To fully destroy OpenDirectory, it is more complex than simply turning off the OpenDirectory and turning it back on again.  Perform the following steps:
Install WorkGroup Manager (it is depricated, but Apple still has a version available for use with OS X Mavericks to handle functions that the Server App does not perform like exporting users and groups).
Sign into WorkGroup Manager as the directory administrator (user name defaults to "diradmin" the password is defined on OpenDirectory creation).
Export the Users, Groups, Computers and Computer Groups to the Desktop or another safe location.
Close WorkGroup Manager
Turn off the OpenDirectory in Server App.
Delete the Server App from the Applications folder and put it in the Trash. (This will disable any active services that are marking various files as being currently in use. Don't worry, we will restore it from the Trash when we are done).
In the terminal, run the following command: sudo slapconfig -destroyldapserver
Make a backup of all website files (just in case)
Navigate to the folder /Library/Server and delete the ProfileManager folder. (If you willing to do so, delete the whole Server folder).
After deleting various folders in the /Library/Server directory, restore the Server.app from the Trash.
Run the Server App.
Set the computer's network connection and host name.
Create a new OpenDirectory.
Use WorkGroup Manager to import any exported files from Step 3.
If you deleted the entire Server directory, use the website backup to retrieve the files that comprise your web site(s) and use the Server App to link the file directories to the Web site's domain name(s).
Personal Note: These instructions got me farther than any other tips I had received previously. After following these instructions, I was able to rebuild my Open Directory. During the process of copying files from the old user home folders into the new user home folders, the computer froze and when it rebooted, all the users and groups I had created during the day had disappeared. Rather than trouble-shooting it again, I decided to do a fresh installation.
A NOTE ON HOME FOLDERS
PER APPLE SUPPORT: Do NOT use the default /Users directory for Network users. Apple Support wanted me to rebuild the home directory, but they noted I was not able to do this, because I had used /Users.  This folder ("/Users") is a critical component of the OS X system, and will cause additional problems if the folder is destroyed and rebuilt.  The directory id and permissions must remain unchanged from the original installation.
For this reason, Server administrators (like yourself) should use File Sharing in the Server App to create a new anchor point for home directories.  Create a shared folder. Ensure that it is shared over the protocols that you will be using (AFP, SMB, WebDav), and then after selecting these values, check the box that allows the folder to be used as a home directory at the bottom of this list.  This box will be greyed out if the system is not already bound to an OpenDirectory. If you have activated OpenDirectory on the same machine, the machine will operate as if bound to itself, and this field will be active.  If the FileShare server is NOT an OpenDirectory master or replica, then bind the machine to an OpenDirectory via the "System Preferences > Users & Groups > Login Options".
If the local area network has FileShares that are enabled for home directory use, the folders will appear in the User Profile editor under the Home Folder list (See image)
In the screenshot above, I have selected a shared directory named "HomeFolders".  By using specially defined home folder directory, the server administrator has the option of deleting and modifying the home folder if necessary.  Creating a home folder directory in a location other than "/Users" is the recommended best practice by Apple Support.
If you are inserting files into the home folders, you will need to change the owner and the group to the new owners names.  I copied files from the old user directories into the new user directories so that the users would have access to their old files.  When my OpenDirectory crashed, and all the users were recreated, they were recreated with different system level user id's.  The system therefore maintains a memory that the file was owned by the original owner, even though the system administrator has put it in the new user profile's folder.  To fix this, do the following:
1. Prior to making the copy, run "ls -al" from the terminal on the new home directory root.  You are looking for the default folder owner and default folder group.  On my system it was the user name and a group named "staff".
2. When making the copy, do not replace the user folder. Copy the files into the file folder, not over it.
3. After you have moved files into the user's folders, you can use "sudo chown -R [owner]:[group] [homeFolderPath]/*" and "sudo chmod -R 700 [homeFolderPath]/*" (replace the [owner] and [group] portions of these commands with the owners and groups identified by the command in step 1, and replace [homeFolderPath] with a path to the user directory created for the specific user.
For example:
For the user johnnybgood, we might see the following:
1. We run "ls -al" on the newly created home folder and find that the folder /Volumes/HomeFolders/johnnybgood is owned by johnnybgood and the group "staff".
2. We copy or move files from the old locations using commands similar to the following:
       sudo mv /OldFolderLocation/johnnybgood/Documents/* /Volumes/HomeFolders/johnnybgood/Documents
       sudo mv /OldFolderLocation/johnnybgood/Desktop/* /Volumes/HomeFolders/johnnybgood/Desktop
       sudo mv /OldFolderLocation/johnnybgood/Music/* /Volumes/HomeFolders/johnnybgood/Music
       ....etc....
       (notice how we are not just moving the old johnnybgood folder to the new location.)
3. Next, we change the ownership and file permissions:
       sudo chown -R johnnybgood:staff /Volumes/HomeFolders/johnnybgood/Documents
       sudo chown -R johnnybgood:staff /Volumes/HomeFolders/johnnybgood/Desktop
       sudo chown -R johnnybgood:staff /Volumes/HomeFolders/johnnybgood/Music
       ...etc...
       sudo chmod -R 700 /Volumes/HomeFolders/johnnybgood/Documents
       sudo chmod -R 700 /Volumes/HomeFolders/johnnybgood/Desktop
       sudo chmod -R 700 /Volumes/HomeFolders/johnnybgood/Music
       ...etc...
4. Let the user log in and use the system normally.