Password service won't start

Question

Following an unplanned hard restart of our server (Mac OS X 10.9.5, Server 3.2.2), the password service won't start, so our staff can't log into mail, calendars or file sharing.
When I launch Open Directory, the main window has the message "Unable to load replica list", then it switches itself off after a minute or so.
I've looked at the certificates in Server and they have the green tick so presumably are OK.
DNS is working OK and running the command "sudo changeip -checkhostname" reports success.
Here's the section of log that repeats every few seconds as PasswordService repeatedly starts up and exits:
13/03/2015 20:55:00.617 com.apple.launchd[1]: (com.apple.PasswordService[5890]) Exited with code: 1
13/03/2015 20:55:00.617 com.apple.launchd[1]: (com.apple.PasswordService) Throttling respawn: Will start in 10 seconds
13/03/2015 20:55:02.540 xscertd[249]: Failed sending LookupCRLByCARecordName command to com.apple.xscertd.helper: The operation couldn’t be completed. (com.apple.certificateserver error 42005.)
13/03/2015 20:55:08.708 com.apple.launchd[1]: (org.openldap.slapd[5894]) Exited with code: 1
13/03/2015 20:55:08.708 com.apple.launchd[1]: (org.openldap.slapd) Throttling respawn: Will start in 7 seconds
13/03/2015 20:55:10.206 xscertd-helper[5897]: ldap_search_ext_s returned -1 - Can't contact LDAP server when searching for bdb suffix, exiting
13/03/2015 20:55:10.207 com.apple.launchd[1]: (com.apple.xscertd-helper[5897]) Exited with code: 1
13/03/2015 20:55:10.207 com.apple.launchd[1]: (com.apple.xscertd-helper) Throttling respawn: Will start in 10 seconds
13/03/2015 20:55:10.639 PasswordService[5901]: -[PasswordServerPrefsObject getSearchBase]: Unable to locate search base: -1 Can't contact LDAP server
13/03/2015 20:55:10.639 PasswordService[5901]: -[PasswordServerPrefsObject loadXMLData]: Unable to locate passwordserver config record's plist attribute: -1 Can't contact LDAP server
13/03/2015 20:55:10.640 PasswordService[5901]: -[PasswordServerPrefsObject getSearchBase]: Unable to locate search base: -1 Can't contact LDAP server
13/03/2015 20:55:10.640 PasswordService[5901]: -[PasswordServerPrefsObject saveXMLData]: ldap_modify_ext_s of the passwordserver config record's plist attribute: -1 Can't contact LDAP server
13/03/2015 20:55:10.684 PasswordService[5901]: int pwsf_GetPublicKey(char *): ldap_search_ext_s cn=authdata for Public Key returned -1
13/03/2015 20:55:10.687 com.apple.launchd[1]: (com.apple.PasswordService[5901]) Exited with code: 1
I have backups (both Time Machine and clones of the hard drive) that I could use to restore the corrupted bit of the configuration, if necessary, but I don't know what to restore.

Answer 1

Many Open Directory problems can be resolved by taking the following steps. Test after each one, and back up all data before making any changes.
1. The OD master must have a static IP address on the local network, not a dynamic address. It must not be connected to the same network with more than one interface; e.g., Ethernet and Wi-Fi.
2. You must have a working DNS service, and the server's hostname must match its fully-qualified domain name. To confirm, select the server by name in the sidebar of the Server application window, then select the Overview tab. Click the Edit button on the Host Name line. On the Accessing your Server sheet, Domain Name should be selected. Change the Host Name, if necessary. The server must have at least a three-level name (e.g. "server.yourdomain.com"), and the name must not be in the ".local" top-level domain, which is reserved for Bonjour.
3. The primary DNS server used by the server must be itself, unless you're using another server for internal DNS. The only DNS server set on the clients should be the internal one, which they should get from DHCP if applicable.
4. If you have accounts with network home directories, make sure the URL's are correct in the user settings. A return status of 45 from the authorizationhost daemon in the log may mean that the URL for mounting the home directory was not updated after a change in the hostname.
5. Only if you're still running Mavericks server, follow these instructions to rebuild the Kerberos configuration on the server.
6. If you use authenticated binding, check the validity of the master's certificate. The common name must match the hostname and domain name. Deselecting and then reselecting the certificate in Server.app has been reported to have an effect in some cases. Otherwise delete all certificates and create new ones.
7. Unbind and then rebind the clients in the Users & Groups preference pane. Use the fully-qualified domain name of the master.
8. Reboot the master and the clients.
9. Don't log in to the server with a network user's account.
10. Disable any internal firewalls in use, including third-party "security" software.
11. If you've created any replica servers, delete them.
12. If OD has only recently stopped working when it was working before, you may be able to restore it from the automatic backup in /var/db/backups, or from a Time Machine snapshot of that backup.
13. As a last resort, export all OD users. In the Open Directory pane of Server, delete the OD server. Then recreate it and import the users. Ensure that the UID's are in the 1001+ range.
If you get this far without solving the problem, then you'll need to examine the logs in the Open Directory section of the log list in the Server app, and also the system log on the clients.