Error creating OD replica

Question

We have some strange behaviour with two 10.8.2 / Server 2.2.1 hosts that were built cleanly as OD master/replica early this year.  After the accidental deletion of some MCX data in Workgroup Manager, the master was restored using the previous days OD archive.  Since then a few issues have been noticed, the main one being the failure when attempting to replicate the directory on the second host.  There seems to be an issue with creation of the intermediate CA for the replica by the root CA on the master:
slapconfig.log:
2013-03-12 00:39:53 +0000 slapconfig -createreplica
2013-03-12 00:39:53 +0000 command: /usr/sbin/sso_util info -r /LDAPv3/ldap://master.domain.com -p
2013-03-12 00:39:53 +0000 1 Creating computer record for replica
2013-03-12 00:39:59 +0000 command: /usr/sbin/slapconfig -delkeychain /LDAPv3/127.0.0.1 replica.domain.com$
2013-03-12 00:39:59 +0000 slapconfig -delkeychain
2013-03-12 00:39:59 +0000 Added computer password to keychain
2013-03-12 00:39:59 +0000 2 Creating ldap replicator user
2013-03-12 00:39:59 +0000 _ldap_replicator exists from previous replica - migrating
2013-03-12 00:39:59 +0000 ServerID for this replica 6
2013-03-12 00:40:00 +0000 command: /usr/bin/sntp -s time.asia.apple.com.
2013-03-12 00:40:00 +0000 3 Updating local replica configuration
2013-03-12 00:40:00 +0000 4 Gathering replication data from the master
2013-03-12 00:40:00 +0000 5 Copying master database to new replica
2013-03-12 00:40:00 +0000 Removed directory at path /var/db/openldap/openldap-data.
2013-03-12 00:40:01 +0000 Starting LDAP server (slapd)
2013-03-12 00:40:01 +0000 Waiting for slapd to start
2013-03-12 00:40:01 +0000 slapd started
2013-03-12 00:40:01 +0000 Stopping LDAP server (slapd)
2013-03-12 00:40:04 +0000 command: /usr/sbin/slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d
2013-03-12 00:40:04 +0000 command: /usr/sbin/slapadd -c -w -l /var/db/openldap/openldap-data/backup.ldif
2013-03-12 00:40:05 +0000 command: /usr/sbin/slapadd -c -w -b cn=authdata -l /var/db/openldap/authdata/authdata.ldif
2013-03-12 00:40:06 +0000
2013-03-12 00:40:06 +0000 513e7965 slapd is running in import mode - only use if importing large data
          513e7965 bdb_monitor_db_open: monitoring disabled; configure monitor database to enable
2013-03-12 00:40:06 +0000 6 Starting new replica
2013-03-12 00:40:06 +0000 Starting LDAP server (slapd)
2013-03-12 00:40:06 +0000 Waiting for slapd to start
2013-03-12 00:40:06 +0000 slapd started
2013-03-12 00:40:06 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2013-03-12 00:40:06 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID
2013-03-12 00:40:06 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2013-03-12 00:40:06 +0000 Starting password server
2013-03-12 00:40:07 +0000 7 Enabling local Kerberos server
2013-03-12 00:40:07 +0000 Configuring Kerberos server, realm is MASTER.DOMAIN.COM
2013-03-12 00:40:07 +0000 command: /usr/sbin/sso_util configure -x -k -r MASTER.DOMAIN.COM -f /LDAPv3/ldapi://%2Fvar%2Frun%2Fldapi -a diradmin -p **** -v 1 all
2013-03-12 00:40:08 +0000 int32_t _createLDAPReplica(const char *, const char *, const char *, const char *): sso_util configure failed 1.  stdout = {  } stderr = { Creating the service list
          Creating the keytab file
2013-03-12 00:40:08 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2013-03-12 00:40:08 +0000 Stopping LDAP server (slapd)
2013-03-12 00:40:09 +0000 Starting LDAP server (slapd)
2013-03-12 00:40:09 +0000 Waiting for slapd to start
2013-03-12 00:40:09 +0000 slapd started
2013-03-12 00:40:09 +0000 8 Enabling syncprov overlay on the replica
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config objectClass=olcSyncProvConfig dn
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2013-03-12 00:40:09 +0000 adding new entry "olcOverlay=syncprov,olcDatabase={1}bdb,cn=config"
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2013-03-12 00:40:09 +0000 adding new entry "olcOverlay=syncprov,olcDatabase={2}bdb,cn=config"
2013-03-12 00:40:09 +0000 9 Adding replica to master
2013-03-12 00:40:09 +0000 Configuring multimaster for (replica.domain.com) with ServerID (6)
2013-03-12 00:40:09 +0000 Remote server (master.domain.com) ID: 1
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b dc=master,dc=domain,dc=com uid=_ldap_replicator dn
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config objectClass=olcSyncProvConfig dn
2013-03-12 00:40:09 +0000 default realm: MASTER.DOMAIN.COM
2013-03-12 00:40:09 +0000 Configuring multimaster
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID
2013-03-12 00:40:09 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2013-03-12 00:40:09 +0000 modifying entry "cn=config"
          modifying entry "olcDatabase={1}bdb,cn=config"
          modifying entry "olcDatabase={2}bdb,cn=config"
2013-03-12 00:40:09 +0000 Stopping LDAP server (slapd)
2013-03-12 00:40:10 +0000 Starting LDAP server (slapd)
2013-03-12 00:40:10 +0000 Waiting for slapd to start
2013-03-12 00:40:10 +0000 slapd started
2013-03-12 00:40:10 +0000 Updating ldapreplicas on master.domain.com as diradmin
2013-03-12 00:40:11 +0000 Updating ldapreplicas record
2013-03-12 00:40:11 +0000 Updating ldapreplicas plist.
2013-03-12 00:40:11 +0000 Binding to 127.0.0.1
2013-03-12 00:40:11 +0000 command: /usr/bin/ldapadd -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2013-03-12 00:40:11 +0000 Could not find root CA certificate in system keychain
2013-03-12 00:40:11 +0000 10 Enabling intermediate CA
2013-03-12 00:40:11 +0000 NSData *_getServerInfoRequestWithNode(ODNode *, NSDictionary *): ODNode - eODCustomCallAppleODClientGetServerInfo - error 10001 (The plugin encountered an error processing request.)
2013-03-12 00:40:11 +0000 int32_t _createLDAPReplica(const char *, const char *, const char *, const char *): Error: Intermediate CA creation failed on replica
2013-03-12 00:40:11 +0000 int32_t _createLDAPReplica(const char *, const char *, const char *, const char *): Error: Intermediate CA creation failed on replica (error = 75)
2013-03-12 00:40:11 +0000 Deleting Cert Authority related data
2013-03-12 00:40:11 +0000 Error deleting IntermediateCA_MASTER.DOMAIN.COM_1 from keychain: -67701
2013-03-12 00:40:11 +0000 No intCAIdentity, not removing int CA from keychain
2013-03-12 00:40:11 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertd.plist
2013-03-12 00:40:11 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertd-helper.plist
2013-03-12 00:40:11 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertadmin.plist
2013-03-12 00:40:11 +0000 Updating ldapreplicas on primary master
2013-03-12 00:40:12 +0000 Removing self from the database
2013-03-12 00:40:12 +0000 Warning: An error occurred while re-enabling GSSAPI.
2013-03-12 00:40:13 +0000 Stopping LDAP server (slapd)
2013-03-12 00:40:14 +0000 Stopping password server
Aside from this replication error and replica creation failure, distinguished bindings of clients (either conducted manually or via DeployStudio) are resulting in two computer records and sometimes a third (seemingly at random).
1.     The expected record of the client's hostname, although with the 'Comments' field populated with the LKDC ID (not normal behaviour)
2.     Another mostly blank record (no MAC address) using the LKDC ID as the computer record name
3.     (Sometimes) another mostly blank record using the client's hostname with a ".local" suffix before the $
Despite this obviously unusual behaviour, the only real clue seems to be the error in the replica creation log.  The servers were intentionally built clean on 10.8 without importing or upgrading anything from the previous 10.6 setup, so it seems strange that there should be such a major issue from a simple archive/restore of a clean setup.  My interpretation of the log contents is that there is a problem with the OD root CA, which is not very well documented to say the least.
At this stage the main concern is isolating whether there is a problem with the current build of the server, a problem with the OD archive data, or just a glitch with the OD archive restoration process which is causing this behaviour.
Does anyone have any clues?  Most specifically for fixing the certificate authority errors in the slapconfig.log?
Any input greatly appreciated.

Answer 1

Hi,
Thanks for your reply.  The server was indeed demoted and this log is of what we experience when attempting to recreate the replica from standalone.
I'm not sure if the replica was demoted before the OD archive was restored, but considering it was an archive from the previous day of the same directory, I wouldn't expect that to break replication.  This is part of my reason for suspecting the problem is with the ODM, more specifically with its root CA based on the log errors.