Deleted failed DC from the domain (Server 2012 R2) - Now after doing metadata and DNS cleanup, I can no longer promote a new DC to the domain

Question

I work for a university and teach IT courses to undergrad and graduate students. The details below are pertaining an isolated lab environment
I had a storage failure in my lab and the DCs became corrupt. This is a university lab environment so there isn't anything crucial on here. I just would rather avoid rebuilding the domain/forest and would rather use this as a learning experience with my
students...
So after the storage failed and was restored, the VMs hosted became corrupt. I did a NTDSUTIL to basically repair the NDTS.dit file but one of my DCs reverted to a state before DC promotion. Naturally, the domain still had this object in AD. After numerous
failed attempts at trying to reinstall the DC on the server through the server manager wizard in 2012 R2, I decided that a metadata cleanup of the old failed object was necessary.
Utilizing this article, I removed all references of the failed DC from both AD and DNS (http://www.petri.com/delete_failed_dcs_from_ad.htm) 
So now that the failed object is removed completely from the domain and the metadata cleanup was successful, I then proceeded to re-install the necessary AD DS role on the server and re-promote to the existing domain. Pre-Requisites pass but generate some
warning around DNS Delgation, and Dynamic Updates (delegation is ignored because the lab is isolated from external comms, and dynamic updates are in fact enabled on both my _msdcs and root domain zones).
Upon the promotion process, I get the following error message (also worth mentioning - the account performing these operations is a member of DA, EA, and Schema Admins)
The operation failed because:
Active Directory Domain Services could not create the NTDS Settings object for this Active Directory Domain Controller CN=NTDS Settings,CN=domainVMDC1,CN=Servers,CN=Default-
First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=school,DC=edu on the remote AD DC domainVMDC2. Ensure the provided network credentials have sufficient permissions.
"While processing a change to the DNS Host Name for an object, the Service Principal Name values could not be kept in sync."
As you can see, this error seems odd considering. Now that I'm down to a single DC and DNS server, the sync should be corrected. I've run a repadmin /syncall and it completed successfully. Since then, I've run dcdiags and dumped those to a text as well and
here are my results...
Directory Server Diagnosis
Performing initial setup:
   Trying to find home server...
   Home Server = domainVMDC2
   * Identified AD Forest. 
   Done gathering initial info.
Doing initial required tests
   Testing server: Default-First-Site-Name\domainVMDC2
      Starting test: Connectivity
         ......................... domainVMDC2 passed test Connectivity
Doing primary tests
   Testing server: Default-First-Site-Name\domainVMDC2
      Starting test: Advertising
         ......................... domainVMDC2 passed test Advertising
      Starting test: FrsEvent
         ......................... domainVMDC2 passed test FrsEvent
      Starting test: DFSREvent
         ......................... domainVMDC2 passed test DFSREvent
      Starting test: SysVolCheck
         ......................... domainVMDC2 passed test SysVolCheck
      Starting test: KccEvent
         ......................... domainVMDC2 passed test KccEvent
      Starting test: KnowsOfRoleHolders
         ......................... domainVMDC2 passed test KnowsOfRoleHolders
      Starting test: MachineAccount
         ......................... domainVMDC2 passed test MachineAccount
      Starting test: NCSecDesc
         ......................... domainVMDC2 passed test NCSecDesc
      Starting test: NetLogons
         ......................... domainVMDC2 passed test NetLogons
      Starting test: ObjectsReplicated
         ......................... domainVMDC2 passed test ObjectsReplicated
      Starting test: Replications
         ......................... domainVMDC2 passed test Replications
      Starting test: RidManager
         ......................... domainVMDC2 passed test RidManager
      Starting test: Services
         ......................... domainVMDC2 passed test Services
      Starting test: SystemLog
         A warning event occurred.  EventID: 0x00001795
            Time Generated: 12/18/2014   00:35:03
            Event String:
            The program lsass.exe, with the assigned process ID 476, could not authenticate locally by using the target name ldap/domainvmdc2.domain.school.edu. The target name used is not valid. A target name should
refer to one of the local computer names, for example, the DNS host name.
         ......................... domainVMDC2 passed test SystemLog
      Starting test: VerifyReferences
         ......................... domainVMDC2 passed test VerifyReferences
   Running partition tests on : ForestDnsZones
      Starting test: CheckSDRefDom
            For the partition
            (DC=ForestDnsZones,DC=domain,DC=school,DC=edu) we encountered
            the following error retrieving the cross-ref's
            (CN=3098109a-ff99-41d4-8926-0e814ac8efde,CN=Partitions,CN=Configuration,DC=domain,DC=school,DC=edu)
             information: 
               LDAP Error 0x52e (1326). 
         ......................... ForestDnsZones failed test CheckSDRefDom
      Starting test: CrossRefValidation
            For the partition
            (DC=ForestDnsZones,DC=domain,DC=school,DC=edu) we encountered
            the following error retrieving the cross-ref's
            (CN=3098109a-ff99-41d4-8926-0e814ac8efde,CN=Partitions,CN=Configuration,DC=domain,DC=school,DC=edu)
             information: 
               LDAP Error 0x52e (1326). 
         ......................... ForestDnsZones failed test
         CrossRefValidation
   Running partition tests on : DomainDnsZones
      Starting test: CheckSDRefDom
            For the partition
            (DC=DomainDnsZones,DC=domain,DC=school,DC=edu) we encountered
            the following error retrieving the cross-ref's
            (CN=2f0b8ac0-2630-441a-891f-b5fcb91498a8,CN=Partitions,CN=Configuration,DC=domain,DC=school,DC=edu)
             information: 
               LDAP Error 0x52e (1326). 
         ......................... DomainDnsZones failed test CheckSDRefDom
      Starting test: CrossRefValidation
            For the partition
            (DC=DomainDnsZones,DC=domain,DC=school,DC=edu) we encountered
            the following error retrieving the cross-ref's
            (CN=2f0b8ac0-2630-441a-891f-b5fcb91498a8,CN=Partitions,CN=Configuration,DC=domain,DC=school,DC=edu)
             information: 
               LDAP Error 0x52e (1326). 
         ......................... DomainDnsZones failed test
         CrossRefValidation
   Running partition tests on : Schema
      Starting test: CheckSDRefDom
         ......................... Schema passed test CheckSDRefDom
      Starting test: CrossRefValidation
            For the partition
            (CN=Schema,CN=Configuration,DC=domain,DC=school,DC=edu) we
            encountered the following error retrieving the cross-ref's
            (CN=Enterprise Schema,CN=Partitions,CN=Configuration,DC=domain,DC=school,DC=edu)
             information: 
               LDAP Error 0x52e (1326). 
         ......................... Schema failed test CrossRefValidation
   Running partition tests on : Configuration
      Starting test: CheckSDRefDom
         ......................... Configuration passed test CheckSDRefDom
      Starting test: CrossRefValidation
            For the partition
            (CN=Configuration,DC=domain,DC=school,DC=edu) we encountered
            the following error retrieving the cross-ref's
            (CN=Enterprise Configuration,CN=Partitions,CN=Configuration,DC=domain,DC=school,DC=edu)
             information: 
               LDAP Error 0x52e (1326). 
         ......................... Configuration failed test CrossRefValidation
   Running partition tests on : domain
      Starting test: CheckSDRefDom
         ......................... domain passed test CheckSDRefDom
      Starting test: CrossRefValidation
            For the partition (DC=domain,DC=school,DC=edu) we encountered
            the following error retrieving the cross-ref's
            (CN=domain,CN=Partitions,CN=Configuration,DC=domain,DC=school,DC=edu)
             information: 
               LDAP Error 0x52e (1326). 
         ......................... domain failed test CrossRefValidation
   Running enterprise tests on : domain.school.edu
      Starting test: LocatorCheck
         ......................... domain.school.edu passed test
         LocatorCheck
      Starting test: Intersite
         ......................... domain.school.edu passed test Intersite
From what I can gather, there is a definite DNS issue but I don't have any stale records to the old DC stored anywhere. I've tried this with a new server as well and get similar errors... 
At this rate I'm ready to rebuild the entire forest over again. I'm just reluctant to do so as I want to make this a learning experience for the students. 
Any help would be greatly appreciated. Thanks!

Answer 1

As you can see, there seems to be some errors. The one that I did correct was the one around the _msdcs NS record being unable to resolve. For whatever, reason the name wasn't resolving the IP but all other NS tabs and records were. Just that one _msdcs
sub-zone. Furthermore, the mentioning of any connections to root hint servers can be viewed as false positives. There is no external comms to this lab so no communication with outside IPs can be expected. Lastly, they mentioned a connectivity issue yet mention
that I should check the firewall settings. All three profiles are disabled in Windows Firewall (as they have been the entire time). Thank you in advance for your help!
C:\Windows\system32>dcdiag /test:dns /v
Directory Server Diagnosis
Performing initial setup:
   Trying to find home server...
   * Verifying that the local machine domainVMDC2, is a Directory Server.
   Home Server = domainVMDC2
   * Connecting to directory service on server domainVMDC2.
   * Identified AD Forest.
   Collecting AD specific global data
   * Collecting site info.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=domain,DC=school,DC=edu,LDAP_SCOPE_SUBTREE,(objectCategory=ntDSSiteSettings),.......
   The previous call succeeded
   Iterating through the sites
   Looking at base site object: CN=NTDS Site Settings,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=school,DC=edu
   Getting ISTG and options for the site
   * Identifying all servers.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=domain,DC=school,DC=edu,LDAP_SCOPE_SUBTREE,(objectClass=ntDSDsa),.......
   The previous call succeeded....
   The previous call succeeded
   Iterating through the list of servers
   Getting information for the server CN=NTDS Settings,CN=domainVMDC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=domain,DC=school,DC=edu
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   * Identifying all NC cross-refs.
   * Found 1 DC(s). Testing 1 of them.
   Done gathering initial info.
Doing initial required tests
   Testing server: Default-First-Site-Name\domainVMDC2
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         The host
         3a38b19c-4bb3-4542-acb6-9e5e97cc15c4._msdcs.domain.school.edu
         could not be resolved to an IP address. Check the DNS server, DHCP,
         server name, etc.
         Got error while checking LDAP and RPC connectivity. Please check your
         firewall settings.
         ......................... domainVMDC2 failed test Connectivity
Doing primary tests
   Testing server: Default-First-Site-Name\domainVMDC2
      Test omitted by user request: Advertising
      Test omitted by user request: CheckSecurityError
      Test omitted by user request: CutoffServers
      Test omitted by user request: FrsEvent
      Test omitted by user request: DFSREvent
      Test omitted by user request: SysVolCheck
      Test omitted by user request: KccEvent
      Test omitted by user request: KnowsOfRoleHolders
      Test omitted by user request: MachineAccount
      Test omitted by user request: NCSecDesc
      Test omitted by user request: NetLogons
      Test omitted by user request: ObjectsReplicated
      Test omitted by user request: OutboundSecureChannels
      Test omitted by user request: Replications
      Test omitted by user request: RidManager
      Test omitted by user request: Services
      Test omitted by user request: SystemLog
      Test omitted by user request: Topology
      Test omitted by user request: VerifyEnterpriseReferences
      Test omitted by user request: VerifyReferences
      Test omitted by user request: VerifyReplicas
      Starting test: DNS
         DNS Tests are running and not hung. Please wait a few minutes...
         See DNS test in enterprise tests section for results
         ......................... domainVMDC2 passed test DNS
   Running partition tests on : ForestDnsZones
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : DomainDnsZones
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : Schema
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : Configuration
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : domain
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running enterprise tests on : domain.school.edu
      Starting test: DNS
         Test results for domain controllers:
            DC: domainVMDC2
            Domain: domain.school.edu
               TEST: Authentication (Auth)
                  Authentication test: Successfully completed
               TEST: Basic (Basc)
                  Error: No LDAP connectivity
                  The OS
                  Microsoft Windows Server 2012 R2 Datacenter (Service Pack level: 0.0)
                  is supported.
                  NETLOGON service is running
                  kdc service is running
                  DNSCACHE service is running
                  DNS service is running
                  DC is a DNS server
                  Network adapters information:
                  Adapter [00000010] vmxnet3 Ethernet Adapter:
                     MAC address is 00:50:56:A2:2C:24
                     IP Address is static
                     IP address: *.*.100.26
                     DNS servers:
                        *.*.100.26 (domainVMDC2) [Valid]
                  No host records (A or AAAA) were found for this DC
                  The SOA record for the Active Directory zone was found
                  The Active Directory zone on this DC/DNS server was found primary
                  Root zone on this DC/DNS server was not found
               TEST: Forwarders/Root hints (Forw)
                  Recursion is enabled
                  Forwarders are not configured on this DNS server
                  Root hint Information:
                     Name: a.root-servers.net. IP: 198.41.0.4 [Invalid (unreachable)]
                     Name: b.root-servers.net. IP: 192.228.79.201 [Invalid (unreachable)]
                     Name: c.root-servers.net. IP: 192.33.4.12 [Invalid (unreachable)]
                     Name: d.root-servers.net. IP: 199.7.91.13 [Invalid (unreachable)]
                     Name: e.root-servers.net. IP: 192.203.230.10 [Invalid (unreachable)]
                     Name: f.root-servers.net. IP: 192.5.5.241 [Invalid (unreachable)]
                     Name: g.root-servers.net. IP: 192.112.36.4 [Invalid (unreachable)]
                     Name: h.root-servers.net. IP: 128.63.2.53 [Invalid (unreachable)]
                     Name: i.root-servers.net. IP: 192.36.148.17 [Invalid (unreachable)]
                     Name: j.root-servers.net. IP: 192.58.128.30 [Invalid (unreachable)]
                     Name: k.root-servers.net. IP: 193.0.14.129 [Invalid (unreachable)]
                     Name: l.root-servers.net. IP: 199.7.83.42 [Invalid (unreachable)]
                     Name: m.root-servers.net. IP: 202.12.27.33 [Invalid (unreachable)]
                  Error: Both root hints and forwarders are not configured or
                  broken. Please make sure at least one of them works.
               TEST: Delegations (Del)
                  Delegation information for the zone: domain.school.edu.
                     Delegated domain name: _msdcs.domain.school.edu.
                        Error: DNS server: domainvmdc2. IP:<Unavailable>
                        [Missing glue A record]
                        [Error details: 9714 (Type: Win32 - Description: DNS name does not exist.)]
               TEST: Dynamic update (Dyn)
                  Test record dcdiag-test-record added successfully in zone domain.school.edu
                  Warning: Failed to delete the test record dcdiag-test-record in zone domain.school.edu
                  [Error details: 13 (Type: Win32 - Description: The data is invalid.)]
               TEST: Records registration (RReg)
                  Network Adapter [00000010] vmxnet3 Ethernet Adapter:
                     Matching CNAME record found at DNS server *.*.100.26:
                     3a38b19c-4bb3-4542-acb6-9e5e97cc15c4._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.a9241004-88ea-422d-a71e-df7b622f0d68.domains._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _kerberos._tcp.dc._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.dc._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _kerberos._tcp.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _kerberos._udp.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _kpasswd._tcp.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.Default-First-Site-Name._sites.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _kerberos._tcp.Default-First-Site-Name._sites.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.gc._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _gc._tcp.Default-First-Site-Name._sites.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.Default-First-Site-Name._sites.gc._msdcs.domain.school.edu
                     Matching  SRV record found at DNS server *.*.100.26:
                     _ldap._tcp.pdc._msdcs.domain.school.edu
               Error: Record registrations cannot be found for all the network
               adapters
         Summary of test results for DNS servers used by the above domain
         controllers:
            DNS server: 128.63.2.53 (h.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 128.63.2.53               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 192.112.36.4 (g.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.112.36.4               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 192.203.230.10 (e.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.203.230.10               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 192.228.79.201 (b.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.228.79.201               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 192.33.4.12 (c.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.33.4.12               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 192.36.148.17 (i.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.36.148.17               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 192.5.5.241 (f.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.5.5.241               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 192.58.128.30 (j.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 192.58.128.30               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 193.0.14.129 (k.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 193.0.14.129               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 198.41.0.4 (a.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 198.41.0.4               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 199.7.83.42 (l.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 199.7.83.42               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 199.7.91.13 (d.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 199.7.91.13               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: 202.12.27.33 (m.root-servers.net.)
               1 test failure on this DNS server
               PTR record query for the 1.0.0.127.in-addr.arpa. failed on the DNS server 202.12.27.33               
[Error details: 1460 (Type: Win32 - Description: This operation returned because the timeout period expired.)]
            DNS server: *.*.100.26 (domainVMDC2)
               All tests passed on this DNS server
               Name resolution is functional._ldap._tcp SRV record for the forest root domain is registered
         Summary of DNS test results:
                                            Auth Basc Forw Del  Dyn  RReg Ext
            Domain: domain.school.edu
               domainVMDC2                 PASS FAIL FAIL FAIL WARN FAIL n/a
         ......................... domain.school.edu failed test DNS
      Test omitted by user request: LocatorCheck
      Test omitted by user request: Intersite