Upgrade to Server 2012 R2 domain controllers from 2003

Question

I am at a loss as to what I did wrong here. Everything seems to be working fine except for one subnet (which is behind a hardware firewall).
We had two Server 2003 domain controllers and one of them was failing.  I raised the forest functional level of our old primary domain controllers to 2003.  I built the first replacement Server 2012 R2 domain controller.  Added the AD DS roles
and promoted it as a domain controller.  I let it sit for a couple days.  The FSMO roles were currently being handled by our other 2003 domain controller.  Once this had been sitting for a while (don't recall how long) I ran dcpromo on the failing
server and demoted it.  Once demoted I shut it down and pulled it out of the rack.  I then built our second 2012 R2 server and gave it the same IP as the failing one.  Installed the AD DS roles and integrated DNS as prompted by the wizard. 
I then made it the operations master for Schema master, Domain naming master, PDC, RID pool manager, and Infrastructure master.  Then I ran dcpromo on the second 2003 domain controller to demote it and removed it from the network.  I then demoted
the first new controller (DC03) changed the hostname and IP to the name and IP of the second 2003 controller and promoted it again.  I'm not sure at what point things broke, but everything works from the same subnet that the domain controllers are in,
just not a second subnet that is through a hardware firewall.  I don't see anything getting blocked while watching firewall logs so I don't think the firewall is the issue.
Here is the dcdiag and ipconfig from the first controller (which has all 5 FSMO roles).
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.
C:\Users\username>dcdiag /v /test:dns
Directory Server Diagnosis
Performing initial setup:
   Trying to find home server...
   * Verifying that the local machine WGDDC01, is a Directory Server.
   Home Server = WGDDC01
   * Connecting to directory service on server WGDDC01.
   * Identified AD Forest.
   Collecting AD specific global data
   * Collecting site info.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=wgd,DC=inet,LD
AP_SCOPE_SUBTREE,(objectCategory=ntDSSiteSettings),.......
   The previous call succeeded
   Iterating through the sites
   Looking at base site object: CN=NTDS Site Settings,CN=Default-First-Site-Name
,CN=Sites,CN=Configuration,DC=wgd,DC=inet
   Getting ISTG and options for the site
   * Identifying all servers.
   Calling ldap_search_init_page(hld,CN=Sites,CN=Configuration,DC=wgd,DC=inet,LD
AP_SCOPE_SUBTREE,(objectClass=ntDSDsa),.......
   The previous call succeeded....
   The previous call succeeded
   Iterating through the list of servers
   Getting information for the server CN=NTDS Settings,CN=WGDDC01,CN=Servers,CN=
Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wgd,DC=inet
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   Getting information for the server CN=NTDS Settings,CN=WGDDC02,CN=Servers,CN=
Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wgd,DC=inet
   objectGuid obtained
   InvocationID obtained
   dnsHostname obtained
   site info obtained
   All the info for the server collected
   * Identifying all NC cross-refs.
   * Found 2 DC(s). Testing 1 of them.
   Done gathering initial info.
Doing initial required tests
   Testing server: Default-First-Site-Name\WGDDC01
      Starting test: Connectivity
         * Active Directory LDAP Services Check
         Determining IP4 connectivity
         * Active Directory RPC Services Check
         ......................... WGDDC01 passed test Connectivity
Doing primary tests
   Testing server: Default-First-Site-Name\WGDDC01
      Test omitted by user request: Advertising
      Test omitted by user request: CheckSecurityError
      Test omitted by user request: CutoffServers
      Test omitted by user request: FrsEvent
      Test omitted by user request: DFSREvent
      Test omitted by user request: SysVolCheck
      Test omitted by user request: KccEvent
      Test omitted by user request: KnowsOfRoleHolders
      Test omitted by user request: MachineAccount
      Test omitted by user request: NCSecDesc
      Test omitted by user request: NetLogons
      Test omitted by user request: ObjectsReplicated
      Test omitted by user request: OutboundSecureChannels
      Test omitted by user request: Replications
      Test omitted by user request: RidManager
      Test omitted by user request: Services
      Test omitted by user request: SystemLog
      Test omitted by user request: Topology
      Test omitted by user request: VerifyEnterpriseReferences
      Test omitted by user request: VerifyReferences
      Test omitted by user request: VerifyReplicas
      Starting test: DNS
         DNS Tests are running and not hung. Please wait a few minutes...
         See DNS test in enterprise tests section for results
         ......................... WGDDC01 failed test DNS
   Running partition tests on : DomainDnsZones
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : ForestDnsZones
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : Schema
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : Configuration
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running partition tests on : wgd
      Test omitted by user request: CheckSDRefDom
      Test omitted by user request: CrossRefValidation
   Running enterprise tests on : wgd.inet
      Starting test: DNS
         Test results for domain controllers:
            DC: WGDDC01.wgd.inet
            Domain: wgd.inet
               TEST: Authentication (Auth)
                  Authentication test: Successfully completed
               TEST: Basic (Basc)
                  The OS
                  Microsoft Windows Server 2012 R2 Standard (Service Pack level:
 0.0)
                  is supported.
                  NETLOGON service is running
                  kdc service is running
                  DNSCACHE service is running
                  DNS service is running
                  DC is a DNS server
                  Network adapters information:
                  Adapter [00000010] Broadcom NetXtreme Gigabit Ethernet:
                     MAC address is B0:83:FE:C1:98:07
                     IP Address is static
                     IP address: 10.240.1.23
                     DNS servers:
                        10.240.1.23 (WGDDC01) [Valid]
                        10.240.1.24 (WGDDC02) [Valid]
                        127.0.0.1 (WGDDC01) [Valid]
                  The A host record(s) for this DC was found
                  The SOA record for the Active Directory zone was found
                  Warning: no DNS RPC connectivity (error or non Microsoft DNS s
erver is running)
                  [Error details: 5 (Type: Win32 - Description: Access is denied
         Summary of test results for DNS servers used by the above domain
         controllers:
            DNS server: 10.240.1.23 (WGDDC01)
               All tests passed on this DNS server
               Name resolution is functional._ldap._tcp SRV record for the fores
t root domain is registered
            DNS server: 10.240.1.24 (WGDDC02)
               All tests passed on this DNS server
               Name resolution is functional._ldap._tcp SRV record for the fores
t root domain is registered
         Summary of DNS test results:
Auth Basc Forw Del  Dyn  RReg Ext
            Domain: wgd.inet
               WGDDC01                      PASS WARN n/a  n/a  n/a 
n/a  n/a
         ......................... wgd.inet passed test DNS
      Test omitted by user request: LocatorCheck
      Test omitted by user request: Intersite
C:\Users\dsmythe>ipconfig /all
Windows IP Configuration
   Host Name . . . . . . . . . . . . : WGDDC01
   Primary Dns Suffix  . . . . . . . : wgd.inet
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : wgd.inet
Ethernet adapter WGD_INET:
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet
   Physical Address. . . . . . . . . : B0-83-FE-C1-98-07
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.240.1.23(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.240.1.1
   DNS Servers . . . . . . . . . . . : 10.240.1.23
                                       10.240.1.24
                                       127.0.0.1
   NetBIOS over Tcpip. . . . . . . . : Enabled
Tunnel adapter isatap.{2C28B0FA-6BF8-4201-A6DA-081AED63B496}:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
When I try to bind a machine to the domain I get an error message that says "
The following error occurred when DNS was queried for the service location (SRV) resource record used to locate an Active Directory Domain Controller (AD DC) for domain "wgd.inet":
The error was: "This operation returned because the timeout period expired."
(error code 0x000005B4 ERROR_TIMEOUT)
The query was for the SRV record for _ldap._tcp.dc._msdcs.wgd.inet
The DNS servers used by this computer for name resolution are not responding. This computer is configured to use DNS servers with the following IP addresses:
10.240.1.24
10.240.1.23
Verify that this computer is connected to the network, that these are the correct DNS server IP addresses, and that at least one of the DNS servers is running.
Please let me know if I'm missing something or if there are other things I can check.
Thanks!
I forgot to mention that after the 2003 domain controllers were out of the environment, I raised the domain and forest functional level to 2012 R2.  All clients in the environment are Windows XP Pro or above.  The XP Pro boxes will be going away as
soon as our vendor supports their software to run on Windows 7.

Answer 1

We now have 2 2012 R2 DCs. The 2003 DCs are gone. Metadata from the old DCs is all cleaned up. DNS seems to be working fine in 3 out of 4 subnets. The 4th is behind a hardware firewall and I can see the IP address of the machine I am trying to bind to the
domain connecting to the two new domain controllers but the client machine that is trying to bind gives an error.  An Active Directory Domain Controller for the domain wgd.inet could not be contacted.  It seems that this is just a DNS issue for one
particular subnet (10.240.2.0/24).  This subnet is setup in AD Sites and Services\Sites\Subnets\10.240.2.0/24 (Site: Default-First-Site-Name).
When trying to do anything with nslookup from the 10.240.2.0/24 subnet it times out.  The route is there and I can watch it connect through our hardware firewall over port 53.
DC01
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.
C:\Users\dsmythe>netdom query fsmo
Schema master               WGDDC01.wgd.inet
Domain naming master        WGDDC01.wgd.inet
PDC                         WGDDC01.wgd.inet
RID pool manager            WGDDC01.wgd.inet
Infrastructure master       WGDDC01.wgd.inet
The command completed successfully.
C:\Users\dsmythe>ipconfig /all
Windows IP Configuration
   Host Name . . . . . . . . . . . . : WGDDC01
   Primary Dns Suffix  . . . . . . . : wgd.inet
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : wgd.inet
Ethernet adapter WGD_INET:
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet
   Physical Address. . . . . . . . . : B0-83-FE-C1-98-07
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.240.1.23(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.240.1.1
   DNS Servers . . . . . . . . . . . : 10.240.1.23
                                       10.240.1.24
   NetBIOS over Tcpip. . . . . . . . : Enabled
Tunnel adapter isatap.{2C28B0FA-6BF8-4201-A6DA-081AED63B496}:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
C:\Users\dsmythe>
DC02
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.
C:\Users\dsmythe>netdom query fsmo
Schema master               WGDDC01.wgd.inet
Domain naming master        WGDDC01.wgd.inet
PDC                         WGDDC01.wgd.inet
RID pool manager            WGDDC01.wgd.inet
Infrastructure master       WGDDC01.wgd.inet
The command completed successfully.
C:\Users\dsmythe>ipconfig /all
Windows IP Configuration
   Host Name . . . . . . . . . . . . : WGDDC02
   Primary Dns Suffix  . . . . . . . : wgd.inet
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : wgd.inet
Ethernet adapter NIC1:
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Broadcom NetXtreme Gigabit Ethernet
   Physical Address. . . . . . . . . : B0-83-FE-C1-9F-74
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.240.1.24(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.240.1.1
   DNS Servers . . . . . . . . . . . : 10.240.1.24
                                       10.240.1.23
   NetBIOS over Tcpip. . . . . . . . : Enabled
Tunnel adapter isatap.{4F45E51E-FC2F-49ED-85CF-0750A9EEECF5}:
   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
C:\Users\dsmythe>