Site to Site Replication only works for a few hours in the morning (each morning)

Question

We have been fighting an odd active directory replication issue for over a month now and I am hoping that someone can provide some insight. We have 5 AD servers in the following orientation...
Site HQ
- PRIME running Windows 2008 R2
- AD2 running Windows 2008 R2
Site COLO
- AD3 running Windows 2008 (not R2)
- AD3NEW running Windows 2008 R2
Site BRANCH
- AD4 running Windows 2008 R2
The domain is at the Windows 2008 Functional Level.
There are always on site to site VPNs between all 3 sites and IP Intersite Transports Site Links defined for all 3 possible connections with Cost of 100 and interval of 15. Each IP site link is configured with a schedule of available all day long.
Every day the following sequence of events happens...
* Somewhere between 6:30 and 7:30am all the servers start to sync with each other perfectly. We can make AD changes and they replicate across all servers without issues. During this time all the repadmin commands work well across all servers.
* Typically somewhere in the 10:30 to 11:30am time frame we start to get errors replicating data - specifically between the HQ and COLO sites. This manifests itself as Event 1232 Call Timeout from the DC RPC Client and and Event 1925 from the KCC. Additionally
repadmin commands fail when attempting to connect to the BRANCH servers.
* For the rest of the day the intra-site replication between PRIME and AD2 work fine - and periodically the BRANCH AD server is updated as well. But the COLO sites remain unreplicated and continue to get errors for the remainder of the day. While this down
- the ability to ping and remote desktop between the servers is perfectly fine - so even if there were a network hiccup that happens - the network is stable for hours without the sites recovering.
* Magically the next morning around 6:30 and 7:30am all the servers are able to replicate without issue and we get 3-5 hours of immediate replication and then it happens again.
As I stated above - there is always on site-to-site VPN connections between all 3 sites that are actively monitored by PRTG. These connections remain open all day long. The Site topology has the COLO servers attempting to replicate with the HQ servers -
and both sites have 100MB data connections that remain active during the entire time. Additionally PRTG bandwidth monitoring shows that these links have no spikes in traffic anywhere near the max capacity of those links during the time that the outages begin
nor during the rest of the day.
Does anyone have any insight as to why these servers would stop communicating with each other about the same time every day and report errors? Also why it would magically start to work again each day without any changes being made to the network or the AD
configuration?
This has been going on for over a month now. When it first started to happen we had 1 Windows 2008 server and 2 Windows 2003 servers in the HQ. We phased out the Windows 2003 servers and upgraded the functional level to Windows 2008 - that did not solve
the problem. We tried to put a new Windows 2008 R2 server out at the COLO site hoping that if it was limited to the other server then only the one server would be impacted. But now they both appear to be having connectivity issues at the same time.
It is as if there is one hung connection that is blocking all the other syncs to this site and then someone each morning that bottleneck is released.
Thank you in advance for any direction you can provide.

Answer 1

As was stated above - ALL Domain Controllers have direct access to each other through Firewall to Firewall site to site VPNs and the Inter-Site Transport Links mirror that setup. So from the OS perspective any of the AD servers can directly connect to any
other one.
There are 3 IP Inter-Site Transport Links defined
HQ < - > COLO   (Contains HQ and COLO sites) Cost 100  Replication Interval 15
HQ < - > BRANCH  (Contains HQ and Branch sites) Cost 100 Replication Interval 15
COLO < - > BRANCH  (Contains COLO and Branch sites) Cost 100 Replication Interval 15
And on IP Inter-Site Transports "Bridge all site links" is enabled (although disabling it doesn't fix this problem as we have already tried that).
Right now the servers are claiming (via Active Directory) to be unable to replicate with each other. But I am able to do direct pings as well as open stream sockets using "telnet <otherserver> <port>" on ports 3268 (gc), 88 (kerberos),
389 (ldap), 135 (replication), 636 (ldap ssl), 53 (DNS). So there is nothing that I can see between the servers that is blocking TCP connectivity.
I cannot seem to make this any clearer. The sites are 100% functional and responsive for several hours per day - and then mysteriously go into a state of complete denial for a lack of a better word for the rest of the day - only to return back to normal
again reliably each morning.
It is as if the sites get into a mode where something in the RPC area are simply refusing to talk to each other despite the servers having full access at the network level.
Another data point to add to this mystery. While it is in the state where the HQ and COLO servers are refusing to sync with each other. You can launch the AD Users and Computers snap-in, right mouse click on the domain, change the Current Directory server
and all 5 servers show up as ONLINE. You can pick any of them (including the one that is unable to replicate with) and make a direct change on that server.
So while the servers are complaining about being unable to talk to each other - the snap in is connecting between those servers and is able to modify it without issue.
Conversely - when the replication is failing the DNS management tool is unable to connect to the remote servers (i.e. COLO can show itself and the other COLO server. HQ can show PRIME, DC2, and DC4 without issue. But no overlap).
Not sure that helps at all - but shows our frustration when two servers refuse to replicate but you can easily remote connect from one to the other and make the change.