Exchange 2010 DAG netowrk replications issues - Causing database copies to be disconnected and resynchronizing.

Question

My
environment is as follows.<o:p></o:p>
We have 3 exchange 2010 servers.<o:p></o:p>
EX1, EX2, DREX1. All servers are exchange 2010 14.2 build 247.5<o:p></o:p>
EX1 and EX2 have two nics, one LAN and one for BACKUP <o:p></o:p>
The dag is configured to to communicate to all three.<o:p></o:p>
The IP for EX1 is 192.168.100.51 - LAN, switch A; 160.100.3.51 - BACKUP, switch B<o:p></o:p>
The IP for EX2 is 192.168.100.52 - LAN, switch A; 160.100.3.52 - BACKUP, switch B<o:p></o:p>
DREX1 is located in another network. It is in another building connected via vpn. The ip
for DREX1 is 192.170.100.51. There is no BACKUP network for this server<o:p></o:p>
(These are not the actual address, but the concept is similar)<o:p></o:p>
The dag network configuration is as follows.<o:p></o:p>
DAGNetworkDR has a subnet of 192.170.100.0/24; Network Interface of 192.170.100.51<o:p></o:p>
DAGNetwork01 (LAN) has a subnet of 192.168.100.0/24; Network Interfaces of 192.168.100.51
and 192.168.100.52<o:p></o:p>
DAGNetwork02 (BACKUP) has a subnet of 160.100.3.0/24; Network Interfaces of
160.100.3.51 and 160.100.3.52<o:p></o:p>
Last Friday, 3/7/14 something happened.<o:p></o:p>
To lead up to that, I noticed the issue because we have DPM 2010 making 15minute express
full backups of our exchange databases when I reviewed the backup jobs, the
exchange database backups using dpm2010 failed all weekend long starting at
11:00pm that Friday.<o:p></o:p>
The issue that occurred was that the database copies went to a status of Disconnected and
Resynchronizing, and after about 30 seconds, the status would show
Resynchronizing.<o:p></o:p>
All attempts to resolve this with the research methods failed.  Deleting the
logs and reseeding failed, suspending, then rebuilding failed.<o:p></o:p>
Creating a new database worked, creating a database copy worked, but seeding the copy
failed. All with the same pattern, a status of Disconnected and
Resynchronizing, and after about 30 seconds, and the status would show
Resynchronizing.<o:p></o:p>
The database was mounted on EX2, and seeding to EX1
DREX1 has no issues.<o:p></o:p>
All the DAG networks were in a good status, all Green and all up.  I could ping from
EX1 to EX2 and from EX2 to EX1 on both networks fine.<o:p></o:p>
In the middle of the week, about 3/12/14, we resolved the issue. We rebooted EX1, no
fix, but after rebooting EX2, it all resolved.  It did a fail over to DREX1. I think it failed to do the failover to EX1. However, when it came back up, it was happy, it was reseeding slowly. After about an hour and a half, it
was all healthy.<o:p></o:p>
We thought it was all good EXCEPT, on Friday night at 11:00pmish on 3/14/14, the same
issue occurred; a status of Disconnected and Resynchronizing, and after about
30 seconds, the status would show Resynchronizing.<o:p></o:p>
The database was mounted on EX2 and seeding to EX1 again. I did not notice the
issue until I performed my review of the backup jobs again on Monday.<o:p></o:p>
Late Monday morning, my boss and I were looking at more articles, but all the same results,
delete the database copy and reseed. Which we knew would not work.<o:p></o:p>
I mentioned that it was strange because the networks were all up. What made us try the next
scenarios was just "luck" I guess. I made a statement about not remembering
which networks DAG replication took place over.  Again, we have a LAN and
a BACKUP network.  Bother were listed to use replication, so we decided to
disable replication over the BACKUP network in the EMC. Immediately, the
database copies switch to healthy and started seeding.<o:p></o:p>
Later that evening I enabled replication over the BACKUP network and it failed again with
a status of Disconnected and Resynchronizing, and after about 30 seconds, the
status would show Resynchronizing.  I disabled the replication over the
BACKUP network and the database copy went back to healthy<o:p></o:p>
The last round of windows updates were pushed to these boxes the Friday before the
issues began, 2/28/14.  I don't believe the updates to be the issue. <o:p></o:p>
Can anyone make any suggestions as to where to look?  It worked with the BACKUP
network enabled to so long that is eluding me as to why the issues would start
all of a sudden. I am also confident that the switch configuration for the
BACKUP network has not been changed at all in at least 2-3 months.  I am
still open to look at the config. Please help if you can.<o:p></o:p>
 <o:p></o:p>
Lastly, this is more of a monitoring thing, if anyone can suggest a good way to monitor
the database copies so that I can be alerted when the status changes.  We
use WhatsUpGold 16.2 to monitor the devices.  <o:p></o:p>
Thanks for any future assistance.<o:p></o:p>
Ian
Ian

Answer 1

You don't have to enable the Replication on Backup Network.
You should collapse your DAG network.
http://blogs.technet.com/b/timmcmic/archive/2011/09/26/exchange-2010-collapsing-dag-networks.aspx
http://blogs.technet.com/b/samdrey/archive/2012/12/04/exchange-2010-dag-collapse-the-mapi-networks-in-a-dag.aspx
Post the result of Below command:
Get-DatabaseAvailabilityGroupNetwork 
Cheers,
Gulab Prasad
Technology Consultant
Blog:
http://www.exchangeranger.com    Twitter:
  LinkedIn:
   Check out CodeTwo’s tools for Exchange admins
Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.