SQL 2005 mirroring : Abrupt Automatic failover

Question

hi All, 
We have a SQL 2005 SP4 mirroring  setup of 15 DBs with Principal(P), Mirror(M) & Witness (W).
We have now seen abrupt DB failovers for some of the databases (yest it was 4 out of 15) from P to M.
Errors were seen on Witness server as follows for all Dbs that failed over:
Date 07/01/2015 11:07:48 PM
Log SQL Server (Current - 08/01/2015 12:00:00 AM)
Source spid19s
Message
The mirroring connection to "TCP://<server.domain.com>:5022" has timed out for database "<DBName>" after 10 seconds without a response.  Check the service and network connections.
Actions taken:
1. Network and Firewall team reverted that no error detected and no network traffic between the witness server and db server during the db auto failover period.
2. On the system side, we have verified that no hardware error found on either VM or SAN storage, and no Symantec SQL backup jobs running nor anti virus scanning during the db auto failover period too.
3. We did see some high amount of IO activity on P server around failover time. Some IO errors similar to below were seen, however point to note is these errors were not only for the DBs that failed over, but for others including TEMPDB:
Date 07/01/2015 11:07:38 PM
Log SQL Server (Current - 08/01/2015 4:06:00 AM)
Source spid2s
Message
SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [R:\SQLDATA\MSSQL.1\MSSQL\Data\<DBName>.mdf] in database [DBName] (5).  The OS file handle is 0x000000000000095C.  The offset of the
latest long I/O is: 0x0000054ff22000
Questions:
1. I assumed that the Witness keeps polling P & M on DB mirroring endpoints (in our case 5022) to check that the DBs are online, but Network team says there is no activity on that port, is my understanding correct?
2. Is there any other reason for DB failover ? 
Link referred:  
http://dba.stackexchange.com/questions/22402/what-can-cause-a-mirroring-session-to-timeout-then-failover-sql-server-2005
http://msdn.microsoft.com/en-us/library/ms179344(v=sql.90).aspx
Any help is highly appreciated!!!
Regards,
Mandar

Answer 1

This is common with Mirroring server it is not as resilient to changes as log shipping. Are you aware about
below fact although not directly related to your question
If you plan to use high-safety mode with automatic failover, the normal load on each failover partner should be less than 50 percent of the CPU. If your work load overloads the CPU, a failover partner might be unable to ping the other server instances in
the mirroring session. This causes a unnecessary failover. If you cannot keep the CPU usage under 50 percent, we recommend that you use either high-safety mode without automatic failover or high-performance mode.
Now to your problem
The mirroring connection to "TCP://<server.domain.com>:5022" has timed out for database "<DBName>" after 10 seconds without a response.  Check the service and network connections.
I would say there was network dip for more than 10 seconds and since default failover time is 10 seconds and for few databases witness thought principal cannot be reached it initiated failover.
Network team is incorrect to say there was no dip (its common with NOC team not to take responsibility)
This Support Article is worth reading specially the network part
Please mark this reply as answer if it solved your issue or vote as helpful if it helped so that other forum members can benefit from it
My Technet Wiki Article
MVP