CAS restart and Mailbox database dismount unexpectedly somtimes

Question

Hi all,
In my environment I have 2 Exchange server 2013 (CAS + Mailbox Role both) CU2 : EX01 & EX02 in DAG01. Sometimes CAS service is restarted or Mailbox Database copy on Exchange servers is dismounted unexpectedly. For example today:
I have "Mailbox Database 01" copies on EX01 & EX02
Get-MailboxDatabaseCopyStatus "Mailbox Database 01"
Name Status
Mailbox Database 01\EX01 Healthy
Mailbox Database 01\EX02 Mounted
"Mailbox Database 01" copy on EX02 is dismounted, of course the other copy on EX01 is mounted automatically
Log Name:      Application
Source:        MSExchangeRepl
Date:          4/3/2015 2:32:19 PM
Event ID:      2153
Task Category: Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EX02.mydomain.com
Description:
The log copier was unable to communicate with server 'EX01.mydomain.com'. The copy of database 'Mailbox Database 01\EX02' is in a disconnected state. The communication error was: An error occurred while communicating with server 'EX01'. Error: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. The copier will automatically retry after a short delay.
CAS service on EX01 , The Microsoft Exchange Mailbox Assistants service  on EX02 are restarted
Log Name: System
Source: Service Control Manager
Date: 4/3/2015 3:25:49 PM
Event ID: 7031
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: EX01.mydomain.com
Description:
The Microsoft Exchange RPC Client Access service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 5000 milliseconds: Restart the service.
Log Name: System
Source: Service Control Manager
Date: 4/3/2015 3:25:51 PM
Event ID: 7036
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: EX01.mydomain.com
Description:
The Microsoft Exchange RPC Client Access service entered the running state.
Log Name: System
Source: Service Control Manager
Date: 4/3/2015 3:25:54 PM
Event ID: 7032
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: EX01.mydomain.com
Description:
The Service Control Manager tried to take a corrective action (Restart the service) after the unexpected termination of the Microsoft Exchange RPC Client Access service, but this action failed with the following error:
An instance of the service is already running.
Log Name: System
Source: Service Control Manager
Date: 4/3/2015 3:30:28 PM
Event ID: 7031
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: EX02.mydomain.com
Description:
The Microsoft Exchange Mailbox Assistants service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 5000 milliseconds: Restart the service.
Log Name: System
Source: Service Control Manager
Date: 4/3/2015 3:30:30 PM
Event ID: 7036
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: EX02.mydomain.com
Description:
The Microsoft Exchange Mailbox Assistants service entered the running state.
Log Name: System
Source: Service Control Manager
Date: 4/3/2015 3:30:33 PM
Event ID: 7032
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: EX02.mydomain.com
Description:
The Service Control Manager tried to take a corrective action (Restart the service) after the unexpected termination of the Microsoft Exchange Mailbox Assistants service, but this action failed with the following error:
An instance of the service is already running.
On EX01 I can see many 1309 , 2009 , 6002 , 4999 in Event Viewer on both servers, and they just used half of RAM
I'm going to upgrade to SP1 this weekend but please help me to figure out really reason of this issue, limit of max connections on Exchange/Window server , or Disk IO or something ?
Thanks for your help.

Answer 1

Last night I tried reseed Mailbox Database 01\EX02 , about 8:15 PM
I checked event viewer, last 2153 event happened on 7:49 PM, check around :
Get-MailboxDatabaseCopyStatus -Server ex02 | fl name,*incoming*,*outgoing*
Name : Mailbox Database 07\EX02
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01,An error occurred while communicating with server
'EX01'. Error: Unable to write data to the transport connection: An established
connection was aborted by the software in your host machine.}
OutgoingConnections :
Name : Mailbox Database 99\EX02
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing*
Name : Mailbox Database 01\EX01
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : Mailbox Database 99\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections :
Name : Mailbox Database 07\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections :
Then I suspend and update Mailbox Database 01\EX02
Suspend-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX02"
Update-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX02" -Network "DAG01\ReplicationDagNetwork01"
Confirm
Are you sure you want to perform this action?
Seeding database copy "Mailbox Database 01\EX02".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"):
The seeding operation failed. Error: An error occurred while running prerequisite checks. Error: Log files exist in
'D:\Exchange Mailbox Database\Mailbox Database 01'. You must remove them before database seeding or reseeding can be
performed. You can use the Update-MailboxDatabaseCopy cmdlet with the -DeleteExistingFiles or -SafeDeleteExistingFiles
parameter to do this. [Database: Mailbox Database 01, Server: EX02.mydomain.com]
+ CategoryInfo : InvalidOperation: (Mailbox Database 01:String) [Update-MailboxDatabaseCopy], SeedPrepare
Exception
+ FullyQualifiedErrorId : [Server=EX02,RequestId=56f71c07-6209-4aae-94cb-bf3cfc0a38fd,TimeStamp=4/22/2015 1:0
5:56 PM] [FailureCategory=Cmdlet-SeedPrepareException] 5F5B4C53,Microsoft.Exchange.Management.SystemConfigurationT
asks.UpdateDatabaseCopy
+ PSComputerName : ex02.mydomain.com
I want they use DAG01\ReplicationDagNetwork01 (1Gbps) for reseeding database, then I tried to add -SafeDeleteExistingFiles
Update-MailboxDatabaseCopy -Identity "Mailbox Database 01\EX02" -Network "DAG01\ReplicationDagNetwork01" -SafeDeleteExistingFiles
Confirm
Are you sure you want to perform this action?
Seeding database copy "Mailbox Database 01\EX02".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"):
The seeding operation failed. Error: An error occurred while running prerequisite checks. Error: An error occurred
when trying to run the database redundancy validation checks for database 'Mailbox Database 01' due to
'SafeDeleteExistingFiles' being specified. Error: There were database redundancy check failures for database 'Mailbox
Database 01' that may be lowering its redundancy and putting the database at risk of data loss. Redundancy Count: 1.
Expected Redundancy Count: 2. Detailed error(s):
ex02:
Database 'Mailbox Database 01' does not have enough copies configured to meet the validation criteria.
[Database: Mailbox Database 01, Server: EX02.mydomain.com]
+ CategoryInfo : InvalidOperation: (Mailbox Database 01:String) [Update-MailboxDatabaseCopy], SeedPrepare
Exception
+ FullyQualifiedErrorId : [Server=EX02,RequestId=48374aa6-1198-4976-a435-3eb034dc5b58,TimeStamp=4/22/2015 1:0
8:44 PM] [FailureCategory=Cmdlet-SeedPrepareException] 458ED9A2,Microsoft.Exchange.Management.SystemConfigurationT
asks.UpdateDatabaseCopy
+ PSComputerName : ex02.mydomain.com
I think I should stop here, search more about -DeleteExistingFiles before continue. So I Resume-MailboxDatabaseCopy "Mailbox Database 01\EX02"
Everything is fine, I checked again
[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus -Server ex02 | fl name,*incoming*,*outgoing*
Name : Mailbox Database 07\EX02
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01}
OutgoingConnections :
Name : Mailbox Database 99\EX02
IncomingLogCopyingNetwork :
OutgoingConnections : {}
[PS] C:\Windows\system32>Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing*
Name : Mailbox Database 01\EX01
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : Mailbox Database 99\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections :
Name : Mailbox Database 07\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections
What is going on :) ? IncomingLogCopyingNetwork for Mailbox Database 01\EX02 is fine :) til mid night
This morning I view events on EX02 and event 2153 appears again on 7:30 AM , I run command et-MailboxDatabaseCopyStatus again
Get-MailboxDatabaseCopyStatus -Server ex02 | fl name,*incoming*,*outgoing*
Name : Mailbox Database 07\EX02
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : Mailbox Database 01\EX02
IncomingLogCopyingNetwork : {EX01,ReplicationDagNetwork01,An error occurred while communicating with server
'EX01'. Error: Unable to write data to the transport connection: An established
connection was aborted by the software in your host machine.}
OutgoingConnections :
Name : Mailbox Database 99\EX02
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Get-MailboxDatabaseCopyStatus -Server ex01 | fl name,*incoming*,*outgoing*
Name : Mailbox Database 01\EX01
IncomingLogCopyingNetwork :
OutgoingConnections : {}
Name : Mailbox Database 99\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections :
Name : Mailbox Database 07\EX01
IncomingLogCopyingNetwork : {EX02,ReplicationDagNetwork01}
OutgoingConnections :
Any suggest ?