Service MSExchangeMailSubmission. There is no available HubTransport server in the local site.

Question

I have found this and a number of other errors in my Exchange Organization after failed attempted DR exercise.
I will list a few:
The Microsoft Exchange Mail Submission service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time.
Attempt to move active database 'XXXMailDB' from 'active.mail.server' to 'failed.mail.server' failed. Move comment: None specified.
Error: An Active Manager operation failed. Error The database action failed. Error: An error occurred while trying to validate the specified database copy for possible activation. Error: Server 'failed.mail.server' has been stopped. To start this server, use
Start-DatabaseAvailabilityGroup..
Database redundancy health check failed.
Database copy: XXXArchiveDB
Redundancy count: 1
Error: Passive copy 'XXXArchiveDB\FAILED.MAIL.SERVER' is not UP according to clustering.
Name                 Status RealCopyQueu InspectorQue  ReplayQueue      CIState
                                       e           ue                        
XXXArchiveDB\   ServiceDown            0            0            0      Unknown
FAILED.MAIL.SERVER                                                                    
XXXArchiveDB\    Dismounted            0            0            0       Failed
ACTIVE.MAIL.SERVER
===============
 Full Status 
===============
Identity                         : XXXArchiveDB\FAILED.MAIL.SERVER
Name                             : XXXArchiveDB\FAILED.MAIL.SERVER
DatabaseName                     : XXXArchiveDB
Status                           : ServiceDown
MailboxServer                    : FAILED.MAIL.SERVER
ActiveDatabaseCopy               : 
ActivationSuspended              : False
ActionInitiator                  : Unknown
ErrorMessage                     : A server-side administrative operation has f
                                   ailed. The Microsoft Exchange Replication se
                                   rvice may not be running on server FAILED.MAIL.SERVER
                                   .XXX.local. Specific RPC error message: Erro
                                   r 0x71a (The remote procedure call was cance
                                   lled) from cli_GetCopyStatusEx2
ErrorEventId                     : 
ExtendedErrorInfo                : 
SuspendComment                   : 
SinglePageRestore                : 0
ContentIndexState                : Unknown
ContentIndexErrorMessage         : 
CopyQueueLength                  : 0
ReplayQueueLength                : 0
LatestAvailableLogTime           : 
LastCopyNotificationedLogTime    : 
LastCopiedLogTime                : 
LastInspectedLogTime             : 
LastReplayedLogTime              : 
LastLogGenerated                 : 0
LastLogCopyNotified              : 0
LastLogCopied                    : 0
LastLogInspected                 : 0
LastLogReplayed                  : 0
LogsReplayedSinceInstanceStart   : 
LogsCopiedSinceInstanceStart     : 
LatestFullBackupTime             : 
LatestIncrementalBackupTime      : 
LatestDifferentialBackupTime     : 
LatestCopyBackupTime             : 
SnapshotBackup                   : 
SnapshotLatestFullBackup         : 
SnapshotLatestIncrementalBackup  : 
SnapshotLatestDifferentialBackup : 
SnapshotLatestCopyBackup         : 
LogReplayQueueIncreasing         : False
LogCopyQueueIncreasing           : False
OutstandingDumpsterRequests      : 
OutgoingConnections              : 
IncomingLogCopyingNetwork        : 
SeedingNetwork                   : 
ActiveCopy                       : False
Identity                         : XXXArchiveDB\ACTIVE.MAIL.SERVER
Name                             : XXXArchiveDB\ACTIVE.MAIL.SERVER
DatabaseName                     : XXXArchiveDB
Status                           : Dismounted
MailboxServer                    : ACTIVE.MAIL.SERVER
ActiveDatabaseCopy               : ACTIVE.MAIL.SERVER
ActivationSuspended              : False
ActionInitiator                  : Administrator
ErrorMessage                     : 
ErrorEventId                     : 
ExtendedErrorInfo                : 
SuspendComment                   : 
SinglePageRestore                : 0
ContentIndexState                : Failed
ContentIndexErrorMessage         : Microsoft Exchange Information Store service
                                    exited for database {XXXXXXX-XXX-XXXX-XXX
                                   X-XXXXXXXXXXX}.
CopyQueueLength                  : 0
ReplayQueueLength                : 0
LatestAvailableLogTime           : 
LastCopyNotificationedLogTime    : 
LastCopiedLogTime                : 
LastInspectedLogTime             : 
LastReplayedLogTime              : 
LastLogGenerated                 : 0
LastLogCopyNotified              : 0
LastLogCopied                    : 0
LastLogInspected                 : 0
LastLogReplayed                  : 0
LogsReplayedSinceInstanceStart   : 0
LogsCopiedSinceInstanceStart     : 0
LatestFullBackupTime             : 
LatestIncrementalBackupTime      : 
LatestDifferentialBackupTime     : 
LatestCopyBackupTime             : 
SnapshotBackup                   : 
SnapshotLatestFullBackup         : 
SnapshotLatestIncrementalBackup  : 
SnapshotLatestDifferentialBackup : 
SnapshotLatestCopyBackup         : 
LogReplayQueueIncreasing         : False
LogCopyQueueIncreasing           : False
OutstandingDumpsterRequests      : {}
OutgoingConnections              : 
IncomingLogCopyingNetwork        : 
SeedingNetwork                   : 
ActiveCopy                       : True
Failed to save admin audit log for this cmdlet invocation. 
Organization:  
Log content:
Subject: DOMAIN.LOCAL/DOMAIN User Accounts/Service Accounts/Systems Admins/USERID : Set-DatabaseAvailabilityGroup
Body: 
Cmdlet Name: Set-DatabaseAvailabilityGroup
Object Modified: VANDAG01
Parameter: WitnessServer = vandagw01.DOMAIN.LOCAL
Parameter: AlternateWitnessServer = edmdagw01.DOMAIN.LOCAL
Parameter: Identity = VANDAG01
Caller: DOMAIN.LOCAL/DOMAIN User Accounts/Service Accounts/Systems Admins/USERID
Succeeded: False
Error: Microsoft.Exchange.Cluster.Replay.DagTaskSetDagNeedsAllNodesUpToChangeQuorumException: A database availability group administrative operation failed. Error: The operation needs to change the quorum type and all database availability group members need
to be running. The following members aren\'t running: FAILED.MAIL.SERVER.
Run Date: 2014-10-08T06:50:32
OriginatingServer: ACTIVE.MAIL.SERVER (14.03.0178.000)
Error:
Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException: Cannot open mailbox /o=DOMAINmail/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=ACTIVE.MAIL.SERVER/cn=Microsoft System Attendant. ---> Microsoft.Mapi.MapiExceptionNetworkError:
MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: ACTIVE.MAIL.SERVER.DOMAIN.LOCAL
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824     
    Lid: 44273  
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=181]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x0][length=56][latency=15]
    Lid: 25964   StoreEc: 0x824     
    Lid: 38951   EMSMDB.EcDoDisconnect called [length=8]
    Lid: 55335   EMSMDB.EcDoDisconnect returned [ec=0x0][length=12][latency=0]
    Lid: 36081  
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065  
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805  
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583  
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778  
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730  
    Lid: 25922   StoreEc: 0x80040115
   at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName,
Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag
storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity
windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.<CreateMailboxSession>b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure
initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString,
PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags
initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsSystemService(ExchangePrincipal mailboxOwner, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.ProvisioningAgent.MailboxLoggerFactory.XsoMailer.Log(AdminLogMessageData data, LogMessageDelegate logMessage)
   at Microsoft.Exchange.ProvisioningAgent.AdminLogProvisioningHandler.OnComplete(Boolean succeeded, Exception e) 
The server could not communicate with the HUB transport running locally. (by locally I mean same server - localhost)
I appreciate any input as to where/what can be the issue. We have had issues in the past while trying to run our yearly DR testing, but none like this.
Thanks!
Ozz.

Answer 1

Hi Ozz,
From your description, I recommend you follow the steps below for troubleshooting:
1. Confirm that the networks are identically configured between all DAG node members.
2. Use the following cmdlet to start the failed Mailbox server and check the result.
Start-DatabaseAvailabilityGroup -Identity "DAG name" -MailboxServer "failed.mail.server"
For your reference:
Start-DatabaseAvailabilityGroup
http://technet.microsoft.com/en-us/library/dd335076(v=exchg.141).aspx
Hope this can be helpful to you.
Best regards,
Amy Wang
TechNet Community Support
First of all thanks for the answer.
Let me give you a better view of our situation.
This was discovered during a staged disaster recovery exercise. All our nodes are VMware VMs.
The (primary site or site A)  FAILED.MAIL.SERVER was shutdown from the VMware console as it would
happen with a power outage or worse.  The Witness server was also shutdown in the same manner. We then had the ACTIVE.MAIL.SERVER
(that meaning the only server running) and the Alternate Witness, on site B.
Initially when the 'failure' was caused the DB mounted right away on the ACTIVE.MAIL.SERVER
then after a bit of time we got all the errors regarding Hub Transport such as:
"The Microsoft Exchange Mail Submission service is currently unable to contact any Hub Transport servers
in the local Active Directory site. The servers may be too busy to accept new connections at this time."
Then after a reboot all the errors persisted and the ACTIVE.MAIL.SERVER
albeit having a running Transport Service still could not communicate via MAPI with the DB. It was still trying to use the FAILED.MAIL.SERVER's
Hub Transport. 
That is my understanding of the errors and issues.
As for item #2 can you please explain in more detail what that would do if the FAILED.MAIL.SERVER
is down?
Does that change anything on the ACTIVE.MAIL.SERVER? 
Thanks for your help!!!
P.S. - One more detail, this is in production so I cannot test during business hours, I tried exactly the
same operation on our test environment and everything came up as expected...  o_0