Mailbox Database Content Index Failed and Suspended

Question

Hello,
     I am having an issue with a Mailbox Database.  The content index shows FailedAndSuspended.  there are no Bad Copy Counts  and it shows as Mounted.  I have no DAG so this is a single database and followed the article
here: 
http://technet.microsoft.com/en-us/library/ee633475(v=exchg.150).aspx
After it crawled for a little while it went to Failed, then unknown, then crawling, and has finally ended up at FailedAndSuspended again.  I am also popping errors as follows:
MSExchangeFastSearch Event ID 1004
MSExchangeIS Event ID 1012
MSExchange  ADAccess Event ID 4020
MSExchangeRepl Event ID 4087 (Though this is weird as it is a stand alone database why would it try to replicate it?)
MSExchange Common Event ID 4999
The following Warnings are hitting at the same time:
MSExchangeFastSearch Event ID 1006, 1009, 1010
MSExchange Mid-Tier Storage Event ID 6002, 10010
Anyone have any thoughts on a fix for this?
Michael R. Mastro II

Answer 1

Ok I did a Set-MailboxDatabase "Mailbox Database #" -IndexEnabled $false, then stopped both the MS Exchange Search and MS Exchange Search Host Controllers. Started them again and it showed disabled when I did a get-mailboxdatabasecopystatus. I then went
Set-MailboxDatabase "Mailbox Database #" -IndexEnabled $true. After about 10 minutes got a FailedAndSuspended Status. So here are the logs from event viewer.
Log Name:      ApplicationSource:        MSExchange Mid-Tier StorageDate:          7/20/2014 7:53:49 PMEvent ID:      6002Task Category: ResourceHealthLevel:         WarningKeywords:      ClassicUser:          N/AComputer:      MRM2EX1.MRM2Inc.comDescription:Ping of mdb '7d98ce58-997c-41c1-abde-06d7eaf3dbdd' timed out after '00:00:00' minutes.  Last successful ping was at '7/20/2014 11:53:49 PM' UTC.Event Xml:<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  <System>    <Provider Name="MSExchange Mid-Tier Storage" />    <EventID Qualifiers="32768">6002</EventID>    <Level>3</Level>    <Task>6</Task>    <Keywords>0x80000000000000</Keywords>    <TimeCreated SystemTime="2014-07-20T23:53:49.000000000Z" />    <EventRecordID>703314</EventRecordID>    <Channel>Application</Channel>    <Computer>MRM2EX1.MRM2Inc.com</Computer>    <Security />  </System>  <EventData>    <Data>7d98ce58-997c-41c1-abde-06d7eaf3dbdd</Data>    <Data>00:00:00</Data>    <Data>7/20/2014 11:53:49 PM</Data>  </EventData></Event>
Log Name:      ApplicationSource:        MSExchangeFastSearchDate:          7/20/2014 7:37:17 PMEvent ID:      1006Task Category: GeneralLevel:         WarningKeywords:      ClassicUser:          N/AComputer:      MRM2EX1.MRM2Inc.comDescription:The FastFeeder component received a connection exception from FAST. Error details: Microsoft.Exchange.Search.Fast.FastConnectionException: Connection to the Content Submission Service has failed. ---> Microsoft.Ceres.External.ContentApi.ConnectionException: Recovery failed after 0 retries   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.CheckRecoveryFailed()   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.SubmitDocument(Document document, TimeSpan timeout)   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.SubmitDocument(Document document)   at Microsoft.Exchange.Search.Fast.FastFeeder.SubmitDocumentInternal(Object state)   --- End of inner exception stack trace ---Event Xml:<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  <System>    <Provider Name="MSExchangeFastSearch" />    <EventID Qualifiers="32772">1006</EventID>    <Level>3</Level>    <Task>1</Task>    <Keywords>0x80000000000000</Keywords>    <TimeCreated SystemTime="2014-07-20T23:37:17.000000000Z" />    <EventRecordID>703192</EventRecordID>    <Channel>Application</Channel>    <Computer>MRM2EX1.MRM2Inc.com</Computer>    <Security />  </System>  <EventData>    <Data>Microsoft.Exchange.Search.Fast.FastConnectionException: Connection to the Content Submission Service has failed. ---&gt; Microsoft.Ceres.External.ContentApi.ConnectionException: Recovery failed after 0 retries   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.CheckRecoveryFailed()   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.SubmitDocument(Document document, TimeSpan timeout)   at Microsoft.Ceres.External.ContentApi.DocumentFeeder.DocumentFeeder.SubmitDocument(Document document)   at Microsoft.Exchange.Search.Fast.FastFeeder.SubmitDocumentInternal(Object state)   --- End of inner exception stack trace ---</Data>  </EventData></Event>
Log Name:      ApplicationSource:        MSExchangeFastSearchDate:          7/20/2014 7:43:22 PMEvent ID:      1010Task Category: GeneralLevel:         WarningKeywords:      ClassicUser:          N/AComputer:      MRM2EX1.MRM2Inc.comDescription:An operation attempted against a FAST endpoint exprienced an exception. This operation may be retried. Error details: Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'. ---> System.IO.IOException: The write operation failed, see inner exception. ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)   at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   --- End of inner exception stack trace ---   at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   at System.ServiceModel.Channels.BufferedConnection.WriteNow(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, BufferManager bufferManager)   at System.ServiceModel.Channels.BufferedConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   at System.ServiceModel.Channels.ConnectionStream.Write(Byte[] buffer, Int32 offset, Int32 count)   at System.Net.Security.NegotiateStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   at System.Net.Security.NegotiateStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   --- End of inner exception stack trace ---   at System.Net.Security.NegotiateStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   at System.Net.Security.NegotiateStream.Write(Byte[] buffer, Int32 offset, Int32 count)   at System.ServiceModel.Channels.StreamConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   --- End of inner exception stack trace ---Server stack trace:    at System.ServiceModel.Channels.StreamConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   at System.ServiceModel.Channels.StreamConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, BufferManager bufferManager)   at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnSendCore(Message message, TimeSpan timeout)   at System.ServiceModel.Channels.TransportDuplexSessionChannel.OnSend(Message message, TimeSpan timeout)   at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout)   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)Exception rethrown at [0]:    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)   at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.GetFlows()   at Microsoft.Exchange.Search.Fast.IndexManager.<GetFlows>b__16()   at Microsoft.Exchange.Search.Fast.IndexManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)   --- End of inner exception stack trace ---Event Xml:<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  <System>    <Provider Name="MSExchangeFastSearch" />    <EventID Qualifiers="32772">1010</EventID>    <Level>3</Level>    <Task>1</Task>    <Keywords>0x80000000000000</Keywords>    <TimeCreated SystemTime="2014-07-20T23:43:22.000000000Z" />    <EventRecordID>703224</EventRecordID>    <Channel>Application</Channel>    <Computer>MRM2EX1.MRM2Inc.com</Computer>    <Security />  </System>  <EventData>    <Data>Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An Exception was received during a FAST operation. ---&gt; System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'. ---&gt; System.IO.IOException: The write operation failed, see inner exception. ---&gt; System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:01:00'. ---&gt; System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)   at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   --- End of inner exception stack trace ---   at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   at System.ServiceModel.Channels.BufferedConnection.WriteNow(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, BufferManager bufferManager)   at System.ServiceModel.Channels.BufferedConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   at System.ServiceModel.Channels.ConnectionStream.Write(Byte[] buffer, Int32 offset, Int32 count)   at System.Net.Security.NegotiateStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   at System.Net.Security.NegotiateStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   --- End of inner exception stack trace ---   at System.Net.Security.NegotiateStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   at System.Net.Security.NegotiateStream.Write(Byte[] buffer, Int32 offset, Int32 count)   at System.ServiceModel.Channels.StreamConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   --- End of inner exception stack trace ---Server stack trace:    at System.ServiceModel.Channels.StreamConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)   at System.ServiceModel.Channels.StreamConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, BufferManager bufferManager)   at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnSendCore(Message message, TimeSpan timeout)   at System.ServiceModel.Channels.TransportDuplexSessionChannel.OnSend(Message message, TimeSpan timeout)   at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout)   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)Exception rethrown at [0]:    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)   at Microsoft.Ceres.ContentEngine.Admin.FlowService.IFlowServiceManagementAgent.GetFlows()   at Microsoft.Exchange.Search.Fast.IndexManager.&lt;GetFlows&gt;b__16()   at Microsoft.Exchange.Search.Fast.IndexManagementClient.PerformFastOperation[T](Func`1 function, String eventLogKey)   --- End of inner exception stack trace ---</Data>  </EventData></Event>
Log Name:      ApplicationSource:        MSExchange Mid-Tier StorageDate:          7/20/2014 7:53:49 PMEvent ID:      6002Task Category: ResourceHealthLevel:         WarningKeywords:      ClassicUser:          N/AComputer:      MRM2EX1.MRM2Inc.comDescription:Ping of mdb '7d98ce58-997c-41c1-abde-06d7eaf3dbdd' timed out after '00:00:00' minutes.  Last successful ping was at '7/20/2014 11:53:49 PM' UTC.Event Xml:<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  <System>    <Provider Name="MSExchange Mid-Tier Storage" />    <EventID Qualifiers="32768">6002</EventID>    <Level>3</Level>    <Task>6</Task>    <Keywords>0x80000000000000</Keywords>    <TimeCreated SystemTime="2014-07-20T23:53:49.000000000Z" />    <EventRecordID>703314</EventRecordID>    <Channel>Application</Channel>    <Computer>MRM2EX1.MRM2Inc.com</Computer>    <Security />  </System>  <EventData>    <Data>7d98ce58-997c-41c1-abde-06d7eaf3dbdd</Data>    <Data>00:00:00</Data>    <Data>7/20/2014 11:53:49 PM</Data>  </EventData></Event>
Log Name:      ApplicationSource:        MSExchange Mid-Tier StorageDate:          7/20/2014 7:58:39 PMEvent ID:      10010Task Category: ResourceHealthLevel:         WarningKeywords:      ClassicUser:          N/AComputer:      MRM2EX1.MRM2Inc.comDescription:Process: MSExchangeDelivery (2824), Db:a433e8a2-ca26-438e-a9a3-d71305c6c266,C:1,BT:00:00:00,Db:7d98ce58-997c-41c1-abde-06d7eaf3dbdd,C:1,BT:00:00:00,Event Xml:<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">  <System>    <Provider Name="MSExchange Mid-Tier Storage" />    <EventID Qualifiers="32768">10010</EventID>    <Level>3</Level>    <Task>6</Task>    <Keywords>0x80000000000000</Keywords>    <TimeCreated SystemTime="2014-07-20T23:58:39.000000000Z" />    <EventRecordID>703327</EventRecordID>    <Channel>Application</Channel>    <Computer>MRM2EX1.MRM2Inc.com</Computer>    <Security />  </System>  <EventData>    <Data>Process: MSExchangeDelivery (2824), Db:a433e8a2-ca26-438e-a9a3-d71305c6c266,C:1,BT:00:00:00,Db:7d98ce58-997c-41c1-abde-06d7eaf3dbdd,C:1,BT:00:00:00,</Data>  </EventData></Event>
Michael R. Mastro II