Exchange backup issues

Question

Exchange 2010 SP3 rollup4
4 Server DAG all servers running Windows 2008 R2 (VMs on VMware esxi5.1) 4 databases (2 mailbox, 2 online archives)
Last week 2 databases backup stopped truncating the logs, Mailbox 1 and Online Archive 1. (Mailbox 2 and Online Archive 2 seem to be OK and are backing up OK)
On investigation there are a number of errors in the application log:
Log Name:     
Application
Source:       
MSExchangeRepl
Date:         
05/06/2014 21:50:48
Event ID:     
2156
Task Category: Service
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
The log copier for database 'Online Archive 1' could not send a log file to server 'EXCHANGESERVER2' because An I/O error occurred while attempting to access file 'L:\Exchange Logs\Online
Archive 1\E05.log' on source server EXCHANGESERVER1. Error: File not found (-1811)
Event Xml:
<Data>An I/O error occurred while attempting to access file 'L:\Exchange Logs\Online Archive 1\E05.log' on source server EXCHANGESERVER1. Error: File not found (-1811)</Data>
Log Name:     
Application
Source:       
MSExchange Search Indexer
Date:         
05/06/2014 21:50:48
Event ID:     
118
Task Category: General
Level:        
Warning
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
Exchange Search Indexer has temporarily disabled indexing of the mailbox database Online Archive 1 (GUID = ee385d93-918f-470e-b57b-a87d7e23ed20) due to an error (Microsoft.Mapi.MapiExceptionMdbOffline:
MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
Lid: 33865 
Lid: 1494    ---- Remote Context Beg ----
Lid: 44215 
Lid: 60049   StoreEc: 0x8004010F
Lid: 49469 
Lid: 65341   StoreEc: 0x8004010F
Lid: 56125 
Lid: 47933   StoreEc: 0x8004010F
Lid: 32829 
Lid: 49213   StoreEc: 0x8004010F
Lid: 48573 
Lid: 64957   StoreEc: 0x8004010F
Lid: 20057   StoreEc: 0x476    
Lid: 1750    ---- Remote Context End ----
Lid: 28777   StoreEc: 0x476    
Lid: 20098 
Lid: 20585   StoreEc: 0x476    
at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
at Microsoft.Exchange.Search.RetriableOperations.ReadEvents(ThreadLocalCrawlData unused1, MapiEventManager eventManager, Int64 watermark, Int32 eventCount, Int64& endCounter)
at Microsoft.Exchange.Search.RetriableOperations.DoRetriableMapiOperation[SourceType,ReturnType,Parameter1Type,Parameter2Type,Parameter3Type](ThreadLocalCrawlData crawlData, SourceType source, Parameter1Type parameter1, Parameter2Type parameter2, Parameter3Type&
parameter3, MapiOperationDelegate`5 operationDelegate)
at Microsoft.Exchange.Search.NotificationWatcher.GetMapiEvents(Int32 maxEvents, NotificationQueue notificationQueue, Int64& endCount)
at Microsoft.Exchange.Search.NotificationWatcher.NotificationWatcherThread()), and Operations Manager would never alert it.
Log Name:     
Application
Source:       
MSExchangeRepl
Date:         
05/06/2014 21:50:52
Event ID:     
4082
Task Category: Service
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
The replication network manager encountered an error while monitoring events. Error: Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error
An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(Exchangeserver1) failed with 0x6d9. Error: There are no more endpoints available from the endpoint mapper"' failed.. ---> System.ComponentModel.Win32Exception:
There are no more endpoints available from the endpoint mapper
--- End of inner exception stack trace ---
at Microsoft.Exchange.Cluster.Replay.NetworkManager.DriveMapRefresh()
at Microsoft.Exchange.Cluster.Replay.NetworkManager.TryDriveMapRefresh()
Event Xml:
Log Name:     
Application
Source:       
ESE
Date:         
05/06/2014 21:51:15
Event ID:     
489
Task Category: General
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
msexchangerepl (2448) An attempt to open the file "O:\Mailbox Databases\Online Archive 1\Online Archive 1.edb" for read only access failed with system error 32 (0x00000020):
"The process cannot access the file because it is being used by another process. ". 
The open file operation will fail with error -1032 (0xfffffbf8).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="ESE" />
<EventID Qualifiers="0">489</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-06-05T20:51:15.000000000Z" />
<EventRecordID>7392312</EventRecordID>
<Channel>Application</Channel>
<Computer>Exchangeserver1</Computer>
<Security />
</System>
<EventData>
<Data>msexchangerepl</Data>
<Data>2448</Data>
<Data>
</Data>
<Data>O:\Mailbox Databases\Online Archive 1\Online Archive 1.edb</Data>
<Data>-1032 (0xfffffbf8)</Data>
<Data>32 (0x00000020)</Data>
<Data>The process cannot access the file because it is being used by another process. </Data>
</EventData>
</Event>
Log Name:     
Application
Source:       
MSExchangeRepl
Date:         
05/06/2014 21:51:15
Event ID:     
4107
Task Category: Service
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
The Microsoft Exchange Replication service detected that database 'Online Archive 1' may be mounted on two servers. The copy on server 'EXCHANGESERVER1' will now be dismounted. Please
review the event logs for more information.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeRepl" />
<EventID Qualifiers="49156">4107</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-06-05T20:51:15.000000000Z" />
<EventRecordID>7392314</EventRecordID>
<Channel>Application</Channel>
<Computer>Exchangeserver1</Computer>
<Security />
</System>
<EventData>
<Data>Online Archive 1</Data>
<Data>EXCHANGESERVER1</Data>
</EventData>
</Event>
Log Name:     
Application
Source:       
MSExchangeRepl
Date:  
       05/06/2014 21:51:15
Event ID:     
2070
Task Category: Service
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
The Microsoft Exchange Replication service encountered an error while inspecting the logs and database for Online Archive 1\EXCHANGESERVER1 on startup. Error: File check failed : File
'O:\Mailbox Databases\Online Archive 1\Online Archive 1.edb' could not be opened. 
The file is locked or in use.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchangeRepl" />
<EventID Qualifiers="49156">2070</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-06-05T20:51:15.000000000Z" />
<EventRecordID>7392315</EventRecordID>
<Channel>Application</Channel>
<Computer>Exchangeserver1</Computer>
<Security />
</System>
<EventData>
<Data>Online Archive 1\EXCHANGESERVER1</Data>
<Data>File check failed : File 'O:\Mailbox Databases\Online Archive 1\Online Archive 1.edb' could not be opened. 
The file is locked or in use.</Data>
</EventData>
</Event>
Log Name:     
Application
Source:       
ESE
Date:         
05/06/2014 21:51:45
Event ID:     
489
Task Category: General
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
msexchangerepl (2448) An attempt to open the file "O:\Mailbox Databases\Online Archive 1\Online Archive 1.edb" for read only access failed with system error 32 (0x00000020):
"The process cannot access the file because it is being used by another process. ". 
The open file operation will fail with error -1032 (0xfffffbf8).
For more information, click http://www.microsoft.com/contentredirect.asp.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="ESE" />
<EventID Qualifiers="0">489</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-06-05T20:51:45.000000000Z" />
<EventRecordID>7392316</EventRecordID>
<Channel>Application</Channel>
<Computer>Exchangeserver1</Computer>
<Security />
</System>
<EventData>
<Data>msexchangerepl</Data>
<Data>2448</Data>
<Data>
</Data>
<Data>O:\Mailbox Databases\Online Archive 1\Online Archive 1.edb</Data>
<Data>-1032 (0xfffffbf8)</Data>
<Data>32 (0x00000020)</Data>
<Data>The process cannot access the file because it is being used by another process. </Data>
</EventData>
</Event>
Log Name:     
Application
Source:       
MSExchange Search Indexer
Date:         
05/06/2014 21:51:50
Event ID:     
108
Task Category: General
Level:        
Information
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
Exchange Search Indexer has enabled indexing for the Mailbox Database Online Archive 1 (GUID = ee385d93-918f-470e-b57b-a87d7e23ed20).
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSExchange Search Indexer" />
<EventID Qualifiers="16388">108</EventID>
<Level>4</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-06-05T20:51:50.000000000Z" />
<EventRecordID>7392320</EventRecordID>
<Channel>Application</Channel>
<Computer>Exchangeserver1</Computer>
<Security />
</System>
<EventData>
<Data>Online Archive 1</Data>
<Data>ee385d93-918f-470e-b57b-a87d7e23ed20</Data>
</EventData>
</Event>
Log Name:     
Application
Source:       
ExchangeStoreDB
Date:         
05/06/2014 21:52:00
Event ID:     
124
Task Category: Database recovery
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
At '05/06/2014 21:51:15' the Exchange Information Store Database 'Online Archive 1' copy on this server encountered an error. Consult the event log on the server for other "ExchangeStoreDb"
or "msexchangeRepl" events that may identify the specific failure. The Replication Service will automatically attempt to retry.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="ExchangeStoreDB" />
<EventID Qualifiers="49156">124</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2014-06-05T20:52:00.000000000Z" />
<EventRecordID>7392327</EventRecordID>
<Channel>Application</Channel>
<Computer>Exchangeserver1</Computer>
<Security />
</System>
<EventData>
<Data>Online Archive 1</Data>
<Data>ee385d93-918f-470e-b57b-a87d7e23ed20</Data>
<Data>Replay</Data>
<Data>AlertOnly</Data>
<Data>Online Archive 1</Data>
<Data>
</Data>
<Data>
</Data>
<Data>05/06/2014 21:51:15</Data>
</EventData>
</Event>
These seem to repeat until a log entry:
Log Name:     
Application
Source:       
MSExchangeIS
Date:         
05/06/2014 22:14:38
Event ID:     
9617
Task Category: Exchange VSS Writer
Level:        
Error
Keywords:     
Classic
User:         
N/A
Computer:     
Exchangeserver1
Description:
Exchange VSS Writer (instance ae29b16e-11b9-4f2d-b2eb-a544d0927d87:2) failed with error code -2147221233 when processing the backup completion event.
I thought it was a backup issue but Veeam have investigated and have said it is a VSS timeout issue (I'm not convinced)
Recently I have had to have a new logs drive and extend the size of the mail box drive but since those changes were made backup has run OK until this issues.
Any advice appreciated.

Answer 1

Hi,
Based on my research, when Veeam takes the snapshot to then do a VSS backup it will cause the databases in the DAG to fail over and cause lots of alerts and we can modify the REM for the backup issue:
http://desktopfeedbag.com/2012/02/23/fixed-exchange-2010-sp1sp2-dag-fail-over-with-veeam/
Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make
sure that you completely understand the risk before retrieving any suggestions from the above link.
Thanks,
Angela Shi
TechNet Community Support