AlwaysOn AG issue during backup

Question

Hi everyone,
I've a strange issue on our AlwaysOn Cluster  Sql Server 2012 Enterprise multi-instance, I'm evaluating to open a support case to MS but I would share the issue with the community before going on.
Sql server are dedicated to host Sharepoint 2010 farm content databases.
Scenario
n.2 VM (hypervisor ESX 5.1) Windows Server 2008 R2; each vm has 2 virtual disk, and 4 raw-mapped disk
each VM has MSFC role installed; quorum model is "file share majority"; no shared disk are mapped to clustered services.
each vm host a Sql Server 2012 enterprise installation; each VM host n.3 SQL named instance (Development, Testing, Production)
there are 3 Availability Group configured, one for each named-instance.
Raw-Mapped disk are configured in this way: raw1(500GB) for sql data, raw2(500GB) for t-log; raw3 and raw4 are dynamic disk configured in spanning mode (500+500GB) as sql backup repository.
Issue
note: This issue came out recently (while entire enviroment is on production since 1 year)
Our full backup job include a backup checksum and VERIFY.
Since a couple of weeks, this verify on a particular contentdb (CrawlStoreDB, around 80GB) cause the availability group to go in an unhealty state (RESOLVING).
From CLUSTER DIAGNOSTIC EXTENDED EVENT LOG I see this entries:
info_message    2013-05-27 15:44:42.4845706    [hadrag] SQLFetch() returns -1 with following information 
info_message    2013-05-27 15:44:42.4845706    [hadrag] ODBC Error: [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Could not serialize the data for node 'filePath' because it contains a character (0x0000)
which is not allowed in XML. To retrieve this data convert it to binary, varbinary or image data type (6842)
info_message    2013-05-27 15:44:42.4845706    [hadrag] No more diagnostics results
info_message    2013-05-27 15:44:42.4845706    [hadrag] Discard the pending result sets  
info_message    2013-05-27 15:44:42.4845706    [hadrag] ODBC Error: [24000] [Microsoft][SQL Server Native Client 11.0]Invalid cursor state (0)  
From AlwaysOn health event file
availability_replica_state_change    2013-05-27 15:45:02.6615881    PRIMARY_NORMAL    RESOLVING_NORMAL    CABD99D4-8591-4B37-9160-6738DAE9C851
From Failover Cluster Manager Events
Cluster resource 'AGPROD' in clustered service or application 'AGPROD' failed.
The Cluster service failed to bring clustered service or application 'AGPROD' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
this situation remains "looping" for 1 hour, unless forcing a manual failover in SMSS; during this hour, the availability group is unreachble (RESOLVING)
If I deactivate the VERIFY on backup job, the issue doesn't happen.
Any idea to point out the problem?

Answer 1

The cluster is running a query that fails with SQL Error 6842 (Could not serialize the data for node 'filePath' because it contains a character (0x0000) which is not allowed in XML. To retrieve this data convert it to binary, varbinary or image data type
(6842)".
My guess is that the cluster is running the SQL Instance health check procedure
sp_server_diagnostics which returns a VARCHAR(MAX) column filled with XML, and that some diagnostic data has a column 'filePath' that has an embedded null value (0x000), which is causing
an XML conversion error.  As to why it happens on backup verification, it's probably because the diagnostics returns information about the backup process, and one of the files involved has the embedded null value.
You can try to run this and see if it repros the issue.
declare @t table
create_time datetime,
component_type varchar(200),
component_name varchar(200),
state int,
state_desc varchar(20),
data xml
insert into @t
exec sp_server_diagnostics
select * from @t
But this is probably a case for support.
David
David http://blogs.msdn.com/b/dbrowne/