Can't back up VMs in 2012 Cluster 0x80042301

Question

I have a 2012 Cluster with a CSV. I have the Dell Equallogic HIT v 4.5 installed on both nodes, as well as the SCDPM 2012 SP1 Agent. When I add a replica to DPM to back up one (or any) of the virtual machines on the cluster the vss writer for Hyper-V is
put in a bad state and DPM gives me the following error:
The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:A function call was made when the object was in an incorrect state
for that function
 (0x80042301))
The following VSS errors are logged in the Eventlog of the Cluster Node:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005, Access is denied.
. This is often caused by incorrect security settings in either the writer or requestor process.
Operation:
   Gathering Writer Data
Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {754c4bfd-91e7-47b5-94a2-565196237f2a}
Things I have done:
Remove DPM SP1 Agent and reinstall on both nodes in cluster
Remove SCVMM SP1 Agent and reinsall on both nodes in cluster
Remove HIT and reinstall on both nodes in cluster
Restart DPM server
Restart both nodes in cluster
Before the DPM agent puts the Hyper-V VSS Writer into a bad state I can create SmartCopies from the HIT Auto-Snapshot Monitor software, but as soon as the VSS Writer is in a bad state ASM no longer works:
Started at 3:32:26 PM
Gathering Information...
Phase 1: Checking pre-requisites... (3:32:26 PM)
Phase 2: Initializing Smart Copy Operation (3:32:26 PM)
Retrieving writer information
Phase 3: Adding Components and Volumes (3:32:28 PM)
Adding components to the Smart Copy Set
Adding volumes to the Smart Copy Set
Phase 4: Creating Smart Copy (3:32:28 PM)
Creating Smart Copy Set
An error occurred:
An error occurred during phase: Creating Smart Copy
    Exception from HRESULT: 0x800423F9.
Creating Smart Copy Set
An error occurred:
An error occurred during phase: Creating Smart Copy
    Exception from HRESULT: 0x800423F9.
An error occurred:
Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_THAW'. Check the application component to verify it is in a valid state for the operation.
An error occurred:
One or more errors occurred during the operation.  Check the detailed progress updates for details.
An error occurred:
Smart Copy creation failed.
Source:  Creating Smart Copy Set
An error occurred:
An error occurred during phase: Creating Smart Copy
    Exception from HRESULT: 0x800423F9.
An error occurred:
Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_THAW'. Check the application component to verify it is in a valid state for the operation.
An error occurred:
One or more errors occurred during the operation.  Check the detailed progress updates for details.
Error:  Exception from HRESULT: 0x800423F9
I also have two more 2-node 2012 Clusters that have the exact same software installed, but
do not exhibit this issue.
Thanks!

Answer 1

I have removed the hardware provider and still no luck. You are correct that the Hyper-V Vss writer seems to be the culprit. Here is my output from vssadmin list wrtiers:
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.
Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error
Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error
Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error
Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {72b4583b-b5d4-4195-90b9-92cc34bb2644}
   State: [7] Failed
   Last error: Timed out
Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {4ad02755-85cd-471e-8360-45545a58d802}
   State: [1] Stable
   Last error: No error
Writer name: 'Cluster Shared Volume VSS Writer'
   Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
   Writer Instance Id: {d9c7650f-24e8-4248-aed3-728abbce6929}
   State: [1] Stable
   Last error: No error
Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {d1b4d6f2-40a8-48cd-bc30-6b15a60e4004}
   State: [1] Stable
   Last error: No error
Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {87d73261-c662-4db8-a5c6-6d9cbbc48e9a}
   State: [1] Stable
   Last error: No error
Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {0113ad9f-591f-4aec-81c6-81d0d4082cea}
   State: [1] Stable
   Last error: No error
Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {fabf8bd5-3c2b-4bc3-836f-760501180f3d}
   State: [1] Stable
   Last error: No error
Writer name: 'Cluster Database'
   Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Writer Instance Id: {136a3871-dd86-4013-9d25-cde67045d6e8}
   State: [1] Stable
   Last error: No error
Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {779a54c2-8f89-4afb-aaea-7977b035650e}
   State: [1] Stable
   Last error: No error
Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {b4e65ba3-e75c-4b49-9fc5-1caf8945247d}
   State: [1] Stable
   Last error: No error
 I will log a support case. If I am provided with a resolution I'll post it back here.