DPM 2012 R2 backup Causes Redirected CSV IO on SOFS Cluster.

Hi, I have a Scale out Storage Spaces Server with 2 nodes, and a 10 node 2012 R2, Hyper-V cluster using this via SMB3.0
I also have installed a DPM2012 R2 backup server.
the DPM agent is installed on all nodes of all servers and I have followed the pre-requisite from Microsoft for setting up DPM backup of SMB Hyper-V machines.
The DPM backups all work fine. but occasionaly I get these errors on the SOFS cluster.
Cluster Shared Volume 'Volume3' ('Cluster Disk 4') has entered a paused state because of '(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.
I really thought this issue had been resolved in this revision, this doesn't seem to cause any issues with my VM's that I can notice. and all DPM backups are working fine, but it still causes me concern.
has anyone else seen this or have any suggestions what I can try to resolve.
Regards
Mark Green

We also encounter this issue. We use Windows Server 2012 R2 and SCVMM 2012 R2 (with RU1). Be carefull with this issue, because it can cause serious issues. Btw, note that Windows Server 2012 R2 used Direct I/O instead of Redirected I/O.
If you can't find a full fix as we are in right now, there are two things that might offer a work-around for you:
Disabled ODX (if your storage system does not support it):
Deploy Windows Offloaded Data Transfers
http://technet.microsoft.com/en-us/library/jj200627.aspx
Serialize virtual machine backups per node
Migrate to a hardware VSS provider
http://technet.microsoft.com/en-us/library/hh758027.aspx
The second option works best, because this issue mostly occurs when you run a backup of many VMs at once. It it not a full fix and makes you backup windows much longer, but can avoid you other problems. Also keep a close eye on this link:
Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters
http://support.microsoft.com/kb/2920151
Boudewijn Plomp, BPMi Infrastructure & Security

Similar Messages

  • DPM 2012 R2 Backup job FAILED for some Hyper-v VMs and Some Hyper-v VMs are not appearing in the DPM

    DPM 2012 R2  Backup job FAILED for some Hyper-v VMs
    DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer experienced a transient error.  If the backup process is retried,
    the error may not reoccur.
     (0x800423F3))
    All the vss Writers are in stable state
    Also Some Hyper-v VMs are not appearing in the DPM 2012 R2 Console When I try to create the Protection Group please note that they are not part of cluster.
    Host is 2012 R2 and The VM is also 2012 R2.

    Hi,
    What update rollup are you running on the DPM 2012 R2 server ?  DPM 2012 R2 UR5 introduced a new refresh feature that will re-enumerate data sources on an individual protected server.
    Check for VSS errors inside the guests that are having problems being backed up.
    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT]
    This posting is provided "AS IS" with no warranties, and confers no rights.

  • DPM Express Full Backups Causing SQL timeouts

    Hi,
    We run a SQL 2008 R2 failover cluster on Server 2008 R2 SP1 connected to an Equallogic SAN. The SQL failover cluster is fully servicepacked and installed the latest cumulative updates. 
    We have all our databases in full recovery mode and protected by DPM 2012 SP1 with latest cumulative updates.
    The protection groups for SQL are configured to synchronise every 60mins and 1 express full backup every night. Every time the SQL express backups run the SQL server experiences timouts and end users report applications timing out. In the SQL server Application
    logs I see Event ID 833 numerous times, these only occur as the express full backups take place. Every time the express full backups run we see the same timeouts and event logged that I/O has taken longer than 15 seconds to complete:
    Log Name:      Application
    Source:        MSSQLSERVER
    Date:          01/04/2014 04:10:10
    Event ID:      833
    Task Category: Server
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      WDCSQL02.local
    Description:
    SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [D:\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\MSDBLog.ldf] in database [msdb] (4).  The OS file handle is 0x00000000000008F8.  The offset
    of the latest long I/O is: 0x000000042eae00
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSSQLSERVER" />
        <EventID Qualifiers="16384">833</EventID>
        <Level>4</Level>
        <Task>2</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2014-04-01T03:10:10.000000000Z" />
        <EventRecordID>14165629</EventRecordID>
        <Channel>Application</Channel>
        <Computer>WDCSQL02.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>1</Data>
        <Data>15</Data>
        <Data>D:\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\MSDBLog.ldf</Data>
        <Data>msdb</Data>
        <Data>4</Data>
        <Data>00000000000008F8</Data>
        <Data>0x000000042eae00</Data>
        <Binary>410300000A0000000600000055004B00530051004C00000000000000</Binary>
      </EventData>
    </Event>
    I was under the impression that DPM SQL backups would remain online during the backup.
    Is this normal? 
    Any way to fix this issue?
    Thanks,
    Microsoft Partner

    Hi
    A copy backup will not truncate the logs but a full backup will and it will also stamp the database once it does a full backup.

  • DPM 2012 R2 backup of Workgroup computer error

    Hi, we have a Workgroup server that we had successfully backed up to a DPM 2012 Server.
    Since then we have built 2 brand new DPM 2012 R2 servers.
    I have uninstalled the DPM 2012 agent from the Workgroup computer and have installed the new DPM 2012 R2 agent successfully.
    However, when i try to Attach the agent I am getting 'Unable to contact the protection agent on server' errors.
    I have run the Set-DPMServer -IsNonDomainServer command correctly so the Firewall rules are there ( I have also turned the FW off to test but same error results). I am using the same account details created via Set-DPMserver to attach the agent on the DPM
    server.
    The Event Logs on the DPM 2012 R2 server show DCOM errors (Event 10028, DCOM was unable to communicate with the computer using any of the configured protocols; requested by PID      5e4 (C:\Program Files\Microsoft System Center 2012
    R2\DPM\DPM\bin\msdpm.exe).)
    I have even uninstalled the 2012 R2 agent and re-installed the 2012 agent and that attaches fine, no problem.
    Can anyone help/advise?
    David

    Hi,
    That is correct, ISCSI / Vfc attached or Pass-through disks are not included in host level backups. You have install a DPM agent inside the guest and protecting the volumes seperatly.
    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT]
    This posting is provided "AS IS" with no warranties, and confers no rights.

  • Can DPM 2012 R2 backup SQL Server 2012 AlwaysOn enabled databases

    Is it possible to use DPM 2012 R2 for backing up
    SQL Server 2012 AlwaysOn enabled databases ?
    Because if Alwayson configured for any Database , particular database is not available to select for backup.

    Hi,
    Yes, starting with DPM 2012 SP1 SQL AlwaysOn enabled databases can be protected as long as they are not part of sharepoint farm. DPM shows the Availability Groups under Cluster Group.
    You must explicitly add the system account NTAuthority\System or NT AUTHORITY\DPMRA to the Sysadmin group on SQL Server.
    Protecting SQL Server Data
    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT]
    This posting is provided "AS IS" with no warranties, and confers no rights.

  • DPM 2012 table backup

    Dear all,
    I have intertied a DPM 2012 server which is perform short (to disk) and long (tape) backups. However the tape backups always seem to fail with various errors / warning. I have noticed that the server has fire streamer and a physical tape drive setup,
    I would like to remove fire streamer and also get some granular logon on the errors so that I can perform some analysis. 

    Hi,
    What update rollup are you running on the DPM 2012 R2 server ?  DPM 2012 R2 UR5 introduced a new refresh feature that will re-enumerate data sources on an individual protected server.
    Check for VSS errors inside the guests that are having problems being backed up.
    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT]
    This posting is provided "AS IS" with no warranties, and confers no rights.

  • DPM 2012 SP1 Backup to Cloud

    I realize I'm going to be a little ahead of where the software is on this one but on the subjects of backups, DPM, Instead of pointing DPM to the azure public cloud can I use our Private Cloud (Windows 2012, System Center 2012 SP1 + Azure Services for
    Windows Server)?
    Thanks
    Si
    Simon Stearn - Militant South - Agile Architecture

    I assume that when you say Azure Services for Windows Server, you are referring to Windows Azure Pack?
    If you are doing this on premise then I think you would need to stick to the conventional DPM-to-DPM model?
    I can't personally imagine that the DPM team will add the ability to use the Cloud Backup feature with an on premise Azure cloud as the capability for on premise off-site backup is already there in DPM-to-DPM.
    Thanks,
    Richard
    Richard Green | MCSA 2012, MCSE 2003, MCTS Desktop Virtualization, VCP5-DV http://richardjgreen.net

  • DPM 2012 R2 Consistency Check (Replica Inconsistent) Error for Failover Cluster

    I have a SCDPM 2012 R2 instance running against a Windows Server 2012 R2 Failover Cluster on the network.
    However, any new/migrated Roles/Virtual machines that I add under an existing protection group always gives me an error saying that the replica is inconsistent.
    I've verified the following : 
    1. All windows updates are applied on the cluster hosts & the DPM server.
    2. The VSS and DPMRA services are running on the hosts & DPM server.
    3. The Windows Server Backup feature has been installed on all the machines as well.
    4. The short term storage disks attached to the DPM server are LUNs and there is enough free space on those.
    What else am I missing that could be causing the Replica to be inconsistent - considering that these are new VMs on the hosts themselves - is there a way to force a fresh replica to be created?

    Do not remove any of the providers they come with the system. There is just something wrong with that VM, your trying to backup, as it could take the initial replica creation.
    Check all of the following
    1) Make sure you have enough C drive space on the guest - need space to create a snapshot
    2) Check the vssadmin list wirters for any errors
    If none of the above, rerun the consistency check when no other backup is running. Check the Application log on the host for any vss failures
    Ensure you have all the DPM hotfixes - I am compiling a list since i've been thru many DPM issues.
    Great product when you get it going but like any product there are bumps in the road :)
    ANNCEX

  • Hyper-V cluster Backup causes virtual machine reboots for common Cluster Shared Volumes members.

    I am having a problem where my VMs are rebooting while other VMs that share the same CSV are being backed up. I have provided all the information that I have gather to this point below. If I have missed anything, please let me know.
    My HyperV Cluster configuration:
    5 Node Cluster running 2008R2 Core DataCenter w/SP1. All updates as released by WSUS that will install on a Core installation
    Each Node has 8 NICs configured as follows:
     NIC1 - Management/Campus access (26.x VLAN)
     NIC2 - iSCSI dedicated (22.x VLAN)
     NIC3 - Live Migration (28.x VLAN)
     NIC4 - Heartbeat (20.x VLAN)
     NIC5 - VSwitch (26.x VLAN)
     NIC6 - VSwitch (18.x VLAN)
     NIC7 - VSwitch (27.x VLAN)
     NIC8 - VSwitch (22.x VLAN)
    Following hotfixes additional installed by MS guidance (either while build or when troubleshooting stability issue in Jan 2013)
     KB2531907 - Was installed during original building of cluster
     KB2705759 - Installed during troubleshooting in early Jan2013
     KB2684681 - Installed during troubleshooting in early Jan2013
     KB2685891 - Installed during troubleshooting in early Jan2013
     KB2639032 - Installed during troubleshooting in early Jan2013
    Original cluster build was two hosts with quorum drive. Initial two hosts were HST1 and HST5
    Next host added was HST3, then HST6 and finally HST2.
    NOTE: HST4 hardware was used in different project and HST6 will eventually become HST4
    Validation of cluster comes with warning for following things:
     Updates inconsistent across hosts
      I have tried to manually install "missing" updates and they were not applicable
      Most likely cause is different build times for each machine in cluster
       HST1 and HST5 are both the same level because they were built at same time
       HST3 was not rebuilt from scratch due to time constraints and it actually goes back to Pre-SP1 and has a larger list of updates that others are lacking and hence the inconsistency
       HST6 was built from scratch but has more updates missing than 1 or 5 (10 missing instead of 7)
       HST2 was most recently built and it has the most missing updates (15)
     Storage - List Potential Cluster Disks
      It says there are Persistent Reservations on all 14 of my CSV volumes and thinks they are from another cluster.
      They are removed from the validation set for this reason. These iSCSI volumes/disks were all created new for
      this cluster and have never been a part of any other cluster.
     When I run the Cluster Validation wizard, I get a slew of Event ID 5120 from FailoverClustering. Wording of error:
      Cluster Shared Volume 'Volume12' ('Cluster Disk 13') is no longer available on this node because of
      'STATUS_MEDIA_WRITE_PROTECTED(c00000a2)'. All I/O will temporarily be queued until a path to the
      volume is reestablished.
     Under Storage and Cluster Shared VOlumes in Failover Cluster Manager, all disks show online and there is no negative effect of the errors.
    Cluster Shared Volumes
     We have 14 CSVs that are all iSCSI attached to all 5 hosts. They are housed on an HP P4500G2 (LeftHand) SAN.
     I have limited the number of VMs to no more than 7 per CSV as per best practices documentation from HP/Lefthand
     VMs in each CSV are spread out amonst all 5 hosts (as you would expect)
    Backup software we use is BackupChain from BackupChain.com.
    Problem we are having:
     When backup kicks off for a VM, all VMs on same CSV reboot without warning. This normally happens within seconds of the backup starting
    What have to done to troubleshoot this:
     We have tried rebalancing our backups
      Originally, I had backup jobs scheduled to kick off on Friday or Saturday evening after 9pm
      2 or 3 hosts would be backing up VMs (Serially; one VM per host at a time) each night.
      I changed my backup scheduled so that of my 90 VMs, only one per CSV is backing up at the same time
       I mapped out my Hosts and CSVs and scheduled my backups to run on week nights where each night, there
       is only one VM backed up per CSV. All VMs can be backed up over 5 nights (there are some VMs that don't
       get backed up). I also staggered the start times for each Host so that only one Host would be starting
       in the same timeframe. There was some overlap for Hosts that had backups that ran longer than 1 hour.
      Testing this new schedule did not fix my problem. It only made it more clear. As each backup timeframe
      started, whichever CSV the first VM to start was on would have all of their VMs reboot and come back up.
     I then thought maybe I was overloading the network still so I decided to disable all of the scheduled backup
     and run it manually. Kicking off a backup on a single VM, in most cases, will cause the reboot of common
     CSV members.
     Ok, maybe there is something wrong with my backup software.
      Downloaded a Demo of Veeam and installed it onto my cluster.
      Did a test backup of one VM and I had not problems.
      Did a test backup of a second VM and I had the same problem. All VMs on same CSV rebooted
     Ok, it is not my backup software. Apparently it is VSS. I have looked through various websites. The best troubleshooting
     site I have found for VSS in one place it on BackupChain.com (http://backupchain.com/hyper-v-backup/Troubleshooting.html)
     I have tested almost every process on there list and I will lay out results below:
      1. I have rebooted HST6 and problems still persist
      2. When I run VSSADMIN delete shadows /all, I have no shadows to delete on any of my 5 nodes
       When I run VSSADMIN list writers, I have no error messages on any writers on any node...
      3. When I check the listed registry key, I only have the build in MS VSS writer listed (I am using software VSS)
      4. When I run VSSADMIN Resize ShadowStorge command, there is no shadow storage on any node
      5. I have completed the registration and service cycling on HST6 as laid out here and most of the stuff "errors"
       Only a few of the DLL's actually register.
      6. HyperV Integration Services were reconciled when I worked with MS in early January and I have no indication of
       further issue here.
      7. I did not complete the step to delete the Subscriptions because, again, I have no error messages when I list writers
      8. I removed the Veeam software that I had installed to test (it hadn't added any VSS Writer anyway though)
      9. I can't realistically uninstall my HyperV and test VSS
      10. Already have latest SPs and Updates
      11. This is part of step 5 so I already did this. This seems to be a rehash of various other stratgies
     I have used the VSS Troubleshooter that is part of BackupChain (Ctrl-T) and I get the following error:
      ERROR: Selected writer 'Microsoft Hyper-V VSS Writer' is in failed state!
      - Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)
      - Writer Failure code: 0x800423f0 (<Unknown error code>)
      - Writer ID: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
      - Instance ID: {d55b6934-1c8d-46ab-a43f-4f997f18dc71}
      VSS snapshot creation failed with result: 8000FFFF
    VSS errors in event viewer. Below are representative errors I have received from various Nodes of my cluster:
    I have various of the below spread out over all hosts except for HST6
    Source: VolSnap, Event ID 10, The shadow copy of volume took too long to install
    Source: VolSnap, Event ID 16, The shadow copies of volume x were aborted because volume y, which contains shadow copy storage for this shadow copy, wa force dismounted.
    Source: VolSnap, Event ID 27, The shadow copies of volume x were aborted during detection because a critical control file could not be opened.
    I only have one instance of each of these and both of the below are from HST3
    Source: VSS, Event ID 12293, Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details RevertToSnashot [hr = 0x80042302, A Volume Shadow Copy Service component encountered an
    unexpected error.
    Source: VSS, Event ID 8193, Volume Shadow Copy Service error: Unexpected error calling routine GetOverlappedResult.  hr = 0x80070057, The parameter is incorrect.
    So, basically, everything I have tried has resulted in no success towards solving this problem.
    I would appreciate anything assistance that can be provided.
    Thanks,
    Charles J. Palmer
    Wright Flood

    Tim,
    Thanks for the reply. I ran the first two commands and got this:
    Name                                                            
    Role Metric
    Cluster Network 1                                              
    3  10000
    Cluster Network 2 - HeartBeat                              1   1300
    Cluster Network 3 - iSCSI                                    0  10100
    Cluster Network 4 - LiveMigration                         1   1200
    When you look at the properties of each network, this is how I have it configured:
    Cluster Network 1 - Allow cluster network communications on this network and Allow clients to connect through this network (26.x subnet)
    Cluster Network 2 - Allow cluster network communications on this network. New network added while working with Microsoft support last month. (28.x subnet)
    Cluster Network 3 - Do not allow cluster network communications on this network. (22.x subnet)
    Cluster Network 4 - Allow cluster network communications on this network. Existing but not configured to be used by VMs for Live Migration until MS corrected. (20.x subnet)
    Should I modify my metrics further or are the current values sufficient.
    I worked with an MS support rep because my cluster (once I added the 5th host) stopped being able to live migrate VMs and I had VMs host jumping on startup. It was a mess for a couple of days. They had me add the Heartbeat network as part of the solution
    to my problem. There doesn't seem to be anywhere to configure a network specifically for CSV so I would assume it would use (based on my metrics above) Cluster Network 4 and then Cluster Network 2 for CSV communications and would fail back to the Cluster Network
    1 if both 2 and 4 were down/inaccessible.
    As to the iSCSI getting a second NIC, I would love to but management wants separation of our VMs by subnet and role and hence why I need the 4 VSwitch NICs. I would have to look at adding an additional quad port NIC to my servers and I would be having to
    use half height cards for 2 of my 5 servers for that to work.
    But, on that note, it doesn't appear to actually be a bandwidth issue. I can run a backup for a single VM and get nothing on the network card (It caused the reboots before any real data has even started to pass apparently) and still the problem occurs.
    As to Backup Chain, I have been working with the vendor and they are telling my the issue is with VSS. They also say they support CSV as well. If you go to this page (http://backupchain.com/Hyper-V-Backup-Software.html)
    they say they support CSVs. Their tech support has been very helpful but unfortunately, nothing has fixed the problem.
    What is annoying is that every backup doesn't cause a problem. I have a daily backup of one of our machines that runs fine without initiating any additional reboots. But most every other backup job will trigger the VMs on the common CSV to reboot.
    I understood about the updates but I had to "prove" it to the MS tech I was on the phone with and hence I brought it up. I understand on the storage as well. Why give a warning for something that is working though... I think that is just a poor indicator
    that it doesn't explain that in the report.
    At a loss for what else I can do,
    Charles J. Palmer

  • Backup fails for a Hyper-V guest with VSS Writer failures using DPM 2012 R2 - Hyper-V guest has Oracle application installed,

    I am trying to backup a VM which before it had Oracle installed backed up with no problems.  It is a Windows 2008 R2 server which sites on an 8 node Hyper-V 2012 R2 cluster (with CSVs).  I am using DPM 2012 R2 to run the backups which have been
    successful for the last few weeks but then Oracle was installed and the backups have failed since then.
    The job fails and a large number of VSS writers go into a 'Failed' state with the 'Last Error' showing as 'Timed out'.  I then get 4 popups appear each referring to a 'temporary' drive which appears briefly in Disk Management with a RAW file system. 
    These pops say "You need to format the disk in drive X: before you can use it.  Do you want to format it?" (where
    X: is replaced by the drive letter assigned to each 'temporary' drive).
    The System event log is populated with a large number of warnings with Event 51, Disk, stating "An error was detected on device \Device\Harddisk<number>\DR<number> during a paging operation".  There are also a
    few warnings for Ntfs (eventid 57) stating that "The system failed to flush data to the transaction log.  Corruption may occur."
    Prior to these warnings there are 5 other warnings for partmgr (eventid 58) stating "The disk signature of disk
    <number x> is equal to the disk signatuire of disk <number y>" and 4 errors (eventid 1, VDS Basic Provider), "Unexpected failure.  Error code: 490@01010004".
    There is a script which is run to stop the Oracle application on the server and if this is run then the backups will complete successfully.  We have been troubleshooting this by running a certain amount of the script and seeing which part affects the
    backup and it seems that if the Weblogic (wls_reports) service is stopped then the backup will succeed but if it is running then the backup will fail and the above symptoms occur.
    Another point which may help is that there is a pre-production server which resides on a Windows 2008 R2 Hyper-V standalone server, has the same scripts and installation of Oracle but backs up without any issues.
    I have experienced VSS writer failures before with VM backups but I have not seen this before.  It is not intermittent and I can find no work around to alleviate the problem of having no backup (except stopping this service or shutting the server down,
    but as it is a production server this is not practical).
    If anyone has experienced this before or have any suggestions / advice it would be much appreciated.
    Thanks
    Chris

    Hi Chris
    I have exactly the same issue.
    2008 R2 Server running Oracle
    6 node 2012 R2 Hyper-V Cluster with CSV
    Exact same errors and popup "format disk" message.
    DPM2012 R2
    I also have some other VM's on the same cluster which do back up through DPM
    The only difference I can see is the server which has the issue has a legacy network adapter.
    Production server so cannot make any changes until later but will add new adapter and report back.
    Interesting about the Weblogic service, may test this also.
    Cheers
    Kev.

  • DPM 2012 R2 Rollup 4, Update Rollup 7 for Exchange Server 2010 SP3 backups with eseutil check fail

    Hi,
    It could be a coincidence, but ever since installing Update rollup 4 onto DPM 2012 R2, (which was at UR3 I believe) I haven't been able to complete an Exchange express full backup or consistency check with verification using Eseutil checked. If I have the
    eseutil check enabled I get the following error:
    Type: Consistency check
    Status: Failed
    Description: Data consistency verification check failed for LOGS of Exchange Mailbox Database Mailbox Database 2011112513 on SERVER.xxx. (ID 30146 Details: The process cannot access the file because it is being used by another process (0x80070020))
    Disabling the eseutil verification allows it to succeed, but I'm not happy to continue with that for too long.
    I've had this running for many years and this is the first real issue with exchange, its a single stand alone exchange database.
    Any ideas?
    Thanks,
    Andreas

    Andreas,
    the error is stating that something else has a handle to a file or files during the ESEUtil check.  Typically this is AV.  I have also seen where the Exchange limits are exceeded and that causes the issue. 
    Please ensure you have the DPM AV exclusions for the DPM server and I would advise the DPMRA.exe on the PS.  You can also trying disabling or removing AV as a test.
    DPM AV exclusions -
    http://technet.microsoft.com/en-us/library/hh757911.aspx
    You may also want to test using eseutil against the logs outside DPM and see if that fails with similar issue.  Let us know.  Thanks
    Regards, A.Nadar, This posting is provided &quot;AS IS&quot; with no warranties, and confers no rights.

  • DPM 2012 R2 crashes during tape backup for a specific server

    Hi Community,
    A customer of ours is experiencing an issue with DPM crashing after a period of time when backing up a large protection group to tape.
    Customer is running DPM 2012 R2 URE 5
    Upon looking in the DPM Logs, we see the following error, immediately prior to the system crashing.
    MSDPMCurr.errlog
    0968 29A8 04/08 12:41:48.218 28 MediaPool.cs(350) NORMAL Found PGSetId a914779a-8847-40e5-90fd-a1ca6cffee05 for MediaPoolId e10c1280-445f-4454-8edd-0cb3c1113c27
    0968 29A8 04/08 12:41:48.221 28 MediaPool.cs(329) WARNING MediaPool 00000000-0000-0000-0000-000000000000 does not exist.
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING Caught unhandled exception : Microsoft.Internal.EnterpriseStorage.Dls.Utils.MediaPoolNotFoundException: Media pool 00000000-0000-0000-0000-000000000000 not found in the database.
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.MediaPool.GetPGSetIdForMediaPool(DbContext sqlContext, Guid mediaPoolId)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.MediaPool.IsMediaPoolCollocated(DbContext sqlContext, Guid mediaPoolId)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.ArchiveMedia.IsMediaCollocated(DbContext sqlContext)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.ArchiveMedia.CheckAndRaiseOffsiteReadyEvent(DbContext sqlContext, Boolean considerOfflineMedia)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.Dataset.RaiseMediaOffsiteReadyEvent(DbContext sqlContext)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.Dataset.SetLifeStatus(DbContext sqlContext, DatasetLifeStatus lifeStatusPassed)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.TapeArchiveMedia.Reclaim(DbContext sqlContext)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.MMCatalog.ReclaimMedia(ArchiveMedia media)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMInterface.ReserveResourcesForBackupHelper.GetRecycedUsableMedia(IList listOfMedia)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMInterface.ReserveResourcesForBackupHelper.GetRecyclableMedia(MediaPool mediaPool)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMInterface.ReserveResourcesForBackupHelper.GetMediaRequiredByArchiveTask(Guid expectedDriveId)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMInterface.ReserveResourcesForBackupHelper.AllocateMedia(Guid expectedDriveId)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMInterface.ReserveResourcesForBackupHelper.QueryResourcePreference()
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.TaskElement.ReserveResources(Dictionary`2 libraryResourcesCache)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.JobElement.ReserveResourcesToOneTask(Dictionary`2 libraryResourcesCache)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.PriorityQueue.ServeTasksInQueue(Message msg, Boolean waitForResources)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.MMCommonBlocks.ReserveResourcesBlock.DoServeTasksInQueueAndPostCheckMessage(Object msgObject)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at Microsoft.Internal.EnterpriseStorage.Dls.EngineUICommon.DpmThreadPool.Function(Object state)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING at System.Threading.ThreadPoolWorkQueue.Dispatch()
    0968 29A8 04/08 12:41:48.236 68 DpmThreadPool.cs(121) WARNING *** Mojito error was: MediaPoolNotFound; 0; None
    0968 29A8 04/08 12:41:48.242 68 DpmThreadPool.cs(149) WARNING Attempting to write error to the event log Media pool 00000000-0000-0000-0000-000000000000 not found in the database.
    0968 29A8 04/08 12:41:48.246 02 EventManager.cs(143) NORMAL Logging event to NT log from DpmThreadPool.cs(163): FatalServiceError
    0968 29A8 04/08 12:41:48.267 68 DpmThreadPool.cs(170) WARNING Attempting to invoke Watson
    0968 29A8 04/08 12:41:48.280 09 everettexception.cpp(761) CRITICAL Exception Message = Media pool 00000000-0000-0000-0000-000000000000 not found in the database. of type Microsoft.Internal.EnterpriseStorage.Dls.Utils.MediaPoolNotFoundException, process will terminate after generating dump
    The server is part of a protection group that is co-located, I am trying the backup not as part of a co-located set to see if this makes any difference.
    Tape backups for other servers, which are part of the same co-location group, work fine and it seems to be just this one causing the issue.
    I have seen a few other posts on the net referencing "WARNING MediaPool 00000000-0000-0000-0000-000000000000 does not exist." where people have similar issue with DPM crashing out but so far nobody seems to have any answers.
    I am debating removing the protection group and trying to 'start from scratch' but wondered if anyone else had any thoughts ?
    Thanks in advance,
    Ed

    Hi,
    Open SQL management studio and connect to the instance used by DPM.
    Run the following Query against the DPMDB and see if it returns any rows, if so please provide the output.
    select MP.mediapoolid, MP.PGSETID from dbo.tbl_MM_MediaPool MP
    Join dbo.tbl_IM_ProtectedGroup PG on PG.ProtectedGroupId = MP.MediaPoolId
    where PG.MediapoolId is NULL
    and MP.PGSetId is not NULL
    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This
    posting is provided "AS IS" with no warranties, and confers no rights.

  • DPM 2012 R2 UR4 and BMR backup from physical 2008 R2 (DC, DNS, DHCP,WSUS, WDS)

    Hi,
    I'm completely lost.
    We updated from DPM2010 (2008 R2) to DPM 2012 R2 RU4 (2012 R2).
    Clean install. Same hardware. Everything went quite smoothly.
    Our new (and also old did the same thing) DMP protects some Hyper-V guest on our 2008 R2 Hyper-V cluster, one Exchange (2010) DB's and SQL 2008 R2 DB's and one physical server - 2008 R2 (DC, DNS, DHCP,WSUS, WDS ~160GB of data).
    Currently we are using some Buffalo Terastation's III (TS-RIXL) iSCSI SAN storage to hold our DPM protected data.
    Configured as RAID 5.
    Our new DPM server is connected to those iSCSI devices using built in MS iSCSI initiator via secondary NIC on this server using dedicated iSCSI subnet.
    Everything seems to be fine unitil System State and BMR (I'v checked both) runs.
    Basically my iSCSI NIC shows Write about ~170Mbit/s and Read about 520Mbit/s when backing up virtual machines, Exchange or SQL.
    When it comes to System State and BMR backup the trouhgput is MUCH slower (about 65Mbit/s) and I can not figure out why. I understand that the BMR backup initiates direct writing to the DPM created and shared replica volume on my iSCSI NAS (my wording
    might be not correct but basically  I understand how this works). But I don't understand why the speed is so much slower. I'v tried everything, googeled around, changed NIC drivers, firmware etc. No luck.
    If I just copy some big ISO file from/to server to DPM or vice versa the speed is arount 100MB/s (as reported by windows itself). So it seems to able to utilize full network speed. But why this backup is so slow? I'm excpecting to see speed about ~170Mbit/s
    because it corresponds my iSCSI Write speed as seen when backing up hyper-v guests and DB's..
    What I'm doing wrong?
    Any help is appreciated :)
    rgds Sven

    Hi Sven,
    Sometimes BMR is slower than backing up virtual machines (VHDs). If there is a free local disk, please test to do a backup locally to see the speed. From your description a common backup works much faster so let's see if it is the backup speed which causes
    the issue or not.
    Please remember to mark the replies as answers if they help and un-mark them if they provide no help. If you have feedback for TechNet Support, contact [email protected]

  • Does DPM 2012 R2 still needs DataSourceGroups.xml and MaxAllowedParallelBackups key when using Hyper-V VSS Software Writer to backup Server 2012 R2 Cluster

    Hi,
    Does DPM 2012 R2 still needs DataSourceGroups.xml and MaxAllowedParallelBackups key when using Hyper-V VSS Software Writer to backup Server 2012 R2 Hyper-V Cluster with CSV's? Can't find a clear answer somewhere.
    Thanx for helping me out.
    DJITS.

    Configure CSV backups
    If you are running Hyper-V on Windows Server 2008 R2 and you’re backing up multiple clusters with DPM, you need to ensure that only one backup uses the same CSV at any one time. To do this, follow the instructions in
    Serialize virtual machine backups.
    If you’re using software VSS with Hyper-V on Windows Server 2008 R2, do the following
    Serialize virtual machine backups per node            
    Serialize virtual machine backups per CSV LUN            
    Have a nice day !!!

  • DPM 2012 R2 long backup to tape job randomly fail after installing SCCM 2012 Client

    Hello,
    I'm managing a two nodes 2012 R2 file server cluster that contains a 16To CSV. I'm using DPM 2012 R2 to backup this entire shared volume directly to LTO 4 tapes, the job last about 55h.
    Since SCCM 2012 client has been installed(I don't manage it), the tape jobs are failing ramdomly after several hours with the error:
    Type: Tape backup
    Status: Failed
    Description: The DPM service was unable to communicate with the protection agent on serverX.xxxx.xxx . (ID 52 Details: The semaphore timeout period has expired (0x80070079))
     More information
    End time: 19/07/2014 03:11:06
    Start time: 18/07/2014 22:00:00
    Time elapsed: 05:11:05
    Data transferred: 768 289,56 MB
    Cluster node serverX.xxxx.xxx
    Source details: G:\
    Protection group members: 1
     Details
    Protection group: File Server Tape Protection
    Library: Quantum PX500 Series Medium Changer
    Tape Label (Barcode): File Server Tape Protection-00000230 (000043L4)
    If I uninstall SCCM 2012 client, no more issue, backups succeed. I've asked our SCCM team, no specific task has been scheduled or deployed in SCCM.
    I can't see anything abnormal in logs.
    Any idea?

    I have disabled "Configuration Manager Maintenance" and I have also tried to set the registry value HKLM\Software\Microsoft\CCM\CcmEval\NotifyOnly to TRUE and still the same issue.
    I can't find any correlated errors in the Windows event logs, task scheduler history neither in the DPM logs.
    I've increased the log level of DPM by following the following procedure
    http://blogs.msdn.com/b/george_bethanis/archive/2013/11/04/how-to-collect-dpm-verbose-logs.aspx
    Now i'm suspecting the maintenance job of Windows 2012 R2, i'll try to disabled this task. But the fact is that I don't have this backup issue if SCCM 2012 client is not installed.
    I'm waiting for next logs and will keep you informed

Maybe you are looking for