2012 r2 file backup causes all VMs to go in to "backing up"

I'm using windows server backup on server 20012 r2, just doing a file backup of the hyper v VMs.  When I do this, all of the child VMs go in to a "backing up" state for the entire backup.  I'm still able to connect to each VM but they
run so slowly they're unusable.  I have it set to do incremental backups, but sometimes it will still take hours to back up, which causes the child VMs to be basically locked up for hours.  Is there any way to set it to only back up one VM at a time
rather than lock them all up throughout the backup process?

Hi, thanks for getting back to me.  
I can select each VM individually when doing the application backup (and they all show as being able to do an online backup) but I just back up the drive rather than the hyper-v vm.
I guess I'm curious if all this is to be expected behavior when doing a drive backup that includes the VM vhd files.  Should all of them go in to a "backing up" state for the entire backup (and causing all VMs to run very poorly).  This seems the
opposite of how it worked in 2k8 where I could schedule the backup for the drive that contained the VMs and still use each and every VM just fine during the backup.  
Also I've noticed there's not really a way to stop the backups at all, I can try to stop the job but nothing ever happens, and i'm forced to do a hard reboot of the machine.  It's acting like it's stuck making the shadow copies until the backup completes.
I've noticed this behavior on 2 different 2012 servers so this makes me think that this is expected behavior for doing a file backup on 2k8 VMs.  
Hi David,
Is this server configured with a CSV volume for storing these VHDs ? I have seen that backups of VMs on a CSV will put entire VMs in backing up status.
Backing up status means that a snap shot is getting created while the backups are happening. A checkpoint is created of every virtual machine that is to be backed up.  Writes are temporarily redirected to the checkpoint which gives the backup requester
a clean & crash consistent copy of the virtual machine’s VHD or VHDX files that are safe to read. The checkpoints are merged with the parent disk once the backup is finished.
I will also test this on my lab.
Cheers.
Shaba
Optimism is the faith that leads to achievement. Nothing can be done without hope and confidence.
InsideVirtualization.com

Similar Messages

  • Creating Poster from File function causes all graphics files to distort.

    Sorry for the long subject title, didn't know how to pare it down...
    What's happening is when I add 3D content in Acrobat 3D V8 I select the option in the Poster Image window to "Creat poster from file", the graphic ALWAYS rotates the image and often distorts it as well.
    Any fix for this?
    Jeff Perkins

    I don't get this error. I've tried both JPG and BMP files, and both work. Hmmm...maybe it's due to the aspect ratio. Try at least matching the layout (portrait/landscape) between the 3D annot and the graphic file.

  • DPM 2012 R2 backup Causes Redirected CSV IO on SOFS Cluster.

    Hi, I have a Scale out Storage Spaces Server with 2 nodes, and a 10 node 2012 R2, Hyper-V cluster using this via SMB3.0
    I also have installed a DPM2012 R2 backup server.
    the DPM agent is installed on all nodes of all servers and I have followed the pre-requisite from Microsoft for setting up DPM backup of SMB Hyper-V machines.
    The DPM backups all work fine. but occasionaly I get these errors on the SOFS cluster.
    Cluster Shared Volume 'Volume3' ('Cluster Disk 4') has entered a paused state because of '(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.
    I really thought this issue had been resolved in this revision, this doesn't seem to cause any issues with my VM's that I can notice. and all DPM backups are working fine, but it still causes me concern.
    has anyone else seen this or have any suggestions what I can try to resolve.
    Regards
    Mark Green

    We also encounter this issue. We use Windows Server 2012 R2 and SCVMM 2012 R2 (with RU1). Be carefull with this issue, because it can cause serious issues. Btw, note that Windows Server 2012 R2 used Direct I/O instead of Redirected I/O.
    If you can't find a full fix as we are in right now, there are two things that might offer a work-around for you:
    Disabled ODX (if your storage system does not support it):
    Deploy Windows Offloaded Data Transfers
    http://technet.microsoft.com/en-us/library/jj200627.aspx
    Serialize virtual machine backups per node
    Migrate to a hardware VSS provider
    http://technet.microsoft.com/en-us/library/hh758027.aspx
    The second option works best, because this issue mostly occurs when you run a backup of many VMs at once. It it not a full fix and makes you backup windows much longer, but can avoid you other problems. Also keep a close eye on this link:
    Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters
    http://support.microsoft.com/kb/2920151
    Boudewijn Plomp, BPMi Infrastructure & Security

  • HyperV 2012 R2 Failover cluster, HV problem, all VMs restart

    Hello, I have 2 node Failover cluster with two nodes, Hyperv 2012, multipath SAS storage MSA2000. But hardware problem with one node (node2). It shutdown unexpectly. When It hapens NODE1 restar all VMs it is normal? It was configured by cluster validation
    tool. There is no witness. I don't clearly understand what happens if one node crash. KR.

    As Eric has said it will start the VM's in a crash consistent state on the non crashed host.
    But from your example I take your seeing your guests on the non crashed host restart. If this is the case I would say yes! I have seen this happen before. It can happen if your not using quorum because only one node has a vote. I would recommend you create
    a witness, on your MSA 2000 carve out 1 GB and do a disk witness. Or if you have a server not in the VM cluster you could do a file share witness, file share is my preferred. Once you have a witness in play you will see all of your hosts having a vote. Look
    in the cluster manager at the nodes section. You should see a vote column. Currently it will say 1/0, once the witness is created it will show 1/1.

  • Temp files eating up all disk space on SCSM 2012 R2 install

    Hello Everyone,
    Problem
    Under this path SCSM is creating a ton of temp files using up all disk space (C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\System Center Service Manager 2010)  All files are .bin and .hash files.  I just cleaned out 120 GBs
    of files which accumulated over the period of 30 days.   
    Does anyone know what could be causing this?  This was not a problem when we were on server 2008 R2.  When we migrated SCSM management server to server 2012 r2 it when problem started.
    Environment
    Management server (server 2012 R2)
    Data warehouse server (server 2008 R2)
    Data on remote SQL server.  SQL 2008 R2
    CU  5 was the last update

    Might be worth checking that the AutoRefresh/Cache policy is set to True via the SDK?
    https://msdn.microsoft.com/en-us/library/gg461136.aspx
    From PowerShell:
    Add-Type -Path "C:\Program Files\Microsoft System Center 2012 R2\Service Manager\SDK Binaries\Microsoft.EnterpriseManagement.Core.dll"
    $NS = "Microsoft.EnterpriseManagement"
    $EMGType = "$NS.EnterpriseManagementGroup"
    $EMG = new-object $EMGType localhost
    $EMG.AutoRefreshCache
    Output True or False
    $EMG.CacheMode
    Configuration
    This is the output from mine:
    Good luck!

  • HELP!!  I viewed my "backup" files, and now ALL of my photos are no longer "linked" to Lightroom.

    I thought I was missing some files, so I viewed my backup files.  I received a "warning" that Lightroom would restart and view my backup files.  I did NOT receive any warning, that there was NO GOING BACK!!  I've tried everything.  I cannot re-link ANY of my photos.  Well, I did just do one, and it told me all the other photos in the folder were corrupt and Lightroom would not be able to edit them (and no, it was not an exported .jpeg file).  Fortunately, I am not concerned about this folder, that's why I tested it.  I was forced to moved all of my photos from my C drive to my D drive because for some weird reason on this computer, my D drive is 5 times bigger and I ran out of room.  When I did that, I was able to relink all the pictures.  Sadly, it didn't keep my Folders, it just relinked them all by date.  That I can deal with, even though it makes finding photos much more difficult, but I wasn't about to try to rename any of them and have to relink again.
    Now, I can't relink any of them because it appears in Lightroom that all photos are back in the C drive and therefore, the new photos that I placed on the D drive are not there, and nothing will relink, period, not even the ones that were originally on my C drive!
    I have never been so aggravated in my life.  I LOVE Lightroom capabilities.  I have version 2.3.  When I look it up on the online help, all I get is how the photos are in whatever location they were saved, but they are linked, etc.  I already know that part.  I seriously wish I had NEVER looked at my backup files.  I don't think I'll bother backing up stuff anymore because apparently looking at the backup messes up everything.
    I fully anticipated that I would be able to go back, choose my folders on my D drive and have all of my current photos available.  Sadly, not.
    My user name should be FREAKED OUT!

    Thanks!  I've been practicing deep breathing between yelling at my computer.  I use Windows 7.  I was in my "regular" Lightroom photos, and I couldn't locate some pictures I took last December.  So, I thought, well, why not look at the back up files?  I'm normally excellent at remembering exact steps, but I'm so worried about all of my thousands of photos, I can only explain what I think I did.  I went into Windows Explorer, and chose the backup files.  When I did, I received the warning about Lightroom restarting and loading the backup photos.  I hesitated for a full minute, but thought, well, isn't that what backup files are for?  So, I selected it. Lightroom opened the backup photos without any problem.  But, when I went back, and tried opening the photos that were linked to Lightroom, they were no longer linked.  When I click on the formerly linked photos, they just open in Windows Explorer, not Lightroom.  When I open Lightroom, it has says all the photos are "missing."  All of the folders are listed in my C drive (in Lightroom), not where they'd been moved to on my D drive.  The one photo I relinked, stated the photos in that folder could not be edited in Lightroom.  So, I'm afraid to try to relink anymore of them. Also, as I said, any photos that I originally imported to my D drive are not showing up in Lightroom at all.
    I would say, I normally learn software quickly. That was my experience with Lightroom.  I installed it, went through all of the online training sessions, and have loved it.  Admittedly, I'm sure there are features that I don't make use of that many people do, but I'm just a passionate photography nut, not a professional.
    The online help basically explains the linking, and even how to relink if you "move" a file, a folder to another location.  It doesn't cover this, or not where I can find it.
    I've think I've given up on yelling at the computer, now I just have knots in my stomach about getting photos relinked.  I have probably 80% of them exported, with the adjustments I made, but that does not make me feel one bit better.
    Thanks!!!

  • I have never updated ios on my iPhone 4.  I am currently running ios 4.  I tried to update yesterday to ios 6 but many hours into the backup I got an error message saying that continuing would cause all of my phone contents to be lost. What should I do?

    I have never updated ios on my iPhone 4.  I am currently running ios 4.  I tried to update yesterday to ios 6 but many hours into the backup I got an error message saying that continuing would cause all of my phone contents to be lost. What should I do?

    Why, if it told you you have insufficient space on your computer, would it suggest making more space available on your phone?
    At a guess, you have insufficient space on your phone. Installing software will typically require more space than the final size of the software, perhaps even twice or three times as much, for the installation process. You can easily dump some videos or music temporarily, and then synch them back in afterwards.

  • Files listed in recent TM backup are all greyed-out, no details?

    A client was looking at their Time Machine backup and all files were greyed-out without any file details (kind, date, size, etc.).
    Steps to reproduce:
    Open Finder window, select All My Files.
    Go to Time Machine icon and select Enter Time Machine.
    In TM, select a recent backup (in this case, from previous day).
    Windows will "scan" backward to display Finder window for that backup, but only the file names are listed and they are greyed-out. No details.
    System is slow to respond, pressing Cancel takes about 20 seconds to respond.
    OS: 10.7.5
    Ext. HDD: LaCie 1TB USB 2.0
    Internal HDD: 1TB, approx 300GB used
    iMac: 24", Approx 2009 (not sure which exact model), 3GHz, 4GB RAM 1333
    Any thoughts? Is this a data transfer issue with the volume of data going across USB 2.0?
    Here is a screen capture:

    Select
              View ▹ Show View Options
    from the Finder menu bar. In the dialog that opens, select
              Arrange by: None

  • I did a backup from all my files saved in old external hd to time capsule. After that double checked and recover one file from TC to mac air and it worked just perfect. Today went through TC to recover another file and none of them were there?

    I did a backup from all my files saved in old external hd to time capsule. After that double checked and recover one file from TC to mac air and it worked just perfect. Today went through TC to recover another file and none of them were there?Anybody has a clue what happened?

    renatocremonese wrote:
    I want to use it for backing up my Mac.
    It's good for that . . . 
    But also I don't keep all my stuff in my Mac.
    But not for that. 
    This older and not day-by-day usage files I want to store in the time machine.
    You can do that (see below), but how are you going to back them up?   If your only copies are on the TC, when (not if) it fails, you risk losing your only copy of the data.
    Is it possible to split TC in two partitions.
    No, but there are some workarounds, including making a fixed-size disk image on it to "reserve" some space.  See #Q3 in Using Time Machine with a Time Capsule.
    But you still won't have backups of the stuff you put there.
    You don't say what kind of Mac you have.  If it's a desktop model, just keep the external HD connected to it, and let Time Machine back it up along with your internal HD.
    However, it sounds like you may have a laptop, where that's not going to be convenient.  In that case, your best bet might be to copy the external HD to a disk image on the TC as above, then keep the HD in a safe place.
    To finish, when i enter the TC and go through the Time line how can I get a file from there and move it back to Mac hd.
    Via the "Star Wars" display, per #15 in Time Machine - Frequently Asked Questions.
    You might also want to review the Time Machine Tutorial, and perhaps browse the rest of the FAQ.

  • My hard wear and soft wear crashed, did a backup on all my files.When I now open them the are all "preview"!!! How do I get my old documents back? it should be pages, and numbers, ight now I can't work with them

    My hard wear and soft wear crashed, did a backup on all my files.
    When I now open them the are all "preview"!!!
    How do I get my old documents back? it should be pages, and numbers, right now I can't work with them

    Reinstall OS X

  • Disk Utility stopped repairing the disk. I was instructed to backup as many of my files as possible.reformat the disk and store my backed files.  I am using an I Mac 2010.  I am not sure what caused this problem.   I use time machine

    Disk Utility stopped repairing the disk. I was instructed to backup as many of my files as possible.reformat the disk and restore my backed up files.  Since I use time machine, can I assume that all my files and programs can be restored from my recent back ups? 

    Hi jeromefrodia, 
    Welcome to the Apple Support Community!
    I understand that you are concerned about your data before reformatting the disk. Because you use Time Machine your information should be safe. Please refer to the attached article for information about what Time Machine backs up and how to restore from that backup. 
    Mac Basics: Time Machine backs up your Mac
    Cheers,
    Joe

  • Server 2012 built in backup. Lost 4+ months of backups.

    I have been running the built in backup software on a 2012 server. Today I noticed that I only have 3 days of backups.
    The server is 2012 running Hyper-V with 6 VMs.  I had the backup software configured to backup all the VMS and a bare metal backup to an external 2 TB USB drive.  I have been looking at the event logs and search the web without much success up
    to this point.
    The event log does have the following information message:
    Backup started a fresh backup for volume '{00000000-0000-0000-0000-000000000000}' ('Esp') : [Reason: 'Volume size changed']. This may cause loss of older backup versions when backup completes.
    Any help would be appreciated.

    Hi,
    From the error message, could you find any volume size is changed before the first "fresh backup"?
    Sometimes we will see similar issue that previous backups are deleted when lacking of free disk space. Please check the vssadmin list shadowstorage for sure. 
    And please check the backup target folder to see if the previous backup copies are all overwritten by the new backup. Backups files are VHD files so try to check if previous VHD files could be found. 
    If you have any feedback on our support, please send to [email protected]

  • Windows Server 2012 - Hyper-V - iSCSI SAN - All Hyper-V Guests stops responding and extensive disk read/write

    We have a problem with one of our deployments of Windows Server 2012 Hyper-V with a 2 node cluster connected to a iSCSI SAN.
    Our setup:
    Hosts - Both run Windows Server 2012 Standard and are clustered.
    HP ProLiant G7, 24 GB RAM, 2 teamed NIC dedicated to Virtual Machines and Management, 2 teamed NIC dedicated to iSCSI storage. - This is the primary host and normaly all VMs run on this host.
    HP ProLiant G5, 20 GB RAM, 1 NIC dedicated to Virtual Machines and Management, 2 teamed NIC dedicated to iSCSI storage. - This is the secondary host that and is intended to be used in case of failure of the primary host.
    We have no antivirus on the hosts and the scheduled ShadowCopy (previous version of files) is switched of.
    iSCSI SAN:
    QNAP NAS TS-869 Pro, 8 INTEL SSDSA2CW160G3 160 GB i a RAID 5 with a Host Spare. 2 Teamed NIC.
    Switch:
    DLINK DGS-1210-16 - Both the network cards of the Hosts that are dedicated to the Storage and the Storage itself are connected to the same switch and nothing else is connected to this switch.
    Virtual Machines:
    3 Windows Server 2012 Standard - 1 DC, 1 FileServer, 1 Application Server.
    1 Windows Server 2008 Standard Exchange Server.
    All VMs are using dynamic disks (as recommended by Microsoft).
    Updates
    We have applied the most resent updates to the Hosts, WMs and iSCSI SAN about 3 weeks ago with no change in our problem and we continually update the setup.
    Normal operation
    Normally this setup works just fine and we see no real difference in speed in startup, file copy and processing speed in LoB applications of this setup compared to a single host with 2 10000 RPM Disks. Normal network speed is 10-200 Mbit, but occasionally
    we see speeds up to 400 Mbit/s of combined read/write for instance during file repair
    Our Problem
    Our problem is that for some reason all of the VMs stops responding or responds very slowly and you can for instance not send CTRL-ALT-DEL to a VM in the Hyper-V console, or for instance start task manager when already logged in.
    Symptoms (i.e. this happens, or does not happen, at the same time)
    I we look at resource monitor on the host then we see that there is often an extensive read from a VHDX of one of the VMs (40-60 Mbyte/s) and a combined write speed to many files in \HarddiskVolume5\System Volume Information\{<someguid and no file extension>}.
    See iamge below.
    The combined network speed to the iSCSI SAN is about 500-600 Mbit/s.
    When this happens it is usually during and after a VSS ShadowCopy backup, but has also happens during hours where no backup should be running (i.e. during daytime when the backup has finished hours ago according to the log files). There is however
    not that extensive writes to the backup file that is created on an external hard drive and this does not seem to happen during all backups (we have manually checked a few times, but it is hard to say since this error does not seem leave any traces in event
    viewer).
    We cannot find any indication that the VMs themself detect any problem and we see no increase of errors (for example storage related errors) in the eventlog inside the VMs.
    The QNAP uses about 50% processing Power on all cores.
    We see no dropped packets on the switch.
    (I have split the image to save horizontal space).
    Unable to recreate the problem / find definitive trigger
    We have not succeeded in recreating the problem manually by, for instance, running chkdsk or defrag in VM and Hosts, copy and remove large files to VMs, running CPU and Disk intensive operations inside a VM (for instance scan and repair a database file).
    Questions
    Why does all VMs stop responding and why is there such intensive Read/Writes to the iSCSI SAN?
    Could it be anything in our setup that cannot handle all the read/write requests? For instance the iSCSI SAN, the hosts, etc?
    What can we do about this? Should we use MultiPath IO instead of NIC teaming to the SAN, limit bandwith to the SAN, etc?

    Hi,
    > All VMs are using dynamic disks (as recommended by Microsoft).
    If this is a testing environment, it’s okay, but if this a production environment, it’s not recommended. Fixed VHDs are recommended for production instead of dynamically expanding or differencing VHDs.
    Hyper-V: Dynamic virtual hard disks are not recommended for virtual machines that run server workloads in a production environment
    http://technet.microsoft.com/en-us/library/ee941151(v=WS.10).aspx
    > This is the primary host and normaly all VMs run on this host.
    According to your posting, we know that you have Cluster Shared Volumes in the Hyper-V cluster, but why not distribute your VMs into two Hyper-V hosts.
    Use Cluster Shared Volumes in a Windows Server 2012 Failover Cluster
    http://technet.microsoft.com/en-us/library/jj612868.aspx
    > 2 teamed NIC dedicated to iSCSI storage.
    Use Microsoft MultiPath IO (MPIO) to manage multiple paths to iSCSI storage. Microsoft does not support teaming on network adapters that are used to connect to iSCSI-based storage devices. (At least it’s not supported until Windows Server 2008 R2. Although
    Windows Server 2012 has built-in network teaming feature, I don’t article which declare that Windows Server 2012 network teaming support iSCSI connection)
    Understanding Requirements for Failover Clusters
    http://technet.microsoft.com/en-us/library/cc771404.aspx
    > I have seen using MPIO suggests using different subnets, is this a requirement for using MPIO
    > or is this just a way to make sure that you do not run out of IP adressess?
    What I found is: if it is possible, isolate the iSCSI and data networks that reside on the same switch infrastructure through the use of VLANs and separate subnets. Redundant network paths from the server to the storage system via MPIO will maximize availability
    and performance. Of course you can set these two NICs in separate subnets, but I don’t think it is necessary.
    > Why should it be better to not have dedicated wireing for iSCSI and Management?
    It is recommended that the iSCSI SAN network be separated (logically or physically) from the data network workloads. This ‘best practice’ network configuration optimizes performance and reliability.
    Check that and modify cluster configuration, monitor it and give us feedback for further troubleshooting.
    For more information please refer to following MS articles:
    Volume Shadow Copy Service
    http://technet.microsoft.com/en-us/library/ee923636(WS.10).aspx
    Support for Multipath I/O (MPIO)
    http://technet.microsoft.com/en-us/library/cc770294.aspx
    Deployments and Tests in an iSCSI SAN
    http://technet.microsoft.com/en-US/library/bb649502(v=SQL.90).aspx
    Hope this helps!
    TechNet Subscriber Support
    If you are
    TechNet Subscription user and have any feedback on our support quality, please send your feedback
    here.
    Lawrence
    TechNet Community Support

  • Hyper-V cluster Backup causes virtual machine reboots for common Cluster Shared Volumes members.

    I am having a problem where my VMs are rebooting while other VMs that share the same CSV are being backed up. I have provided all the information that I have gather to this point below. If I have missed anything, please let me know.
    My HyperV Cluster configuration:
    5 Node Cluster running 2008R2 Core DataCenter w/SP1. All updates as released by WSUS that will install on a Core installation
    Each Node has 8 NICs configured as follows:
     NIC1 - Management/Campus access (26.x VLAN)
     NIC2 - iSCSI dedicated (22.x VLAN)
     NIC3 - Live Migration (28.x VLAN)
     NIC4 - Heartbeat (20.x VLAN)
     NIC5 - VSwitch (26.x VLAN)
     NIC6 - VSwitch (18.x VLAN)
     NIC7 - VSwitch (27.x VLAN)
     NIC8 - VSwitch (22.x VLAN)
    Following hotfixes additional installed by MS guidance (either while build or when troubleshooting stability issue in Jan 2013)
     KB2531907 - Was installed during original building of cluster
     KB2705759 - Installed during troubleshooting in early Jan2013
     KB2684681 - Installed during troubleshooting in early Jan2013
     KB2685891 - Installed during troubleshooting in early Jan2013
     KB2639032 - Installed during troubleshooting in early Jan2013
    Original cluster build was two hosts with quorum drive. Initial two hosts were HST1 and HST5
    Next host added was HST3, then HST6 and finally HST2.
    NOTE: HST4 hardware was used in different project and HST6 will eventually become HST4
    Validation of cluster comes with warning for following things:
     Updates inconsistent across hosts
      I have tried to manually install "missing" updates and they were not applicable
      Most likely cause is different build times for each machine in cluster
       HST1 and HST5 are both the same level because they were built at same time
       HST3 was not rebuilt from scratch due to time constraints and it actually goes back to Pre-SP1 and has a larger list of updates that others are lacking and hence the inconsistency
       HST6 was built from scratch but has more updates missing than 1 or 5 (10 missing instead of 7)
       HST2 was most recently built and it has the most missing updates (15)
     Storage - List Potential Cluster Disks
      It says there are Persistent Reservations on all 14 of my CSV volumes and thinks they are from another cluster.
      They are removed from the validation set for this reason. These iSCSI volumes/disks were all created new for
      this cluster and have never been a part of any other cluster.
     When I run the Cluster Validation wizard, I get a slew of Event ID 5120 from FailoverClustering. Wording of error:
      Cluster Shared Volume 'Volume12' ('Cluster Disk 13') is no longer available on this node because of
      'STATUS_MEDIA_WRITE_PROTECTED(c00000a2)'. All I/O will temporarily be queued until a path to the
      volume is reestablished.
     Under Storage and Cluster Shared VOlumes in Failover Cluster Manager, all disks show online and there is no negative effect of the errors.
    Cluster Shared Volumes
     We have 14 CSVs that are all iSCSI attached to all 5 hosts. They are housed on an HP P4500G2 (LeftHand) SAN.
     I have limited the number of VMs to no more than 7 per CSV as per best practices documentation from HP/Lefthand
     VMs in each CSV are spread out amonst all 5 hosts (as you would expect)
    Backup software we use is BackupChain from BackupChain.com.
    Problem we are having:
     When backup kicks off for a VM, all VMs on same CSV reboot without warning. This normally happens within seconds of the backup starting
    What have to done to troubleshoot this:
     We have tried rebalancing our backups
      Originally, I had backup jobs scheduled to kick off on Friday or Saturday evening after 9pm
      2 or 3 hosts would be backing up VMs (Serially; one VM per host at a time) each night.
      I changed my backup scheduled so that of my 90 VMs, only one per CSV is backing up at the same time
       I mapped out my Hosts and CSVs and scheduled my backups to run on week nights where each night, there
       is only one VM backed up per CSV. All VMs can be backed up over 5 nights (there are some VMs that don't
       get backed up). I also staggered the start times for each Host so that only one Host would be starting
       in the same timeframe. There was some overlap for Hosts that had backups that ran longer than 1 hour.
      Testing this new schedule did not fix my problem. It only made it more clear. As each backup timeframe
      started, whichever CSV the first VM to start was on would have all of their VMs reboot and come back up.
     I then thought maybe I was overloading the network still so I decided to disable all of the scheduled backup
     and run it manually. Kicking off a backup on a single VM, in most cases, will cause the reboot of common
     CSV members.
     Ok, maybe there is something wrong with my backup software.
      Downloaded a Demo of Veeam and installed it onto my cluster.
      Did a test backup of one VM and I had not problems.
      Did a test backup of a second VM and I had the same problem. All VMs on same CSV rebooted
     Ok, it is not my backup software. Apparently it is VSS. I have looked through various websites. The best troubleshooting
     site I have found for VSS in one place it on BackupChain.com (http://backupchain.com/hyper-v-backup/Troubleshooting.html)
     I have tested almost every process on there list and I will lay out results below:
      1. I have rebooted HST6 and problems still persist
      2. When I run VSSADMIN delete shadows /all, I have no shadows to delete on any of my 5 nodes
       When I run VSSADMIN list writers, I have no error messages on any writers on any node...
      3. When I check the listed registry key, I only have the build in MS VSS writer listed (I am using software VSS)
      4. When I run VSSADMIN Resize ShadowStorge command, there is no shadow storage on any node
      5. I have completed the registration and service cycling on HST6 as laid out here and most of the stuff "errors"
       Only a few of the DLL's actually register.
      6. HyperV Integration Services were reconciled when I worked with MS in early January and I have no indication of
       further issue here.
      7. I did not complete the step to delete the Subscriptions because, again, I have no error messages when I list writers
      8. I removed the Veeam software that I had installed to test (it hadn't added any VSS Writer anyway though)
      9. I can't realistically uninstall my HyperV and test VSS
      10. Already have latest SPs and Updates
      11. This is part of step 5 so I already did this. This seems to be a rehash of various other stratgies
     I have used the VSS Troubleshooter that is part of BackupChain (Ctrl-T) and I get the following error:
      ERROR: Selected writer 'Microsoft Hyper-V VSS Writer' is in failed state!
      - Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)
      - Writer Failure code: 0x800423f0 (<Unknown error code>)
      - Writer ID: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
      - Instance ID: {d55b6934-1c8d-46ab-a43f-4f997f18dc71}
      VSS snapshot creation failed with result: 8000FFFF
    VSS errors in event viewer. Below are representative errors I have received from various Nodes of my cluster:
    I have various of the below spread out over all hosts except for HST6
    Source: VolSnap, Event ID 10, The shadow copy of volume took too long to install
    Source: VolSnap, Event ID 16, The shadow copies of volume x were aborted because volume y, which contains shadow copy storage for this shadow copy, wa force dismounted.
    Source: VolSnap, Event ID 27, The shadow copies of volume x were aborted during detection because a critical control file could not be opened.
    I only have one instance of each of these and both of the below are from HST3
    Source: VSS, Event ID 12293, Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details RevertToSnashot [hr = 0x80042302, A Volume Shadow Copy Service component encountered an
    unexpected error.
    Source: VSS, Event ID 8193, Volume Shadow Copy Service error: Unexpected error calling routine GetOverlappedResult.  hr = 0x80070057, The parameter is incorrect.
    So, basically, everything I have tried has resulted in no success towards solving this problem.
    I would appreciate anything assistance that can be provided.
    Thanks,
    Charles J. Palmer
    Wright Flood

    Tim,
    Thanks for the reply. I ran the first two commands and got this:
    Name                                                            
    Role Metric
    Cluster Network 1                                              
    3  10000
    Cluster Network 2 - HeartBeat                              1   1300
    Cluster Network 3 - iSCSI                                    0  10100
    Cluster Network 4 - LiveMigration                         1   1200
    When you look at the properties of each network, this is how I have it configured:
    Cluster Network 1 - Allow cluster network communications on this network and Allow clients to connect through this network (26.x subnet)
    Cluster Network 2 - Allow cluster network communications on this network. New network added while working with Microsoft support last month. (28.x subnet)
    Cluster Network 3 - Do not allow cluster network communications on this network. (22.x subnet)
    Cluster Network 4 - Allow cluster network communications on this network. Existing but not configured to be used by VMs for Live Migration until MS corrected. (20.x subnet)
    Should I modify my metrics further or are the current values sufficient.
    I worked with an MS support rep because my cluster (once I added the 5th host) stopped being able to live migrate VMs and I had VMs host jumping on startup. It was a mess for a couple of days. They had me add the Heartbeat network as part of the solution
    to my problem. There doesn't seem to be anywhere to configure a network specifically for CSV so I would assume it would use (based on my metrics above) Cluster Network 4 and then Cluster Network 2 for CSV communications and would fail back to the Cluster Network
    1 if both 2 and 4 were down/inaccessible.
    As to the iSCSI getting a second NIC, I would love to but management wants separation of our VMs by subnet and role and hence why I need the 4 VSwitch NICs. I would have to look at adding an additional quad port NIC to my servers and I would be having to
    use half height cards for 2 of my 5 servers for that to work.
    But, on that note, it doesn't appear to actually be a bandwidth issue. I can run a backup for a single VM and get nothing on the network card (It caused the reboots before any real data has even started to pass apparently) and still the problem occurs.
    As to Backup Chain, I have been working with the vendor and they are telling my the issue is with VSS. They also say they support CSV as well. If you go to this page (http://backupchain.com/Hyper-V-Backup-Software.html)
    they say they support CSVs. Their tech support has been very helpful but unfortunately, nothing has fixed the problem.
    What is annoying is that every backup doesn't cause a problem. I have a daily backup of one of our machines that runs fine without initiating any additional reboots. But most every other backup job will trigger the VMs on the common CSV to reboot.
    I understood about the updates but I had to "prove" it to the MS tech I was on the phone with and hence I brought it up. I understand on the storage as well. Why give a warning for something that is working though... I think that is just a poor indicator
    that it doesn't explain that in the report.
    At a loss for what else I can do,
    Charles J. Palmer

  • Will Log Backup cause issue?

    Hi Experts,
    we have been getting complaints from application users about slowness on and off. tried all the ways, no clues. finally, one business user is helping me with the exact timings of slowness in the application. the trend is exactly at the time of log backup
    happening time. log backups are completing in less 30 sec. even the slowness is only at that time.
    Now I didn't understand, will log backups cause slowness to the databases access? I believe, full backups will cause a bit.
    please help.
    thanks in advance
    Vinod Mallolu
    Cheers, Vinod Mallolu

    Hi Vinod, 
    It really depends on I/O subsystem and your server configuration. For example:
    Poorly performing I/O subsystem can cause poor during log backups. 
    Poor performance can occur if your database log drive is same as your backup drive. If you are using SAN then make sure that both drives are not commissioned from same LUN. 
    Transaction Log activity and the size of your log files. If your database is highly transactional database than you should experience some slowness if you are performing Log backups. If you are using SQL Server 2012 or about with Enterprise edition and
    have standby server than use Availability Groups to offload your transaction log backups. 
    Increasing the frequency of log backup also helps. 
    I hope this information is useful for you to rectify the issue. 
    Regards,
    Basit A. Farooq (MSC Computing, MCITP SQL Server 2005 & 2008, MCDBA SQL Server 2000)
    http://basitaalishan.com
    Please remember to click "Mark as Answer" on the post that helps you, and to click
    "Unmark as Answer" if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

Maybe you are looking for