Windows Server 2012 - Hyper-V - iSCSI SAN - All Hyper-V Guests stops responding and extensive disk read/write

Question

We have a problem with one of our deployments of Windows Server 2012 Hyper-V with a 2 node cluster connected to a iSCSI SAN.
Our setup:
Hosts - Both run Windows Server 2012 Standard and are clustered.
HP ProLiant G7, 24 GB RAM, 2 teamed NIC dedicated to Virtual Machines and Management, 2 teamed NIC dedicated to iSCSI storage. - This is the primary host and normaly all VMs run on this host.
HP ProLiant G5, 20 GB RAM, 1 NIC dedicated to Virtual Machines and Management, 2 teamed NIC dedicated to iSCSI storage. - This is the secondary host that and is intended to be used in case of failure of the primary host.
We have no antivirus on the hosts and the scheduled ShadowCopy (previous version of files) is switched of.
iSCSI SAN:
QNAP NAS TS-869 Pro, 8 INTEL SSDSA2CW160G3 160 GB i a RAID 5 with a Host Spare. 2 Teamed NIC.
Switch:
DLINK DGS-1210-16 - Both the network cards of the Hosts that are dedicated to the Storage and the Storage itself are connected to the same switch and nothing else is connected to this switch.
Virtual Machines:
3 Windows Server 2012 Standard - 1 DC, 1 FileServer, 1 Application Server.
1 Windows Server 2008 Standard Exchange Server.
All VMs are using dynamic disks (as recommended by Microsoft).
Updates
We have applied the most resent updates to the Hosts, WMs and iSCSI SAN about 3 weeks ago with no change in our problem and we continually update the setup.
Normal operation
Normally this setup works just fine and we see no real difference in speed in startup, file copy and processing speed in LoB applications of this setup compared to a single host with 2 10000 RPM Disks. Normal network speed is 10-200 Mbit, but occasionally
we see speeds up to 400 Mbit/s of combined read/write for instance during file repair
Our Problem
Our problem is that for some reason all of the VMs stops responding or responds very slowly and you can for instance not send CTRL-ALT-DEL to a VM in the Hyper-V console, or for instance start task manager when already logged in.
Symptoms (i.e. this happens, or does not happen, at the same time)
I we look at resource monitor on the host then we see that there is often an extensive read from a VHDX of one of the VMs (40-60 Mbyte/s) and a combined write speed to many files in \HarddiskVolume5\System Volume Information\{<someguid and no file extension>}.
See iamge below.
The combined network speed to the iSCSI SAN is about 500-600 Mbit/s.
When this happens it is usually during and after a VSS ShadowCopy backup, but has also happens during hours where no backup should be running (i.e. during daytime when the backup has finished hours ago according to the log files). There is however
not that extensive writes to the backup file that is created on an external hard drive and this does not seem to happen during all backups (we have manually checked a few times, but it is hard to say since this error does not seem leave any traces in event
viewer).
We cannot find any indication that the VMs themself detect any problem and we see no increase of errors (for example storage related errors) in the eventlog inside the VMs.
The QNAP uses about 50% processing Power on all cores.
We see no dropped packets on the switch.
(I have split the image to save horizontal space).
Unable to recreate the problem / find definitive trigger
We have not succeeded in recreating the problem manually by, for instance, running chkdsk or defrag in VM and Hosts, copy and remove large files to VMs, running CPU and Disk intensive operations inside a VM (for instance scan and repair a database file).
Questions
Why does all VMs stop responding and why is there such intensive Read/Writes to the iSCSI SAN?
Could it be anything in our setup that cannot handle all the read/write requests? For instance the iSCSI SAN, the hosts, etc?
What can we do about this? Should we use MultiPath IO instead of NIC teaming to the SAN, limit bandwith to the SAN, etc?

Answer 1

Hi,
> All VMs are using dynamic disks (as recommended by Microsoft).
If this is a testing environment, it’s okay, but if this a production environment, it’s not recommended. Fixed VHDs are recommended for production instead of dynamically expanding or differencing VHDs.
Hyper-V: Dynamic virtual hard disks are not recommended for virtual machines that run server workloads in a production environment
http://technet.microsoft.com/en-us/library/ee941151(v=WS.10).aspx
> This is the primary host and normaly all VMs run on this host.
According to your posting, we know that you have Cluster Shared Volumes in the Hyper-V cluster, but why not distribute your VMs into two Hyper-V hosts.
Use Cluster Shared Volumes in a Windows Server 2012 Failover Cluster
http://technet.microsoft.com/en-us/library/jj612868.aspx
> 2 teamed NIC dedicated to iSCSI storage.
Use Microsoft MultiPath IO (MPIO) to manage multiple paths to iSCSI storage. Microsoft does not support teaming on network adapters that are used to connect to iSCSI-based storage devices. (At least it’s not supported until Windows Server 2008 R2. Although
Windows Server 2012 has built-in network teaming feature, I don’t article which declare that Windows Server 2012 network teaming support iSCSI connection)
Understanding Requirements for Failover Clusters
http://technet.microsoft.com/en-us/library/cc771404.aspx
> I have seen using MPIO suggests using different subnets, is this a requirement for using MPIO
> or is this just a way to make sure that you do not run out of IP adressess?
What I found is: if it is possible, isolate the iSCSI and data networks that reside on the same switch infrastructure through the use of VLANs and separate subnets. Redundant network paths from the server to the storage system via MPIO will maximize availability
and performance. Of course you can set these two NICs in separate subnets, but I don’t think it is necessary.
> Why should it be better to not have dedicated wireing for iSCSI and Management?
It is recommended that the iSCSI SAN network be separated (logically or physically) from the data network workloads. This ‘best practice’ network configuration optimizes performance and reliability.
Check that and modify cluster configuration, monitor it and give us feedback for further troubleshooting.
For more information please refer to following MS articles:
Volume Shadow Copy Service
http://technet.microsoft.com/en-us/library/ee923636(WS.10).aspx
Support for Multipath I/O (MPIO)
http://technet.microsoft.com/en-us/library/cc770294.aspx
Deployments and Tests in an iSCSI SAN
http://technet.microsoft.com/en-US/library/bb649502(v=SQL.90).aspx
Hope this helps!
TechNet Subscriber Support
If you are
TechNet Subscription user and have any feedback on our support quality, please send your feedback
here.
Lawrence
TechNet Community Support