Upgrading a 3-node Hyper-V clusters storage for £10k and getting the most bang for our money.

Question

Hi all, looking for some discussion and advice on a few questions I have regarding storage for our next cluster upgrade cycle.
Our current system for a bit of background:
3x Clustered Hyper-V Servers running Server 2008 R2 (72TB Ram, dual cpu etc...)
1x Dell MD3220i iSCSI with dual 1GB connections to each server (24x 146GB 15k SAS drives in RAID 10) - Tier 1 storage
1x Dell MD1200 Expansion Array with 12x 2TB 7.2K drives in RAID 10 - Tier 2 storage, large vm's, files etc...
~25 VM's running all manner of workloads, SQL, Exchange, WSUS, Linux web servers etc....
1x DPM 2012 SP1 Backup server with its own storage.
Reasons for upgrading:
Storage though put is becoming an issue as we only get around 125MB/s over the dual 1GB iSCSI connections to each physical server.  (tried everything under the sun to improve bandwidth but I suspect the MD3220i Raid is the bottleneck here.
Backup times for vm's (once every night) is now in the 5-6 hours range.
Storage performance during backups and large file syncronisations (DPM)
Tier 1 storage is running out of capacity and we would like to build in more IOPS for future expansion.
Tier 2 storage is massively underused (6tb of 12tb Raid 10 space)
Migrating to 10GB server links.
Total budget for the upgrade is in the region of £10k so I have to make sure we get absolutely the most bang for our buck.  
Current Plan:
Upgrade the cluster to Server 2012 R2
Install a dual port 10GB NIC team in each server and virtualize cluster, live migration, vm and management traffic (with QoS of course)
Purchase a new JBOD SAS array and leverage the new Storage Spaces and SSD caching/tiering capabilities.  Use our existing 2TB drives for capacity and purchase sufficient SSD's to replace the 15k SAS disks.
On to the questions:
Is it supported to use storage spaces directly connected to a Hyper-V cluster?  I have seen that for our setup we are on the verge of requiring a separate SOFS for storage but the extra costs and complexity are out of our reach. (RDMA, extra 10GB NIC's
etc...)
When using a storage space in a cluster, I have seen various articles suggesting that each csv will be active/passive within the cluster.  Causing redirected IO for all cluster nodes not currently active?
If CSV's are active/passive its suggested that you should have a csv for each node in your cluster?  How in production do you balance vm's accross 3 CSV's without manually moving them to keep 1/3 of load on each csv?  Ideally I would like just
a single CSV active/active for all vm's to sit on.  (ease of management etc...)
If the CSV is active/active am I correct in assuming that DPM will backup vm's without causing any re-directed IO?
Will DPM backups of VM's be incremental in terms of data transferred from the cluster to the backup server?
Thanks in advance for anyone who can be bothered to read through all that and help me out!  I'm sure there are more questions I've forgotten but those will certainly get us started.
Also lastly, does anyone else have a better suggestion for how we should proceed?
Thanks

Answer 1

Current Plan:
Upgrade the cluster to Server 2012 R2
Install a dual port 10GB NIC team in each server and virtualize cluster, live migration, vm and management traffic (with QoS of course)
Purchase a new JBOD SAS array and leverage the new Storage Spaces and SSD caching/tiering capabilities.  Use our existing 2TB drives for capacity and purchase sufficient SSD's to replace the 15k SAS disks.
On to the questions:
Is it supported to use storage spaces directly connected to a Hyper-V cluster?  I have seen that for our setup we are on the verge of requiring a separate SOFS for storage but the extra costs and complexity are out of our reach. (RDMA, extra 10GB NIC's
etc...)
When using a storage space in a cluster, I have seen various articles suggesting that each csv will be active/passive within the cluster.  Causing redirected IO for all cluster nodes not currently active?
If CSV's are active/passive its suggested that you should have a csv for each node in your cluster?  How in production do you balance vm's accross 3 CSV's without manually moving them to keep 1/3 of load on each csv?  Ideally I would like just
a single CSV active/active for all vm's to sit on.  (ease of management etc...)
If the CSV is active/active am I correct in assuming that DPM will backup vm's without causing any re-directed IO?
Will DPM backups of VM's be incremental in terms of data transferred from the cluster to the backup server?
Thanks in advance for anyone who can be bothered to read through all that and help me out!  I'm sure there are more questions I've forgotten but those will certainly get us started.
Also lastly, does anyone else have a better suggestion for how we should proceed?
Thanks
1) You can use direct connection to SAS with a 3-node cluster of course (4-node, 5-node etc). Sure it would be much faster then running with an additional SoFS layer (with SAS fed directly to your Hyper-V cluster nodes all reads and writes would be local
travelling down the SAS fabric and with SoFS layer added you'll have the same amount of I/Os targeting SAS + Ethernet with its huge compared to SAS latency sitting in between a requestor and your data residing on SAS spindles, I/Os overwrapped into SMB-over-TCP-over-IP-over-Etherent
requests at the hypervisor-SoFS layers). Reason why SoFS is recommended - final SoFS-based solution would be cheaper as SAS-only is a pain to scale beyond basic 2-node configs. Instead of getting SAS switches, adding redundant SAS controllers to every hypervisor
node and / or looking for expensive multi-port SAS JBODs you'll have a pair (at least) of SoFS boxes doing a file level proxy in front of a SAS-controlled back end. So you'll compromise performance in favor of cost. See:
http://davidzi.com/windows-server-2012/hyper-v-and-scale-out-file-cluster-home-lab-design/
Used interconnect diagram within this design would actually scale beyond 2 hosts. But you'll have to get a SAS switch (actually at least two of them for redundancy as you don't want any component to become a single point of failure, don't you?)
2) With 2012 R2 all I/O from a multiple hypervisor nodes is done thru the storage fabric (in your case that's SAS) and only metadata updates would be done thru the coordinator node and using Ethernet connectivity. Redirected I/O would be used in a two cases
only a) no SAS connectivity from the hypervisor node (but Ethernet one is still present) and b) broken-by-implementation backup software would keep access to CSV using snapshot mechanism for too long. In a nutshell: you'll be fine :) See for references:
http://www.petri.co.il/redirected-io-windows-server-2012r2-cluster-shared-volumes.htm
http://www.aidanfinn.com/?p=12844
3) These are independent things. CSV is not active/passive (see 2) so basically with an interconnection design you'll be using there's virtually no point to having one-CSV-per-hypervisor. There are cases when you'd still probably do this. For example if
you'd have all-flash and combined spindle/flash LUNs and you know for sure you want some VMs to sit on flash and others (no so I/O hungry) to stay within "spinning rust". One more case is many-node cluster. With it multiple nodes basically fight for a single
LUN and a lot of time is wasted for SCSI reservation conflicts resove (ODX has no reservation offload like VAAI has so even if ODX is present its not going to help). Again it's a place where SoFS "helps" as having intermediate proxy level turns block I/O into
file I/O triggering SCSI reservation conflicts for a two SoFS nodes only instead of an evey node in a hypervisor cluster. One more good example is when you'll have a mix of a local I/O (SAS) and Ethernet with a Virtual SAN products. Virtual SAN runs directly
as part of the hypervisor and emulates high performance SAN using cheap DAS. To increase performance it DOES make sense to create a  concept of a "local LUN" (and thus "local CSV") as reads targeting this LUN/CSV would be passed down the local storage
stack instead of hitting the wire (Ethernet) and going to partner hypervisor nodes to fetch the VM data. See:
http://www.starwindsoftware.com/starwind-native-san-on-two-physical-servers
http://www.starwindsoftware.com/sw-configuring-ha-shared-storage-on-scale-out-file-servers
(feeding basically DAS to Hyper-V and SoFS to avoid expensive SAS JBOD and SAS spindles). The same thing as VMware is doing with their VSAN on vSphere. But again that's NOT your case so it DOES NOT make sense to keep many CSVs with only 3 nodes present or
SoFS possibly used. 
4) DPM is going to put your cluster in redirected mode for a very short period of time. Microsoft says NEVER. See:
http://technet.microsoft.com/en-us/library/hh758090.aspx
Direct and Redirect I/O
Each Hyper-V host has a direct path (direct I/O) to the CSV storage Logical Unit Number (LUN). However, in Windows Server 2008 R2 there are a couple of limitations:
For some actions, including DPM backup, the CSV coordinator takes control of the volume and uses redirected instead of direct I/O. With redirection, storage operations are no longer through a host’s direct SAN connection, but are instead routed
through the CSV coordinator. This has a direct impact on performance.
CSV backup is serialized, so that only one virtual machine on a CSV is backed up at a time.
In Windows Server 2012, these limitations were removed:
Redirection is no longer used. 
CSV backup is now parallel and not serialized.
5) Yes, VSS and CBT would be used so data would be incremental after first initial "seed" backup. See:
http://technet.microsoft.com/en-us/library/ff399619.aspx
http://itsalllegit.wordpress.com/2013/08/05/dpm-2012-sp1-manually-copy-large-volume-to-secondary-dpm-server/
I'd look at some other options. There are few good discussion you may want to read. See:
http://arstechnica.com/civis/viewtopic.php?f=10&t=1209963
http://community.spiceworks.com/topic/316868-server-2012-2-node-cluster-without-san
Good luck :)
StarWind iSCSI SAN & NAS