Guide to migrating a HV 2012 cluster managed by VMM to HV 2012 R2 + VMM R2

Question

Hello everyone,
I have a task on hand and even after searching the ‘net for quite some time, I have not come across a solution listed on one page. I hope this thread becomes a “Guide to migrating a Hyper-V 2012 cluster managed by VMM 2012 to Hyper-V 2012 R2 + VMM 2012
R2”.
My searches did find these useful threads but they do not have all the information I need:
http://social.technet.microsoft.com/Forums/systemcenter/en-US/eb824c20-9632-4ec9-9a71-2d1717e44a81/migrate-from-2012-to-2012-r2-cluster-with-vmm-2012-r2?forum=virtualmachinemgrclustering
http://social.technet.microsoft.com/Forums/windowsserver/en-US/34412c47-3a37-4ead-aa67-f5119382ae9d/upgrade-from-server-2012-clusterhyperv-to-server-2012-r2-clusterhyperv?forum=winserverClustering
http://social.technet.microsoft.com/Forums/systemcenter/en-US/5d4103b5-6491-40d2-b52e-7fffde9580ff/vmm-2012-sp1-supported-to-manage-windows-server-2012-r2-hyperv-cluster?forum=virtualmachingmgrhyperv
So let me start by listing what I have right now:
I have a 4 node cluster running Hyper-V 2012 SP1 (the free, core edition).
It is currently being managed with SC VMM 2012 SP1.
99% of the VMs are running on CSVs cut off a Fiber Channel SAN. I have some VMs on local drives and they are not HA.
Everything is in production. I have around 90 VMs total.
I do have capacity to take up to two HV hosts out of the cluster and still survive just fine.
I also have two other spare servers which have no OS on them.
What I don’t have:
I do not have SCOM or DPM or any other SC component.
Well, I used to have SCOM 2012 but I uninstalled it and removed it. Long story, don’t ask! I do plan to install SCOM 2012 R2 at some point soon.
Here is what I want to do:
Upgrade both the cluster and VMM to 2012 R2 version.
This time, I want to use Server 2012 Datacenter edition (with GUI).
I plan to join the spare servers to the 2012 R2 cluster.
Here is what I have learnt up until now (and correct me if I am wrong):
The steps I need to take to accomplish all this are:
Upgrade VMM 2012 to VMM 2012 R2 version
Eject at least one host from the cluster
Install Server 2012 R2 on that server.
Configure a one server (or more, if possible) cluster running 2012 R2
Give that server the same old CSVs
Use the “Copy cluster roles Wizard” to copy VMs from old cluster to new cluster. However this will incur downtime.
I have not started doing all this because I am not 100% sure on some things.
Here is the list of my questions:
1. About step #1: Upgrade VMM to VMM R2 version
Is there any reliable / official guide out there for this? I have seen these on the net:
http://technet.microsoft.com/en-us/library/dn469607.aspx
http://www.systemcentercentral.com/migrating-from-vmm-2012-sp1-to-vmm-2012-r2-step-by-step-upgrading-to-2012-r2-series/
The problem I see is that none of them talk about upgrading a clustered scenario. Do I assume that the steps of upgrading VMM 2012 to 2012 R2 are the same with or without a cluster?
On the server I eject from the cluster, do I upgrade the OS or do a fresh install? I prefer doing a fresh install.
2. Is there a way of doing all this with no downtime?
I am worried about needing downtime for the last step “copy roles…”. I wish there was a way of migrating the roles (VMs) without any downtime.
3. Anything I need to worry about?
I wonder if I need to be careful about granting access to the CSVs to the new R2 Cluster. Will that cause any problem?
Any inputs on this topic are highly appreciated! I promise to update the first post so that it becomes a guide as I get corrections and inputs from everyone :-)
Thanks!
-Rajeev rajdude.com

Answer 1

Time to tackle the actual Hyper-V Cluster migration:
So by this time, I have two nodes in the old Hyper-V 2012 cluster managed by SC VMM 2012 R2. I wiped out the OS on the other 2 nodes and installed Windows 2012
Datacenter on them. This time I am not installing the Core edition for Hyper-V. After working with Core for a year or so, I feel Core gives way too much trouble for very little gain.
Now I plan to follow a technet document which describes how to
migrate virtual machines from a Hyper-V cluster using Cluster Shared Volumes (CSV). Although this document does not describe VMM I think VMM should be able to use those migrated VMs. Here is this migration strategy in a nutshell:
Create a new cluster using 2012 R2
Present the existing CSVs to the new cluster.
Leave the disk offline on the new cluster.
Use the copy cluster roles wizard in Failover Cluster Manager to copy VMs from old cluster to new cluster.
Virtual machines are created on the new cluster but are not turned on.
Turn off the virtual machines and take the disk offline on the old cluster and then on the new cluster you need to enable the disk and start the virtual machines.
Hmmm…sounds pretty involved to me. Well let’s get on with it. First things first…in order to use this migration strategy we need to create a new cluster.
Here are the steps I took to create a new cluster:
Install 2 Windows 2012 R2 on the same 2 servers I ejected from the old cluster.
Correct drive letters of local hard disk drives and optical drives, if needed.
Update them.
Setup networking. Take care of main network, heartbeat for cluster and iSCSI networks (if applicable). Here are some best practices:
http://blogs.technet.com/b/askcore/archive/2014/02/20/configuring-windows-failover-cluster-networks.aspx
http://blogs.msdn.com/b/clustering/archive/2011/06/17/10176338.aspx
https://social.technet.microsoft.com/wiki/contents/articles/9711.hyper-v-network-design-configuration-and-prioritization-guidance.aspx
http://technet.microsoft.com/en-us/library/dn550728.aspx
Make sure your bindings order is correct. I had to correct mine because by default, the heartbeat network was at the top!
I also followed this checklist of Hyper-v’s best practices.
http://blogs.technet.com/b/askpfeplat/archive/2013/03/10/windows-server-2012-hyper-v-best-practices-in-easy-checklist-form.aspx
Much of the stuff in there is very relevant.
Add the domain account used for host management and the SCVMM service account to the local admin group on the 2 new servers.
Use SC VMM manager to install Hyper-V role on the two new servers. VMM can do that as a part of the “Add Host process”.
In VMM, right click on “All Hosts” and click on “Add Hyper-V hosts and clusters”. Go through the wizard and add the two new servers.
When I did this, it failed. I got three errors, Error 10441, Warning 10444 and warning 26179.
I logged into one of the new servers locally and from Server Manager >> Dashboard >> Add roles and features.
On the “create virtual switch page, I selected the NIC connected to the main company network (as this time I have only one NIC for the company network, you should have at least two).
Unfortunately, even this time I get an error: Access is denied: 0x800700005
This is very strange because I am already logged on as an admin here.
Now I started checking for windows updates. I remembered that the checklist above also has a set of recommended updates. There was a huge 890MB), cumulative update available KB2919355. I installed and restarted after that.
I also installed the feature “Multipath I/O” since I was getting the “warning 26179”.
After this installation, I ran windows update again. No available updates.
Now I went back to VMM. I really want to enable the Hyper-V role using VMM. So I repeated step # 8. Interestingly, this time I did not get any error. VMM rebooted the host automatically. Yay!
Now I repeated steps 14 to 16 on the second server. Worked fine!
At this point we are ready to create a new cluster using the two nodes. I followed this document:
http://technet.microsoft.com/en-us/library/gg610576.aspx
At this point I ran into a problem. I talked about it on my thread: When I try to create a new HV cluster using the "Create HV Cluster" wizard in VMM, it comes to a screen titled "Storage" where it shows all the existing LUNs as disks and
it says "select the discs you want to cluster". That is all good but at the bottom it says "Any data on the disks will be overwritten”. Of course I want to keep my data intact so I can use the copy cluster roles wizard later on! How can I create a new cluster
which uses the same disks as the old cluster WITHOUT wiping out the disks?
Ok so I did not get an answer on the forum, so I decided NOT to take a risk and do a quick test. Here is what I did: I masked/unexported all CSVs. Then I created a test CSV, put some data on it and proceeded to create a cluster using that
CSV and a quorum/witness disk.
This time on the storage page, For the test volume I checked the checkbox under CSV. I left the format boxes unchecked << THIS IS WRONG read below for updates
The attempt failed. Somehow the test CSV got configured as a local disk on host #1, it was showing up as NTFS instead of CSVFS. The VMM cluster creation process stopped. I found out that both the CSV and the quorum disk were online on the
node #1. I made them offline.
Retried , got the same error as last time : Mounts storage disk on (servername).
FINALLY I realized that step # 21 bove is WRONG! You need to select “Do not format” under the file system column. This will leave the CSV untouched.
Unfortunately, even this did not work. So I just took away ALL CSVs and tried to recreate the Cluster. It STILL did not work! LOL! The error is 25303
I corrected the permissions as this:
ON the HV hosts, inside local administrators group add VMM service account, Host admin account and VMM server’s computer account
Inside “Virtual Machine Manager Servers” group add VMM server’s computer account
Unfortunately, it still did not work! The error is 25303 (same). I looked everywhere including all the logs (system and security) nothing there. I searched on the net but never found a solution. Eventually I accepted defeat and tried to
create a cluster from outside VMM…using “Failover cluster manager”
Even then I got an error! It seems to me that it failed on creating the computer name of the cluster in AD.
I tried it again with a different computer name in AD…something which had never been created earlier (I had mistakenly created a computer name for the cluster and then deleted it). This time it worked! However I got an error that the witness
disk was not created because it does not have an acceptable file system. Hmmmm.,….I had not formatted the witness disk since I thought the cluster wizard will do it. I created a Witness disk later, manually.)
Interestingly, when I opened VMM again, I see the cluster in there and I am able to manage the cluster from within VMM…interesting!
-Rajeev rajdude.com