Windows 2008 R2 Failover Cluster - Netapp FAS6280 Black Screen on Restart when in cluster IGROUP

Question

Hello everyone,
I've been working on this issue for a month now and have tickets open with Netapp and Microsoft.  We have not been able to resolve the issue as of yet and since I'm losing all my hair I thought I should post it...
Scenario:
Two Windows 2008 R2 Enterprise SP Fail-Over Clusters (C1 & C2)
Two Netapp FAS6280 Filers for SAN storage
8 DELL R815 servers w/ 2x QLE2562 8GB FC HBAs all paths used (9.1.9.47 - newer did not resolve)
Netapp DSM 4.0 & SnapManager 6.4.2 (we also tried 1 rev newer, 1 rev older without luck)
C1 has 2 nodes & C2 has 4 nodes.
We are trying to add new nodes into the clusters.  We were originally on EMC Clariion arrays and have been migrated to the netapp.  Prior to moving we had 3 nodes in C2 and added node 4.
Now we want to add 2 more nodes, one to each cluster.  We were able to configure the OS and get everything installed and matching the other nodes.  we can assign HBAs to a test IGROUP and assign a lun, format it use it restart all is well and normal. 
As soon as we change the HBAs to the IGROUP for C1 or C2's LUNs the servers don't want to restart properly.  They hang at a black screen.  The little green windows start splash screen runs the bar at the bottom.  it then is supposed to fade
out and come back with the grey screen saying starting windows but that doesn't always happen.  Sometimes it can take 4 hours to restart, or the machine will not come back onine at all.  The pre-existing nodes *do not* experience this issue. 
They restart in 5-10 minutes without issue.
We know drivers are part of the issue, but the maddening part is that it all works fine until we try to add the HBAs to the existing cluster storage group.  We have a suggestion to create separate IGROUPs for each cluster node and add all luns to the
separate igroups.  We configured this on one node and it still restarts long.  We plan to modify the other nodes during our maintenance window and try again.
On the other cluster we were able to successfully complete the validate wizard and join.  everything fails over and works except the server does not come up upon restart.  If I unplug the HBAs it will come up fine, then I can plug them in and the
cluster functions.
Does anyone have any other ideas for us to try?  Currently support for both netapp and Microsoft appear stumped.  Sunday is our Window and we plan to try reconfiguring but I'm losing optimism...
Thanks for any feedback/ideas!
-Ryan

Answer 1

Thanks everyone who looked at this and offered feedback.
The issue appears to be resolved after the configuration change suggested by Netapp support. 
The originally recommended configuration for our storage migration was similar to how I've seen storage groups configured on EMC arrays in the past.  1 storage group / igroup with all hbas/initiators for each host in the WFC/SQL cluster added to this
group.  that group then gets all LUNs assigned to it. 
The change in configuration that was recommended is to have 1 storage group / igroup for each host and put all initiators/hbas for each host in their own separate group.  Then assign the all LUNs to each of the individual host igroups. 
There is a lack of documentation on this, but it appears to work.  We had 1 hit for separate igroups but had to go through a couple of support techs to get one to confirm this.
After applying this configuration in our change window today the issue is resolved.  we were able to fail all resources onto the new node and restart it without issue.  The question remaining for us is why the old systems continued to work while
the new nodes did not.  The old nodes were originally migrated off an EMC and the version of ONTapp at that time was 8.1.2 and they worked fine.
We had upgraded to 8.1.3 to resolve an issue with flash cache and it seems like machines that we tried to attach to this igroup after that time did not work.  all drivers software etc the same.  We are wondering if this is somehow related and have
asked about it.
I wanted to update this thread, and perhaps it can save someone else some trouble down the line.
-Ryan