Very Strange Network Issue With Two Guests on 2012 R2 Hyper-V Failover Cluster

Hi all. We're having a odd issue with two guests on our 2012 R2 failover cluster.
In a nutshell, if we shutdown a particular server (I'll call it Server A) another totally different server (Server B) on the same node loses it's network connectivity to the domain. If we start server A back up, network connectivity returns on server B.
At first I thought server A might be running a service that was somehow linked to server B, so I decided to disable server A's NIC. Interestingly, that had no affect on server B's connectivity.
The next step I tried was pausing server A and again, no adverse affect on server B's connectivity.
Next step was to live migrate server A to another node. This action did
cause server B to lose its network connection.
One other clue is that if I ping server B from either of the Hyper-V hosts in the cluster, I never lose network connection to server B.
So I would suspect this is some network issue on the cluster, but I'm kind of at a loss where to go from here.
Has anyone seen this behavior before or does anyone have any troubleshooting suggestions I can try?
Thanks!
George Moore

Hi Sir,
I'v never seen this before .
>>Next step was to live migrate server A to another node. This action did
cause server B to lose its network connection.
They are connecting to same virtual switch ?
First please run cluster validation to check if there is any error .
If it is ok , please try the following items for troubleshooting :
1. shutdown serverA serverB
2. then add another virtual NIC for serverB
3. start server B check if the issue happens to both "old" and "new" virtual NIC .
In addition , you can live migrate both A and B to another node , then try to live migrate A to the original node .
If the issue persists , I would suggest you to remove that virtual switch on both nodes then re-create them .
Best Regards,
Elton Ji
If it is not the answer please unmark it to continue
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected] .

Similar Messages

Strange Networking Issue with OES

Since moving our entire infrastructure from Netware to OES, I've been trying to figure out a very strange issue. We are a K-12 school system with a data center at the central offices and a single server at each school. The data center has the servers that run all the centralized stuff, including the Master replicas for [root], the DA's for SLP, etc. We noticed that sometimes a school's server will drop out of the SLP tables. When that happens, we also cannot SSH to the school's server from the central servers. We can SSH to it from any other machine on the same subnet as the central servers. The central servers can ping it, but no UDP/TCP traffic can pass between them (no web traffic, etc). We finally realized it only happens when the ISP that serves that school has a brief blip or outage of some sort. The only way we have found to get the servers talking again is to reboot the central servers.
We have tried:
Restarting almost all services on central servers
Stopping and restarting networking
Switching to runlevel 1 and then back
Stopping firewall and/or shutting it off completely
Checking hosts.deny files
Disabling virus scanner (McAfee Enterprise)
Changing vmware network drivers from E1000 (known issue with vSphere 5.1) to VMXNET3
All servers run on HP Proliant servers (blades at central office, towers at schools) running vSphere 5.1. Links between the central office and schools are all 100Mbps fiber via two different ISP (depending on the school's location). A restart of the central office server fixes it every time. I am out of ideas now and at a loss as to why this happens when a WAN link goes down. We thought it was just isolated to one ISP, but it was just that that particular ISP is not quite as reliable as the other. We had an extended power outage at a school served by another ISP over the weekend and saw the same issue with that server after power was restored.
Any ideas or suggestions at where else we should be looking? We also run ZCM 11 at each school and the central office. When this happens, those servers never lose connectivity with each other and with the OES servers on each end. The ZCM servers run on the same vSphere host as the OES servers.

On 04/08/2014 12:26 PM, jmlester wrote:
>
> Since moving our entire infrastructure from Netware to OES, I've been
> trying to figure out a very strange issue. We are a K-12 school system
> with a data center at the central offices and a single server at each
> school. The data center has the servers that run all the centralized
> stuff, including the Master replicas for [root], the DA's for SLP, etc.
> We noticed that sometimes a school's server will drop out of the SLP
> tables. When that happens, we also cannot SSH to the school's server
To be clear, while i think this correlation is possible, I do not see any
possibility of causality in here unless your SSH client is somehow using
SLP to resolve the SSH server, which seems very unlikely. I'm not sure if
you're trying to state that SLP is the cause of the SSH stuff, but I do
not think you are.
> from the central servers. We can SSH to it from any other machine on
> the same subnet as the central servers. The central servers can ping
> it, but no UDP/TCP traffic can pass between them (no web traffic, etc).
Hmmm........... so routing works, and the server is even responding to
requests made to it (presumably, or something else is), but SSH is not
responding, or something is preventing it from responding.
> We finally realized it only happens when the ISP that serves that school
> has a brief blip or outage of some sort. The only way we have found to
> get the servers talking again is to reboot the central servers.
>
> We have tried:
>
> - Restarting almost all services on central servers
> - Stopping and restarting networking
> - Switching to runlevel 1 and then back
> - Stopping firewall and/or shutting it off completely
> - Checking hosts.deny files
> - Disabling virus scanner (McAfee Enterprise)
> - Changing vmware network drivers from E1000 (known issue with vSphere
> 5.1) to VMXNET3
>
>
> All servers run on HP Proliant servers (blades at central office, towers
> at schools) running vSphere 5.1. Links between the central office and
> schools are all 100Mbps fiber via two different ISP (depending on the
> school's location). A restart of the central office server fixes it
> every time. I am out of ideas now and at a loss as to why this happens
> when a WAN link goes down. We thought it was just isolated to one ISP,
> but it was just that that particular ISP is not quite as reliable as the
> other. We had an extended power outage at a school served by another
> ISP over the weekend and saw the same issue with that server after power
> was restored.
>
> Any ideas or suggestions at where else we should be looking? We also
> run ZCM 11 at each school and the central office. When this happens,
> those servers never lose connectivity with each other and with the OES
> servers on each end. The ZCM servers run on the same vSphere host as
> the OES servers.
I'd get a LAN trace from both sides at the same time. At the very least,
run tcpdump as shown below on both the central office machine as well as
the remote machine to see how the packets differ when trying different
things. If you do a test and do not see output on both sides, do it again
writing it to a file and post somewhere for us to review. Note: since
it's an SSH problem I would either NOT SSH into the system, or I'd be sure
to limit the tcpdump capture/output to something specific to the
connection being tested. Tracing something over which you are sending the
tracing data is never fun since every packet picked up causes another
packet to be generated, to be picked up, to be generated.... ad infinitum.
To see tcpdump output with limited details on the screen:
Code:
sudo /usr/sbin/tcpdump -n -s 0 -i any -vv
To write tcpdump output to a file:
Code:
sudo /usr/sbin/tcpdump -n -s 0 -i any -w /tmp/problem-20140408.cap
While filtering when capturing is typically not desired, you can do it by
adding something like 'host 10.1.2.3' to the list of tcpdump parameters at
the end of either of those two commands. The reason filtering during
capturing is not desired is that you may miss something.... the SLP
traffic, DNS, ARP, ICMP error showing the exact problem, etc.
Good luck.
If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below...

Strange Printing Issue With New HP & Pages

Hello!
I have a very strange printing issue with a new HP printer. I have written pages documents heavy in Texts which are located within tables. The documents print fine on a Cannon and Epson printer and look great but when I print the document on my new HP Laserjet Pro 400 Colour M451nw some of the text on some rows come out of the printer looking like they are bold / slightly fuzzy. This only happens with some of the rows of text, not all of them. The font is the same in every row with the same format.
You can see this in the attached image. The first line (Thanks very much) has printed correctly but the subsequent lines look bold / slightly fuzzy.
After a lot of testing the only way I can resolve this is if I copy the content into a new pages document and only then it will print and look ok with the HP printer.
This must be a problem within my origional document. I would simply copy and paste everything into a new document however this would take a very, very long time (over 5 years of work producing teaching materials and over 100 documents and thousands of tables and formatting that cannot be copied and pasted).
Also, on other printers you can see the lines seperating each row of the table. These lines are not visible on the HP print outs, even when the table is copied into a new document.
Has anyone come accross an issue like this before? Is there a way I can fix this without creating a new document? Do I need to get a new another new printer (the 4th printer this week!)?
Any advise, wise words would be greatly apprechiated!
Thanks!

Sorry, Fruhulda. After 4 hours on the phone with Apple care they are taking this issue further themselves. It seems to be a problem with our pages document and the compatibility with certain drivers. We have still not gotton to the bottom of it and Apple are going to get back to me when they have an answer.
We also worked out that the issue only happens when you print via Ethrenet & USB. Printing via Wifi is ok and there are no quality issues.
Very Strange!

Very strange wifi happenings with my iPhone 5

Here's what's up:
I have an iPhone 5 and it is updated to the latest iOS release. The WiFi connection does something very strange.
There are two saved networks in the phone. One is a hidden wifi network from work (a public school) that also uses an auto proxy (a URL address). The other is my home wifi. I have Verizon Fios.
When only my home network is saved, the wifi at home works perfectly. However, when both networks are saved, the home wifi pretty much doesn't work. It seems to constantly disconnect and connect with the network; web pages do not load, etc. At work, with both networks saved in the phone, it works perfectly.
For the life of me I cannot figure out what is wrong. Now, to make matters even more mysterious, I have an iPad2 with the exact same network settings and with both networks saved in that, it works perfectly no matter where I go- home or work.
Please, does anyone have any ideas as to what is messed up?

https://www.youtube.com/watch?v=iVQcFIHWbQ8
here is a fix i have followed some of the instructions and i am now back at 27down 13 up

Really strange network issue that is specific to OS X 10.6.8

Really strange network issue that is specific to OS X 10.6.8 on both 2 iMacs and an older intel Mac Pro.
Any system running 10.6.8 is achieving on average 15 –20 mb a sec transfer rates regardless of drive speeds, cables, network activity etc.. I've spent some time removing any cable , drive speed, router issues to the point where one machine dual boots into both 10.6.8 and 10.6.5
The 10.6.5 system has network speeds of 60mb – 80mb p/sec transfer rates.
Our older 10.5.8 PPC mac gets 80mb and my 2011 Macbook Pro 10.7.2 hits 60mb – 80mb consistently as well.
Any idea what the cause is?
Cheers

Since your other Mac running 10.5.8 is not affected, you know that your AirPort router is functioning correctly.
What changed on your Mac? This seems like a post for either the Mac OS X v10.6 Snow Leopard support area, or the support area for your model of Mac.
We normally handle questions about the AirPort Express, AirPort Extreme or Time Capsule in this area, but you can certainly wait to see if you receive a response here.

Heartbeat Network inside Hyper-V Failover Cluster

Dear All,
I need to build an SQL Server 2012 SP1 Two-node cluster inside Hyper-V Failover Cluster. There is only one virtual switch in the Hyper-V Failover Cluster environment which is being
used for communication to the outside world using VLAN Tagging. Now, since SQL Server 2012 Failover Cluster would require a heartbeat network as well and although possible to assign VLAN tag to the heartbeat adapter and use that for hearbeat, but since there
would not be any gateway on the heartbeat network, it would render the VLAN tagging useless, so is following plan good enough:
1. Create an Internal Virtual Switch on all nodes of the cluster with the same name
2. Link the heartbeat adapter of the virtual machines to the Internal Virtual Switch
Is it good enough? or is there any other better way?
Thanks in advance.

I am not an expert on this, but here is what I would do.
Create the VLAN and use that to keep the network setup as easy to understand as possible. Our Hyper-V cluster is only using IPv6 on the heartbeat network, and that is working like a charm. I would do the same inside the hyper-v hosts if I need to builld
a virtual cluster.
What is the reason for deploying a failover cluster for SQL inside Hyper-V? Wouldn't a log shipping "cluster" provide a more secure solution for your SQL?
/Martin
Exchange is a passion not just a collaboration software.

Hyper-V Failover Cluster Networking Configuration After Install

Hello All,
Is it possible to install hyper-v and failover or in other words create a hyper-v failover cluster and then configure the networking part of the solution later? As I am coming into
terms with the networking part of it, wanted to do it later after the install. Is it possible?
And from later configuration, I am trying to say, creation of NIC Team, Virtual NICs, VLAN tagging, etc.

Hi,
Failover cluster deployment requires network connectivity between cluster nodes. You can't create a cluster without properly configured TCP\IP on cluster nodes.
http://OpsMgr.ru/

Guest cluster on Hyper-V failover cluster

I want to implement Hyper-V failover cluster on Windows Server 2012 R2. Then Install Windows 2003 R2 as Hyper-V guest OS.
Do you know whether Windows 2003 R2 guest cluster support on Hyper-V on Windows Server 2012 R2?

Hi steventsf,
Windows Server 2003 R2 with Service Pack 2 (SP2) is in the supported guest operation system list :
http://technet.microsoft.com/library/hh831531.aspx
The guest OS is independent of hyper-v host , I think windows server 2003 R2 SP2 guest cluster is a supported scenario .
Best Regards
Elton Ji
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

Hyper-V Failover Cluster virtual guests suddenly reboots

The environment is Server 2012 R2 using dual clusters--a Hyper-V Failover Cluster running guest application virtual machines and a Scale-Out File Server Cluster using Tiered Storage Spaces which are used to supply SMB3 shares
for Quorum and CSV. Has anyone had this problem?

Anything relevant in the host or guest event logs? I would also check the cluster event logs to see if there are any indications there as well.
Does the guest go down hard or gracefully reboot?
Need more info.
Andy Syrewicze
Come talk more about Hyper-V and the Microsoft Server Stack at
Syrewiczeit.com and the Altaro Hyper-V Hub!
Post are my own and in no way reflect the views of my employer or any other entity in which I produce technical content for.

Issues with two time capsules in one network

I have two time machines in my environment and issues with dropped connections.
As long as I use only my iMac or Apple MacBook Pro the connections work 90% of the time, I do have some issues with dropped (afp/smb) connections and unable to reconnect every now and then.
However, invariably when I connect a windows machine to the network both time machine connections just die, including the internet connection, I had to restart one of the time machines in order to reconnect. I do have a Guest Network and connecting the windows machine to that network does not affect the time machines availability.
Is having two time machines in the environment not supported?
Why do I experience these drops in connections to the online disks?
Why can't I have a mixed environment (win/mac) connect to the time machines at the same time?
One time machine is the AP and the other one is just extending that network.
I have seen issues in the following scenarios
Copying large amounts of data from iMac to time machine
Copying large amounts of data from one time machine to the other
Connecting a windows computer to the network
Any ideas?

Yes

Strange performance issue with 3510/3511 SAM-FS disk cache

Hi there!
I'm running a small SAM-QFS environment and have some strange performance issue on the disk storage part, which somebody here might be able to explain.
Configuration: one 3510, dual controller, RAID-5 9+1, one hot spare and one disk not configured for whatever reason. The R5 logical drive hosts a 150GB LUN for SAM-QFS metadata (mm in SAM-FS speak) and a 1TB LUN for data (mr in SAM-FS speak). Further, there are two small LUNs (2GB, 100GB) for some other purpose. Those two LUNs have nearly no I/O. All disks are SUN146G. Host connection is 2GBit, multipathing enabled and working.
Then the disk cache became too small, and the customer added a 3511 expansion unit with SUN300G disks. One logical drive is a RAID-1, 1+1, used for NetBackup catalog. The other is a RAID-5, 8+1, providing two LUNs: 260GB SAM-FS metadata (mm) and 1.999TB SAM-FS data (mr).
For SAM-FS, the LUNs form two file systems: one "residing" in the 3510, the other "residing" in the 3511 expansion. Cabling is according to the manual and checked several times by several independant people. Operating system is Solaris 10, hardware is a V880.
The problem we observe: SAM-FS I/O on LUNs on disks inside the 3510 is fine. With iostat, I see 100MB/s read and 50MB/s write at the same time. On the SAM-FS file system which is running on the two LUNs in the 3511, the limit seems to be at 40MB/s read/write. Both SAM-FS file systems are configured the same in regards of block size.
In case I have activity on both SAM-FS file systems, I see 100MB/s+ on the LUN running inside the controller shelf and another 40MB/s on the disk runnin in the 3511 expansion chassis. So, the controller is easily capable of handling 150MB/s.
Cache settings in the 3510 controller are default I think (wasn't installed by me), batteries are fine.
Is this 40MB/s we experience a limitation by the expansion shelf? Don't think so. Anybody has any ideas on this? What parameters to check or to change? Any hint appreciated. I can also provide further details if needed. Thank you.
wolfgang

SUN300G disks sound like 300GB FC disks.
Depending on how many files are in the SAMFS file system, sharing the mm and mr devices on the same RAID array can be a pretty horrible idea. In my opinion and experience, it's almost always better to NEVER put more than one LUN on a RAID array. Period. Putting more than one LUN on an array results in IO contention on that array. And large, unnaturally configured (9+1? Why?) RAID arrays will have problems from the start.
What are the block sizes used on the RAID arrays? It wouldn't surprise me to see that the RAID array on the expansion tray has a very large block size. Larger block sizes are, in general, not better. Especially for SAMFS metadata - which IIRC is something like 8k or 16k blocks.
I suspect what is happening is most of the metadata updates are going to the mm device on the new array, contending with the IO operations on the file data.
How much space is left on each mm device? What does "iostat -sndxz 2" show when you're having the IO problems?

Very slow network performance with Intel NIC when TCP Large Send Offload is enabled

I have a Windows 2008 server with two NIC's (Intel PRO/1000 PL and Intel PRO/1000 PM), and one virtual external switch connected to each NIC.
With the 01/14/08 network driver, I get very low throughput on both NIC's (~10KB/sec when downloading from my server when it should be around 11MB/sec).
I've found that disabling "TCP Large Send Offload (IPv4)" on the virtual switch and "Large Send Offload v2 (IPv4)" on the physical adapters solves the problem.
Is this a bug in the driver?
Thank you,
Ricardo Costa

2 years later, I was able to get that machine to work with Hyper-V under Windows Server 2008 R2, but only after disabling everything that had anything to do with TCP or IP offloading. I think that I had to turn off 4 settings on one controller,
and 5 settings on the other controller. Once that was done, everything worked flawlessly. It is very apparent that the Virtual Controllers do not properly use many of the features of non-Intel devices (not all Intel devices work either), and
that Hyper-V does not completely uninstall if you remove it.
If you have any problems, start turning off the features that your adapter supports, until it starts working. You'll need to weight the cost of replacing the controller with a fully supported controller, of which there is no list, or using your existing
controller at a capacity lower than it physically supports. This has been an elusive issue with Microsoft's Virtual Server projects since they were first in beta, so don't expect a fix anytime soon.
P.S. For those who don't know, the settings are on the actual device and can be changed in the Device Manager. If you are still having trouble after turning off all the settings, then you will need to replace the controller(s) with a different
make/model.
Bill Bosacker's Blog @ http://www.openSourceC.org/

Network issue with initial connections on Hyper-V

This is the opposite of all the other networking issues I have seen with Hyper-V- anyone run into this?
I am running Hyper-V on a Server Core installation, and have had the same issue when I used Server 2012R2 as when I used Server 2008R2 on the same server as a virtual host. The server is an HP ProLiant DL360 Gen8 with 2x 8-core processors, 32 GB RAM. The
VM is sized and configured to stay within 1 NUMA node. It uses a single port of the 4-port HP/Broadcom 331 NIC, with all the TCP LSO settings disable at the switch, host, and guest, same with the power management settings. We have ProxyARP on the network,
so I have set the ArpRetryCount key to 0.
I have seen it happening with and without SR-IOV enabled. The guest OS is an RDS server, so for the sake of our CAL's it has to stay at Server 2008R2. I have set up a HOSTS file to mitigate this problem, as it seems to be a DNS issue- Here's what happens:
The first time I visit a resource, the server will timeout. An example not in our HOSTS file does this consistently: If I go to https://mail.mydomain.com/OutlookWebAccess, it spins for a couple of minutes before timing out. I hit refresh and the page loads
immediately. This server is colocated in-house, as is the mailserver. I'd say it is a DNS issue at some level, but I'd like to know what level it is- Is it just timing out trying to cache the DNS request, or is there a performance issue with a DNS server on
the network? (All of our DNS is AD-integrated running on DC's).
Here is where I'm seeing this:
Internet Explorer- described above
Salient Interactive Miner- at the login screen, it searches for a database and times out. If I go to the setup and enter the IP address, it still times out. It will not find the database server until I quit and restart the program. If this is the same issue,
it would obviously not be a DNS problem.
I DON'T see it in GP9 connecting to a datasource
Prior to setting up the HOSTS file, I saw this in Windows Explorer connecting to network shares on the same LAN segment.
This appears to only happen to the Hyper-V guest OS. As a best practice, whenever Windows Update runs, I update the Integration Components.
I have the bindings properly ordered on the guest VM, with the network adapter the goes to the physical NIC first, and IPv4 ordered before IPv6.
All updates are isntalled, and the firmware is at it's latest revision.
Any ideas?

Hi Daniel,
" This appears to only happen to the Hyper-V guest OS. "
Do you mean that any other hosts can access the network resource normally except these VMs ?
If yes , my suggestion is to disable the VMQ on the physical NIC for the VMs , then try test again .
Best Regards
Elton Ji
If it is not the answer please unmark it to continue
We
are trying to better understand customer views on social support experience, so your participation in this
interview project would be greatly appreciated if you have time.
Thanks for helping make community forums a great place.

Network Issues with Visual Studio Online Application Insgihts

My team is using Visual Studio Application Insights to monitor availability of our web server (we use cloud service)by making a call to it every 5 minutes from different locations around the world including Singapore, Japan, France and Switzerland.
We experienced continuous failures of this monitoring from Singapore from September, 19-th and the issue became very big on September 21-st, September, 22-nd. After that on September, 23-rd and 24-th we have failures form Paris. Then starting from September,
24-th to September, 26 we have very high percent of failures from Japan. Our problems finished with one more failure coming from Switzerland on September, 26.
I wanted to find what the source of the problem is since we checked our code and it very simple to cause network issues. However, it is very difficult to reproduce the issue. My questions is - were all these problems coming from Visual Studio Online Application
Insights?
One example error that it is tracked in Visual Studio Online Application Insights is:
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because
connected host has failed to respond 138.91.9.51:8080
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.VisualStudio.TestTools.WebStress.WebTestTransaction.ResponseReceived(IAsyncResult result)
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
Thank you!
Kanio

Hello Kanio,
There is nothing AI-specific in the stack, and, since the exception was reported to the portal, I would say that AI was working that time. However, if the issues goes away as soon as you disable AI - then it might mean that the issue is induced by AI presence.
Also, you can try the latest AI SDK (0.13 as of today) and see if the issue still happens.
Dmitry Matveev

Network issues with Apple products?

I've got a weird issue with my Apple products connecting to certain networks. Two different instances:
1) My parents house, my iPhone, iPad, and MacBook Pro all have issues in contacting certain sites and apps. Things from google or apple servers wont work. Some things do, if I VPN to an outside network, everything works perfectly fine. I haven't been able to figure out what the root of the issue is. I've switched routers and contacted the network provider but they said it wasn't there fault. This has been going on for a bit. All non-Apple products seem to be fine.
2) My office, I know even less details about the network here, but I'm having a similar issue where my Apple devices get connectivity for a bit then stop and then come back later. If I VPN again, everything works perfectly fine. All non-Apple products work fine.
Anyone have any idea as to what this could be from?

I am having the same problem since upgrading to Snow Leopard. Really need an update from Apple or a solution. Been reading a lot of posts on several discussion boards but nobody has come up with a solution yet. Apple really needs to send us a fix for this. I understand it is also hapenning in Lion.

Very Strange Network Issue With Two Guests on 2012 R2 Hyper-V Failover Cluster

Similar Messages

Maybe you are looking for