Weird Networking problem with Windows 2008 R2 VMs

[taylorb]

I am not 100% sure this is VMware related, but I am having this weird networking problem with my 2008 R2 VMs.    On VMs with a static IP, about every 2nd or 3rd time I reboot them, they come up with no network connectivity and a 169.x.x.x autoconfig address.   Rebooting the VM brings back the static IP and full network function.  If it was just a network connectivity or driver issue, you'd think my IP would still be there, so I am baffled. Mostly the fact that a reboot breaks it and then a reboot fixes it is what is perplexing.  It would make more sense if it just stayed broke and then I could troubleshoot.     I even totally redid my VMware image and newly deployed VMs have this issue, too.   So either I am systemically doing something wrong, or I have a weird bug.   Patching is a nightmare because of this, as you can imagine. 

I thought maybe this KB article would have something to do with it:Virtual machine is assigned an invalid IP address after a reboot

But it mentions "The guest operating system network interface with a static IP address fails to communicate with the network until the IP address is changed to DHCP and then back to a static IP address".   These work as soon as I reboot, every time.  Also, I have 200 VMs and the problem is random so I can't very well fix every single one with a unique registry key (well I'd rather not). 

Any ideas? 

[WasimShaikh]

All 200 VMs are newly deployed from same master Template? does this issue arises on VM which is not created from Template? what is the Network card type selected for VM (E1000, VMXNET2-3), all VMs connected to same vSwitch / vDS portgroup?

[admin]

Please try the steps mentioned in the Below link

Windows Server 2008 Failover Clusters: Networking (Part 4) - Ask the Core Team - Site Home - TechNet Blog

http://blogs.technet.com/b/askcore/archive/2010/04/15/windows-server-2008-failover-clusters-networking-part-4.aspx

s

[Dany Demers]

I have the same problem VM randomly loose their IP address after reboot, rebooting the VM again fix the problem. Most VM where deployed from template. I looked at the same VMware KB mentioned in the original post VMware KB: Virtual machine is assigned an invalid IP address after a reboot  but the GUID of the network card in the registry is displaying the good information, there is no GUID with the autoconfigured IP. Vm are all 2008r2 with vmxnet3 VM adapter running virtual machine version 8 on Esx 5.5 with vmware tool 9.4.6. These server are not part of any cluster so the post about cluster is not relevant in my case. Anyone else having the same issue ?

[Dany Demers]

More info, when I do start -> cmd -> ipconfig /all

I see that the assigned static IP address is marked as duplicate hence why the server goes into autoconfiguration IP. I found this VMware article http://kb.vmware.com/kb/1028373 that explain that this can be caused by gratuitous ARP by physical Cisco Swith or ASA firewall. We do have cisco swith so we are investigating on this issue.

[JPM300]

Hey DanyDemers,

I think your on the write track with your latest KB find, however just out of curiousity was the Windows Server 2008 R2 systems that are having the IP problems created fresh in VMware or where they P2V'ed from physical?

[Dany Demers]

All of the VM are built from a template and the template was a fresh install in VMware. I'm still discussing with the networking team to see what we can do about this gratuitous ARP.

[JPM300]

Just out of curiousity if you check your VM's that was deployed with that template with this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1179

Essentially in a command prompt that is running as administrator run

set devmgr_show_nonpresent_devices=1

devmgmt.msc

and show hidden devices

Look at your nics and see if there is any grayed out legacy nics that may have been added during the template creation, it shouldn't cause any issues if there is but you never know.  Just something I figured I would bring up.

Let us know if you get it resloved with the KB your currently working on.

Cheers

[Dany Demers]

Hi, I'm familiar with these command as I used them to remove floppy to VM that were created before my template :smileyhappy: and I had already ruled that out as there are no greyed out network adapter. Also my VM doesn't return any message about IP already configured on another adapter.

[Dany Demers]

Latest update on what we found. Using windows 2012r2 VM to reproduce the problem generate an event id 99 from source Tcpip in the system event viewer with the MAC address causing the conflict. We found that this MAC address was one of the switch in the LAB that was upgraded to cisco IOS 15.1.x and that even with the no gratuitous ARP setting it was still generating conflicting IP address. This switch was downgraded to the same level of the other switch in the building at IOS 15.0.x and since the switch has been downgraded I can't reproduce the problem after 60+ reboot of the the VM. So it looks like this problem comes from the cisco switch.

[derekwarner]

I am having the same issue as well. Here is my situation

Env:

ESXI 5

Vcenter 5

Windows 2008 R2 Domain Controller (Running DHCP) P2Vd

Centos 6.5 workstations

We have a W2K8R2 domain controller running DHCP. The domain controller was P2Vd. We have 64 Centos 6.5 workstations which receive reserved addresses via DHCP. All the MACs are correct and using DHCP only and not BOOTP.

When we boot at least 11 Linux machines, 2 out of ten times, a few machines will fail to get DHCP addresses. Doing a service network restart on the Centos WS allows it to then get an IP from DHCP. This happens all the time when we reboot 64 workstations as part of a large simulation (modeling and simulation).

I am thinking its related to the virtual environment as this issue did not occur when the Domain Controller was physical.

Any solutions would be great. Figured I would post here so we can work together and find a solution.

V/R

Derek Warner

[dskelt]

I am experiencing the same issue on several VMs. 3 of the Guests were built from OVM Templates and 2 of the other machines were built from scratch.

OS: Windows 2008 R2 SP1 and Windows 7 SP1

Verified that the Ghost NICs were gone.

Cleared TCP Stack with the netsh winsock reset and the netsh interface IPV4 reset commands

After 7-10 reboots the Guest will get an IP of 169.254.34.203 (Duplicate) and the IPV4 address will also show as (Duplicate)

ESXi Version: 5.1 update 3

vNIC VMXNET3

I have tried removing the vNICs, rebooting, running the set_devmgr_show_nonpresent_devices=1, start devmgmt.msc, show hidden devices, and removed the vNICs, rebooted, and then readded a vNIC and set the Static IP address. I then rebooted the machine 7 times and then it failed again.

At this point I am at a loss.

[Dany Demers]

After more diagnostic the problem for us occur from the cisco switch, but it's not related to the IOS version. the problem is about the duplicate ip detection between Windows and the switch. I strongly suggest that if you are experiencing this problem to build a test VM in windows 2012r2 as this will provide you with a event 99 saying which mac address report the conflict. The problem as it was explained to me is the fact then when the VM reboot the switch power off and on the port and send a packet but since the Os is virtual the boot time is so fast that windows seen a packet almost simultaneously to check for duplicate IP since windows receive a packet from the switch it thinks that this packet is a response to his demand and then trigger the duplicate IP. What we did is increase the IP device tracking proble delay to 10 and since then we have been unable to reproduce the problem.

This cisco document explain in detail the problem and the solution.

www.cisco.com/image/gif/paws/116529/116529-problemsolution-product-00.pdf

[tallenmcp]

I'm sure that you probably figured this out but I was having the same issue.  Here is the fix that was given to me.  Once tested, I created a GPO to add this key to all of my Windows Servers.  To this day, I haven't had any issues.

VMware KB: False duplicate IP address detected on Microsoft Windows Vista and later virtual machines on ESX/ESXi whe…

[Edaigle]

In my case the solution was to add the following Network configuration manually in the registry:

You have to found the correct interface under: Computer\HKLM\SYSTEM\CurentControlSet\TCPIP\Parameters\Interfaces\(your interface)

Ensure the following value are there, if not just add it manually:

DefaultGateway = your default Gateway

DefaultGatewayMetric = 0

IPAddress = your IP Address