VM obtain APIPA IP after OS reboot

[MRoushdy]

Hello,

VMWare support team are doing their best to solve my issue, but I've thought about sharing my issue here, maybe somebody here had experienced the same issue, let me tell the story:

Hypervisor: vSphere 6.5 U1.

Hardware: HPE blade system C7000.

Number of blades: 26 blades - HPE BL460c Gen8.

Connectivity: FCoE.

Network layer: Cisco, Cisco FEX modules are installed into the blade chassis and directly connected to Nexus 5K switches.

VM Network: Distributed switches, load balancing mode: IP hash.

VM Guest OS network config: static IP addresses are configured.

Problem description:

when a Windows VM is rebooted it loses network connectivity, (I am 80% sure that only Windows VMs suffer from this, not Linux, it's not easy to reboot Linux Vms to test as they are hosting the ERP system) by examining the IP of the VM during the issue we find that it gets APIPA address, but it's configured with static IP already (which is nice :smileyhappy: ), this happens on a random-fashion, I am almost sure that this occurs only with Windows VMs (all editions, Windows server editions 2008 R2, 2012 R2, and 2016). The logs of ESXi hosts says nothing about this issue.

Error message in Windows eventvwr:

"The system detected an address conflict for IP address 0.0.0.0 with the system having MAC address xx-xx-xx-xx-xx-xx-xx. Network operations on this system maybe disrupted as a result"

How we solve the issue:

we solve it by rebooting again, disconnecting the vnic and reconnecting it, changing the distributed switch port number, or disabling and then reenabling the TCP/IP connection from the OS itself.

Solution proposed by VMware support:

According to the following KB, solving the issue requires disabling an option on Cisco infrastructure, or adding a registry key in Windows.

https://kb.vmware.com/s/article/1028373

But this KB is talking about DHCP, we don't use DHCP. We've checked this thing with Cisco technical support, but they said that it has nothing to do with Cisco equipment.

Now, I hope that one of you have experienced this issue and would help us.

Thanks,

[mprazeres183]

Hi MRoushdy,

I had a similar issue, this was then related to another issue with Networking in my case it was a VIN Fault ( https://vmtn.blog/2017/11/24/error-vim-fault-notfound/ ) however, also the APIPA was triggered. What I did was to create a new vDS and I moved the Ports to the new one, ironicly the issue disaperead. I know it dosn't make any sense. However try it once, it maight also be the solution at your end.

Best regards,

Marco

[MRoushdy]

Thanks for the info, I may try your suggestion. You know what? I think it's a "caching" issue, like a cached data in Dvs, but VMW support told me that there's no "cache" to flush in DVS. I will take your suggestion into consideration if my trials fail to solve the issue.

Thanks,

[mprazeres183]

That's the same response I got from them.

However I'm pretty sure that somewhere informations are stored and keeped.

Otherwise this "trick" would not have solved the issue, it's just sad, not to know where and what to clean so that it would work without a vds rebuild.

Good luck,
Best regards,
Marco

[MRoushdy]

I've solved the problem, the root cause is a port-channel group on a Cisco Switch that was installed recently in my environment, the networking team has fixed the problem and the issue never occured since then.

Thanks,

[dx12180]

Hi MRoushdy,

Can you tell me what your Networking Team do to fix the Port-Channel. I am having the same issue as well.

Thanks

[ErgenKaya]

Hi MRoushdy,

Can you tell me what your Networking Team do to fix the Port-Channel. I am having the same issue as well.

I have over 400 virtual computers and I'm afraid to reboot.

Your answer too important for me :smileyhappy: