Strange ESXi 4.1 IP Addressing Issue

[gogogo5]

Hello

We have experienced a strange ESXi 4.1 IP addressing anomaly where the ESXi host's management IP address is changed to the vMotion address.

In our lab we have 2 ESXi hosts with 6 pNICs. All pNICS are configured on a single standard vSwitch. All pNICS are Active (no Standby or Unused) and configured for Route Based on IP hash. The 6 pNICS are uplinked to a Cisco 3120 stack where Etherchannel is configured. All works ok so far.

We have found that if the ESXi host has assigned vmk0 to vMotion and vmk1 to Management, upon rebooting the ESXi host the vMotion IP address is shown as the IP address (i.e. the management address) on the DCUI!!!! When using the DCUI to change the IP address back to its correct IP address, when exiting from the IP configuration screen and restarting the management network the IP address changes back to the vMotion address.

As mentioned earlier, we have narrowed this down to the vmk numbering. If vmk0 is assigned to vMotion and vmk1 assigned to Management then this issue occurs. For ESXi hosts that have assigned vmk0 to Management and vmk1 to vMotion this issue does not occur. To fix the issue we have to remove the vMotion and Management portgroups, then re-create the Management port group which then claims vmk0 and all is ok.

So the question we are trying to answer is why/how ESXi changes the Management vmkernel port from vmk0 to vm1. We built the hosts ESXi manually so its safe to assume that vmk0 would have been allocated to the Management vmkernel port as this is the first vmkernel port to be created. Is this expected behaviour to have vmk0 take higher priority? Should it matter which vmk number is assigned to Management anyway?

I have seen other users post similar observations and it seems Host Profiles is the culprit. I am surprised VMware have not chimed in to these previous posts considering the impact. See:

http://communities.vmware.com/message/1521966#1521966

and

http://communities.vmware.com/message/1597396#1597396

I will be logging a call with VMware but would like to hear if anyone who has ESX 4.1 running and has vmk0 assigned to vMotion and vmk1 assigned to Management to reboot the host and report the IP address on the DCUI.

Cheers

gogogo5

[bulletprooffool]

Is this on all hosts, or hosts one host?

it could be that the bootbank files on the host are corrupt and you making the change, which continues to work until reboot. The reboot then reverts to previous settings as the changes are effectively not saved.

have a look at the size of your bootbank volume and the amount of space

available on it.

It may well be that the filesystem has a problem and so your hourly

backups are unable to write to bootbank, so configs are lost at reboot.

Have a look at this SUPER useful doc . .

solved my near identical issue.

[gogogo5]

Hello bulletprooffool

The problem is related to the order behaviour of vmk0 and vmk1. When the vmkernel port is assigned to vmk0, it takes precedence over vmk1 i.e. if the vMotion vmkernel port is assigned vmk0 then the vMotion IP address becomes your Management IP address, as verified on the DCUI logon screen.

It is easy to reproduce on a test ESXi 4.1 host:

1. Using the vSphere Client, delete the vMotion vmkernel port.

2. Using Local TSM delete the Management vmkernel port.

3. Using Local TSM, create a new vMotion vmkernel port and assign the original IP address and enable it for vMotion. This step will assign vmk0 as this is the first vmkernel port on the host.

4. Using Local TSM, create a new Management vmkernel port, assign the original IP address and enable it for Management. This step will assign vmk1.

5. Reboot your host.

6. Look at the DCUI and see what IP address shows.

Can anyone help verify this?

[hharold]

We are facing the same issue on ESXi 4.0 hosts.

We do the configuration through Host Profiles, and are also getting vmk0 bound to the VMotion port group and vmk1 to the Management interface.

The DCUI is showing the VMotion address and not the management address.

If we go into the Configure Management Network option, (in the DCUI) we see the addresses and settings from the VMotion port group.

Perhaps more important the host registers its VMotion IP-address in the /etc/hosts file

We suspect that this is causing some strange HA errors occuring.

Anyone has any idea how to solve this?

Regards,

Harold

[gogogo5]

hharold - we are getting the same thoughts its Host Profiles that cause this. We did a few tests in our lab and found that if you make a change to both the vMotion and Management ports (admittedly not a regular occurrence but that's besides the point) in the Host Profile and apply the profile the vMotion and Management ports get destroyed and re-created. Its this re-creation process that binds vMotion to vmk0 and the ensuing IP addressing anomaly.

Want to see something even weirder? When you have ESXi hosts exhibiting this behaviour, go to the DCUI and set the Management IP to DHCP. This clears the IP settings and assuming you don't actually have DHCP on your management network, you'll end up with an address of 0.0.0.0. Then, in the vSphere Client (either connected to VC or host directly) you'll magically see the vMotion IP address go to 0.0.0.0. But, didn't we just change the Management IP address??

I have also seen HA issues. Come into the office and all your VMs are dimmed with "not responding" or "disconnected" status.....sigh....(we have HA set to Leave VMs Powered On to give us some hope).

I have logged a SR with VMware so I'll keep you informed....

-gogogo5

[gogogo5]

I logged a SR with VMware and received this reply:

-

---

Thank you for your Support Request.

As we discussed in the call, this is a known issue with the product ESXi 4.1, ie DCUI shows the IP of vmk0 as Management IP, though vmk0 is not configure for Management traffic.

Vmware product engineering team identified the root cause and fixed the issue in the upcoming major release for ESXi. As of now we don't have ETA on release for this product.

The details on product releases are available in http://www.vmware.com.

Based on our last communication, it appears that this case is ready to be closed. I will proceed with closing your Support Request today.

-

---

I then asked if there was a workaround and received the following (but it doesn't work, for me anyway):

-

---

Here is the workaround to get correct IP reflected as Management IP in DCUI.

By design DCUI looks for the "ManagementIface" from esx.conf file. To fix the issue "DCUI always shows vmk0 IP address as Management IP"

We have to go for the following workaround.

1. Keep ESXi host has assigned vmk0 to vMotion and vmk1 to Management Network.

2. Execute the following command.

esxcfg-advcfg -s vmk1 /Net/ManagementIface 3. reboot the host.

4. Ensure that the management IP in DCUI is the IP of vmk1.

-

---

Have fun!

gogogo5

[mjamal]

gogogo5. so vmware support acknowledges that there is an issue in ESXi 4.1 but no word on when it will be fixed!!. that workaround you provided also didn't work for us either!! It will be good to hear on this further from everyone else.

Mo

[kresimir_pirkl]

Hi everyone.

I had the same issue today.

After applying host profile, vmk0 becomes VMkernel port and vmk1 becomes Management port.

I've managed to solve the problem by manually editing Host Profile and reapplying it after that.

The problem is order that vmk ports are created and applied.

I've renamed VMkernel port to Management and management to VMkernel port, and edited option with IP address setting in "new" VMkernel port.

Important is that Management port is the First port in order (so it could get vmk0 port); VMkernel should be second (so it could get vmk1 port).

After that, I've put host in maintenance mode, reapply host profile, and the problem was solved.

Find some screenshots attached.

Regards, Kresimir.

[mjamal]

Excellent work around. Well done. We fixed it by assigning vmk nics manually to mgmt and vmk ports.

[RParker]

The problem is order that vmk ports are created and applied.

This looks more like a host profiles bug rather than ESXi itself. We have a few ESXi servers, this has never been a problem, so it must a result of setting / configuring host profiles for those servers.

[gogogo5]

RParker - this is actually a bug in the way ESXi treats the assignment of vmk ports and their purpose. It shouldn't matter whether vmk0,vmk5 or vmk12 etc is assigned to "management" but we have found that if "management" is not assigned to vmk0 then problems ensue. As detailed in my previous replies, when vMotion assigned to vmk0 then the IP address of vMotion is shown at the DCUI. This clearly shouldn't happen and in my eyes is a bug.

Your ESXi hosts you refer to that have never had this problem, check out the vmk assignment and I bet all your Management vmkernel ports have claimed vmk0?

Depending on what your host profiles will do to bring your host(s) into compliance, if the profile needs to destroy and recreate both the "management" and "vMotion" port groups at the same time then Host Profiles will create the vMotion port first which, since no vmks have been assigned at this stage, will be assigned vmk0 followed by Management on vmk1. This causes the DCUI to show the vMotion IP address and causes HA havoc :smileywink:

This is easy to replicate even without using Host Profiles - if you have a test ESXi 4.1 host, use the CLI to remove Management and vMotion vmkernel ports then create the vMotion vmkernel first followed by Management so that when you run a esxcfg-vmknic -l you see vMotion on vmk0 and Management on vmk1. Once this is done, flick to your DCUI screen and what do you see?

So, whichever vmk is assigned to Management, the fact that it is "management" should be the dictating/priority thing here not what vmk number is allocated.

Cheers

gogogo5

[davidw_davis]

I'm also having issues with this but don't have Host Profiles licensing.

I've got two ESXi hosts with 8 network adapters each. I assign vSwitch0 to vmnic0 and vmnic1 with the VM Network role. I assign vmnic2 and vmnic3 to vSwitch1 for iSCSI traffic, vmnic4 and vmnic5 to vSwitch2 for vMotion traffic and vmnic6 and vmnic7 for VM Management traffic. DCUI is showing vSwitch1 ip info as the management interface on both hosts. I haven't been able to manually change this. When I edit the host via DCUI and change the management network adapters from vmnic2 and vmnic3 to vmnic6 and vmnic7, in vSphere vmnic6 and vmnic7 get assigned to vSwitch1 while vSwitch3 has no adapters.

So it seems the best workaround is to assign vSwitch1 containing vmnic2 and vmnic3 to the VM Management network. Hopefully this will clear up my HA issues!

[SuperSpike]

vSwitch order and vmnic uplink order really shouldn't have anything to do with it. The key is the order in which the vmkernel ports are created. The "Management Network" vmkernel port for your ESXi host has to be vmk0 (check the Config tab, Networking, Virtual Switch). If your iSCSI IP is vmk0, that's what's going to show up in the DCUI. If it's not in that order, blow all of the vmkernel ports away and configure networking from scratch in the DCUI. This assumes that the ESXi hosts aren't in production of course :smileylaugh:

[davidw_davis]

vSwitch0 is vmk0, correct?

My iSCSI IP is configured on vSwitch1 but the DCUI shows the iSCSI IP as the management IP and also shows the management network as vmnic2 and vmnic3 which are assigned to vSwitch1.

[SuperSpike]

No, the vSwitch number has nothing to do with vmkernel (vmk) port number. NICs are uplinks for virtual switches. Think of vmkernel ports as ports on a physical switch, except in this case they are virtual switches.

Use the vcenter client to connect to your ESXi host. Then click the Configuration tab, then the Networking link under "Hardware". You should see all of your vswitches listed here. For each vSwitch, you should see different port groups or vmkernel ports. Each vmkernel port will have a name. Beneath the name you should see vmk0, vmk1, etc. depending on how many vmkernel ports you have.

I suspect that your iSCSI vmkernel port is assigned to vmk0, which is why it's showing up as the host's IP in the DCUI. Typically the first vmkernel port that you add is set as vmk0.

Unfortunately the only way to correct this is to blow the vmkernel ports away and start from scratch, making sure you create the "Management Network" vmkernel port first.

[davidw_davis]

I see what you're saying. I think my vmk0 was associated with my vSwitch0 but I removed that vmkernel network to make it a vm only network. Perhaps this is why vmk1 is being used for managment? I currently don't have a vmk0 network. Just vmk1, vmk2 and vmk3.

[DyJohnnY]

We are seeing the same problem, the ESXi hosts have no fault, the host profiles are the root cause of the problem.

Obviously the ramifications of this are much greater than simply having the vmk ports out of order. We have had nexus 1000V switches report the IP address of the host they were managing as the vMotion address. i guess it has to do more with the etc/hosts file, than the switches actually asking for the management IP adddress.

Also when ESXi hosts registered on our Clariion SAN they registered with the vMotion IP address (it was more a cosmetic issues but strange to see it there).

___________________

IonutN, I blog here - www.rivnet.ro

Message was edited by: DyJohnnY I removed the workaround we proposed since we discovered it doesn't really work as we expected it to

[Datto]

Just wanted to mention this problem still exists in VSphere 5.0 so whatever coding errors existing in ESX 4.x that were causing this problem more than a year ago must still exist in ESXi 5.0.

It appears to me the problem is caused by applying the profile (I'm positive all the ESXi 5.0 hosts were setup correctly with Management on vmk0 with the correct IP prior to applying the host profile). However, after Host Profiles were applied, most (but not all) ESXi 5.0 hosts had their vmk0 and vmk1 assignments flipped (ie, Management became vmk1 and vmotion became vmk0 after applying host profile to the ESX 5.0 hosts). The ESXi 5.0 reference host that was the example used to create the host profile remained with Management on vmk0 and VMotion on vmk1 while all the other ESXi 5.0 hosts in the cluster got the vmk0 and vmk1 assignment flipped. One of the negative results of this, as mentioned by others above, is the on-screen address of the ESXi 5.0 console is shown wrong -- it shows the vmotion IP instead of the mangement IP.

Two other clusters in the same VC datacenter that had host profiles applied to the cluster and associated cluster members didn't have this problem during the process of applying the host profile (each cluster had it's own reference host used to create the reference host profile).

Also, these were all fresh ESXi 5.0 installs that were installed from media, not upgrades. I used kresimir.pirkl workaround to fix it with these ESXi 5.0 hosts. We should officially call that the "KP workaround".

Hope that helps someone who many be looking into this.

Datto

[gogogo5]

Hello Datto - thanks for providing an update and its disappointing this issue still occurs in vSphere 5.0.  Do you have a support agreement with VMware and if so, would you be willing to log a SR for this as I was told by VMware this issue was going to be fixed in v5.0.

Cheers

gogogo5

[hharold]

Very disappointed that this still has not been fixed.

We had the same issue in 4.1, and were hoping that his would be fixed in 5.0

In vSphere 5.0 VMware has introduced another dependency on vmk numbering...

The VMware Dump collector has to be configured on a fixed vmk port (eg. vmk0, vmk1 etc)

If you want to do this with Host Profiles, you must be able to rely on consistent vmk numbering, which we can't.

[SuperSpike]

The steps to recreate the issue posted by Datto is spot on. That's exactly how to recreate the issue in 4.1 as well.

I've had an SR open with VMware Support on this issue since at least July 2011! The PR associated with the bug is 580080. The latest update I've received from Support is that the fix will be implemented in version 5.1. Keep on waiting!

It is pretty simple to fix by manually modifying the offending host profile (as previously documented in this thread). However, if you've gone ahead and applied the bad profile to many hosts already, it could take a long time to resolve due to each host having to be in maintenance mode. All the more reason I feel host profiles are a huge waste of money. Everything they do can be done with PowerCLI for nothing. Plus, they don't do anything with your storage config. Worthless!

[MichaelSteg]

Thank you for your e-mail.

I'm currently out of office till Monday, 23th of April 2012. During this time I will have no access to my email.

Your e-mail will not be forwarded. For urgent concerns please feel free to contact the main office on [phone]

Best regards,

Michael Steg

Michael Steg

Systems Engineer

Bechtle Regensdorf AG

Bahnstrasse 58/60

CH-8105 Regensdorf

tel: [phone]

gsm: [phone]

fax: [phone]

[e-mail]<mailto:[e-mail]>

www.bechtle.ch<http://www.bechtle.ch/

[danzig69]

problem always there with ESXi 5.0.0 Build 702118 !! In short, leave Management traffic at vSwitch0 and vmk0 after the installation.

[erickmiller]

Anybody try this in 5.1?  I just ran into this issue with 5.0 Update 1b.  Fun fun.

Eric

[gogogo5]

kresimir.pirkl - this sounds promising and thanks for your contribution. Can you do me a favour and update your profile from your reference host and let us know if this changes or resets anything in the now, updated, host profile.

Cheers

gogogo5

[kresimir_pirkl]

Hi gogogo5.

There is my "edited" host profile. I've changed port groups so that Management Network is firstly created and then the other two port groups.

During the "apply host profile" wizard you will be prompted to manually enter IP addresses (this can be also edited in host profile template).

Verify all "configuration changes that will be applied on the host" before you click "finish" button.

I've reapply host profile on 8 production hosts, without any problem, only order of vmk ports was changed.

Please be advised to change NTP server to your custom IP (in host profile template).

This host profile template is made for Test environment only, and I take no responsibility for any misuse.

Some screenshots are attached.

I hope this will help you to properly edit your own host profile.

Regards, Kresimir

[SuperSpike]

Wow. Great job on this, everyone. VMware needs to write a KB article on this issue ASAP if they don't plan on fixing it until the next major release. Given that they have the audacity to charge extra for host profiles, it's completely unacceptable that they don't plan to fix it right away. Just another VMware product rushed to market.

[mtnbiker5]

This works great! I had 13 hosts that had vmk0 and vmk1 flipped thus setting the management network to my vMotion network. I placed the hosts in Maintenance mode and applied the "manipulated" Host Profile and everything is back to normal. Thanks to everyone!

[AndreTheGiant]

If you add / remove / move some PCI cards this can generate some changes in NIC enumeration.

But I suppose that is not your case, so it's a strange problem.

Should it matter which vmk number is assigned to Management anyway?

No.. but it's nice to have homogeneous configuration (to keep simple document and manage).

Andre

[haroldf]

@kresimir

Still seeing the issue on 5.0 and this corrected it. Thanks for the fix!