VMware vSphere

 View Only
  • 1.  ESXi 7 to 8 upgrade and NIC reordering with Cisco UCS

    Posted 24 days ago

    I have just completed an upgrade on several esxi hosts from 7 to 8 on some Cisco UCS blades using the Cisco vendor ESXi iso. 

    They looked fine. MGMT still worked, vmotion still worked. I thought things were fine until i moved a vm to an upgraded host. It would not communicate with anything. Not things on its own vlan or gateway. Nothing. The nic was connected but just would not talk. 

    Come to find out, i think the vmnics reordered themselves. 

    This is cisco ucs and normally your vnic and your vmnic when loaded are aligned

    vnic0 = vmnic0

    vnic1 = vmnic1

    vnic2 =vmnic2

    vnic3 = vmnic3

    etc. 

    I show my vmnics named correctly on the upgraded host and bound to my dvswitch. For example, vmnic2 and vmnic3 are bound to my dvs. Same names and everything as a non upgraded working host. 

    It appears as if it kept the same names in esxi but reordered the nics. Somehow the mgmt and vmotion nics are still passing traffic and work.  

    It looks like the nics are ordered differently though

    vnic0 = vmnic2

    vnic1 = vmnic3

    vnic2 = vmnic0

    vnic4 = vmnic1

    Why would a straight upgrade of a host do this? I see others have had this issue and there is some sort of process were you can manually remap pci addreses to the proper vmnic names but that seems crazy. It seems easier to reload the entire host so that it puts things back in order....although i havent tried this to confirm it will work. 

    Also....i havent tired to roll back the upgrade with Shift+r on boot. I am hoping that will work.  It should correct?

    How is anyone supposed to upgrade at scale with many hosts using lifecycle manager or automation when  the nics change on a simple upgrade of versions? 

    What other recourse is there to correct this?

    Here are the only options i can think of:

    1. Reconfigure upgraded esxi  hots so that the vnmics are all redone after the upgrade.........but if that works the PCI order would still be off. 

    2. I could remap the pci order or attempt to reorder via the cli  per host and reboot but that also seems risky as a typo or doing it incorrectly will lead to other issues. 

    3. Reload from scratch and reconfigure the host settings identical. This would likely be the cleanest option that i can see. 

    What am i missing. How is anyone supposed to upgrade at scale if this is the issue?

    This article  (How VMware ESXi determines the order in which names are assigned to devices) seems to suggest that it shouldnt be an issue when upgrading:

    "ESXi Upgrades
    Upgrades from one release of ESXi to the next, including autodeploy configurations, do not change the aliases previously assigned to the system's ports. This includes upgrading from prior releases to ESXi 5.5 and later."

    Then it immediately says that it is/can be a problem:

    " In rare cases, aliases can change across an ESXi upgrade or patch installation, due to ESXi parsing the firmware device location information. For example, in ESXi, NICs are named in vmnicN format, where N is a number ranging from 0 upward, where other vendors may name them different: ESXi vmnic order changed after adding more vNICs in UCS Service Profile.
    If you need to correct a changed alias, follow the procedure in "Changing the names assigned by the ESXi host" below. As well, reinstalling ESXi on UCS hosts can re-apply the UCS profile correctly in the event a single host has the wrong enumeration for it's vmnics. "

    Per that same article it explains how to change the alias around but that seems ridiculous to have to try and figure that out. At that point there is too much room for error and would likely be easier to reload from scratch so that the order is again correct in the first place. 



    -------------------------------------------


  • 2.  RE: ESXi 7 to 8 upgrade and NIC reordering with Cisco UCS

    Posted 24 days ago

    Hi there,

    Have you set up the placement policy for UCS templates?

    -------------------------------------------



  • 3.  RE: ESXi 7 to 8 upgrade and NIC reordering with Cisco UCS

    Posted 23 days ago

    Good question. I assume so but i have inheirited the environment from others who are no longer here. 

    Where would i check this on the policy?

    I have upgraded these same host and patched them in the past and never ran into this issue. It seems like its something specific to esxi8.  All my VHBAs for storage still look correct, its just the vmnics that appear to have changed....i think.

    -------------------------------------------



  • 4.  RE: ESXi 7 to 8 upgrade and NIC reordering with Cisco UCS

    Posted 23 days ago

    https://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/141/UCSM_GUI_Configuration_Guide_141_chapter28.html

    Placement policies are set in the service profile template.  See the documentation above for more info.

    -------------------------------------------



  • 5.  RE: ESXi 7 to 8 upgrade and NIC reordering with Cisco UCS

    Posted 22 days ago

    Thanks. 

    nic1 points to vnic0 which points to a mac address

    nic2 points to vnic1 which points to a mac address

    These mappings didnt change. UCS didint change.  ESX did. It thinks vnic0 is now vmnic2 in vmware where it used to be vmnic0 and the mac address in UCS again hasnt change or moved, but now in vmware shows that mac is tied to vmnic2. 

    I would say this is not a jumbling of mappings in UCS at all but ESX from nothing more than a straight upgrade. ITs like 7 detected the ports correctly and somehow 8 has decided to detect them in a different way entirely putting them in a differernt order.  It makes no sense. 

    I am working on on getting a case open with that vendor as well but im not holding my breath....especially not these days.  I am at my wits end trying to sort out what happened here. Its very odd. 

    -------------------------------------------