ESXi 7.0.3 Upgrade Errors?

[DanMc85]

 

I am attempting to upgrade a Dell R740xd cluster from 7.0.2 to 7.0.3.

I have attempted multiple ways... Lifecycle Manager Baseline Remediation, manual SSH "software profile update", and a regular Dell custom ISO. All have failed with an error one way or another.

When I use putty to do an SSH update I see:

[ValueError]
Expected 1 component, found 2

 

Lifecycle Remediation gives

 Cannot execute upgrade script on host.

 

The ISO method, gives this known issue error (from release notes):

If a live VIB install, upgrade, or remove operation immediately precedes an interactive or scripted upgrade to ESXi 7.0 Update 3 by using the installer ISO, the upgrade fails

When a VIB install, upgrade, or remove operation immediately precedes an interactive or scripted upgrade to ESXi 7.0 Update 3 by using the installer ISO, the ConfigStore might not keep some configurations of the upgrade. As a result, ESXi hosts become inaccessible after the upgrade operation, although the upgrade seems successful. To prevent this issue, the ESXi 7.0 Update 3 installer adds a temporary check to block such scenarios. In the ESXi installer console, you see the following error message: Live VIB installation, upgrade or removal may cause subsequent ESXi upgrade to fail when using the ISO installer.

 

Anyone have any ideas or tips that I may not have thought of to get this update installed.   - vCenter updated to 7.0.3 perfectly, vSphere not so much.

 

[DanMc85]

2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 794, in <module>
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: main()
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 785, in main
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: ret = CMDTABLE[command](options)
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/usr/lib/vmware/esxcli-software", line 252, in ProfileUpdateCmd
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: res = t.UpdateProfileFromDepot([], opts.profile, depotUrls=depoturls,
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 365, in UpdateProfileFromDepot
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: return t.InstallVibsFromProfile(metadataUrls, profileName, depotUrls,
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/tmp/esx-update-2248676/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1771, in InstallVibsFromProfile
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/tmp/esx-update-2248676/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1207, in _installVibs
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/tmp/esx-update-2248676/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1122, in checkFdmConfigDowngrade
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/tmp/esx-update-2248676/lib64/python3.8/site-packages/vmware/esximage/ImageProfile.py", line 2416, in GetCompsDowngradeInfo
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: File "/tmp/esx-update-2248676/lib64/python3.8/site-packages/vmware/esximage/Bulletin.py", line 1276, in GetComponent
2021-10-06T13:04:09Z esxupdate: 2248676: root: ERROR: ValueError: Expected 1 component, found 2

[Rajesh Petchimuthu]

Refer to ESXi 7.0 Update 3 release note, where there is mentioned of this point:

• When patching ESXi hosts by using VMware Update Manager from a version prior to ESXi 7.0 Update 2, it is strongly recommended to use the rollup bulletin in the patch baseline. If you cannot use the rollup bulletin, be sure to include all of the following packages in the patching baseline. If the following packages are not included in the baseline, the update operation fails:
  • VMware-vmkusb_0.1-1vmw.701.0.0.16850804 or higher
  • VMware-vmkata_0.1-1vmw.701.0.0.16850804 or higher
  • VMware-vmkfcoe_1.0.0.2-1vmw.701.0.0.16850804 or higher
  • VMware-NVMeoF-RDMA_1.0.1.2-1vmw.701.0.0.16850804 or higher

Can you please confirm any of the above mentioned ESXi components versions are lower in the source ESXi host.

[DanMc85]

That says from a version prior to update 2 in the first sentence. I am currently running VMware ESXi, 7.0.2, 18538813 on the cluster.

Hosts currently have installed:

vmkusb USB Native Driver for VMware 0.1-4vmw.702.0.20.18426014 VMW Wednesday, August 25, 2021, 15:17:50 -0400
vmkata Native ATA driver for VMware 0.1-1vmw.702.0.0.17867351 VMW Tuesday, May 25, 2021, 12:39:16 -0400
vmkfcoe Native Software FCoE Driver for VMware ESX 1.0.0.2-1vmw.702.0.0.17867351 VMW Tuesday, May 25, 2021, 12:39:16 -0400

nvmerdma VMware NVME over RDMA Driver1.0.2.1-1vmw.702.0.0.17867351VMWTuesday, May 25, 2021, 12:39:15 -0400

 

Attempted to use ZIP and ISOs from:

https://customerconnect.vmware.com/downloads/details?downloadGroup=OEM-ESXI70U3-DELLEMC&productId=974

• DellEMC Custom Image for ESXi 7.0 U3 Install CD
• DellEMC Custom Image for ESXi 7.0 U3 Offline Bundle

[DanMc85]

 

I was able to fix that first error by running:

esxcli software vib remove --vibname=i40enu               (due to two versions, one with u at the end due to renaming)
esxcli software vib remove --vibname=brcmnvmefc    (deprecated)

 

My hardware isn't actively using either of these VIBs, so they weren't an issue to remove.

[polppol]

Same problem here,
For some reason in my case vSphere HA stage also failed to start for all Dell machines
(because of vsphere-fdm_7.0.3-18649296 and VMware-VM-Tools_11.3.0.18090558-18644231 failed to install)

I able to fix it by follow DanMc85  answer and re-install failed VIBs again as following:

#put host in Mamtance mode

esxcli software vib remove --vibname=i40enu 

esxcli network firewall ruleset set -e true -r httpClient
esxcli software profile update -p ESXi-7.0U3-18644231-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml 

#restart

Install following Ext. by using Lifecycle Manager Baseline:
VMware-NVMeoF-RDMA_1.0.3.5-1vmw.703.0.0.18644231
VMware-vmkata_0.1-1vmw.703.0.0.18644231
VMware-vmkfcoe_1.0.0.2-1vmw.703.0.0.18644231
VMware-vmkusb_0.1-6vmw.703.0.0.18644231

#restart

Install following Ext. by using Lifecycle Manager Baseline:
VMware-VM-Tools_11.3.0.18090558-18644231 (or 11.3.5.18557794)
vsphere-fdm_7.0.3-18649296

#restart again and it should working now

If got "Expected 1 component, found 2" error after some installation
just run "esxcli software vib remove --vibname=i40enu" and repeat that step again.

[Rajesh Petchimuthu]

For i40en duplicate vibs , there is a KB article published  https://kb.vmware.com/s/article/85982.

[DanMc85]

Thanks... looks like Dell should then remove "INT_bootbank_i40en_1.14.1.0-1OEM.700.1.0.15843807.vib" from their "VMware-VMvisor-Installer-7.0.0.update03-18644231.x86_64-DellEMC_Customized-A00" customized ISO/ZIP. As it will reinstall the 15843807 version, if used.

[DanMc85]

 

Also, since the other VIB is still present in lifecycle manager, what is the best way to remove\block\stop the Non-Critical Host Patches (Predefined) from reinstalling the unwanted i40en vib from 15843807? Or reinstalling it from a previous Dell Rollup that is present in the repository?

 

See attached screenshot.

[Rajesh Petchimuthu]

One option that worked for us was upgrade to 7.0U2c first and then upgrade 7.0U3. 

[DanMc85]

That is the build the hosts were running pre-U3 upgrade, so that was done.

ESXi-7.0U2c-18426014-standard (Build 18426014)

[Rajesh Petchimuthu]

Ok, got it. Other option that worked for us is - create upgrade(using ISO) baseline and use this baseline for upgrading to 7.0U3.

[montagna]

Same here trying to use ESXi 7.03 standard ( NO OEM ) image to install a fresh copy of 7.0.3 . No upgrade but there is 7.0.2 present on local disks on the server .

The server is an HPE Proliant DL580 G10 ...

cannot believe that until I saw this post...and there is no official solution from VMware 

[vjrk83]

Breaking my head and finally found this. Upgrade to ESXi70U2c-18426014 and then to ESXi 7.0 Update 3. 

[Aaron Rombaut]

Ran into this exact error -- same version on same model hardware. I have more than a few hosts, so I decided to write a small PowerCLI script to do this for me. Please review before running in your environment, your hosts may be using these drivers! If you have already removed the driver or it does not exist, you will receive a PowerShell fault, but it is safe to ignore as it is only informational. Also note, this script is not fast because it is waiting for each driver to be removed from the host before proceeding. While slow to run, it was still a lot faster than me manually performing this on each host.

$VCHosts = Get-VMHost
foreach ($VCHost in $VCHosts)
{ 
	$esxcli_v2 = Get-EsxCli -VMHost $vcHost -V2
	$vib_remove_args = $esxcli_v2.software.vib.remove.CreateArgs()
	$vib_remove_args.vibname = "i40enu"
	$esxcli_v2.software.vib.remove.Invoke($vib_remove_args)

	$vib_remove_args.vibname = "brcmnvmefc"
	$esxcli_v2.software.vib.remove.Invoke($vib_remove_args)
}

  After the script completes, you will have to reboot the hosts for it to take effect. Luckily, I am in a greenfield deployment and there are no VMs on my VSAN so I can safely reboot all the hosts I need after migrating the vSphere Cluster Service VM (vCLS) to an online host.

I am managing the cluster with a single image, but after rebooting, the errors go away and Lifecyle Manager reports the host is out of compliance -- exactly what I was expecting and wanting to see in the first place. Go about updating using Lifecycle Manager!

[srodenburg]

Had major issues with basically everything involving VIB's. HA would not work anymore after upgrading from 7 U2b to U3 and U3a (vCenter). LCM had trouble installing patches. I too saw that error "[ValueError]  Expected 1 component, found 2"

HA would not work because the fdm agent VIB did not install (it crapped out with said error). After removing that Intel-i40en VIB on all hosts, the HA FDM VIB would install normally and HA is working since then again. LCM also has no trouble doing it's thing.

Unbelievable what problems a driver rename can cause...

[ccaldwell]

Same issues here with duplicate VIBs. After updating to vCenter U3a, HA was constantly trying to configure itself and failing the agent install. After removing the offending i40en VIB, all was well. 

As someone else mentioned, even after removing the vib and installing the latest driver from the U3 depot, the same offending duplicate vib driver wants to be reinstalled inside of Lifecycle manager, thus causing the issue again if you mistakenly install it. I did a test to see if I can get my host fully up to date after fixing the duplicate vib issue and sure enough, after running all Lifecycle updates, the issue occurred again and I was getting the dreaded error "[ValueError]  Expected 1 component, found 2. 

Hopefully VMware will come up with a way to block or remove this offending driver for folks that have updated to U3. 

[mfirth]

Seems the same problem applies to HPE systems too - got exactly the same error, and removing the i40enu driver resolved it.

But, as also mentioned here, it came back after applying some updates

Hopefully VMware get this resolved soon

[Aaron Rombaut]

Question for those of you who are still experiencing the error after updating and trying to update again. Are you using Baseline updates or Image updates? I am using Image updates and this is no longer an issue for me and just want to know if I am missing something. Once I "set" the image, the offending components should not be able to be reinstalled. With my set up, I was also getting offending Mellanox drivers, but that was from a previous administrator performing Baseline updates and upgrading the drivers higher than the Dell driver upgrade.

It's painful, because the drivers have to get removed before I can update them via Image. Once updated via image, though, I should no longer experience this since I am setting a desired state configuration at the image level and no longer at the individual component level.

 

Aaron

[mattgyver]

I'm assuming that if the second method of upgrading to 7.0.2c then 7.0.3 works, then upgrading to 7.0.2d instead of 7.0.2c would also suffice?

[srodenburg]

Like most people, I'm using Baseline Patch updates as Image Updates are bit like "taking an Abrams Tank to shoot at a fly" if you know what I mean. Images are useful with major releases but it for small incremental updates they are total overkill.

VMware made quite a mess this time (U3 thin-disk PSOD is even worse) and I think I speak for many that we would appreciate it when we can simply patch using baselines again without the wheels coming off and the cluster blowing up in our faces (bad analogy, I know).

[ccaldwell]

I'm using baselines as well.

[dgrove12]

I'm having a similar issue but with upgrading vCenter to u3a.  This patch completely hoses HA, and LCM cluster scans result in unknown results.  I do not have a duplicate intel driver on my hosts... just i40en.  I can't disable/enable HA at the cluster level... fails.  I guess I'll try removing that driver as it's not in use by our DL380s.