vSAN HCL DB update issues since vCenter 7.0.3 U3r update

[Mach6]

Ever since we updated our vCenter to 7.0.3 U3r from 7.0.3 U3p this week, we're starting to see warnings concerning the vSAN HCL DB Auto update from all vSan clusters and not being able to access the DB online. We're seeing new denied requests from our firewall coming from the vCenter IP. We're seeing IPs coming from cloudfront, cloudflare and others, whenever we try to update it. Is there any documentation out there with all the IPs, domain names etc. that need to be added to our firewall's ****?

Everything from vmware.com was already whitelisted before but looks like some these requests our coming from elsewhere with the new update.

[Ronald Laarman]

We're seeing the samen issues on 7.03 U3q. Downloading is performed through a chain of proxies and a firewall. No denies or failures anywhere. Downloading the HCL file from https://partnerweb.vmware.com/service/vsan/all.json and installing it manually works without error. Letting vCenter download the HCL will fail with the following error: Unable to get the latest HCL database version online. Check the vSphere Client logs for details.

The following error occurs in /var/log/vmware/vsan-health/vsanvcmgmtd.log when pressing the download button under vCenter > Configure > VSAN > Update.

2024-07-03T11:53:14.589Z verbose vsanvcmgmtd[11558] [vSAN@6876 sub=PyBackedMO opId=0b8a007a] Invoke vim.cluster.VsanVcClusterHealthSystem.updateHclDbFromWeb failed: (vim.fault.VsanFault) {
-->   msg = 'Failed to replace old HCL DB with new one.',
-->   faultMessage = (vmodl.LocalizableMessage) [
-->     (vmodl.LocalizableMessage) {
-->       key = 'com.vmware.vsan.health.msg.hclupdatefailed',
-->       arg = (vmodl.KeyAnyValue) [],
-->       message = 'Failed to replace old HCL DB with new one.'
-->     }
-->   ]
--> }. Traceback:
--> #0 UpdateHclDbFromWeb() at VsanVcClusterHealthSystemImpl.py:12166

We can successfully download the HCL thought the vCenter shell.

wget -e use_proxy=yes -e https_proxy=***** https://partnerweb.vmware.com/service/vsan/all.json
--2024-07-03 12:25:42--  https://partnerweb.vmware.com/service/vsan/all.json
Proxy request sent, awaiting response... 200 OK
Length: 18403444 (18M) [application/json]
Saving to: ‘all.json’

all.json                               100%[===========================================================================>]  17.55M  10.6MB/s    in 1.7s

2024-07-03 12:25:44 (10.6 MB/s) - ‘all.json’ saved [18403444/18403444]

[Duncan Epping]

please file a support request, I am guessing this has something to do with the "day 2" changes moving from VMware to Broadcom. Although we also had some issues with the HCL DB file being incorrectly stages earlier this week, so if the problem has been resolved since that was probably it.

[Mach6]

I've already opened support case. No news as of yet after sending vCenter logs. I believe this is a just a matter of knowing which subnets and/or domain names are now used for cloudfront, cloudflare and aws, that we need to add to our firewall rules, beside the Vmware.com domain.

[Ronald Laarman]

Please keep me posted.  

I added vmw-vsan-healthcheck-resources.s3.us-west-2.amazonaws.com to our proxy and firewall.

It seems download the HCL through vCenter is working again, at least no error is showing.

Running the Skyline Health check keeps showing the vSAN HCL DB Auto Update warning.

[Ronald Laarman]

The Skyline Health warning is gone. I guess there had to be a successful auto download for the warning to disappear.

[Benoit Berthiaume]

Thanks to Ronald, you save me a lot of time ! The support asked for a lot of logs and told us that this discussion had nothing to do with our problem. However, opening the new URL solved all the problems.

[Ronald Laarman]

You're welcome. Glad I could help.