ESXi 7 "Host Certificate Status", expiration imminent - renewal doesn't update "valid to" date

[mgregory83]

We have a vCenter 7 U3 environment with several ESXi 7 hosts. Recently all hosts are now showing a red exclamation mark and alert that says "ESXi Host Certificate Status". Going to the Certificate section in Configure tab shows the cert is valid until 03 February this year. I've tried doing both Renew and Refresh CA Certificate buttons but the only thing that changed was the Valid from date, not the Valid to date.

I did some research and found that we may need to update the VCSA certs (under vCenter Administration Certificates section) which is done via SSH to the vCenter. I've tried this and while the commands completed successfully, the valid to dates did not.

I've also tried using the command /sbin/generate-certificates on one of the ESXi hosts, restarting services, then trying again but it still isn't working.

I'm not really familiar with certs, these are all self-signed as far as I'm aware. I don't really know how to fix this and i've run out of ideas from research.

We are paying for VMware licenses, and when I called Broadcom support I was told we didn't have any active contract for support and that we have to buy a new one. I don't know if my boss is going to do that or not. I'm not sure what will happen after these certs expire.

Does anyone know what is going on here and how to fix it?

[Andrea Consalvi]

This sounds like a certificate renewal issue where ESXi hosts aren't correctly receiving or recognizing updated certs. 

Check ESXi host certificate mode

Run this on vCenter via SSH:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT 

Does the output show an expired cert?

Also, check the mode with:

esxcli system settings advanced list | grep Cert 

Look for Config.HostAgent.plugins.certificates.mode. If it's set to vmca, the hosts should be using vCenter CA-issued certs. If it's thumbprint, the hosts are likely not trusting vCenter's CA.

[mgregory83]

Thanks for the information. The first command ran against my vCenter via SSH comes back saying there is 1 private key in store and lists two large alphanumeric keys.

However whenever I try to run the other command I get an error saying esxcli command is not found.

[Andrea Consalvi]

t looks like the issue is that vCenter's CA itself is about to expire, which means it can't issue ESXi certificates with a validity beyond its own expiration date. To fix this, the CA on vCenter needs to be renewed first, and then the ESXi host certificates can be regenerated.

The first thing to do is check the status of vCenter's CA. Running this command on the VCSA over SSH:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT 

will show the current certificate and its expiration date. If it's about to expire, that confirms the issue.

To renew it, you'll need to regenerate a new VMCA root certificate. That can be done by running:

/usr/lib/vmware-vmca/bin/certificate-manager 

From there, selecting Option 4 will regenerate a new CA certificate. The process will ask for details like organization name, country, etc., but if everything should remain the same, the default values can just be accepted.

Once the CA is renewed, checking again with the first command should show an updated expiration date. At this point, the ESXi host certificates will also need to be refreshed. This can be done through vCenter's web UI by navigating to vCenter > Configure > Certificates, selecting the affected hosts, and choosing to renew their certificates.

Alternatively, logging into each ESXi host and running:

/sbin/generate-certificates
/etc/init.d/hostd restart 

should trigger a certificate renewal as well.

If the hosts still aren't picking up the new certs, it's worth checking their configuration by running:

esxcli system settings advanced list | grep Cert 

If it's set to vmca, they should automatically use vCenter's CA, so everything should start working. If not, they might be using the thumbprint mode, which could require additional steps.

Finally, if the issue persists, restarting vCenter's services might help:

service-control --stop --all
service-control --start --all 

That should get everything back on track, with valid certificates for both vCenter and the ESXi hosts.

[mgregory83]

Thank you for the guidance. I am not very familiar with certs so that was very helpful. All I needed to do was generate new CA certificates using option 4. I had used previously used only option 6. I then renewed certificates on each host and they are now showing expiration in 2030. So the hosts are happy now.

However in vSphere Administration Certificate Management section, the STS Signing Certificate and 3 of 4 trusted root certificates are still old... maybe I need to do more options. I'm not sure which to do?

[Andrea Consalvi]

First, you need to check the current status of the certificates. Access vCenter via SSH and run the command /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS. This will list all the trusted root certificates. If some appear outdated, they will need to be regenerated.

The STS Signing Certificate is managed separately from the VMCA certificates. To regenerate it, start the Certificate Manager using the command /usr/lib/vmware-vmca/bin/certificate-manager. Once started, select option 6 to replace the STS Signing Certificate and follow the on-screen instructions.

If some root certificates are still outdated after this operation, you may need to regenerate the VMCA root certificate. To do this, restart the Certificate Manager with /usr/lib/vmware-vmca/bin/certificate-manager and select option 2 to replace the VMCA Root Certificate.

Once the certificates have been regenerated, restart the vCenter services to apply the changes. To do this, run the command service-control --stop --all, wait for all services to stop, and then restart them with service-control --start --all.

At this point, log into the vCenter web interface and navigate to vSphere Administration > Certificate Management to check if the STS and Root certificates have been successfully updated.

If the list of trusted root certificates still contains outdated certificates, you may need to remove them manually. To do this, use the command /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <certificate_alias>, replacing <certificate_alias> with the name of the certificate you want to delete.

After removing the obsolete certificates, it is advisable to perform a final check to ensure that all certificates are up to date and that there are no more errors in the vSphere certificate management section.

[mgregory83]

When I run the first command it comes back showing there are 4 entries, but it doesn't list any dates.

As for the STS signing cert I was able to refresh that from within vSphere Administration. 

I did try to regenerate the VMCA root certs using option 2, but it didn't make a difference and I still have 3 of 4 trust root certs that are about to expire.

Should I really delete these three certs? I don't know what will happen if I do.

[Andrea Consalvi]

Hi,

for the VMCA root cert, running option 2 should have updated everything, but since three of your trusted root certificates are still close to expiring, we should double-check if they're actually in use before deleting anything.

First, try running:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS

This will show all trusted root certs, but unfortunately, it won't display expiration dates. To get more details on each one, use:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <certificate_alias>

Replace <certificate_alias> with the actual alias from the list. That should give you the expiration date and other useful info.

The next thing to check is whether your ESXi hosts are still relying on those old certificates. You can do this by running the following directly on an ESXi host:

esxcli system settings advanced list | grep Cert

If Config.HostAgent.plugins.certificates.mode is set to vmca, then your hosts should already be using the new VMCA-issued certificates, meaning those expiring ones aren't needed anymore. If it's set to thumbprint, though, it's possible that some hosts or services are still referencing the old certs.

If everything checks out and those three expiring certs aren't being used anywhere, you can remove them safely with:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <certificate_alias>

That said, deleting a trusted root cert is irreversible, so unless you're 100% sure they're not needed, it might be better to leave them for now just to avoid any authentication issues with vCenter or connected ESXi hosts.

Once that's all sorted, a full restart of vCenter services can help apply the changes:

service-control --stop --all

service-control --start --all

After that, hop back into vSphere Administration > Certificate Management and check if everything looks good. If anything still seems off, let me know what you see!

[mgregory83]

Thanks for responding!

Running the second command still doesn't reveal any more information unfortunately.

And the esxcli command ran against an ESXi host just displays a path. /UserVars/ActiveDirectoryVerifyCAMCertificate What am I do to with that information? I can't change directory to that path, it says it doesn't exist.

[mgregory83]

Well one of the three nearly expired certs has actually expired and I've not seen any ill effects. The other two expire in a few days so hopefully nothing happens. I'm not even sure what they are for really...

[RaoulSchaffner]

"""
I've tried doing both Renew and Refresh CA Certificate buttons but the only thing that changed was the Valid from date, not the Valid to date.
"""

This sounds to me as if the ESXi host is actively renewing the certificate from its vCenter CA, but the lifetime of the vCenter CA is reached, i.e. its certificate will expire on 2025-02-03. A CA cannot issue a certificate that would live past its own existance.

Which commands did you try on the VCSA to renew the vCenter CA self-signed certificate?

[Nicholas Fletcher]

Are you using your VCSA as an Intermediate CA? If so, you may want to check the expiry date of their certificates because they cannot issue a certificate for beyond their own expiry date.