VCenter 7 Update Manager not Working Error 500

[tobistar3]

Hey,

 

i have Updated my VCenter and also put Lets Encrypt certs in. The VCenter self works fine an say that all certs are vailed. But the Update Manager doesent work and the log show that:

2021-08-26T12:57:41.042Z error vmware-vum-server[07663] [Originator@6876 sub=VumVapiEndpoint] Caught unexpected exception 'SSL Exception: Verification parameters:
--> PeerThumbprint: 5A:6A:84:39:1E:24:CE:46:4C:94:6C:AB:B9:F4:08:97:B8:43:15:6A
--> ExpectedThumbprint:
--> ExpectedPeerName: vcenter.my.net
--> The remote host certificate has these problems:
-->
--> * unable to get issuer certificate' while setting up Esx Health Perspectives service. Backtrace: [backtrace begin] product: VMware Update Manager, version: 7.0.2, build: build-18355805, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00347672]
--> backtrace[01] libvmacore.so[0x0029AA01]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[02] libvmacore.so[0x002A9F09]: Vmacore::System::SystemFactory::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvmacore.so[0x002F6943]: Vmacore::Throwable::Throwable(std::string&&)
--> backtrace[04] libvmacore.so[0x0028F5F2]
--> backtrace[05] libvmacore.so[0x002946A5]
--> backtrace[06] libvmacore.so[0x0028F941]
--> backtrace[07] libvmacore.so[0x0028FEC2]
--> backtrace[08] libvmacore.so[0x0020A6A2]
--> backtrace[09] libvmacore.so[0x002041E1]
--> backtrace[10] libvmacore.so[0x00209DF2]
--> backtrace[11] libvmacore.so[0x00340546]
--> backtrace[12] libpthread.so.0[0x00007F87]
--> backtrace[13] libc.so.6[0x000F35BF]
--> [backtrace end]

 

I dont know what i can do now because a replace and update all certs dosent help. Also the Update Manager service stops immediately.

 

Can anyone help?

[smile__2008]

Hi, I also have this problem, is there any resolutution how-to?

[tobistar3]

Yeah kinda. I revertet all certs to the maschine self signed one. So i dont get it working with lets encrypt certs.

[smile__2008]

I spent half day and also done self certs) other ways fails to start vum(

Please write there if you got letsencrypt to work

[leonhedding]

I am also seeing this problem, but I am not using the LE certificate. I am using the Dehydrated ACME client to get an RSA certificate from ZeroSSL. (The elliptical that default from ZeroSSL are not suported by vCenter.) I can get the certificate to install on the server, but the VUM service no longer starts and I get "unable to get issuer certificate" from the VUM logs too.

I thought maybe I should try to add the root signed certificate to  /usr/lib/python3.7/site-packages/certifi/cacert.pem and restart the vum service, but that did not fix the problem. 

 

2021-10-06T10:43:29.229Z info vmware-vum-server[70996] [Originator@6876 sub=httpDownload] [httpDownloadPosix 691] curl_easy_perform() succeeded - url: http://localhost:1080/idm/tenant/vsphere.local/certificates?scope=TENANT
2021-10-06T10:43:29.229Z info vmware-vum-server[70996] [Originator@6876 sub=CertsCache] [CertsCache 224] Parsing STS Certificates
2021-10-06T10:43:29.229Z info vmware-vum-server[70996] [Originator@6876 sub=CertsCache] [CertsCache 290] 2 encoded certificate chunks extracted
2021-10-06T10:43:29.229Z info vmware-vum-server[70996] [Originator@6876 sub=CertsCache] [CertsCache 320] Done parsing STS certificates.
2021-10-06T10:43:29.229Z info vmware-vum-server[70996] [Originator@6876 sub=CertsCache] [CertsCache 207] STS Certs successfully downloaded at time : 4238329249
2021-10-06T10:43:29.246Z warning vmware-vum-server[71057] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00007fd424078118, h:29, <TCP '127.0.0.1 : 45272'>, <TCP '127.0.0.1 : 443'>>), e: 336134278(certificate verify failed), duration: 13msec
2021-10-06T10:43:29.246Z warning vmware-vum-server[71057] [Originator@6876 sub=HttpConnectionPool-000000] Failed to get pooled connection; <cs p:00007fd438938180, SsoCustomConnectionSpec:[vcenter.domain.tld]:443>, SSL(<io_obj p:0x00007fd424078118, h:29, <TCP '127.0.0.1 : 45272'>, <TCP '127.0.0.1 : 443'>>), duration: 14msec, N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: FC:F0:74:6B:38:EC:DE:40:F8:9C:ED:5E:F9:95:45:14:E9:A6:27:AD
--> ExpectedThumbprint:
--> ExpectedPeerName: <vcenter.domain.tld>
--> The remote host certificate has these problems:
-->
--> * unable to get issuer certificate)

 

What I find very curious is that when I download the cert from http://localhost:1080/idm/tenant/vsphere.local/certificates?scope=TENANT the certificate is not the custom signed certificate. It is the self-signed ssoserverSign certificate created during the installation of the server.

[ShadesJeff]

Is there an update on this? I am experiencing the same problem after following the instructions at:

https://kb.vmware.com/s/article/2150895

Even though I am running vCenter 7.0u3a.

The only step I was not able to complete is running this command:

/usr/lib/vmware-updatemgr/bin/updatemgr-util refresh-certs

as there does not appear to be an equivalent command in my version that I could find.

Thanks,

Jeff

[kdelucca]

Also looking for an update on this. Encountering the same issue after upgrading to 7.0.3 and using ZeroSSL certs.

[durdin]

Hi, Inspect your certificate chain. You need to have all certificates in chain in vmware trusted store to be able to verify the certificate even though vCenter trusts them, LCM/VUM is more picky. Once all relevant certificates are imported to vCenter trusted root certificates store, Lifecycle manager starts working again.

If you run openssl s_client against vCenter with ZeroSSL certificate, you could see the chain provided:

0 s:/CN=...thevcenter...
i:/C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA
1 s:/C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA
i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services

 

ZeroSSL RSA Domain Secure Site CA / USERTrust RSA Certification Authority - You would find them eg in chain.cer received through ACME, or by xca from fullchain, etc. (but you probably have it already there at this point)

It's not visible here, but also SHA-2 Root USERTrust RSA Certification Authority - was needed to be downloaded from ZeroSSL KB/site and appended to chain to be able to install ZeroSSL certificate to vCenter (so you probably have it there).

But the last important is Comodo CA Limited/CN=AAA Certificate Services. vCenter itself works fine without this one, but as it's used to cross sign the USERTRUST cert, LCM/VUM needs to have it and it's missing, so import this one as well. It could be grabbed eg through eg https://ssl-tools.net or grepping it from The Mozilla CA certificate store (see below).

 

You should be ok at this point, but if it's still not working, look also for workaround section of: https://kb.vmware.com/s/article/74844 . If still not working after that you could look on couple of interesting files but seems they have no impact on LCM:

1) /usr/lib/vmware-updatemgr/bin/ssl/vmware-vum.keystore
Inspecting this (using java keytool) reveals the LCM/VUM already has stored the machine certificate into it's keystore even though it does not trust it.

2) /usr/lib/vmware-updatemgr/bin/RootCert.pem
Seems format is just export of the
"openssl x509 -in certificate.pem -text -fingerprint" with headers, and I guess it's the The Mozilla CA certificate store in
PEM format.

Good luck

[stevesunny]

I'm having the same exact issue. It's weird, if I reset all certificates using the cert-manager, I can use the vcenter just fine, take backups and VUM works but if I replace it with the signed certs, I can get vcenter in general to work, but VUM doesn't work and because of that backups don't work either. 

I'm also getting a "Certificate Status" warning on my vCenter as well even though none of the certificates in the vCenter are going to expire anytime soon. 

 

If the latest commentor can explain what he means a little more simply that would be great. 

[Steve Sunny]

I guess I didn't understand what durdin was saying but I got it to work now. I was using Digicert certificates personally so I opened the certificate in chrome and checked the name of the top root CA in the trust chain and then I added that to the vmware trusted root certificate store. This can be done by going to Digicert's website to first download the top root CA file and then going to the administration -> certificate in the vcenter and adding the root ca to the trusted list. 

Once that is done I was able to start the update manager service and everything started working again. 

[Navin A]

Download the Root cert from

https://comodoca.my.salesforce.com/sfc/p/1N000002Ljih/a/3l000000VZ4M/ie5Sho19m8SLjTZkH_VL8efOD1qyGFt9h5Ju1ddtbKQ

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --cert /tmp/SHA-2RootUSERTrustRSACertificationAuthority.cer

/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

Create a chain for Machine_SSL

0 s:/CN=...thevcenter...
i:/C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA
1 s:/C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA
i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
2 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services

By this, you can get rid of the SHA-2 cert in the chain.
https://communities.vmware.com/t5/vCenter-Server-Discussions/SSL-Certificate-Error-vCenter-8/td-p/2933907