SSL Certificate Error vCenter 8

[AHMNco]

Hi Everyone, the surprisingly new version of vCenter does not work with my current SSL from vCenter 7

here are the errors:

1. When trying to insert Sectigo 1yr (Error occurred while fetching tls: Provided certificate using the weak signature algorithm. Please provide the strong signature algorithm certificate)

2. When trying to insert Let's Encrypt (Error occurred while fetching tls: the trustAnchors parameter must be non-empty)

3. When trying to SSL.com 90-days (Error occurred while fetching tls: 0)

4. When trying to insert wildcard not work as well (Wildcard SSL working well with ESXi but not working with Center)

================

I tried to re-issue, I changed the SSL provider, I read every article, and none of them is working

since I was at vCenter 7 all of them except Let's Encrypt working fine

but now none of them working

please give me a solution, appreciate it

Best Regards

[chall32]

Confirmed. vCenter 8.0, I'm seeing same error:

Error occurred while fetching tls:0 

when trying to replace machine certificate with certificate genrated using a CSR generated by vCenter itself.

[AHMNco]

After thousand years nobody even replied, where are the VMware Experts????

[Maksym Roshchyna]

I will not be so original and will say to generate a new certificate and to check a certificate template for vCenter7

Highly possible that maybe changes are needed

[chall32]

The problem is not with the certificate, it is with the application of the newly generated certificate into vCenter 8.

[AHMNco]

so where is the resolution?

where are the VMWare experts??

is nobody gonna come and answer this problem?

[AHMNco]

Dear  

Thank you for your great effort, with your CA Bundle edited in your blog now I Installed PositiveSSL based on your research

thanks a lot

[lasersword]

same problem!

[Tokiha]

Same problem and Auto Gen Certificate is Sha256 with rsa:3084  and my certs with Sectigo Sha384 with rsa:8192

[tim427]

Same problem; tried "ecdsa-with-SHA256 + id-ecPublicKey (384 bit)" and "sha384WithRSAEncryption + rsaEncryption (2048 bit)", both without luck.

Default certificate is: "sha256WithRSAEncryption + rsaEncryption (2048 bit)"...

[AHMNco]

It seems none of the VMWare Experts care about our problem!!!!!

[BrianCunnie]

This is what I had to do to fix it for my Sectigo/Comodo certificate:

• edit the .ca-bundle
• replace the bad PEM with the good PEM (see attached files)

Longer story: the bad & good certificates have the same key (their RSA Modulus is the same) and the same CN ("USERTrust RSA Certification Authority"), so they can be interchanged, but the bad PEM has been cross-signed (issued) by the old, bad "AAA Certificate Services" (which is self-signed with the weak SHA1 algorithm). The good cert is self-signed with the strong SHA-384 algorithm.

If you feel uncomfortable downloading the cert from a forum (and you should feel uncomfortable), you can view the details of the good certificate here: https://crt.sh/?id=1199354

And you can download a copy of the PEM here: https://crt.sh/?d=1199354

 

Attachment(s)

good.pem.txt   2 KB 1 version

bad.pem.txt   1 KB 1 version

[AHMNco]

No, not working

[ASMITH_77]

I had similar issues and in my case it was due to the path to root certificated being incomplete. Also make sure that any cert in path is at least above sha 1 as version 8 rejects any sha 1 cert. Another approach i took was to delete all related trusted root certs to make sure there is no conflict's. That has t be done from CLI and a slight pain in ass.

 

[BrianCunnie]

Hey :

FYI, I wrote a blog post describing how I was able to get past this error (use a different cert in the CA Bundle). You may want to skip to the Troubleshooting section.

[chall32]

OK good to see some have got it working using public certs.  I'm still struggling to replace the machine cert with a cert generated by an internal CA.

acursory look through the logs in /var/log/vmware/certificatemanagement and /var/log/vmware/certificateauthority isn't providing much help either!

[chall32]

Found the error in /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log:

[2022-11-03T15:31:03.619Z] [ERROR] http-nio-5090-exec-4          com.vmware.vise.mvc.exception.GlobalExceptionHandler              Exception handled while processing request for /ui/certificate-ui/ctrl/certificates/tls:  com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.certificatemanagement.error,
    defaultMessage = Exception found (0),
    args = [0],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = ERROR
}

Full error attached.

Attachment(s)

vC80-Import-Cert-Error.txt   26 KB 1 version

[obsidianindy]

Hey   I followed your instructions from your blog post, even purchasing the exact certificate you purchased and attempted this with vCenter 8.

First, I created via 

 

CN=vcenter-80.nono.io # "CN" is the abbreviation for "Common Name"
openssl genrsa -out $CN.key 3072
openssl req \
  -new \
  -key $CN.key \
  -out $CN.csr \
  -sha256 \
  -subj "/C=US/ST=California/L=San Francisco/O=nono.io/OU=homelab/CN=${CN}/emailAddress=[e-mail]" \
  -config <(cat <<EOF
[ req ]
distinguished_name = req_distinguished_name
req_extensions     = req_ext
[ req_distinguished_name ]
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = ${CN}
EOF
)

 

(obviously, I changed the values).

I then requested a certificate from  SSls.com, and we purchased their least-expensive offering, the PositiveSSL 1 domain Comodo SSL.

(using the same disclaimer):

[We do not endorse either SSLs.com or Sectigo (formerly Comodo); We encourage you to use the reseller and the Certificate Authority (CA) with which you are most comfortable].

They then provided me with the two files.  vcenter.domain.co.crt and vcenter_domain_co.ca-bundle

Then we followed the instructions from your blog post:

• • On your vCenter, navigate to Menu → Administration → Certificates → Certificate Management
  • On the __MACHINE_CERT tile, click Actions, select Import and Replace Certificate.
  • Select Replace with external CA certificate(requires private key).
    • Machine SSL Certificate: click Browse File and select vcenter.domain.crt
    • Chain of trusted root certificates: click Browse File and select vcenter_domain_co.ca-bundle
    • Private Key: click Browse File and select vcenter_domain_co.ca-bundle
  • Click Replace.

  ---

  After doing this, vCenter reports: "Error occurred while fetching tls: Invalid input, not a valid PEM formatted Primary Key"

I've been beating my head all day with this.  vCenter logs aren't much of a help.  Did you have to do anything else, or am I just missing a step?

[chall32]

It would appear that vCenter 8 is very fussy when it comes to importing certificates CSR derived or otherwise.

I managed to get my CSR derived internally signed cert accepted by vCenter by not adding or changing any subject alternative name information. For me the issue looked like it was something to do with this. If I added any, the import would fail.

I’ve documented the process I followed here https://polarclouds.co.uk/vsphere-certificates-vmca-hybrid-mode/#configuring-vmca-hybrid-mode

From memory, when signing the CSR, the subject alternative info from the CSR had an “IP Address” field added that was blank (that is no IP address set).  Deleting the IP address field and continuing to sign the CSR resulted in a certificate vCenter was happy to import.

Finally, when importing the signed certificate and the root certificates, try copying and pasting the vCenter certificate and CA certificate crt file contents into step 2 of the replace certificate wizard, rather than using the browse file buttons.

Hope this helps!

[BrianCunnie]

 

Oops—my bad. The last step should be the key, not bundle:

 

• Private Key: click Browse File and select vcenter_domain_co.ca-bundle vcenter_domain_co.key
• Click Replace.

 

Thanks for finding it—I'll update the instructions next break.

[BAUERAG]

Problem also exists when configuring vCenter login with OpenID Connect in Azure. To access login.microsoft.com, both CA certs from Digicert are needed, but "DigiCert Global Root CA" use "SHA-1 with RSA Encryption" signature algorithm. Importing fails, means also configuring OIDC fails.

It's not that VMware supports 100s of different IDPs. The only one is ADFS and using Microsofts cloud service isn't uncommon.

When checking login.microsoft.com with ssllabs.com, they wrote about the Root CA from Digicert: Weak or insecure signature, but no impact on root certificate

VMware, please fix it. Thanks.

 

[donaldsteele]

Same issue here. Reverting back to vCenter 7....

[des1]

not sure if you got this fixed but i fell into the same problem with a 

basic ov digcert bought  in march 2023

contacted support and they temporarily enabled an  "additonal options" in our request template

this gives you the option to choose  "digicert global g2 " which is sha256

so i revoked the current cert (money back if revoked in first 30 days )

and issued a new one with this root 

steps to install 

1. request the csr from the vcenter 

2. buy this customer cert from digicert with global g2 as root

3. import the .pem root file

4. import using the "cert option where csr was generated by vcenter"

5. use the entity pem file ,   downloaded the full pem file and chopped out the entity, so leaving the intermediate and the root certs only

6. vcenter immediately kicks you off and restarts 

7. wait a few minutes , you can check the vcsa page comes up first

8. happy days 

[des1]

not sure if you got this fixed but i fell into the same problem with a 

basic ov digcert bought  in march 2023

contacted support and they temporarily enabled an  "additonal options" in our request template

this gives you the option to choose  "digicert global g2 " which is sha256

so i revoked the current cert (money back if revoked in first 30 days )

and issued a new one with this root 

steps to install 

1. request the csr from the vcenter 

2. buy this custom cert from digicert with global g2 as root

3. import the .pem root file

4. import using the "cert option where csr was generated by vcenter"

5. use the entity pem file ,   downloaded the full pem file and chopped out the entity, so leaving the intermediate and the root certs only

6. vcenter immediately kicks you off and restarts 

7. wait a few minutes , you can check the vcsa page comes up first

8. happy days 

[Anil0210]

vSphere 8.0 (vCenter Server and ESXi )  do not support certificate with weak signature algorithms, such as sha1WithRSAEncryption. 

Just check if the certificate you are using is with weak signature algorithms. Check the below KB for more details.

 

https://kb.vmware.com/s/article/89424 

[rwattuab]

This thread pops up in the search results for problems with the VCSA 8 certificate errors problem, I thought I would add my experience for future reference in case it helps point others to their solution.

 

I was trying to upgrade 7.0.3.01700 (22357613) to 8.0.2 (22617221), but it was failing with weak signature algorithm errors. I know about the main support article: https://kb.vmware.com/s/article/89424. However, it didn't address all the issues and potential troubleshooting steps. I would suggest testing your certificates with the vsphere8_upgrade_certificate_checks.py Python script at the bottom of that article link, since you can make changes and re-test quickly without going through the upgrade process.

 

2023-11-08 10:24:58.823Z ERROR #################### Errors Found ####################
2023-11-08 10:24:58.823Z ERROR
2023-11-08 10:24:58.823Z ERROR Support for certificates with weak signature algorithms has been removed in vSphere 8.0. Weak signature algorithm certificates must be replaced before upgrade. Refer to the vSphere release notes and VMware KB 89424 for more details. Correct the following 2 issues before proceeding with upgrade.
2023-11-08 10:24:58.823Z ERROR
2023-11-08 10:24:58.823Z ERROR 1. The certificate with subject '/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services' in VECS store MACHINE_SSL_CERT has weak signature algorithm sha1WithRSAEncryption. The certificate thumbprint is D1:EB:23:A4:6D:17:D6:8F:D9:25:64:C2:F1:F1:60:17:64:D8:E3:49. The certificate Subject Key Identifier is A0:11:0A:23:3E:96:F1:07:EC:E2:AF:29:EF:82:A5:7F:D0:30:A4:B4.
2023-11-08 10:24:58.823Z ERROR
2023-11-08 10:24:58.823Z ERROR 2. The certificate with subject '/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services' in VECS store TRUSTED_ROOTS has weak signature algorithm sha1WithRSAEncryption. The certificate thumbprint is D1:EB:23:A4:6D:17:D6:8F:D9:25:64:C2:F1:F1:60:17:64:D8:E3:49. The certificate Subject Key Identifier is A0:11:0A:23:3E:96:F1:07:EC:E2:AF:29:EF:82:A5:7F:D0:30:A4:B4. Caution: Verify that any certificates signed by the problematic certificate are not in use by vCenter Server.
2023-11-08 10:24:58.823Z ERROR
2023-11-08 10:24:58.823Z ERROR ######################################################

 

Our leaf certificate was issued by "InCommon ECC Server CA" (https://crt.sh/?id=12722102) which was issued by "USERTrust ECC Certification Authority" (https://crt.sh/?id=1282303296) which was issued by "AAA Certificate Services" (https://crt.sh/?id=331986). The last one is the problem, because its signature algorithm is sha1WithRSAEncryption. The "USERTrust ECC Certification Authority" is also a problem, because it's issued by the bad root.

 

[*] Store : TRUSTED_ROOTS
Alias: d1eb23a46d17d68fd92564c2f1f1601764d8e349
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services
Subject: C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services
Subject Key Identifier: A0:11:0A:23:3E:96:F1:07:EC:E2:AF:29:EF:82:A5:7F:D0:30:A4:B4

[*] Store : TRUSTED_ROOTS
Alias: ca7788c32da1e4b7863a4fb57d00b55ddacbc7f9
Signature Algorithm: sha384WithRSAEncryption
Issuer: C=GB, ST=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=AAA Certificate Services
Subject: C=US, ST=New Jersey, L=Jersey City, O=The USERTRUST Network, CN=USER Trust ECC Certification Authority
Subject Key Identifier: 3A:E1:09:86:D4:CF:19:C2:96:76:74:49:76:DC:E0:35:C6:63:63:9A

 

Based on 's reply and website, I knew I needed to remove not only the root certificate, but also remove & replace the "USERTrust ECC Certification Authority" at the next level down with its newer self-signed version (https://crt.sh/?id=2841410) that expires in 2038.

 

At that point, I used the common commands to list, unpublish, and publish.

 

 

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

# This is the AAA Certificate Services root
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id A0110A233E96F107ECE2AF29EF82A57FD030A4B4 --outcert /certs/A0110A233E96F107ECE2AF29EF82A57FD030A4B4.pem

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certs/A0110A233E96F107ECE2AF29EF82A57FD030A4B4.pem

# This is the USERTrust ECC Certification Authority issued by AAA Certificate Services
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id 3AE10986D4CF19C29676744976DCE035C663639A --outcert /certs/3AE10986D4CF19C29676744976DCE035C663639A.pem

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /certs/3AE10986D4CF19C29676744976DCE035C663639A.pem

 

 

I then uploaded the new self-signed "USERTrust ECC Certification Authority" (https://crt.sh/?id=2841410) through the vSphere Certificate Manager GUI. I had to do that after the above, because it has the same Subject Key Identifier as the other version, otherwise vSphere would complain that it was already in the store.

 

At this point, I was still having problems. The VCSA 8 certificate check was still failing. Hmmmm??? I started looking and remembered about /etc/vmware-rhttpproxy/ssl/rui.crt and /etc/vmware-vpx/ssl/rui.crt. These files had the old intermediate+root chain in them, so I removed that (i.e., the "-----BEGIN CERTIFICATE-----" sections) and added the new certificate information to them and restarted the services. I went back to the GUI and got an error: "Error occurred while fetching machine certificates: com.vmware.vcenter.certificate_management.vcenter.tls". This was solved with a full VCSA reboot. For some reason stopping and starting the services wouldn't fix it.

 

After the reboot, everything looks great. The correct root is there and no errors in VCSA. BUT!!! The VCSA 8 certificate check still fails with: "The certificate with subject '/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services' in VECS store MACHINE_SSL_CERT has weak signature algorithm sha1WithRSAEncryption." WHY???!!!

 

I figured that somewhere the old root information was still in VCSA, but I've replaced everything. Not so fast. Whenever you upload a new leaf certificate, VMware tells us to append the full chain to the end of that certificate. So when it's saying the problem is in MACHINE_SSL_CERT, it's talking about this. But this isn't mention anywhere in the notes and you can't easily troubleshoot it, at least I couldn't. I thought the easiest would be to create a new file that contained the old/current leaf, but with the new root chain appended. But VCSA won't let you do that, because: “MACHINE_SSL_CERT certificate replacement failed. SerialNumber and Thumbprint not changed after replacement, certificates are same before and after.” I understand the error, because the leaf is not changing. But the chain is changing. I kind of feel like I should be able to perform this action.

 

While reviewing https://kb.vmware.com/s/article/83276, it showed the procedure for extracting the current certificate and private key from the MACHINE_SSL_CERT. When I did that, I confirmed that the “__MACHINE_CERT” alias contained the WHOLE certificate chain (leaf, intermediates, root). So I created a new file that contained the old leaf, intermediate, and NEW root chain. I deleted and recreated “__MACHINE_CERT” and restarted VCSA services. That finally fixed it! The upgrade certificate check script succeeds.

 

 

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output ~/entry__MACHINE_CERT-getcert.txt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output ~/entry__MACHINE_CERT-getkey.txt

openssl pkey -in entry__MACHINE_CERT-getkey.txt -pubout -outform pem | sha256sum

openssl x509 -in leaf_MACHINE_CERT.pem -pubkey -noout -outform pem | sha256sum

 

 

I manually created my own leaf_chain_MACHINE_CERT.pem with the right certificates.

 

 

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACINE_CERT --cert leaf_chain_MACHINE_CERT.pem --key entry__MACHINE_CERT-getkey.txt

 

 

No more errors with the certificate checks.

[markwpumphrey]

My fix, slightly related to https://uit.stanford.edu/service/ssl/chain

copy & paste 8079908730.crt 1199354.crt vcenter.key to "Chain of trusted root certificates"

that was it, frustrating as anything though.