Certificate Management

[rbjohnson78]

Anyone ever come across this message when trying to import a cert into "Machine SSL Certificate" in vcenter 7: Error occurred while fetching tls: the trustAnchors parameter must be non-empty

[MikeDoug]

Beating my head against this now. I'm a total newb to PKI and we just installed a Microsoft Offline RootCA and a Subordinate/Issuing CA to our infrastructure for these types of things.  I have found several blogs that walk through this process, however each one of them leads me right to this error.  Frustrating.  Any guidance would be greatly appreaciated.

[iandekit]

I'm seeing this also on recent vCenter's; tried again on the latest vCenter 7.0.3 18778458 and it still occurs

"Error occurred while fetching tls: the trustAnchors parameter must be non-empty"

I'm feeding VCSA a recently generated Let Encrypt certificate.   Originally tried wildcard but said that wasn't supported (Doh!)

so made a non-wildcard one...

 

I also tried loading it via the shell, using /usr/lib/vmware-vmca/bin/certificate-manager  but after much churning it fails too.

-------

You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? : Y
Command Output: cert1.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup:certificate has expired
OK

Get site nameCompleted [Replacing Machine SSL Cert...]                  

... 

Updated 43 service(s)
Status : 85% Completed [starting services...]                   
Error while starting services, please see service-control log for more details

Status : 0% Completed [Operation failed, performing automatic rollback]
                 
Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

Performing rollback of Machine SSL Cert...
Get site nameus : 0% Completed [Rollback Machine SSL Cert...]   

---------

/var/log/vmware/vmcad/certificate-manager.log didn't really say a lot other than this:

----

2021-10-25T00:52:15.433Z INFO certificate-manager Running command :- service-control --start  --all
2021-10-25T00:52:15.434Z INFO certificate-manager please see service-control.log for service status
Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

2021-10-25T00:59:16.445Z ERROR certificate-manager None
2021-10-25T00:59:16.446Z ERROR certificate-manager Error while starting services, please see service-control log for more details
2021-10-25T00:59:16.446Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

2021-10-25T00:59:16.446Z ERROR certificate-manager {
   "detail": [
       {
           "id": "install.ciscommon.command.errinvoke",
           "translatable": "An error occurred while invoking external command : '%(0)s'",
           "args": [
               "None"
           ],
           "localized": "An error occurred while invoking external command : 'None'"
       },
       "Error while starting services, please see service-control log for more details"
   ],
   "componentKey": null,
   "problemId": null,
   "resolution": null
}
2021-10-25T00:59:16.446Z INFO certificate-manager Performing rollback of Machine SSL Cert...

---

It did roll back ok though.

[RobinPlomp01]

Hi guys,

Anny update on this, I have the same issue?

Robin

[julferts]

Experienced the same issue with a "cheap" cert. The reason at least for my issue was the chain of trusted root certificates was not complete.

Most ca's will give you an intermediate-ca chain. Whats missing in most cases is the root certificate of this chain. In my case the root CA was unknown by vSphere so it must be part of the chain.

To do so find the root cert of the intermediate-ca chain and add it as the first PEM section to the intermediate-ca. That will most probably solve your issue.

[iandekit]

Hi,

The LetsEncrypt fullchain.pem contains the site certificate and two other CA certificates.

Both the CA certificates show up in VCSA in the Trusted Root Certificates list (checked the hex signatures and dates).

Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Validity
Not Before: Sep 4 00:00:00 2020 GMT
Not After : Sep 15 16:00:00 2025 GMT
Subject: C = US, O = Let's Encrypt, CN = R3

Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After : Sep 30 18:14:03 2024 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

I tried replacing the Machine SSL Cert with the LE cert for the site from this, along with the private key but the

form demands the chain be supplied for the submit button to work, and supplying both or either of the CA certs 

still results in the same error.

 

"Error occurred while fetching tls: the trustAnchors parameter must be non-empty"

[iandekit]

Problem still exists in latest VCSA 7.0.3.00300 (U3c)

[hakanlund]

I hade the same issue with letsencrypt certificates, but found this blog post https://virtuallywired.io/2021/11/29/replace-default-vcenter-certificate-with-a-free-lets-encrypt-ssl/ wich helped med solve the issue.

Also (I don't remember if this was relevant) the certificate can NOT be a wildcard cert, I had issues with that as well.

[iandekit]

Alas that was exactly what I was doing, and it didn't work for me.

Yes I know wildcards aren't supported ... would also be nice if they were, but not critical.

[gabriel_pujols]

Hello,

Did you resolve the issue? I have the same problem 

[rbjohnson78]

I ended up calling VMware and they were able to help me fix this. My issue was, that I didn't have my chain cert correct. Being new to how the certs work, I didn't know I had to concatenate the cert files and make sure they were listed in the correct order. This is what I followed. https://docs.vmware.com/en/vRealize-Log-Insight/8.6/com.vmware.log-insight.administration.doc/GUID-1203BA5A-3E23-463A-8297-BCD3A9D380E1.html

[ggeddes]

This worked for me as well after several frustrating attempts of importing it every other way. Thanks!

[Tesfaye Assefa]

Anny update on this, I have the same issue?

[iandekit]

No, alas.

[hakanlund]

My working solution on vCenter 7.0.2 for Letsencrypt certificate's.

I start with creating a new cert.pem file that i call cert_combined.pem containing the cert.pem cert and after that the two certs from chain.pem.
I uploaded cert_combined.pem, chain.pem and privkey.pem to the vcenter server and executed:

/usr/lib/vmware-vmca/bin/certificate-manager

I chose option 1 (Replace Machine SSL certificate with Custom Certificate),
provided my credentials and the chose option 2 (Replace Machine SSL certificate with Custom Certificate)

Please provide valid custom certificate for Machine SSL.
File : /root/certs/cert_combined.pem

Please provide valid custom key for Machine SSL.
File : /root/certs/privkey.pem

Please provide the signing certificate of the Machine SSL certificate
File : /root/certs/chain.pem

I hops this is of some help.

[hakanlund]

Update to my post, the cert_combined.pem file i created is the same as the fullchain.pem file that certbot generates.

[iandekit]

Thanks, but trying that on 7.0.3 build 19234570 doesn't work for me...

Ran /usr/lib/vmware-vmca/bin/certificate-manager, selected option 1 (Replace Machine SSL certificate with Custom Certficate) ... and then authenticated... it then says this and stops:

 

Certificate Manager tool do not support vCenter HA systems

It doesn't get as far asking for the cert files.

 

 

[iandekit]

I should add that this is running with an Essentials licence, and there is no HA.

[iandekit]

(back originally I was running under an Eval licence for Standard)

[Francois Corfdir]

I resolve this one with importing all the intermediate certificate in one file. I use xolphin certificate for my vcenter. intermediate public is sectigo - and usertrust. so in the "cahin of trusted root certificates" i use both stacked like that

 

-----BEGIN CERTIFICATE-----
MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTgx
MTAyMDAwMDAwWhcNMzAxMjMxMjM1OTU5WjCBjzELMAkGA1UEBhMCR0IxGzAZBgNV
BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEYMBYGA1UE
ChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQDEy5TZWN0aWdvIFJTQSBEb21haW4g
VmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENBMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1nMz1tc8INAA0hdFuNY+B6I/x0HuMjDJsGz99J/LEpgPLT+N
TQEMgg8Xf2Iu6bhIefsWg06t1zIlk7cHv7lQP6lMw0Aq6Tn/2YHKHxYyQdqAJrkj
eocgHuP/IJo8lURvh3UGkEC0MpMWCRAIIz7S3YcPb11RFGoKacVPAXJpz9OTTG0E
oKMbgn6xmrntxZ7FN3ifmgg0+1YuWMQJDgZkW7w33PGfKGioVrCSo1yfu4iYCBsk
Haswha6vsC6eep3BwEIc4gLw6uBK0u+QDrTBQBbwb4VCSmT3pDCg/r8uoydajotY
uK3DGReEY+1vVv2Dy2A0xHS+5p3b4eTlygxfFQIDAQABo4IBbjCCAWowHwYDVR0j
BBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFI2MXsRUrYrhd+mb
+ZsF4bgBjWHhMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAbBgNVHSAEFDASMAYGBFUdIAAw
CAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0
LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDB2Bggr
BgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRydXN0LmNv
bS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZaHR0cDov
L29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAMr9hvQ5Iw0/H
ukdN+Jx4GQHcEx2Ab/zDcLRSmjEzmldS+zGea6TvVKqJjUAXaPgREHzSyrHxVYbH
7rM2kYb2OVG/Rr8PoLq0935JxCo2F57kaDl6r5ROVm+yezu/Coa9zcV3HAO4OLGi
H19+24rcRki2aArPsrW04jTkZ6k4Zgle0rj8nSg6F0AnwnJOKf0hPHzPE/uWLMUx
RP0T7dWbqWlod3zu4f+k+TY4CFM5ooQ0nBnzvg6s1SQ36yOoeNDT5++SR2RiOSLv
xvcRviKFxmZEJCaOEDKNyJOuB56DPi/Z+fVGjmO+wea03KbNIaiGCpXZLoUmGv38
sbZXQm2V0TP2ORQGgkE49Y9Y3IBbpNV9lXj9p5v//cWoaasm56ekBYdbqbe4oyAL
l6lFhd2zi+WJN44pDfwGF/Y4QA5C5BIG+3vzxhFoYt/jmPQT2BVPi7Fp2RBgvGQq
6jG35LWjOhSbJuMLe/0CjraZwTiXWTb2qHSihrZe68Zk6s+go/lunrotEbaGmAhY
LcmsJWTyXnW0OMGuf1pGg+pRyrbxmRE1a6Vqe8YAsOf4vmSyrcjC8azjUeqkk+B5
yOGBQMkKW+ESPMFgKuOXwIlCypTPRpgSabuY0MLTDXJLR27lk8QyKGOHQ+SwMj4K
00u/I5sUKUErmgQfky3xxzlIPK1aEn8=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw
MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV
BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU
aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK
AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B
3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY
tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/
Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2
VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT
79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6
c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT
Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l
c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee
UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE
Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd
BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G
A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF
Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO
VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3
ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs
8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR
iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze
Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ
XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/
qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB
VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB
L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG
jjxDah2nGN59PRbxYvnKkKj9
-----END CERTIFICATE-----

and I was able to import my machine ssl certificate and the private key nicely.

[iandekit]

Just tried again using VCSA web ui with 7.0.3 00500 (u3d) ... (with Essentials licence) ... no better.

- pasted machine specific cert into first box; rest of LE chain in 2nd box and private key in 3rd box

"Error occurred while fetching tls: the trustAnchors parameter must be non-empty"

However, via shell, it seems to work better this time...

/usr/lib/vmware-vmca/bin/certificate-manager 

Option[1 to 8]: 1

Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [[e-mail]]:[e-mail]
Enter password:
1. Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate

2. Import custom certificate(s) and key(s) to replace existing Machine SSL certificate

Option [1 or 2]: 2

Please provide valid custom certificate for Machine SSL.
File : mach.pem

Please provide valid custom key for Machine SSL.
File : privkey.pem

Please provide the signing certificate of the Machine SSL certificate
File : chain.pem

You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? : Y
Command Output: mach.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup:certificate has expired
OK

Get site nameCompleted [Replacing Machine SSL Cert...]
default-first-site
Lookup all services
Get service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3
Update service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3; spec: /tmp/svcspec_gk1j6msc
Get service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875
Update service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875; spec: /tmp/svcspec_nkgb1iw5
Get service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f
Update service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f; spec: /tmp/svcspec_uxp7q_b0
Get service 4893d3bb-13af-444d-960f-9921f78b46ef
Update service 4893d3bb-13af-444d-960f-9921f78b46ef; spec: /tmp/svcspec__vuw_yab
Get service bedd2511-d562-4651-9b3e-396262ebb7e2
Update service bedd2511-d562-4651-9b3e-396262ebb7e2; spec: /tmp/svcspec_99mvk_4m
Get service d2edb72a-c5b2-4d36-b70c-601bfb138c93
Update service d2edb72a-c5b2-4d36-b70c-601bfb138c93; spec: /tmp/svcspec_lg44eljk
Get service d1b9b4bf-d248-4e06-b717-f48778fd5df6
Update service d1b9b4bf-d248-4e06-b717-f48778fd5df6; spec: /tmp/svcspec_mk3cux2o
Get service 909ce2ce-df14-485b-a3a8-c9dc84384200

Update service 909ce2ce-df14-485b-a3a8-c9dc84384200; spec: /tmp/svcspec_va2x08s3
Get service ddff5bef-f435-4c1a-88b9-d6629c7b0c78
Update service ddff5bef-f435-4c1a-88b9-d6629c7b0c78; spec: /tmp/svcspec_f7l8u632
Get service a6a8f33a-492d-43b5-b5ff-91de64d816a3
Update service a6a8f33a-492d-43b5-b5ff-91de64d816a3; spec: /tmp/svcspec_xlen8yjv
Get service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc
Update service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc; spec: /tmp/svcspec_94d0er4o
Get service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b
Update service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b; spec: /tmp/svcspec_1r2hh7x_
Get service e4eb4b29-53b3-4208-8561-70d183c3f790
Update service e4eb4b29-53b3-4208-8561-70d183c3f790; spec: /tmp/svcspec_yla75a4o
Get service 549b49fb-2991-4c49-8ddc-e9ccff4905aa
Update service 549b49fb-2991-4c49-8ddc-e9ccff4905aa; spec: /tmp/svcspec_msrozdz2
Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7
Update service 8505c9d2-faf9-4306-9034-e38fbc7224e7; spec: /tmp/svcspec_d902vatn
Get service 072493ad-d644-4817-a7c0-75c9111a6155
Update service 072493ad-d644-4817-a7c0-75c9111a6155; spec: /tmp/svcspec_xft2na_v
Get service 36ca1bc5-4a55-4175-8f31-361215949e8c
Update service 36ca1bc5-4a55-4175-8f31-361215949e8c; spec: /tmp/svcspec_v4hui5wd
Get service 2c847253-2e8e-4705-ad7d-1f671039b7ca
Update service 2c847253-2e8e-4705-ad7d-1f671039b7ca; spec: /tmp/svcspec_go_71a3k
Get service 70906193-2f34-415b-a8cf-e1212e18e93b
Update service 70906193-2f34-415b-a8cf-e1212e18e93b; spec: /tmp/svcspec_e8s6_h5y
Get service de956063-1652-4215-b6ae-ec5e6b4ba272
Don't update service de956063-1652-4215-b6ae-ec5e6b4ba272
Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34
Update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34; spec: /tmp/svcspec_o532w120
Get service 6bbf80db-8685-4867-8c55-59d3ba4799f1
Update service 6bbf80db-8685-4867-8c55-59d3ba4799f1; spec: /tmp/svcspec_q6g8h42h
Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv
Update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv; spec: /tmp/svcspec_lassjge8
Get service 757f879d-67d9-4d5d-bb27-01287ec25585
Update service 757f879d-67d9-4d5d-bb27-01287ec25585; spec: /tmp/svcspec_3fr1lp04
Get service 5b04af9d-524f-4368-a93d-2e09ae43b2aa
Update service 5b04af9d-524f-4368-a93d-2e09ae43b2aa; spec: /tmp/svcspec_z22yuja9
Get service fb7108d7-0f76-4ea2-8bac-e7c514157c0a
Update service fb7108d7-0f76-4ea2-8bac-e7c514157c0a; spec: /tmp/svcspec_8u_li1zr
Get service 248c802f-aecd-4769-92b5-5da089a802e4
Update service 248c802f-aecd-4769-92b5-5da089a802e4; spec: /tmp/svcspec_md1w6guz
Get service 756da5b5-8635-42b3-9d78-c532a56d1aaf
Update service 756da5b5-8635-42b3-9d78-c532a56d1aaf; spec: /tmp/svcspec_7isozv20
Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
Don't update service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz
Update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz; spec: /tmp/svcspec_sw8u6e_x
Get service da4612c2-ab08-4f5e-8d1a-7bd0057b4608
Update service da4612c2-ab08-4f5e-8d1a-7bd0057b4608; spec: /tmp/svcspec_9jhd5vmv
Get service a5941153-af0b-4c69-a9e3-68bc4c396e5e
Update service a5941153-af0b-4c69-a9e3-68bc4c396e5e; spec: /tmp/svcspec_upe42sfk
Get service c1cc086c-692e-400c-b89b-a08a83029652
Update service c1cc086c-692e-400c-b89b-a08a83029652; spec: /tmp/svcspec_1wk8g0k6
Get service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb
Update service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb; spec: /tmp/svcspec_1_v6oglz
Get service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5
Update service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5; spec: /tmp/svcspec_3r3mg1vv
Get service 3c89ddcd-3ead-4242-be83-079f1357e8b2
Update service 3c89ddcd-3ead-4242-be83-079f1357e8b2; spec: /tmp/svcspec_4oqyzyvv
Get service 63951d4b-811a-4444-9a90-d31c1257a49b
Update service 63951d4b-811a-4444-9a90-d31c1257a49b; spec: /tmp/svcspec_ynbu47jf
Get service c14db7f5-b259-42f7-aeb0-e14eac50e94d
Update service c14db7f5-b259-42f7-aeb0-e14eac50e94d; spec: /tmp/svcspec_bxdlnalf
Get service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec
Update service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec; spec: /tmp/svcspec_bzr02o51
Get service b2821cf5-81f6-4f15-841c-0badf2ce9c73
Update service b2821cf5-81f6-4f15-841c-0badf2ce9c73; spec: /tmp/svcspec_337fmjqa
Get service 982b69fa-07ec-4219-9f5b-ae629aa5136f
Update service 982b69fa-07ec-4219-9f5b-ae629aa5136f; spec: /tmp/svcspec_obfkk20u
Get service 0c795821-c2e0-4dfd-8537-3e1740722583
Update service 0c795821-c2e0-4dfd-8537-3e1740722583; spec: /tmp/svcspec_mguwb7ig
Get service 931f3c0e-3fbb-40a4-991e-01275f1aa000
Update service 931f3c0e-3fbb-40a4-991e-01275f1aa000; spec: /tmp/svcspec_gimbuirw
Get service 8b5c0028-95c5-48f1-a356-1764c9441d46
Update service 8b5c0028-95c5-48f1-a356-1764c9441d46; spec: /tmp/svcspec_yzhs35e8
Get service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd
Update service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd; spec: /tmp/svcspec_6olyr8vl

... about 12 minutes later ... (OMG why is this so slow...)

Updated 43 service(s)

Status : 85% Completed [starting services...]
Error while starting services, please see service-control log for more details
Status : 0% Completed [Operation failed, performing automatic rollback]

Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

Performing rollback of Machine SSL Cert...
Get site nameus : 0% Completed [Rollback Machine SSL Cert...]
default-first-site
Lookup all services
Get service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3
Don't update service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3
Get service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875
Don't update service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875
Get service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f
Don't update service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f
Get service 4893d3bb-13af-444d-960f-9921f78b46ef
Don't update service 4893d3bb-13af-444d-960f-9921f78b46ef
Get service bedd2511-d562-4651-9b3e-396262ebb7e2
Don't update service bedd2511-d562-4651-9b3e-396262ebb7e2
Get service d2edb72a-c5b2-4d36-b70c-601bfb138c93
Don't update service d2edb72a-c5b2-4d36-b70c-601bfb138c93
Get service d1b9b4bf-d248-4e06-b717-f48778fd5df6
Don't update service d1b9b4bf-d248-4e06-b717-f48778fd5df6
Get service 909ce2ce-df14-485b-a3a8-c9dc84384200
Don't update service 909ce2ce-df14-485b-a3a8-c9dc84384200
Get service ddff5bef-f435-4c1a-88b9-d6629c7b0c78
Don't update service ddff5bef-f435-4c1a-88b9-d6629c7b0c78
Get service a6a8f33a-492d-43b5-b5ff-91de64d816a3
Don't update service a6a8f33a-492d-43b5-b5ff-91de64d816a3
Get service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc
Don't update service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc
Get service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b
Don't update service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b
Get service e4eb4b29-53b3-4208-8561-70d183c3f790
Don't update service e4eb4b29-53b3-4208-8561-70d183c3f790
Get service 549b49fb-2991-4c49-8ddc-e9ccff4905aa
Don't update service 549b49fb-2991-4c49-8ddc-e9ccff4905aa
Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7
Don't update service 8505c9d2-faf9-4306-9034-e38fbc7224e7
Get service 072493ad-d644-4817-a7c0-75c9111a6155
Don't update service 072493ad-d644-4817-a7c0-75c9111a6155
Get service 36ca1bc5-4a55-4175-8f31-361215949e8c
Don't update service 36ca1bc5-4a55-4175-8f31-361215949e8c
Get service 2c847253-2e8e-4705-ad7d-1f671039b7ca
Don't update service 2c847253-2e8e-4705-ad7d-1f671039b7ca
Get service 70906193-2f34-415b-a8cf-e1212e18e93b
Don't update service 70906193-2f34-415b-a8cf-e1212e18e93b
Get service de956063-1652-4215-b6ae-ec5e6b4ba272
Don't update service de956063-1652-4215-b6ae-ec5e6b4ba272
Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34
Don't update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34
Get service 6bbf80db-8685-4867-8c55-59d3ba4799f1
Don't update service 6bbf80db-8685-4867-8c55-59d3ba4799f1
Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv
Don't update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv
Get service 757f879d-67d9-4d5d-bb27-01287ec25585
Don't update service 757f879d-67d9-4d5d-bb27-01287ec25585
Get service 5b04af9d-524f-4368-a93d-2e09ae43b2aa
Don't update service 5b04af9d-524f-4368-a93d-2e09ae43b2aa
Get service fb7108d7-0f76-4ea2-8bac-e7c514157c0a
Don't update service fb7108d7-0f76-4ea2-8bac-e7c514157c0a
Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
Don't update service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
Get service 248c802f-aecd-4769-92b5-5da089a802e4
Don't update service 248c802f-aecd-4769-92b5-5da089a802e4
Get service 756da5b5-8635-42b3-9d78-c532a56d1aaf
Don't update service 756da5b5-8635-42b3-9d78-c532a56d1aaf
Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz
Don't update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz
Get service da4612c2-ab08-4f5e-8d1a-7bd0057b4608
Don't update service da4612c2-ab08-4f5e-8d1a-7bd0057b4608
Get service a5941153-af0b-4c69-a9e3-68bc4c396e5e
Don't update service a5941153-af0b-4c69-a9e3-68bc4c396e5e
Get service c1cc086c-692e-400c-b89b-a08a83029652
Don't update service c1cc086c-692e-400c-b89b-a08a83029652
Get service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb
Don't update service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb
Get service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5
Don't update service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5
Get service 3c89ddcd-3ead-4242-be83-079f1357e8b2
Don't update service 3c89ddcd-3ead-4242-be83-079f1357e8b2
Get service 63951d4b-811a-4444-9a90-d31c1257a49b
Don't update service 63951d4b-811a-4444-9a90-d31c1257a49b
Get service c14db7f5-b259-42f7-aeb0-e14eac50e94d
Don't update service c14db7f5-b259-42f7-aeb0-e14eac50e94d
Get service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec
Don't update service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec
Get service b2821cf5-81f6-4f15-841c-0badf2ce9c73
Don't update service b2821cf5-81f6-4f15-841c-0badf2ce9c73
Get service 982b69fa-07ec-4219-9f5b-ae629aa5136f
Don't update service 982b69fa-07ec-4219-9f5b-ae629aa5136f
Get service 0c795821-c2e0-4dfd-8537-3e1740722583
Don't update service 0c795821-c2e0-4dfd-8537-3e1740722583
Get service 931f3c0e-3fbb-40a4-991e-01275f1aa000
Don't update service 931f3c0e-3fbb-40a4-991e-01275f1aa000
Get service 8b5c0028-95c5-48f1-a356-1764c9441d46
Don't update service 8b5c0028-95c5-48f1-a356-1764c9441d46
Get service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd
Don't update service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd
Updated 0 service(s)

BTW none of the LE certificates (mach.pem, chain.pem) have expired; not sure what that's about

(mach.pem was the first cert in fullchain.pem from certbot; chain.pem was the rest)

The funny thing is, before the rollback, I accessed the VCSA via the web ui and it had my LE cert on it!!  (and it showed as Secure in the browser)

/var/log/vmware/vmcad/certificate-manager.log  has these tidbits in it

 

--

2022-04-07T07:11:16.685Z INFO certificate-manager all services stopped successfully.
2022-04-07T07:11:16.685Z INFO certificate-manager None
2022-04-07T07:11:26.696Z INFO certificate-manager Running command :- service-control --start --all
2022-04-07T07:11:26.697Z INFO certificate-manager please see service-control.log for service status
Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

2022-04-07T07:18:16.888Z ERROR certificate-manager None
2022-04-07T07:18:16.889Z ERROR certificate-manager Error while starting services, please see service-control log for more details
2022-04-07T07:18:16.889Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
2022-04-07T07:18:16.889Z ERROR certificate-manager {
"detail": [
{
"id": "install.ciscommon.command.errinvoke",
"translatable": "An error occurred while invoking external command : '%(0)s'",
"args": [
"None"
],
"localized": "An error occurred while invoking external command : 'None'"
},
"Error while starting services, please see service-control log for more details"
],
"componentKey": null,
"problemId": null,
"resolution": null
}
2022-04-07T07:18:16.890Z INFO certificate-manager Performing rollback of Machine SSL Cert...

...

----

I give up; this shouldn't be that hard.

[hakanlund]

I found my self having this exact issue again and found another work around.
My guess is that vmware has an issue with the last cert in the chain,

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

So I tested replacing it with the ISRG ROOT X1 cert from: https://letsencrypt.org/certs/isrgrootx1.pem.txt
and that worked for me.

TLDR; remove the last cert in the the fullchain file and the chain file, add the cert from https://letsencrypt.org/certs/isrgrootx1.pem.txt at the end to both files.

[TValis]

The answer provided by hakanlund resolved it for me. Here is how I implemented it.

In the GUI I selected to "Import and replace certificate" under the machine cert and chose the option to replace with external CA certificate(requires private key). When presented with the three boxes, I uploaded the following files provided by LetsEncrypt certbot:

Machine SSL Certificate:  cert.pem

Chain of trusted root certificates: chain.pem

Private Key:  privkey.pem

Then I opened isrgrootx1.pem with a text editor and copied all to the clipboard and then in the box for the Chain of trusted root certificates I scrolled down to the end of the first cert and beginning of the next. I held shift while paging down and selected all of the second cert and deleted it and pasted the contents of my clipboard which contained the isrgrootx1.pem certificate.

I then clicked replace and it was successful.

 

[Tesfaye Assefa]

Any update on this, I have the same issue?

[Tesfaye Assefa]

The LetsEncrypt fullchain.pem contains the site certificate and two other CA certificates.

Both the CA certificates show up in VCSA in the Trusted Root Certificates list (checked the hex signatures and dates).

Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Validity
Not Before: Sep 4 00:00:00 2020 GMT
Not After : Sep 15 16:00:00 2025 GMT
Subject: C = US, O = Let's Encrypt, CN = R3

Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After : Sep 30 18:14:03 2024 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

I tried replacing the Machine SSL Cert with the LE cert for the site from this, along with the private key but the

form demands the chain be supplied for the submit button to work, and supplying both or either of the CA certs 

still results in the same error.

 

"Error occurred while fetching tls: the trustAnchors parameter must be non-empty"

[rbjohnson78]

This is what I did for the full chain. Hopefully this helps. Took me a long time to figure this out.

Issue a new CSR...

1. Open WinSCP
   1. Use SFTP file protocol
   2. FQDN for host name
   3. root for user name
2. After login, navigate to the /tmp folder or the folder you chose when exporting the csr and key
3. Copy the files below to a directory on your local PC
   1. vmca_issued_csr.csr
   2. vmca_issued_key.key
4. Use the copied csr file to submit to the CA authority
5. Generate the cert for Apache use
6. Once the cert is generated, download the file type as
   1. A P7B bundle of all the certs in a .p7b file
7. Open the bundle after download
8. Right click each cert > all tasks > export
9. Click next to get you to the format options
10. Select Base-64 encoded x.509 (.CER)
11. Browse to a folder to export the .cer files too
12. Use the following naming convention for each cert to make it easier to identify
    1. Vcenterhostname.domain.com: MachineSSL.cer
    2. DigiCert Global Root CA: Root.cer
    3. DigiCert TLS RSA SHA256 2020: Intermediate.cer
       (Cert names should be similar depending on your CA)

Create the chain...

1. Open the newly created Intermediate.cer file with notepad
2. Highlight and Copy everything in the open doc
3. Open the Root.cer file with notepad
4. Paste the information from the Intermediate.cer file to the top of the root.cer file.
5. Save the Root.cer file but don’t close it
6.  Select and copy all text in the Root.cer
7. Open the MachineCert.cer file with notepad
8. Paste the copied text from Root.cer file to the bottom of the doc
9. You will now have the full chain and three certs embedded
10. Save the machineSSL.cer file

[iandekit]

Thanks people, but I'm hesitant to spend more time on testing possible solutions here.

Last test I ran effectively destroyed my 7.0.3 u3d VCSA but I didn't know that until a few weeks later when I rebooted; and VPXd wouldn't start anymore and config backups had cycled away by then.

I had to do a fresh install of u3e as the upgrade failed to work either.

I should have snapshotted it before testing.   Reboots take 15 minutes before its usable which is quite annoying.

 

[hakanlund]

I'm running two different vcenter servers and on the first one I can import Let's Encrypt certificates without any issues, on the second one I could not. On the second vcenter server I could import certificates if I replaced the last certificate in the chain as explained earlier.

How ever I found that the second vcenter server I had a trusted root certificate with ID 79B459E67BB6E5E40173800888C81A58F6E99B6E that was issued by DST Root CA X3 and valid until Sep 30, 2024. but on the first vcenter server the trusted root certificate with the same ID is issued by ISRG Root X1 and valid until June 4, 2035.
The cert on my first vcenter server is the same I used when replacing the cert in the chain, acquired from https://letsencrypt.org/certificates/, https://letsencrypt.org/certs/isrgrootx1.pem

On the second vcenter server i followed this guide https://kb.vmware.com/s/article/2146011 on how to remove a certificate from the store, to remove the certificate with ID 79B459E67BB6E5E40173800888C81A58F6E99B6E and then through the UI I added the https://letsencrypt.org/certs/isrgrootx1.pem certificate to the Trusted Root Certificates.

After I hade replaced the trusted root certificate on the second vcenter server it also accepted Let's Encrypt certificates with out issue.

 

[rcolliard]

Hi

Just to let you know that on my side it had to do with windows encoding. I did a "dos2unix" on my files and I was able to import everything without this annoying error about trusted anchor.

Raphael