VMware vSphere

 View Only
Expand all | Collapse all

Certificate Management

  • 1.  Certificate Management

    Posted Apr 06, 2021 01:37 PM

    Anyone ever come across this message when trying to import a cert into "Machine SSL Certificate" in vcenter 7: Error occurred while fetching tls: the trustAnchors parameter must be non-empty



  • 2.  RE: Certificate Management

    Posted May 18, 2021 04:01 PM

    Beating my head against this now. I'm a total newb to PKI and we just installed a Microsoft Offline RootCA and a Subordinate/Issuing CA to our infrastructure for these types of things.  I have found several blogs that walk through this process, however each one of them leads me right to this error.  Frustrating.  Any guidance would be greatly appreaciated.



  • 3.  RE: Certificate Management

    Posted Oct 25, 2021 08:20 AM

    I'm seeing this also on recent vCenter's; tried again on the latest vCenter 7.0.3 18778458 and it still occurs

    "Error occurred while fetching tls: the trustAnchors parameter must be non-empty"

    I'm feeding VCSA a recently generated Let Encrypt certificate.   Originally tried wildcard but said that wasn't supported (Doh!)

    so made a non-wildcard one...

     

    I also tried loading it via the shell, using /usr/lib/vmware-vmca/bin/certificate-manager  but after much churning it fails too.

    -------

    You are going to replace Machine SSL cert using custom cert
    Continue operation : Option[Y/N] ? : Y
    Command Output: cert1.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
    error 10 at 3 depth lookup:certificate has expired
    OK

    Get site nameCompleted [Replacing Machine SSL Cert...]                  

    ... 

    Updated 43 service(s)
    Status : 85% Completed [starting services...]                   
    Error while starting services, please see service-control log for more details

    Status : 0% Completed [Operation failed, performing automatic rollback]
                     
    Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

    Performing rollback of Machine SSL Cert...
    Get site nameus : 0% Completed [Rollback Machine SSL Cert...]   

    ---------

    /var/log/vmware/vmcad/certificate-manager.log didn't really say a lot other than this:

    ----

    2021-10-25T00:52:15.433Z INFO certificate-manager Running command :- service-control --start  --all
    2021-10-25T00:52:15.434Z INFO certificate-manager please see service-control.log for service status
    Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

    2021-10-25T00:59:16.445Z ERROR certificate-manager None
    2021-10-25T00:59:16.446Z ERROR certificate-manager Error while starting services, please see service-control log for more details
    2021-10-25T00:59:16.446Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

    2021-10-25T00:59:16.446Z ERROR certificate-manager {
       "detail": [
           {
               "id": "install.ciscommon.command.errinvoke",
               "translatable": "An error occurred while invoking external command : '%(0)s'",
               "args": [
                   "None"
               ],
               "localized": "An error occurred while invoking external command : 'None'"
           },
           "Error while starting services, please see service-control log for more details"
       ],
       "componentKey": null,
       "problemId": null,
       "resolution": null
    }
    2021-10-25T00:59:16.446Z INFO certificate-manager Performing rollback of Machine SSL Cert...

    ---

    It did roll back ok though.



  • 4.  RE: Certificate Management

    Posted Dec 13, 2021 07:55 AM

    Hi guys,

    Anny update on this, I have the same issue?

    Robin



  • 5.  RE: Certificate Management

    Posted Jan 12, 2022 08:04 PM

    Experienced the same issue with a "cheap" cert. The reason at least for my issue was the chain of trusted root certificates was not complete.

    Most ca's will give you an intermediate-ca chain. Whats missing in most cases is the root certificate of this chain. In my case the root CA was unknown by vSphere so it must be part of the chain.

    To do so find the root cert of the intermediate-ca chain and add it as the first PEM section to the intermediate-ca. That will most probably solve your issue.



  • 6.  RE: Certificate Management

    Posted Jan 13, 2022 05:38 AM

    Hi,

    The LetsEncrypt fullchain.pem contains the site certificate and two other CA certificates.

    Both the CA certificates show up in VCSA in the Trusted Root Certificates list (checked the hex signatures and dates).


    Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
    Validity
    Not Before: Sep 4 00:00:00 2020 GMT
    Not After : Sep 15 16:00:00 2025 GMT
    Subject: C = US, O = Let's Encrypt, CN = R3

    Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
    Validity
    Not Before: Jan 20 19:14:03 2021 GMT
    Not After : Sep 30 18:14:03 2024 GMT
    Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

    I tried replacing the Machine SSL Cert with the LE cert for the site from this, along with the private key but the

    form demands the chain be supplied for the submit button to work, and supplying both or either of the CA certs 

    still results in the same error.

     

    "Error occurred while fetching tls: the trustAnchors parameter must be non-empty"



  • 7.  RE: Certificate Management

    Posted Feb 08, 2022 03:12 AM

    Problem still exists in latest VCSA 7.0.3.00300 (U3c)



  • 8.  RE: Certificate Management

    Posted Feb 09, 2022 09:00 AM

    I hade the same issue with letsencrypt certificates, but found this blog post https://virtuallywired.io/2021/11/29/replace-default-vcenter-certificate-with-a-free-lets-encrypt-ssl/ wich helped med solve the issue.

    Also (I don't remember if this was relevant) the certificate can NOT be a wildcard cert, I had issues with that as well.



  • 9.  RE: Certificate Management

    Posted Feb 09, 2022 10:38 AM

    Alas that was exactly what I was doing, and it didn't work for me.

    Yes I know wildcards aren't supported ... would also be nice if they were, but not critical.



  • 10.  RE: Certificate Management

    Posted Mar 09, 2022 04:07 PM

    Hello,

    Did you resolve the issue? I have the same problem 



  • 11.  RE: Certificate Management

    Posted Mar 09, 2022 05:48 PM

    I ended up calling VMware and they were able to help me fix this. My issue was, that I didn't have my chain cert correct. Being new to how the certs work, I didn't know I had to concatenate the cert files and make sure they were listed in the correct order. This is what I followed. https://docs.vmware.com/en/vRealize-Log-Insight/8.6/com.vmware.log-insight.administration.doc/GUID-1203BA5A-3E23-463A-8297-BCD3A9D380E1.html



  • 12.  RE: Certificate Management

    Posted Jun 15, 2022 01:28 AM

    This worked for me as well after several frustrating attempts of importing it every other way. Thanks!



  • 13.  RE: Certificate Management

    Posted Jun 28, 2022 03:15 PM

    Anny update on this, I have the same issue?



  • 14.  RE: Certificate Management

    Posted Mar 10, 2022 12:01 AM

    No, alas.



  • 15.  RE: Certificate Management

    Posted Mar 29, 2022 12:28 PM

    My working solution on vCenter 7.0.2 for Letsencrypt certificate's.

    I start with creating a new cert.pem file that i call cert_combined.pem containing the cert.pem cert and after that the two certs from chain.pem.
    I uploaded cert_combined.pem, chain.pem and privkey.pem to the vcenter server and executed:

    I chose option 1 (Replace Machine SSL certificate with Custom Certificate),
    provided my credentials and the chose option 2 (Replace Machine SSL certificate with Custom Certificate)

    I hops this is of some help.



  • 16.  RE: Certificate Management

    Posted Apr 01, 2022 08:30 AM

    Update to my post, the cert_combined.pem file i created is the same as the fullchain.pem file that certbot generates.



  • 17.  RE: Certificate Management

    Posted Apr 05, 2022 07:56 AM

    Thanks, but trying that on 7.0.3 build 19234570 doesn't work for me...

    Ran /usr/lib/vmware-vmca/bin/certificate-manager, selected option 1 (Replace Machine SSL certificate with Custom Certficate) ... and then authenticated... it then says this and stops:

     

    Certificate Manager tool do not support vCenter HA systems

    It doesn't get as far asking for the cert files.

     

     



  • 18.  RE: Certificate Management

    Posted Apr 05, 2022 07:58 AM

    I should add that this is running with an Essentials licence, and there is no HA.



  • 19.  RE: Certificate Management

    Posted Apr 05, 2022 08:01 AM

    (back originally I was running under an Eval licence for Standard)



  • 20.  RE: Certificate Management

    Posted Apr 07, 2022 01:52 AM

    I resolve this one with importing all the intermediate certificate in one file. I use xolphin certificate for my vcenter. intermediate public is sectigo - and usertrust. so in the "cahin of trusted root certificates" i use both stacked like that

     

    -----BEGIN CERTIFICATE-----
    MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
    iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
    cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
    BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTgx
    MTAyMDAwMDAwWhcNMzAxMjMxMjM1OTU5WjCBjzELMAkGA1UEBhMCR0IxGzAZBgNV
    BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEYMBYGA1UE
    ChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQDEy5TZWN0aWdvIFJTQSBEb21haW4g
    VmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENBMIIBIjANBgkqhkiG9w0BAQEFAAOC
    AQ8AMIIBCgKCAQEA1nMz1tc8INAA0hdFuNY+B6I/x0HuMjDJsGz99J/LEpgPLT+N
    TQEMgg8Xf2Iu6bhIefsWg06t1zIlk7cHv7lQP6lMw0Aq6Tn/2YHKHxYyQdqAJrkj
    eocgHuP/IJo8lURvh3UGkEC0MpMWCRAIIz7S3YcPb11RFGoKacVPAXJpz9OTTG0E
    oKMbgn6xmrntxZ7FN3ifmgg0+1YuWMQJDgZkW7w33PGfKGioVrCSo1yfu4iYCBsk
    Haswha6vsC6eep3BwEIc4gLw6uBK0u+QDrTBQBbwb4VCSmT3pDCg/r8uoydajotY
    uK3DGReEY+1vVv2Dy2A0xHS+5p3b4eTlygxfFQIDAQABo4IBbjCCAWowHwYDVR0j
    BBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFI2MXsRUrYrhd+mb
    +ZsF4bgBjWHhMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0G
    A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAbBgNVHSAEFDASMAYGBFUdIAAw
    CAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0
    LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9yaXR5LmNybDB2Bggr
    BgEFBQcBAQRqMGgwPwYIKwYBBQUHMAKGM2h0dHA6Ly9jcnQudXNlcnRydXN0LmNv
    bS9VU0VSVHJ1c3RSU0FBZGRUcnVzdENBLmNydDAlBggrBgEFBQcwAYYZaHR0cDov
    L29jc3AudXNlcnRydXN0LmNvbTANBgkqhkiG9w0BAQwFAAOCAgEAMr9hvQ5Iw0/H
    ukdN+Jx4GQHcEx2Ab/zDcLRSmjEzmldS+zGea6TvVKqJjUAXaPgREHzSyrHxVYbH
    7rM2kYb2OVG/Rr8PoLq0935JxCo2F57kaDl6r5ROVm+yezu/Coa9zcV3HAO4OLGi
    H19+24rcRki2aArPsrW04jTkZ6k4Zgle0rj8nSg6F0AnwnJOKf0hPHzPE/uWLMUx
    RP0T7dWbqWlod3zu4f+k+TY4CFM5ooQ0nBnzvg6s1SQ36yOoeNDT5++SR2RiOSLv
    xvcRviKFxmZEJCaOEDKNyJOuB56DPi/Z+fVGjmO+wea03KbNIaiGCpXZLoUmGv38
    sbZXQm2V0TP2ORQGgkE49Y9Y3IBbpNV9lXj9p5v//cWoaasm56ekBYdbqbe4oyAL
    l6lFhd2zi+WJN44pDfwGF/Y4QA5C5BIG+3vzxhFoYt/jmPQT2BVPi7Fp2RBgvGQq
    6jG35LWjOhSbJuMLe/0CjraZwTiXWTb2qHSihrZe68Zk6s+go/lunrotEbaGmAhY
    LcmsJWTyXnW0OMGuf1pGg+pRyrbxmRE1a6Vqe8YAsOf4vmSyrcjC8azjUeqkk+B5
    yOGBQMkKW+ESPMFgKuOXwIlCypTPRpgSabuY0MLTDXJLR27lk8QyKGOHQ+SwMj4K
    00u/I5sUKUErmgQfky3xxzlIPK1aEn8=
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB
    iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
    cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
    BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw
    MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV
    BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU
    aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy
    dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK
    AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B
    3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY
    tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/
    Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2
    VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT
    79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6
    c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT
    Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l
    c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee
    UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE
    Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd
    BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G
    A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF
    Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO
    VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3
    ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs
    8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR
    iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze
    Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ
    XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/
    qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB
    VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB
    L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG
    jjxDah2nGN59PRbxYvnKkKj9
    -----END CERTIFICATE-----

    and I was able to import my machine ssl certificate and the private key nicely.



  • 21.  RE: Certificate Management

    Posted Apr 07, 2022 07:28 AM

    Just tried again using VCSA web ui with 7.0.3 00500 (u3d) ... (with Essentials licence) ... no better.

    - pasted machine specific cert into first box; rest of LE chain in 2nd box and private key in 3rd box

    "Error occurred while fetching tls: the trustAnchors parameter must be non-empty"

    However, via shell, it seems to work better this time...

    /usr/lib/vmware-vmca/bin/certificate-manager 

    Option[1 to 8]: 1

    Please provide valid SSO and VC privileged user credential to perform certificate operations.
    Enter username [Administrator@vsphere.local]:administrator@vsphere1.local
    Enter password:
    1. Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate

    2. Import custom certificate(s) and key(s) to replace existing Machine SSL certificate

    Option [1 or 2]: 2

    Please provide valid custom certificate for Machine SSL.
    File : mach.pem

    Please provide valid custom key for Machine SSL.
    File : privkey.pem

    Please provide the signing certificate of the Machine SSL certificate
    File : chain.pem

    You are going to replace Machine SSL cert using custom cert
    Continue operation : Option[Y/N] ? : Y
    Command Output: mach.pem: O = Digital Signature Trust Co., CN = DST Root CA X3
    error 10 at 3 depth lookup:certificate has expired
    OK

    Get site nameCompleted [Replacing Machine SSL Cert...]
    default-first-site
    Lookup all services
    Get service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3
    Update service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3; spec: /tmp/svcspec_gk1j6msc
    Get service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875
    Update service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875; spec: /tmp/svcspec_nkgb1iw5
    Get service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f
    Update service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f; spec: /tmp/svcspec_uxp7q_b0
    Get service 4893d3bb-13af-444d-960f-9921f78b46ef
    Update service 4893d3bb-13af-444d-960f-9921f78b46ef; spec: /tmp/svcspec__vuw_yab
    Get service bedd2511-d562-4651-9b3e-396262ebb7e2
    Update service bedd2511-d562-4651-9b3e-396262ebb7e2; spec: /tmp/svcspec_99mvk_4m
    Get service d2edb72a-c5b2-4d36-b70c-601bfb138c93
    Update service d2edb72a-c5b2-4d36-b70c-601bfb138c93; spec: /tmp/svcspec_lg44eljk
    Get service d1b9b4bf-d248-4e06-b717-f48778fd5df6
    Update service d1b9b4bf-d248-4e06-b717-f48778fd5df6; spec: /tmp/svcspec_mk3cux2o
    Get service 909ce2ce-df14-485b-a3a8-c9dc84384200

    Update service 909ce2ce-df14-485b-a3a8-c9dc84384200; spec: /tmp/svcspec_va2x08s3
    Get service ddff5bef-f435-4c1a-88b9-d6629c7b0c78
    Update service ddff5bef-f435-4c1a-88b9-d6629c7b0c78; spec: /tmp/svcspec_f7l8u632
    Get service a6a8f33a-492d-43b5-b5ff-91de64d816a3
    Update service a6a8f33a-492d-43b5-b5ff-91de64d816a3; spec: /tmp/svcspec_xlen8yjv
    Get service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc
    Update service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc; spec: /tmp/svcspec_94d0er4o
    Get service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b
    Update service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b; spec: /tmp/svcspec_1r2hh7x_
    Get service e4eb4b29-53b3-4208-8561-70d183c3f790
    Update service e4eb4b29-53b3-4208-8561-70d183c3f790; spec: /tmp/svcspec_yla75a4o
    Get service 549b49fb-2991-4c49-8ddc-e9ccff4905aa
    Update service 549b49fb-2991-4c49-8ddc-e9ccff4905aa; spec: /tmp/svcspec_msrozdz2
    Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7
    Update service 8505c9d2-faf9-4306-9034-e38fbc7224e7; spec: /tmp/svcspec_d902vatn
    Get service 072493ad-d644-4817-a7c0-75c9111a6155
    Update service 072493ad-d644-4817-a7c0-75c9111a6155; spec: /tmp/svcspec_xft2na_v
    Get service 36ca1bc5-4a55-4175-8f31-361215949e8c
    Update service 36ca1bc5-4a55-4175-8f31-361215949e8c; spec: /tmp/svcspec_v4hui5wd
    Get service 2c847253-2e8e-4705-ad7d-1f671039b7ca
    Update service 2c847253-2e8e-4705-ad7d-1f671039b7ca; spec: /tmp/svcspec_go_71a3k
    Get service 70906193-2f34-415b-a8cf-e1212e18e93b
    Update service 70906193-2f34-415b-a8cf-e1212e18e93b; spec: /tmp/svcspec_e8s6_h5y
    Get service de956063-1652-4215-b6ae-ec5e6b4ba272
    Don't update service de956063-1652-4215-b6ae-ec5e6b4ba272
    Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34
    Update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34; spec: /tmp/svcspec_o532w120
    Get service 6bbf80db-8685-4867-8c55-59d3ba4799f1
    Update service 6bbf80db-8685-4867-8c55-59d3ba4799f1; spec: /tmp/svcspec_q6g8h42h
    Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv
    Update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv; spec: /tmp/svcspec_lassjge8
    Get service 757f879d-67d9-4d5d-bb27-01287ec25585
    Update service 757f879d-67d9-4d5d-bb27-01287ec25585; spec: /tmp/svcspec_3fr1lp04
    Get service 5b04af9d-524f-4368-a93d-2e09ae43b2aa
    Update service 5b04af9d-524f-4368-a93d-2e09ae43b2aa; spec: /tmp/svcspec_z22yuja9
    Get service fb7108d7-0f76-4ea2-8bac-e7c514157c0a
    Update service fb7108d7-0f76-4ea2-8bac-e7c514157c0a; spec: /tmp/svcspec_8u_li1zr
    Get service 248c802f-aecd-4769-92b5-5da089a802e4
    Update service 248c802f-aecd-4769-92b5-5da089a802e4; spec: /tmp/svcspec_md1w6guz
    Get service 756da5b5-8635-42b3-9d78-c532a56d1aaf
    Update service 756da5b5-8635-42b3-9d78-c532a56d1aaf; spec: /tmp/svcspec_7isozv20
    Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
    Don't update service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
    Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz
    Update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz; spec: /tmp/svcspec_sw8u6e_x
    Get service da4612c2-ab08-4f5e-8d1a-7bd0057b4608
    Update service da4612c2-ab08-4f5e-8d1a-7bd0057b4608; spec: /tmp/svcspec_9jhd5vmv
    Get service a5941153-af0b-4c69-a9e3-68bc4c396e5e
    Update service a5941153-af0b-4c69-a9e3-68bc4c396e5e; spec: /tmp/svcspec_upe42sfk
    Get service c1cc086c-692e-400c-b89b-a08a83029652
    Update service c1cc086c-692e-400c-b89b-a08a83029652; spec: /tmp/svcspec_1wk8g0k6
    Get service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb
    Update service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb; spec: /tmp/svcspec_1_v6oglz
    Get service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5
    Update service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5; spec: /tmp/svcspec_3r3mg1vv
    Get service 3c89ddcd-3ead-4242-be83-079f1357e8b2
    Update service 3c89ddcd-3ead-4242-be83-079f1357e8b2; spec: /tmp/svcspec_4oqyzyvv
    Get service 63951d4b-811a-4444-9a90-d31c1257a49b
    Update service 63951d4b-811a-4444-9a90-d31c1257a49b; spec: /tmp/svcspec_ynbu47jf
    Get service c14db7f5-b259-42f7-aeb0-e14eac50e94d
    Update service c14db7f5-b259-42f7-aeb0-e14eac50e94d; spec: /tmp/svcspec_bxdlnalf
    Get service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec
    Update service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec; spec: /tmp/svcspec_bzr02o51
    Get service b2821cf5-81f6-4f15-841c-0badf2ce9c73
    Update service b2821cf5-81f6-4f15-841c-0badf2ce9c73; spec: /tmp/svcspec_337fmjqa
    Get service 982b69fa-07ec-4219-9f5b-ae629aa5136f
    Update service 982b69fa-07ec-4219-9f5b-ae629aa5136f; spec: /tmp/svcspec_obfkk20u
    Get service 0c795821-c2e0-4dfd-8537-3e1740722583
    Update service 0c795821-c2e0-4dfd-8537-3e1740722583; spec: /tmp/svcspec_mguwb7ig
    Get service 931f3c0e-3fbb-40a4-991e-01275f1aa000
    Update service 931f3c0e-3fbb-40a4-991e-01275f1aa000; spec: /tmp/svcspec_gimbuirw
    Get service 8b5c0028-95c5-48f1-a356-1764c9441d46
    Update service 8b5c0028-95c5-48f1-a356-1764c9441d46; spec: /tmp/svcspec_yzhs35e8
    Get service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd
    Update service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd; spec: /tmp/svcspec_6olyr8vl

    ... about 12 minutes later ... (OMG why is this so slow...)

    Updated 43 service(s)

    Status : 85% Completed [starting services...]
    Error while starting services, please see service-control log for more details
    Status : 0% Completed [Operation failed, performing automatic rollback]

    Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

    Performing rollback of Machine SSL Cert...
    Get site nameus : 0% Completed [Rollback Machine SSL Cert...]
    default-first-site
    Lookup all services
    Get service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3
    Don't update service default-first-site:c9925fb8-fde7-4fad-9375-8e72435507e3
    Get service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875
    Don't update service default-first-site:b93d77a3-5740-4201-9e9a-53941a5ad875
    Get service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f
    Don't update service default-first-site:d4c06daa-21e9-4f2a-a66e-d5939d5fd05f
    Get service 4893d3bb-13af-444d-960f-9921f78b46ef
    Don't update service 4893d3bb-13af-444d-960f-9921f78b46ef
    Get service bedd2511-d562-4651-9b3e-396262ebb7e2
    Don't update service bedd2511-d562-4651-9b3e-396262ebb7e2
    Get service d2edb72a-c5b2-4d36-b70c-601bfb138c93
    Don't update service d2edb72a-c5b2-4d36-b70c-601bfb138c93
    Get service d1b9b4bf-d248-4e06-b717-f48778fd5df6
    Don't update service d1b9b4bf-d248-4e06-b717-f48778fd5df6
    Get service 909ce2ce-df14-485b-a3a8-c9dc84384200
    Don't update service 909ce2ce-df14-485b-a3a8-c9dc84384200
    Get service ddff5bef-f435-4c1a-88b9-d6629c7b0c78
    Don't update service ddff5bef-f435-4c1a-88b9-d6629c7b0c78
    Get service a6a8f33a-492d-43b5-b5ff-91de64d816a3
    Don't update service a6a8f33a-492d-43b5-b5ff-91de64d816a3
    Get service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc
    Don't update service 20df66ab-8d43-4965-b13b-b11e1e4f6cfc
    Get service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b
    Don't update service c43dba09-a988-4f8b-aa9e-a4fea1de3b5b
    Get service e4eb4b29-53b3-4208-8561-70d183c3f790
    Don't update service e4eb4b29-53b3-4208-8561-70d183c3f790
    Get service 549b49fb-2991-4c49-8ddc-e9ccff4905aa
    Don't update service 549b49fb-2991-4c49-8ddc-e9ccff4905aa
    Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7
    Don't update service 8505c9d2-faf9-4306-9034-e38fbc7224e7
    Get service 072493ad-d644-4817-a7c0-75c9111a6155
    Don't update service 072493ad-d644-4817-a7c0-75c9111a6155
    Get service 36ca1bc5-4a55-4175-8f31-361215949e8c
    Don't update service 36ca1bc5-4a55-4175-8f31-361215949e8c
    Get service 2c847253-2e8e-4705-ad7d-1f671039b7ca
    Don't update service 2c847253-2e8e-4705-ad7d-1f671039b7ca
    Get service 70906193-2f34-415b-a8cf-e1212e18e93b
    Don't update service 70906193-2f34-415b-a8cf-e1212e18e93b
    Get service de956063-1652-4215-b6ae-ec5e6b4ba272
    Don't update service de956063-1652-4215-b6ae-ec5e6b4ba272
    Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34
    Don't update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34
    Get service 6bbf80db-8685-4867-8c55-59d3ba4799f1
    Don't update service 6bbf80db-8685-4867-8c55-59d3ba4799f1
    Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv
    Don't update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_kv
    Get service 757f879d-67d9-4d5d-bb27-01287ec25585
    Don't update service 757f879d-67d9-4d5d-bb27-01287ec25585
    Get service 5b04af9d-524f-4368-a93d-2e09ae43b2aa
    Don't update service 5b04af9d-524f-4368-a93d-2e09ae43b2aa
    Get service fb7108d7-0f76-4ea2-8bac-e7c514157c0a
    Don't update service fb7108d7-0f76-4ea2-8bac-e7c514157c0a
    Get service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
    Don't update service 8505c9d2-faf9-4306-9034-e38fbc7224e7_com.vmware.vsphere.client
    Get service 248c802f-aecd-4769-92b5-5da089a802e4
    Don't update service 248c802f-aecd-4769-92b5-5da089a802e4
    Get service 756da5b5-8635-42b3-9d78-c532a56d1aaf
    Don't update service 756da5b5-8635-42b3-9d78-c532a56d1aaf
    Get service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz
    Don't update service 4efec6ec-3a5f-432c-ba4a-5deb39d45f34_authz
    Get service da4612c2-ab08-4f5e-8d1a-7bd0057b4608
    Don't update service da4612c2-ab08-4f5e-8d1a-7bd0057b4608
    Get service a5941153-af0b-4c69-a9e3-68bc4c396e5e
    Don't update service a5941153-af0b-4c69-a9e3-68bc4c396e5e
    Get service c1cc086c-692e-400c-b89b-a08a83029652
    Don't update service c1cc086c-692e-400c-b89b-a08a83029652
    Get service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb
    Don't update service c6845dc9-386d-4b2f-bd0f-d8df62cdbfbb
    Get service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5
    Don't update service 90dcbf11-e60d-4d7a-b22f-e4fff798d1f5
    Get service 3c89ddcd-3ead-4242-be83-079f1357e8b2
    Don't update service 3c89ddcd-3ead-4242-be83-079f1357e8b2
    Get service 63951d4b-811a-4444-9a90-d31c1257a49b
    Don't update service 63951d4b-811a-4444-9a90-d31c1257a49b
    Get service c14db7f5-b259-42f7-aeb0-e14eac50e94d
    Don't update service c14db7f5-b259-42f7-aeb0-e14eac50e94d
    Get service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec
    Don't update service 5ae858a8-297d-4d5c-9e2b-33b6c09da4ec
    Get service b2821cf5-81f6-4f15-841c-0badf2ce9c73
    Don't update service b2821cf5-81f6-4f15-841c-0badf2ce9c73
    Get service 982b69fa-07ec-4219-9f5b-ae629aa5136f
    Don't update service 982b69fa-07ec-4219-9f5b-ae629aa5136f
    Get service 0c795821-c2e0-4dfd-8537-3e1740722583
    Don't update service 0c795821-c2e0-4dfd-8537-3e1740722583
    Get service 931f3c0e-3fbb-40a4-991e-01275f1aa000
    Don't update service 931f3c0e-3fbb-40a4-991e-01275f1aa000
    Get service 8b5c0028-95c5-48f1-a356-1764c9441d46
    Don't update service 8b5c0028-95c5-48f1-a356-1764c9441d46
    Get service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd
    Don't update service 67e643fa-a31f-48d4-8a88-7c18e37cd9dd
    Updated 0 service(s)

    BTW none of the LE certificates (mach.pem, chain.pem) have expired; not sure what that's about

    (mach.pem was the first cert in fullchain.pem from certbot; chain.pem was the rest)

    The funny thing is, before the rollback, I accessed the VCSA via the web ui and it had my LE cert on it!!  (and it showed as Secure in the browser)

    /var/log/vmware/vmcad/certificate-manager.log  has these tidbits in it

     

    --

    2022-04-07T07:11:16.685Z INFO certificate-manager all services stopped successfully.
    2022-04-07T07:11:16.685Z INFO certificate-manager None
    2022-04-07T07:11:26.696Z INFO certificate-manager Running command :- service-control --start --all
    2022-04-07T07:11:26.697Z INFO certificate-manager please see service-control.log for service status
    Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

    2022-04-07T07:18:16.888Z ERROR certificate-manager None
    2022-04-07T07:18:16.889Z ERROR certificate-manager Error while starting services, please see service-control log for more details
    2022-04-07T07:18:16.889Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
    2022-04-07T07:18:16.889Z ERROR certificate-manager {
    "detail": [
    {
    "id": "install.ciscommon.command.errinvoke",
    "translatable": "An error occurred while invoking external command : '%(0)s'",
    "args": [
    "None"
    ],
    "localized": "An error occurred while invoking external command : 'None'"
    },
    "Error while starting services, please see service-control log for more details"
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
    }
    2022-04-07T07:18:16.890Z INFO certificate-manager Performing rollback of Machine SSL Cert...

    ...

    ----

    I give up; this shouldn't be that hard.



  • 22.  RE: Certificate Management

    Posted Jun 28, 2022 01:05 PM

    I found my self having this exact issue again and found another work around.
    My guess is that vmware has an issue with the last cert in the chain,

    So I tested replacing it with the ISRG ROOT X1 cert from: https://letsencrypt.org/certs/isrgrootx1.pem.txt
    and that worked for me.

    TLDR; remove the last cert in the the fullchain file and the chain file, add the cert from https://letsencrypt.org/certs/isrgrootx1.pem.txt at the end to both files.



  • 23.  RE: Certificate Management

    Posted Oct 31, 2022 02:11 AM

    The answer provided by hakanlund resolved it for me. Here is how I implemented it.

    In the GUI I selected to "Import and replace certificate" under the machine cert and chose the option to replace with external CA certificate(requires private key). When presented with the three boxes, I uploaded the following files provided by LetsEncrypt certbot:

    Machine SSL Certificate:  cert.pem

    Chain of trusted root certificates: chain.pem

    Private Key:  privkey.pem

    Then I opened isrgrootx1.pem with a text editor and copied all to the clipboard and then in the box for the Chain of trusted root certificates I scrolled down to the end of the first cert and beginning of the next. I held shift while paging down and selected all of the second cert and deleted it and pasted the contents of my clipboard which contained the isrgrootx1.pem certificate.

    I then clicked replace and it was successful.

     



  • 24.  RE: Certificate Management

    Posted Jun 28, 2022 03:18 PM

    Any update on this, I have the same issue?



  • 25.  RE: Certificate Management

    Posted Jun 28, 2022 03:19 PM

    The LetsEncrypt fullchain.pem contains the site certificate and two other CA certificates.

    Both the CA certificates show up in VCSA in the Trusted Root Certificates list (checked the hex signatures and dates).


    Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
    Validity
    Not Before: Sep 4 00:00:00 2020 GMT
    Not After : Sep 15 16:00:00 2025 GMT
    Subject: C = US, O = Let's Encrypt, CN = R3

    Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
    Validity
    Not Before: Jan 20 19:14:03 2021 GMT
    Not After : Sep 30 18:14:03 2024 GMT
    Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

    I tried replacing the Machine SSL Cert with the LE cert for the site from this, along with the private key but the

    form demands the chain be supplied for the submit button to work, and supplying both or either of the CA certs 

    still results in the same error.

     

    "Error occurred while fetching tls: the trustAnchors parameter must be non-empty"



  • 26.  RE: Certificate Management
    Best Answer

    Posted Jun 28, 2022 03:36 PM

    This is what I did for the full chain. Hopefully this helps. Took me a long time to figure this out.

    Issue a new CSR...

    1. Open WinSCP
      1. Use SFTP file protocol
      2. FQDN for host name
      3. root for user name
    2. After login, navigate to the /tmp folder or the folder you chose when exporting the csr and key
    3. Copy the files below to a directory on your local PC
      1. vmca_issued_csr.csr
      2. vmca_issued_key.key
    4. Use the copied csr file to submit to the CA authority
    5. Generate the cert for Apache use
    6. Once the cert is generated, download the file type as
      1. A P7B bundle of all the certs in a .p7b file
    7. Open the bundle after download
    8. Right click each cert > all tasks > export
    9. Click next to get you to the format options
    10. Select Base-64 encoded x.509 (.CER)
    11. Browse to a folder to export the .cer files too
    12. Use the following naming convention for each cert to make it easier to identify
      1. Vcenterhostname.domain.com: MachineSSL.cer
      2. DigiCert Global Root CA: Root.cer
      3. DigiCert TLS RSA SHA256 2020: Intermediate.cer
        (Cert names should be similar depending on your CA)

    Create the chain...

    1. Open the newly created Intermediate.cer file with notepad
    2. Highlight and Copy everything in the open doc
    3. Open the Root.cer file with notepad
    4. Paste the information from the Intermediate.cer file to the top of the root.cer file.
    5. Save the Root.cer file but don’t close it
    6.  Select and copy all text in the Root.cer
    7. Open the MachineCert.cer file with notepad
    8. Paste the copied text from Root.cer file to the bottom of the doc
    9. You will now have the full chain and three certs embedded
    10. Save the machineSSL.cer file


  • 27.  RE: Certificate Management

    Posted Jun 29, 2022 03:30 AM

    Thanks people, but I'm hesitant to spend more time on testing possible solutions here.

    Last test I ran effectively destroyed my 7.0.3 u3d VCSA but I didn't know that until a few weeks later when I rebooted; and VPXd wouldn't start anymore and config backups had cycled away by then.

    I had to do a fresh install of u3e as the upgrade failed to work either.

    I should have snapshotted it before testing.   Reboots take 15 minutes before its usable which is quite annoying.

     



  • 28.  RE: Certificate Management

    Posted Jun 30, 2022 01:01 PM

    I'm running two different vcenter servers and on the first one I can import Let's Encrypt certificates without any issues, on the second one I could not. On the second vcenter server I could import certificates if I replaced the last certificate in the chain as explained earlier.

    How ever I found that the second vcenter server I had a trusted root certificate with ID 79B459E67BB6E5E40173800888C81A58F6E99B6E that was issued by DST Root CA X3 and valid until Sep 30, 2024. but on the first vcenter server the trusted root certificate with the same ID is issued by ISRG Root X1 and valid until June 4, 2035.
    The cert on my first vcenter server is the same I used when replacing the cert in the chain, acquired from https://letsencrypt.org/certificates/https://letsencrypt.org/certs/isrgrootx1.pem

    On the second vcenter server i followed this guide https://kb.vmware.com/s/article/2146011 on how to remove a certificate from the store, to remove the certificate with ID 79B459E67BB6E5E40173800888C81A58F6E99B6E and then through the UI I added the https://letsencrypt.org/certs/isrgrootx1.pem certificate to the Trusted Root Certificates.

    After I hade replaced the trusted root certificate on the second vcenter server it also accepted Let's Encrypt certificates with out issue.

     



  • 29.  RE: Certificate Management

    Posted Mar 28, 2023 07:43 AM

    Hi

    Just to let you know that on my side it had to do with windows encoding. I did a "dos2unix" on my files and I was able to import everything without this annoying error about trusted anchor.

    Raphael