vCenter

 View Only
  • 1.  vCenter fails to start after certificate replacement

    Posted Dec 19, 2019 05:32 PM

    Using vcenter 6.7 Administration - > Certificates have added root CA certificate of Letsencrypt and replaced Machine certificate with signed one provide certificate and key

    After reboot vcenter doesn`t start anymore:

    2019-12-19T17:22:23.429Z info vpxd[05606] [Originator@6876 sub=ThreadPool] Entering worker thread loop

    2019-12-19T17:22:23.430Z info vpxd[05605] [Originator@6876 sub=ThreadPool] Thread enlisted

    2019-12-19T17:22:23.430Z info vpxd[05605] [Originator@6876 sub=ThreadPool] Entering worker thread loop

    2019-12-19T17:22:23.459Z error vpxd[05321] [Originator@6876 sub=Main opID=CheckCertificateExpiry-6058ed8] Unable to get certificate count for APPLMGMT_PASSWORD from VECS localhost, error: 0

    2019-12-19T17:22:23.548Z info vpxd[05332] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 144, idle: 19

    2019-12-19T17:22:23.553Z info vpxd[05617] [Originator@6876 sub=ThreadPool] Thread enlisted

    2019-12-19T17:22:23.553Z info vpxd[05617] [Originator@6876 sub=ThreadPool] Entering worker thread loop

    2019-12-19T17:22:23.572Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while getting service with Id :e2136204-f25b-4a2b-a5ac-67b473cfd253. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

    --> [context]zKq7AVECAAAAAGC34QAOdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAHOlWUB9qFlASkvoAIqhQJsaWJhdXRoemNsaWVudC5zbwABvdeeAToJVAGKaFQBGcZSA5AFAmxpYmMuc28uNgABpb5S[/context]

    2019-12-19T17:22:23.573Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

    --> [context]zKq7AVECAAAAAGC34QAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAE4l2UBKJllASuiZQEpL6ACKoUCbGliYXV0aHpjbGllbnQuc28AAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]

    2019-12-19T17:22:23.574Z warning vpxd[05113] [Originator@6876 sub=LSClient] endpoint not found for Product: com.vmware.cis, Type: cs.inventory

    2019-12-19T17:22:23.574Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while getting service with Id :e2136204-f25b-4a2b-a5ac-67b473cfd253. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

    --> [context]zKq7AVECAAAAAGC34QAOdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAHOlWUB9qFlASkvoAI3hQJsaWJhdXRoemNsaWVudC5zbwABvdeeAToJVAGKaFQBGcZSA5AFAmxpYmMuc28uNgABpb5S[/context]

    2019-12-19T17:22:23.575Z warning vpxd[05113] [Originator@6876 sub=LSClient] Caught exception while retrieve endpoint. N7Vmacore9ExceptionE(Cannot initialize service registration stub)

    --> [context]zKq7AVECAAAAAGC34QAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGAH5kWV2cHhkAAE4l2UBKJllASuiZQEpL6ACN4UCbGliYXV0aHpjbGllbnQuc28AAb3XngE6CVQBimhUARnGUgOQBQJsaWJjLnNvLjYAAaW+Ug==[/context]

    2019-12-19T17:22:23.597Z warning vpxd[05113] [Originator@6876 sub=LSClient] endpoint not found for Product: com.vmware.cis, Type: cs.inventory

    2019-12-19T17:22:23.718Z warning vpxd[05113] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

    --> PeerThumbprint: 49:68:90:15:2C:75:C6:7C:C7:B4:55:EB:87:E2:E6:29:92:21:A8:72

    --> ExpectedThumbprint:

    --> ExpectedPeerName: localhost

    --> The remote host certificate has these problems:

    -->

    --> * Host name does not match the subject name(s) in certificate.)

    --> [context]zKq7AVECAAAAAGC34QANdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbAP6dGACeQCIAaXEiABtFIgDTSSIAOaIjAHFvIwA6ciMAnVYrAdRzAGxpYnB0aHJlYWQuc28uMAAC3Y4ObGliYy5zby42AA==[/context]

    2019-12-19T17:22:23.719Z info vpxd[05113] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate

    2019-12-19T17:22:23.729Z error vpxd[05113] [Originator@6876 sub=ServerAccess] Remote login failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

    --> PeerThumbprint: 49:68:90:15:2C:75:C6:7C:C7:B4:55:EB:87:E2:E6:29:92:21:A8:72

    --> ExpectedThumbprint:

    --> ExpectedPeerName: localhost

    --> The remote host certificate has these problems:

    -->

    --> * Host name does not match the subject name(s) in certificate.)

    When resetting certificates using /usr/lib/vmware-vmca/bin/certificate-manager it works again

    There is no ESXi host connected to vCenter just in case...



  • 2.  RE: vCenter fails to start after certificate replacement

    Posted Dec 19, 2019 07:00 PM

    Looks like the cert is incorrectly configured

    * Host name does not match the subject name(s) in certificate.



  • 3.  RE: vCenter fails to start after certificate replacement

    Posted Dec 19, 2019 11:57 PM

    Run the below commands and make sure all 3 gives you hostnmae of vCSA

    1. PNID of the vCenter server: # /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

    2. Hostname of vCenter server: # hostname -f

    3. And SAN (Subject Alternative Name) field of machine ssl cert: # openssl x509 -in machine.cer -noout -text | grep DNS:



  • 4.  RE: vCenter fails to start after certificate replacement

    Posted Jan 19, 2020 06:22 PM

    same here... cant figure out why

    tail -f /var/log/vmware/vpxd/vpxd.log

    --> ExpectedThumbprint:

    --> ExpectedPeerName: localhost

    --> The remote host certificate has these problems:

    -->

    --> * Host name does not match the subject name(s) in certificate.)

    the following command gives same result

    /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative

    openssl x509 -in <path_to_certificate_file> -noout -text | grep -A1 Alternative

    hostname -f



  • 5.  RE: vCenter fails to start after certificate replacement

    Posted May 13, 2020 02:51 AM

    Did you ever find a fix for this issue?  Having the exact same issue here.



  • 6.  RE: vCenter fails to start after certificate replacement

    Posted May 17, 2020 05:26 AM

    I had the same issue for past few weeks. Updated to 6.7.0.44000 and looks like this is resolved. I've run my playbook for renewing letsencrypt certificates a bunch of times, rebooted vcsa and everything seems to be stable so far.



  • 7.  RE: vCenter fails to start after certificate replacement

    Posted May 19, 2020 06:41 PM

    It could be also due to duplicate certificate in the trusted root store . Try running the below command and match the serial numbers . If you find duplicate serial numbers then you would have to remove them.

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text| grep -A 6 -i alias |less



  • 8.  RE: vCenter fails to start after certificate replacement

    Posted Feb 18, 2021 10:23 PM

    Was this fixed?? 

    I am facing the same issue. Followed the thread but nothing seems to work



  • 9.  RE: vCenter fails to start after certificate replacement

    Posted May 19, 2022 01:29 PM

    vcenter 6.7.0.52000

    have the same issue after renewal letencrypt certificate:

    vcenter.yyy.com

    # /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
    vcenter.yyy.com
    # hostname -f
    vcenter.yyy.com
    # /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative
    X509v3 Subject Alternative Name:
    DNS:vcenter.yyy.com


    --> PeerThumbprint: 98:FE:16:42:E3:CF:43:2B:63:C5:9D:79:9C:77:FB:BD:B2:2A:07:FA
    --> ExpectedThumbprint:
    --> ExpectedPeerName: localhost
    --> The remote host certificate has these problems:
    -->
    --> * Host name does not match the subject name(s) in certificate.)
    --> [context]zKq7AVECAAAAAAt9JgENdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAAehyIA9b4iAJuLIgBTkCIAie8jAMG8IwCKvyMA+asrAdRzAGxpYnB0aHJlYWQuc28uMAAC7Y8ObGliYy5zby42AA==[/context]
    2022-05-19T13:40:08.483Z info vpxd[36481] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
    2022-05-19T13:40:08.487Z error vpxd[36481] [Originator@6876 sub=ServerAccess] Remote login failed: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
    --> PeerThumbprint: 98:FE:16:42:E3:CF:43:2B:63:C5:9D:79:9C:77:FB:BD:B2:2A:07:FA
    --> ExpectedThumbprint:
    --> ExpectedPeerName: localhost
    --> The remote host certificate has these problems:
    -->
    --> * Host name does not match the subject name(s) in certificate.)
    --> [context]zKq7AVECAAAAAAt9JgENdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAAehyIA9b4iAJuLIgBTkCIAie8jAMG8IwCKvyMA+asrAdRzAGxpYnB0aHJlYWQuc28uMAAC7Y8ObGliYy5zby42AA==[/context]
    2022-05-19T13:40:08.488Z error vpxd[36481] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
    --> )
    --> [context]zKq7AVECAAAAAAt9JgESdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAEGEFR2cHhkAAGu9FoBmOJjAV+voAG6mKACru4BbGliYXV0aHpjbGllbnQuc28AAlcHAgLSjgICsoYCAdkvnwFSJ1QBooZUAfnjUgPgBgJsaWJjLnNvLjYAAYXcUg==[/context]>
    2022-05-19T13:40:08.490Z error vpxd[36481] [Originator@6876 sub=Default] Failed to instantiate AuthzStorageProvider: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
    --> )
    --> [context]zKq7AVECAAAAAAt9JgESdnB4ZAAAPFYrbGlidm1hY29yZS5zbwAAJEUbAFqxGAEGEFR2cHhkAAGu9FoBmOJjAV+voAG6mKACru4BbGliYXV0aHpjbGllbnQuc28AAlcHAgLSjgICsoYCAdkvnwFSJ1QBooZUAfnjUgPgBgJsaWJjLnNvLjYAAYXcUg==[/context]



    any thoughts?