certificate-manager 'lstool reregister' failed | Fresh Install | SSL certificate verification failed

[TryllZ]

Hi,

I have trying to get this work for some days now, have installed fresh numerous times, it just won't work.

vCenter Appliance version is 6.7.0 build-16709110 (6.7 U3 J, tried latest version U3 L as well), changed browser as well none of it helped.

Once vCenter is installed I am only able to login once, the 1st time, and thats it. Once the vCenter is restart the HTML UI shows "Failed to connect to VMware Lookup Service - SSL Certificate Verification Failed"

I have tried Reset All Certifications (option 8 in Certificate Manager) which was successful but upon vCenter restart the same error again.

If someone can help as this has been going on for days now trying to troubleshooting one method after another from kb.vmware.com.

I attempted a CA certificate using option 1 in Certificate manager and using the CSR to get the certificate from Windows Server CA (no intermediate CA) but that method as instructed on https://kb.vmware.com/s/article/2112014 does not work either.

Any help is appreciated.

2021-01-03T23:23:43.575Z INFO certificate-manager ['__MACHINE_CERT']
2021-01-03T23:23:43.623Z INFO certificate-manager lstool command currently being executed is : ['/usr/java/jre-vmware/bin/java', '-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', '-cp', '/usr/lib/vmidentity/tools/lib/lookup-client.jar:/usr/lib/vmidentity/tools/lib/*', '-Dlog4j.configuration=tool-log4j.properties', 'com.vmware.vim.lookup.client.tool.LsTool', 'get-site-id', '--no-check-cert', '--url', 'https://vCSA.vlab.lab:443/lookupservice/sdk']
2021-01-03T23:23:44.634Z INFO certificate-manager lstool command currently being executed is : ['/usr/java/jre-vmware/bin/java', '-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', '-cp', '/usr/lib/vmidentity/tools/lib/lookup-client.jar:/usr/lib/vmidentity/tools/lib/*', '-Dlog4j.configuration=tool-log4j.properties', 'com.vmware.vim.lookup.client.tool.LsTool', 'list', '--no-check-cert', '--url', 'https://vCSA.vlab.lab:443/lookupservice/sdk', '--id-only']
2021-01-03T23:23:45.874Z INFO certificate-manager lstool command currently being executed is : ['/usr/java/jre-vmware/bin/java', '-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', '-cp', '/usr/lib/vmidentity/tools/lib/lookup-client.jar:/usr/lib/vmidentity/tools/lib/*', '-Dlog4j.configuration=tool-log4j.properties', 'com.vmware.vim.lookup.client.tool.LsTool', 'get55', '--no-check-cert', '--url', 'https://vCSA.vlab.lab:443/lookupservice/sdk', '--id', 'default-site:ead635c0-b556-45eb-9646-d1c917913c06', '--as-spec']
2021-01-03T23:23:46.846Z INFO certificate-manager lstool command currently being executed is : ['/usr/java/jre-vmware/bin/java', '-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', '-cp', '/usr/lib/vmidentity/tools/lib/lookup-client.jar:/usr/lib/vmidentity/tools/lib/*', '-Dlog4j.configuration=tool-log4j.properties', 'com.vmware.vim.lookup.client.tool.LsTool', 'reregister55', '--no-check-cert', '--url', 'https://vCSA.vlab.lab:443/lookupservice/sdk', '--id', 'default-site:ead635c0-b556-45eb-9646-d1c917913c06', '--spec', '/tmp/svcspec_esplbyv7', '--user', '[e-mail]', '--password', '*****']
2021-01-03T23:23:48.98Z ERROR certificate-manager 'lstool reregister' failed: 1
2021-01-03T23:23:48.98Z INFO certificate-manager Error while reverting certificate for store : MACHINE_SSL_CERT
2021-01-03T23:23:48.99Z ERROR certificate-manager Error while performing rollback operation, please try Reset operation...
2021-01-03T23:23:48.99Z ERROR certificate-manager please see /var/log/vmware/vmcad/certificate-manager.log for more information.

[Alex_Romeo]

Hi,

Before installation, did you create the DNS record for the vCenter on your DNS?
Did you enter DNS parameters when installing vCenter?

in the test environment i had the same problem in the past and it was the missing DNS record during installation.

Alex_Romeo

 

[TryllZ]

The DNS entries are in place, and DNS resolves fine both from the HTML UI and CLI.

The error is once vCenter is installed completely.

[Alex_Romeo]

Hi,

it also happened to me that after installation I only logged into vCenter once and then it also gave me the error.
I remember running a couple of tests including DNS and changing the vCenter 6.7 U3L ISO image to 6.7 U3 ... in the end it worked.

 

Alex_Romeo

[TryllZ]

Thanks  

I have been through 6.7 U3, U3 J, U3 L with the same issue.

Currently I'm looking at the below for the same issue, one relates to vCenter and the other to Windows Server CA certificate algorithm.

https://mueller-tech.com/2019/06/28/replacing-expired-certificates/

https://www.reddit.com/r/vmware/comments/6vlk1l/psc_vmca/

[TryllZ]

Hi,

I manage to get the issue resolved following the explanation on https://mueller-tech.com/2019/06/28/replacing-expired-certificates/

The vCenter is up and running, have restarted and checked as well.

Thanks for all the help.

[lmaxfield]

I've run into many vcenter/VMware product certificate replacement issues over the years, so I'm sure I've seen various permutations of this same error with different causes. None of the fixes out there, including the one in this thread, addressed the issue I've been dealing with for the past couple of days, so I'm putting what I found in this thread in case it helps someone else.

I am hitting this issue on a vCenter v6.7 server. Some of the registered services show v6.5.0, but the server version is v6.7. This appears to be normal.

In the cert replacement log (/var/log/vmware/vmcad/certificate-manager.log) I saw the following error, essentially exactly the same series of messages as appears in the OP's post. Note the 'ID' of the service that is failing the reregister.

 

2022-02-07T15:18:55.671Z INFO certificate-manager lstool command currently being executed is : ['/usr/java/jre-vmware/bin/java', '-Djava.security.properties=/etc/vmware/java/vmware-override-java.security', '-cp', '/usr/lib/vmidentity/tools/lib/lookup-client.jar:/usr/lib/vmidentity/tools/lib/*', '-Dlog4j.configuration=tool-log4j.properties', 'com.vmware.vim.lookup.client.tool.LsTool', 'reregister55', '--no-check-cert', '--url', 'https://<VCENTER_FQDN>/lookupservice/sdk', '--id', 'local:7', '--spec', '/tmp/svcspec_9f34k7xn', '--user', '<SSO_ADMIN>', '--password', '*****']
2022-02-07T15:18:58.6Z ERROR certificate-manager 'lstool reregister' failed: 1
2022-02-07T15:18:58.6Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
2022-02-07T15:18:58.6Z ERROR certificate-manager 'lstool reregister' failed: 1
2022-02-07T15:18:58.6Z INFO certificate-manager Performing rollback of Machine SSL Cert...

 

Unfortunately, certificate-manager.log isn't capturing the output of the commands it's running. However, you can take the command that appears here and convert all of the delimiters (', ') to spaces to get a command that you can run yourself. When the tool fails, the "spec" that is created is not deleted, so the "spec" file should still be in /tmp/, allowing you to try the command again and view its output. For example:

 

/usr/java/jre-vmware/bin/java -Djava.security.properties=/etc/vmware/java/vmware-override-java.security -cp /usr/lib/vmidentity/tools/lib/lookup-client.jar:/usr/lib/vmidentity/tools/lib/* -Dlog4j.configuration=tool-log4j.properties com.vmware.vim.lookup.client.tool.LsTool reregister55 --no-check-cert --url https://<VCENTER_FQDN>/lookupservice/sdk --id local:7 --spec /tmp/svcspec_9f34k7xn --user <SSO_ADMIN> --password <PASSWORD>

 

When executing this command, I'd get a bunch of output, including an error message at the end:

 

(lookup.fault.ServiceFault) {
faultCause = null,
faultMessage = null,
errorMessage = Already exists
}

 

This was actually vaguely useful, unlike the output in certificate-manager.log and made me interested to check the registered services on my vcenter. You can do that with the commands in this KB: Process to view the List of Services Registered with Single Sign-On (2043509)

On vcenter 6.7, for example, you can view the services registered with SSO with this command, on the local shell:

python /usr/lib/vmidentity/tools/scripts/lstool.py list --url http://localhost:7080/lookupservice/sdk | less

 

In the output of "lstool.py list", I searched for "local:7". I could see that it appeared to be "Service Type: vsphereclient". I searched the rest of the output for another service with type "vsphereclient" and found there were two services with this type:

 

-------------------------------------------------------
Name: cis.vsphere-client.ServiceName
Description: cis.vsphere-client.ServiceDescription
Service Product: com.vmware.cis
Service Type: vsphereclient
Service ID: local:7
Site ID: local
Owner ID: vsphere-client-<VCENTER_NAME>[e-mail]
Version: 6.5.0
-------------------------------------------------------
Name: cis.vsphere-client.ServiceName
Description: cis.vsphere-client.ServiceDescription
Service Product: com.vmware.cis
Service Type: vsphereclient
Service ID: de1da181-6a32-4841-a406-d230f61e86e7
Site ID: local
Node ID: b4e06b6d-f5b6-41b1-adf4-f37d0bf20d5a
Owner ID: [e-mail]
Version: 6.5.0

 

Note that the failing "local:7" service has no "Node ID", and the "Service ID" is not a GUID, unlike all of the other services. Referencing another vcenter in my environment, I found that there should only be one service with type "vsphereclient", the service should likely have a "Node ID" listed, and the "Service ID" should likely be a GUID.

I unregistered the "bad", duplicate "local:7" vsphereclient service using this command:

 

python /usr/lib/vmidentity/tools/scripts/lstool.py unregister --url https://<VCENTER_FQDN>/lookupservice/sdk --id local:7 --user <SSO_ADMIN> --password <PASSWORD> --no-check-cert

 

I verified it was gone using the above "lstool.py list" command; I now have only one service with type "vsphereclient". After unregistering the duplicate service, I was able to successfully replace the certificate.