An error occurred when processing the metadata during vCenter Single Sign-On setup

[RaoulSchaffner]

Dear all,

after the weekend, my test vCenter no longer shows the vSphere-UI login screen, but this:
[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - java.lang.reflect.InvocationTargetException.

Googling does not show meaningful results, log files on the vCenter indicate a credential problem with the SSO admin server:

File: /var/log/vmware/vapi/endpoint/endpoint.log

2025-05-19T08:55:29.133Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking keystore-builder
2025-05-19T08:55:29.133Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking endpoint-settings-builder
2025-05-19T08:55:29.135Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking tracer-builder
2025-05-19T08:55:29.136Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking lookup-service-client-builder
2025-05-19T08:55:29.136Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking http-settings-builder
2025-05-19T08:55:29.136Z | INFO  | state-manager1            | HttpSettingsBuilder            |                                      | HTTP Endpoint default=http://127.0.0.1, ::1:12346
2025-05-19T08:55:29.136Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking static-configuration-utilities
2025-05-19T08:55:29.136Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking configuration-utilities
2025-05-19T08:55:29.144Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking http-server
2025-05-19T08:55:29.144Z | INFO  | state-manager1            | BaseServerBuilder              |                                      | Server instances already started. Do nothing.
2025-05-19T08:55:29.144Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking trusted-certificates-cache-builder
2025-05-19T08:55:29.144Z | INFO  | state-manager1            | CertificateUtil                |                                      | Creating anonymous SSO Admin Client for URI http://localhost:1080/sso-adminserver/system-sdk
2025-05-19T08:55:29.162Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking cis-sso-settings-builder
2025-05-19T08:55:29.171Z | INFO  | state-manager1            | DefaultStateManager            |                                      | Invoking sts-builder
2025-05-19T08:55:29.184Z | ERROR | state-manager1            | SoapBindingImpl                |                                      | SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure
at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178) ~[?:1.8.0_422
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116) ~[?:1.8.0_422]
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259) ~[?:1.8.0_422]
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289) ~[?:1.8.0_422]
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:198) [wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:120) [wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983
[wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902
[wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509) [wstClient.jar:?]
at com.vmware.vapi.endpoint.sso.context.SsoUtil.createToken(SsoUtil.java:75) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.cis.LocalStsConfigurator.configure(LocalStsConfigurator.java:146) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.cis.StsBuilder.configureNoCache(StsBuilder.java:134) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:57) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:374) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:170) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:153) [vapi-endpoint-1.0.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_422]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_422]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_422]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]
2025-05-19T08:55:29.184Z | INFO  | state-manager1            | StatusInfoFactory              |                                      | HEALTH ORANGE Failed to login in SSO.
2025-05-19T08:55:29.184Z | ERROR | state-manager1            | DefaultStateManager            |                                      | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Cannot initalize STS and obtain token on any of the alternative endpoints. The last error if any is included
at com.vmware.vapi.endpoint.cis.LocalStsConfigurator.configure(LocalStsConfigurator.java:153) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.cis.StsBuilder.configureNoCache(StsBuilder.java:134) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:57) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:374) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:170) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:153) [vapi-endpoint-1.0.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_422]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_422]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_422]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]
Caused by: com.vmware.vapi.endpoint.sso.StsException: Error acquiring token
at com.vmware.vapi.endpoint.sso.context.SsoUtil.createToken(SsoUtil.java:78) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.cis.LocalStsConfigurator.configure(LocalStsConfigurator.java:146) ~[vapi-endpoint-1.0.0.jar:?]
... 12 more
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:1066) ~[wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:988) ~[wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) ~[wstClient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509) ~[wstClient.jar:?]
at com.vmware.vapi.endpoint.sso.context.SsoUtil.createToken(SsoUtil.java:75) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.cis.LocalStsConfigurator.configure(LocalStsConfigurator.java:146) ~[vapi-endpoint-1.0.0.jar:?]
... 12 more
2025-05-19T08:55:29.184Z | INFO  | state-manager1            | StatusInfoFactory              |                                      | HEALTH ORANGE Application error has occurred. Please check log files for more information.
2025-05-19T08:55:29.184Z | INFO  | state-manager1            | StatusInfoFactory              |                                      | HEALTH GREEN Configuration health status is created between 2025-05-19T08:55:29UTC and 2025-05-19T08:55:29UTC.
2025-05-19T08:55:29.184Z | INFO  | state-manager1            | HealthStatusCollectorImpl      |                                      | Computed health status is ORANGE.

Questions:
- Am I looking at the right place?
- Should I try to reset the SSO admin user [e-mail]?
- There seems to be another user involved as per the config file, should I look there?

#******************************************************************************
# Endpoint Instance
#******************************************************************************
endpoint.instance.id=32130288-cb20-4d8b-9197-5974181f87c4
endpoint.instance.ldu.guid=23a06a56-894f-4340-b363-4cdc5a07624f
endpoint.instance.user=[e-mail]
endpoint.instance.key.keystore=private
endpoint.instance.key.alias=vsphere-webclient
#******************************************************************************

Thank you very much,
Raoul.

[RaoulSchaffner]

answering my own post for others to learn: it was triggered by expired solution user certificates. i could renew the certificates with the vCert utility provided here: https://knowledge.broadcom.com/external/article/385107

before:

Checking Certificate Status
-----------------------------------------------------------------
Checking Machine SSL certificate                            VALID
Checking Solution User certificates:
   machine                                                EXPIRED
   vsphere-webclient                                      EXPIRED
   vpxd                                                   EXPIRED
   vpxd-extension                                         EXPIRED
   hvc                                                    EXPIRED
   wcp                                                      VALID

after:

Checking Certificate Status
-----------------------------------------------------------------
Checking Machine SSL certificate                            VALID
Checking Solution User certificates:
   machine                                                  VALID
   vsphere-webclient                                        VALID
   vpxd                                                     VALID
   vpxd-extension                                           VALID
   hvc                                                      VALID
   wcp                                                      VALID

cheers,
Raoul.

[ChrisLeblanc]

Thank you for replying to educate others. I wish more did that. I've become a big fan of the vCert tool. Its now part of every cert change I do.