VMware vSphere

 View Only
  • 1.  503 Service Unavailable

    Posted May 08, 2020 01:06 PM

    Hello,

    I have 2 vcenter 6.5 servers in one sso domain.

    Some time ago I have seen alarm about certificate. So i renewed all from gui, but alarm persisted.

    Today I could not log in to vspere web ui, had error "User name and password are required" on the first server, and blank screen on the second one. So i restarted both. After reboot

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00005567895eb3d0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

    error is displayed.

    In /var/log/vmware/messages there was error:

    faultstring: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 06:42:07 UTC 2020, endTime=Fri May 08 06:52:07 UTC 2020] :: Signing certificate is not valid at Fri May 08 06:42:07 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]

    So I have decided to generate new certificates from cli with /usr/lib/vmware-vmca/bin/certificate-manager. Tried regenerating and resetting, but both operations failed when manager tried to start services. Anyway, using cmd /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text showed newly generated certificates.

    # service-control --status

    Running:

    applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

    Stopped:

    pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

    Don't know where to look at, but below are some logs. Seems like something with SSO.

    grep -i error /var/log/vmware/vpxd/vpxd.log

    2020-05-08T12:24:43.997Z error vpxd[7F6D42D58800] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)

    2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Main] Init failed. SystemError: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)

    2020-05-08T12:24:43.999Z error vpxd[7F6D42D58800] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down

    vapi/endpoint/endpoint.log

    2020-05-08T09:35:46.938Z | ERROR | state-manager1            | ComponentManagerClientWrapper  | SSO lookup failed.

    java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

            at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)

            at com.vmware.cis.cm.client.ComponentManagerClient.lookup(ComponentManagerClient.java:876)

            at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:939)

            at com.vmware.cis.cm.client.ComponentManagerClient$3.call(ComponentManagerClient.java:933)

            at com.vmware.cis.cm.client.ComponentManagerClient.retry(ComponentManagerClient.java:548)

            at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:929)

            at com.vmware.cis.cm.client.ComponentManagerClient.cachedLookup(ComponentManagerClient.java:908)

            at com.vmware.cis.cm.client.ComponentManagerClient.lookupSso(ComponentManagerClient.java:993)

            at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:171)

            at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

            at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

            at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

            at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

            at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

            at java.util.concurrent.FutureTask.run(FutureTask.java:266)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

            at java.lang.Thread.run(Thread.java:748)

    Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

            at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

            at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

            ... 3 more

    Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

            at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

            at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

            at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

            at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

            at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

            at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

            at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

            at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

            at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

            ... 3 more

    Caused by: java.net.ConnectException: Connection refused (Connection refused)

            at java.net.PlainSocketImpl.socketConnect(Native Method)

            at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

            at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

            at java.net.Socket.connect(Socket.java:589)

            at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

            at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

            ... 13 more

    2020-05-08T09:35:46.941Z | ERROR | state-manager1            | SsoSettingsBuilder             | Failded to retrieve SSO settings.

    com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.

            at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)

            at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

            at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

            at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

            at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

            at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

            at java.util.concurrent.FutureTask.run(FutureTask.java:266)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

            at java.lang.Thread.run(Thread.java:748)

    Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

            at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

            at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

            ... 3 more

    Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

            at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

            at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

            at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

            at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

            at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

            at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

            at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

            at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

            at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

            ... 3 more

    Caused by: java.net.ConnectException: Connection refused (Connection refused)

            at java.net.PlainSocketImpl.socketConnect(Native Method)

            at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

            at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

            at java.net.Socket.connect(Socket.java:589)

            at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

            at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

            ... 13 more

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Failed to retrieve SSO settings from component manager.

    2020-05-08T09:35:46.941Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.

    com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.

            at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:63)

            at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:354)

            at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:168)

            at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:151)

            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

            at java.util.concurrent.FutureTask.run(FutureTask.java:266)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

            at java.lang.Thread.run(Thread.java:748)

    Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: SSO lookup failed.

            at com.vmware.vapi.endpoint.cis.ComponentManagerClientWrapper.lookupSso(ComponentManagerClientWrapper.java:174)

            at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:171)

            at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:56)

            ... 10 more

    Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

            at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

            at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

            ... 3 more

    Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:18090 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

            at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

            at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

            at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

            at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

            at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

            at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

            at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

            at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

            at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

            ... 3 more

    Caused by: java.net.ConnectException: Connection refused (Connection refused)

            at java.net.PlainSocketImpl.socketConnect(Native Method)

            at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

            at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

            at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

            at java.net.Socket.connect(Socket.java:589)

            at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

            at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

            ... 13 more

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Failed to retrieve SSO settings from component manager.

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Application error has occurred. Please check log files for more information.

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | HealthConfigurationEventListener | HEALTH Current vApi Endpoint health status is created between 2020-05-08T09:35:46UTC and 2020-05-08T09:35:46UTC.

    2020-05-08T09:35:46.941Z | INFO  | state-manager1            | DefaultStateManager            | lock

    2020-05-08T09:35:46.942Z | INFO  | state-manager1            | DefaultStateManager            | Initial state build failed. Will retry after 5 seconds.

    2020-05-08T09:35:46.942Z | INFO  | state-manager1            | DefaultStateManager            | unlock

    2020-05-08T09:35:50.607Z | INFO  | shutdown-hook             | ApiEndpointServer              | Start shutting down...

    2020-05-08T09:35:50.607Z | INFO  | shutdown-hook             | DefaultStateManager            | shutdown

    2020-05-08T09:35:50.614Z | INFO  | shutdown-hook             | ApiEndpointServer              | Shutdown.

    But couldn't find nothing intresting in sso

    sso/ssoAdminServer.log

    [2020-05-08T12:24:43.988Z pool-6-thread-5 opId=21190014-abf6-4825-8234-ea672d2cdbb0 ERROR com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask] Failed to serialize response

    com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response

            at com.vmware.vim.vmomi.server.exception.ExceptionUtil.buildFaultForInternalException(ExceptionUtil.java:22) ~[vlsi-server.jar:?]

            at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:95) [vlsi-server.jar:?]

            at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:63) [vlsi-server.jar:?]

            at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.run(CompletionContinuerTask.java:53) [vlsi-server.jar:?]

            at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]

            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]

    Caused by: com.vmware.vim.vmomi.core.exception.MarshallException: Missing non-optional return value

            at com.vmware.vim.vmomi.server.impl.SoapBindingImpl.serializeResponse(SoapBindingImpl.java:168) ~[vlsi-server.jar:?]

            at com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask.complete(CompletionContinuerTask.java:84) ~[vlsi-server.jar:?]

            ... 6 more

    sso/vmware-identity-sts.log //still some problem with certificate?

    [2020-05-08T12:58:34.733Z tomcat-http--39 vsphere.local        97b81203-5e98-4499-b11f-1f5b6ade0789 INFO  com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:InvalidTimeRange and description: The token authority rejected an issue request for TimePeriod [startTime=Fri May 08 12:58:34 UTC 2020, endTime=Fri May 08 13:08:34 UTC 2020] :: Signing certificate is not valid at Fri May 08 12:58:34 UTC 2020, cert validity: TimePeriod [startTime=Tue May 08 20:01:11 UTC 2018, endTime=Thu May 07 20:01:11 UTC 2020]

    sso/lookupServer.log

    [2020-05-08T08:09:40.314Z ERROR] [OpenLdapClientLibrary] Exception when calling ldap_search_s: base=cn=4cea3f17-670c-4ee6-938c-c7e1aaec7cfe,cn=ServiceRegistrations,cn=LookupService,cn=silp,cn=sites,cn=configuration,dc=vsphere,dc=local, scope=2, filter=(objectclass=*), attrs=null, attrsonly=0

    com.vmware.identity.interop.ldap.NoSuchObjectLdapException: No such object

    Any help would be appreciated



  • 2.  RE: 503 Service Unavailable

    Posted May 08, 2020 01:08 PM

    Have you opened a support request with VMware GSS?



  • 3.  RE: 503 Service Unavailable

    Posted May 08, 2020 05:08 PM

    not yet, as I don't have required permissions. Will have to wait for that until Monday.



  • 4.  RE: 503 Service Unavailable

    Posted May 08, 2020 02:31 PM

    Moderator: Thread moved to the vCenter Server area.



  • 5.  RE: 503 Service Unavailable

    Posted May 09, 2020 08:04 AM

    Looks like STS certificate is expired on this.



  • 6.  RE: 503 Service Unavailable
    Best Answer

    Posted May 09, 2020 08:40 AM

    one option is to set wrong date on vcsa, start services and follow the below process:

    Generate a New STS Signing Certificate on the Appliance

    Refresh the Security Token Service Certificate

    GSS has an script to automate this process. You may open a SR.



  • 7.  RE: 503 Service Unavailable

    Posted May 11, 2020 07:18 PM

    You were right, the cause of the issue was expired STS certificate. Problem resolved by vmware support.



  • 8.  RE: 503 Service Unavailable

    Posted May 11, 2020 08:16 PM

    Good to know :smileyhappy: