VMware vSphere

 View Only
  • 1.  vmware-vpxd: vpxd failed to initialize in time.

    Posted Feb 08, 2023 09:16 PM

    Hello, I am here because I am try a lot of KBs and didn't find a way to make vcenter back to work, I hope one of you could help me.

    I have my vCenter Server Appliance 6.0.0 workin on a ESXi 6.0.0, I am using a VMware vSphere Client to connect to vCenter and ESXi.

    Now when I tey to connect to vCenter I get this message:

    Celso_Finti_0-1675889669213.png

     

    In the browser I get this message:

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7fadaf0c8130] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

     

    Stopping, Starting or Restarting VMware vCenter Server Appliance 6.x & above services (2109887)

    When I check for status:

     

     

     

    service-control --all --status
    INFO:root:Service: vmafdd, Action: status
    Service: vmafdd, Action: status
    2023-02-08T20:49:52.025Z Running command: ['/sbin/service', u'vmafdd', 'status']
    2023-02-08T20:49:52.041Z Done running command
    INFO:root:Service: vmware-rhttpproxy, Action: status
    Service: vmware-rhttpproxy, Action: status
    2023-02-08T20:49:52.042Z Running command: ['/sbin/service', u'vmware-rhttpproxy', 'status']
    2023-02-08T20:49:52.077Z Done running command
    INFO:root:Service: vmdird, Action: status
    Service: vmdird, Action: status
    2023-02-08T20:49:52.078Z Running command: ['/sbin/service', u'vmdird', 'status']
    2023-02-08T20:49:52.093Z Done running command
    INFO:root:Service: vmcad, Action: status
    Service: vmcad, Action: status
    2023-02-08T20:49:52.094Z Running command: ['/sbin/service', u'vmcad', 'status']
    2023-02-08T20:49:52.109Z Done running command
    INFO:root:Service: vmware-sts-idmd, Action: status
    Service: vmware-sts-idmd, Action: status
    2023-02-08T20:49:52.110Z Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']
    2023-02-08T20:49:52.122Z Done running command
    INFO:root:Service: vmware-stsd, Action: status
    Service: vmware-stsd, Action: status
    2023-02-08T20:49:52.122Z Running command: ['/sbin/service', u'vmware-stsd', 'status']
    2023-02-08T20:49:52.153Z Done running command
    INFO:root:Service: vmware-cm, Action: status
    Service: vmware-cm, Action: status
    2023-02-08T20:49:52.153Z Running command: ['/sbin/service', u'vmware-cm', 'status']
    2023-02-08T20:49:52.268Z Done running command
    INFO:root:Service: vmware-cis-license, Action: status
    Service: vmware-cis-license, Action: status
    2023-02-08T20:49:52.268Z Running command: ['/sbin/service', u'vmware-cis-license', 'status']
    2023-02-08T20:49:52.383Z Done running command
    INFO:root:Service: vmware-sca, Action: status
    Service: vmware-sca, Action: status
    2023-02-08T20:49:52.384Z Running command: ['/sbin/service', u'vmware-sca', 'status']
    2023-02-08T20:49:52.491Z Done running command
    INFO:root:Service: applmgmt, Action: status
    Service: applmgmt, Action: status
    2023-02-08T20:49:52.491Z Running command: ['/sbin/service', u'applmgmt', 'status']
    2023-02-08T20:49:52.537Z Done running command
    INFO:root:Service: vmware-netdumper, Action: status
    Service: vmware-netdumper, Action: status
    2023-02-08T20:49:52.537Z Running command: ['/sbin/service', u'vmware-netdumper', 'status']
    2023-02-08T20:49:52.582Z Done running command
    INFO:root:Service: vmware-syslog, Action: status
    Service: vmware-syslog, Action: status
    2023-02-08T20:49:52.583Z Running command: ['/sbin/service', u'vmware-syslog', 'status']
    2023-02-08T20:49:52.634Z Done running command
    INFO:root:Service: vmware-syslog-health, Action: status
    Service: vmware-syslog-health, Action: status
    2023-02-08T20:49:52.634Z Running command: ['/sbin/service', u'vmware-syslog-health', 'status']
    2023-02-08T20:49:52.759Z Done running command
    INFO:root:Service: vmware-vapi-endpoint, Action: status
    Service: vmware-vapi-endpoint, Action: status
    2023-02-08T20:49:52.760Z Running command: ['/sbin/service', u'vmware-vapi-endpoint', 'status']
    2023-02-08T20:49:52.811Z Done running command
    INFO:root:Service: vmware-vpostgres, Action: status
    Service: vmware-vpostgres, Action: status
    2023-02-08T20:49:52.811Z Running command: ['/sbin/service', u'vmware-vpostgres', 'status']
    2023-02-08T20:49:52.897Z Done running command
    INFO:root:Service: vmware-invsvc, Action: status
    Service: vmware-invsvc, Action: status
    2023-02-08T20:49:52.898Z Running command: ['/sbin/service', u'vmware-invsvc', 'status']
    2023-02-08T20:49:53.012Z Done running command
    INFO:root:Service: vmware-mbcs, Action: status
    Service: vmware-mbcs, Action: status
    2023-02-08T20:49:53.013Z Running command: ['/sbin/service', u'vmware-mbcs', 'status']
    2023-02-08T20:49:53.447Z Done running command
    INFO:root:Service: vmware-vpxd, Action: status
    Service: vmware-vpxd, Action: status
    2023-02-08T20:49:53.448Z Running command: ['/sbin/service', u'vmware-vpxd', 'status']
    2023-02-08T20:49:53.787Z Done running command
    INFO:root:Service: vmware-eam, Action: status
    Service: vmware-eam, Action: status
    2023-02-08T20:49:53.788Z Running command: ['/sbin/service', u'vmware-eam', 'status']
    2023-02-08T20:49:53.849Z Done running command
    INFO:root:Service: vmware-rbd-watchdog, Action: status
    Service: vmware-rbd-watchdog, Action: status
    2023-02-08T20:49:53.850Z Running command: ['/sbin/service', u'vmware-rbd-watchdog', 'status']
    2023-02-08T20:49:53.947Z Done running command
    INFO:root:Service: vmware-sps, Action: status
    Service: vmware-sps, Action: status
    2023-02-08T20:49:53.948Z Running command: ['/sbin/service', u'vmware-sps', 'status']
    2023-02-08T20:49:54.080Z Done running command
    INFO:root:Service: vmware-vdcs, Action: status
    Service: vmware-vdcs, Action: status
    2023-02-08T20:49:54.081Z Running command: ['/sbin/service', u'vmware-vdcs', 'status']
    2023-02-08T20:49:54.181Z Done running command
    INFO:root:Service: vmware-vpx-workflow, Action: status
    Service: vmware-vpx-workflow, Action: status
    2023-02-08T20:49:54.181Z Running command: ['/sbin/service', u'vmware-vpx-workflow', 'status']
    2023-02-08T20:49:54.233Z Done running command
    INFO:root:Service: vmware-vsm, Action: status
    Service: vmware-vsm, Action: status
    2023-02-08T20:49:54.233Z Running command: ['/sbin/service', u'vmware-vsm', 'status']
    2023-02-08T20:49:54.299Z Done running command
    INFO:root:Service: vsphere-client, Action: status
    Service: vsphere-client, Action: status
    2023-02-08T20:49:54.300Z Running command: ['/sbin/service', u'vsphere-client', 'status']
    2023-02-08T20:49:54.412Z Done running command
    INFO:root:Service: vmware-perfcharts, Action: status
    Service: vmware-perfcharts, Action: status
    2023-02-08T20:49:54.412Z Running command: ['/sbin/service', u'vmware-perfcharts', 'status']
    2023-02-08T20:49:54.466Z Done running command
    INFO:root:Service: vmware-vws, Action: status
    Service: vmware-vws, Action: status
    2023-02-08T20:49:54.467Z Running command: ['/sbin/service', u'vmware-vws', 'status']
    2023-02-08T20:49:54.534Z Done running command
    INFO:root:Running:
    applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Framework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vmware-cis-license (VMware License Service) vmware-cm (VMware Component Manager) vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager) vsphere-client ()
    Running:
    applmgmt (VMware Appliance Management Service) vmafdd (VMware Authentication Framework) vmcad (VMware Certificate Service) vmdird (VMware Directory Service) vmware-cis-license (VMware License Service) vmware-cm (VMware Component Manager) vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-sts-idmd (VMware Identity Management Service) vmware-stsd (VMware Security Token Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vpostgres (VMware Postgres) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager) vsphere-client ()
    INFO:root:Stopped:
    vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-perfcharts (VMware Performance Charts) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server)
    Stopped:
    vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-perfcharts (VMware Performance Charts) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server)

     

     

    If I try start vmware-vpxd I get this error:

     

    service-control --start vmware-vpxd
    INFO:root:Service: vmware-vpxd, Action: start
    Service: vmware-vpxd, Action: start
    2023-02-08T20:55:19.781Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
    2023-02-08T20:55:19.816Z   Done running command
    2023-02-08T20:55:19.817Z   Running command: ['/sbin/service', u'vmware-vpxd', 'status']
    2023-02-08T20:55:20.153Z   Done running command
    2023-02-08T20:55:20.154Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-vpxd', 'on']
    2023-02-08T20:55:20.189Z   Done running command
    2023-02-08T20:55:20.189Z   Running command: ['/sbin/service', u'vmware-vpxd', 'start']
    2023-02-08T21:01:34.114Z   Done running command
    2023-02-08T21:01:34.114Z   Invoked command: ['/sbin/service', u'vmware-vpxd', 'start']
    2023-02-08T21:01:34.114Z   RC = 1
    Stdout = vmware-vpxd: VC SSL Certificate does not exist, it will be generated by vpxd
    Waiting for the embedded database to start up: success
    Executing pre-startup scripts...
    vmware-vpxd: Starting vpxd by administrative request.
    success
    vmware-vpxd: Waiting for vpxd to start listening for requests on 8089
    Waiting for vpxd to initialize: .....................................failed
    failed
    vmware-vpxd: vpxd failed to initialize in time.
    vpxd is already starting up. Aborting the request.
    
    Stderr =
    2023-02-08T21:01:34.114Z   {
        "resolution": null,
        "detail": [
            {
                "args": [
                    "Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: "
                ],
                "id": "install.ciscommon.command.errinvoke",
                "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-vpxd', 'start']\nStderr: '",
                "translatable": "An error occurred while invoking external command : '%(0)s'"
            }
        ],
        "componentKey": null,
        "problemId": null
    }
    ERROR:root:Unable to start service vmware-vpxd, Exception: {
        "resolution": null,
        "detail": [
            {
                "args": [
                    "vmware-vpxd"
                ],
                "id": "install.ciscommon.service.failstart",
                "localized": "An error occurred while starting service 'vmware-vpxd'",
                "translatable": "An error occurred while starting service '%(0)s'"
            }
        ],
        "componentKey": null,
        "problemId": null
    }
    Unable to start service vmware-vpxd, Exception: {
        "resolution": null,
        "detail": [
            {
                "args": [
                    "vmware-vpxd"
                ],
                "id": "install.ciscommon.service.failstart",
                "localized": "An error occurred while starting service 'vmware-vpxd'",
                "translatable": "An error occurred while starting service '%(0)s'"
            }
        ],
        "componentKey": null,
        "problemId": null
    }

     

     

    vsphere_client_virgo.log

     

    tail /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            at java.lang.reflect.Method.invoke(Unknown Source)
            at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
            at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)
            at org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:70)
            at java.util.TimerThread.mainLoop(Unknown Source)
            at java.util.TimerThread.run(Unknown Source)
    
    [2023-02-08T21:03:36.329Z] [ERROR] ing.timer.TimerFactoryBean#0  com.vmware.vise.vim.cm.healthstatus.InventoryServiceHealth        Search failed for Health Status Endpointjava.lang.IllegalArgumentException: SAML token is required for signing.
    [2023-02-08T21:03:36.329Z] [ERROR] ing.timer.TimerFactoryBean#0  com.vmware.vise.vim.cm.healthstatus.InventoryServiceHealth        Could not find Health Status EndPoint for Inventory Service

     

     

     

    vpxd.log

     

    tail /var/log/vmware/vpxd/vpxd.log
    2023-02-08T20:59:32.595Z info vpxd[7F53340AB7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.
    2023-02-08T20:59:32.622Z info vpxd[7F53340AB7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Retrieved trusted STS certificate: O=vcenter55,C=US,DC=local,DC=vsphere,CN=CA, TP = 3C:05:E1:FD:96:3E:4F:B3:0F:87:61:8E:BC:D5:49:BC:03:C6:21:6A
    2023-02-08T20:59:32.622Z info vpxd[7F53340AB7A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] Retrieved trusted STS certificate: emailAddress=ssl-certificates@vmware.com,CN=vcenter.canall.com.br CA 7c9fd84b,O=VMware\, Inc.,L=Palo Alto,ST=California,C=US, TP = 29:DA:40:E0:27:A2:CC:90:BA:E3:21:0A:CC:6C:62:3B:15:BA:D7:72
    2023-02-08T20:59:32.648Z error vpxd[7F53340AB7A0] [Originator@6876 sub=componentManagerUtil] AcquireToken for solution (vpxd-06887578-2176-4dec-a243-127ebd2fd6af@vsphere.local): SsoException: Unexpected SOAP fault: ns0:InvalidTimeRange; request failed.
    2023-02-08T20:59:32.648Z error vpxd[7F53340AB7A0] [Originator@6876 sub=licenseServiceDiscovery] Failed to discover license service endpoint: N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:InvalidTimeRange; request failed.)
    2023-02-08T20:59:32.651Z warning vpxd[7F53340AB7A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 2988 ms
    2023-02-08T20:59:32.651Z error vpxd[7F53340AB7A0] [Originator@6876 sub=Main] [VpxdMain] Failed to initialize N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:InvalidTimeRange; request failed.)
    2023-02-08T20:59:32.651Z error vpxd[7F53340AB7A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...
    2023-02-08T20:59:32.651Z info vpxd[7F53340AB7A0] [Originator@6876 sub=SupportMgr] Wrote uptime information
    2023-02-08T21:01:32.652Z info vpxd[7F53340AB7A0] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now

     

     

    I put the date manually on the ESXi and vCenter, so could be some difference in seconds.

    https://kb.vmware.com/s/article/2118563

    Already update my password...

    I have space on the discs...

    I do know what is the problem neither how to solve, please, someone?

     

     



  • 2.  RE: vmware-vpxd: vpxd failed to initialize in time.
    Best Answer

    Broadcom Employee
    Posted Feb 14, 2023 04:26 PM

    Check for expired certs and renew



  • 3.  RE: vmware-vpxd: vpxd failed to initialize in time.

    Posted Feb 22, 2023 06:58 PM

    Hi, for some reason I was unable to answer my own post so I create another one to continue explain what I did next.

    https://communities.vmware.com/t5/VMware-vCenter-Discussions/The-token-authority-rejected-an-issue-request-for-TimePeriod/m-p/2954069