vCenter

 View Only
Expand all | Collapse all

Cannot start vCenter: 503 Service unavailable

ssSFrankSss

ssSFrankSssJun 16, 2020 10:27 AM

ssSFrankSss

ssSFrankSssJun 17, 2020 05:00 PM

  • 1.  Cannot start vCenter: 503 Service unavailable

    Posted Jun 16, 2020 10:25 AM

    Hi,

    I get this message from web browser, while I try to connect to vCenter:

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000557c588763f0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

    And I have this image from shell that shows which services have started:

    There are like many solutions online, but I do not know where to start...

    I appreciate your time trying to help me,

    Frank



  • 2.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 16, 2020 10:27 AM



  • 3.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 16, 2020 10:42 AM

    I hope this helps also in order to find a solution

    Command> service-control --status

    Running:

    applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

    Stopped:

    pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

    Command> service-control --start --all

    Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True

    2020-06-16T10:39:26.127Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']

    2020-06-16T10:39:26.136Z   Done running command

    2020-06-16T10:39:26.144Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.144Z   Running command: ['/sbin/service', u'lwsmd', 'status']

    2020-06-16T10:39:26.210Z   Done running command

    Successfully started service lwsmd

    2020-06-16T10:39:26.221Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']

    2020-06-16T10:39:26.236Z   Done running command

    2020-06-16T10:39:26.245Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.245Z   Running command: ['/sbin/service', u'vmafdd', 'status']

    2020-06-16T10:39:26.308Z   Done running command

    Successfully started service vmafdd

    2020-06-16T10:39:26.317Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']

    2020-06-16T10:39:26.332Z   Done running command

    2020-06-16T10:39:26.341Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.341Z   Running command: ['/sbin/service', u'vmdird', 'status']

    2020-06-16T10:39:26.404Z   Done running command

    Successfully started service vmdird

    2020-06-16T10:39:26.413Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']

    2020-06-16T10:39:26.428Z   Done running command

    2020-06-16T10:39:26.436Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.437Z   Running command: ['/sbin/service', u'vmcad', 'status']

    2020-06-16T10:39:26.500Z   Done running command

    Successfully started service vmcad

    2020-06-16T10:39:26.509Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']

    2020-06-16T10:39:26.524Z   Done running command

    2020-06-16T10:39:26.533Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.533Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']

    2020-06-16T10:39:26.595Z   Done running command

    Successfully started service vmware-sts-idmd

    2020-06-16T10:39:26.604Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']

    2020-06-16T10:39:26.618Z   Done running command

    2020-06-16T10:39:26.626Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.626Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']

    2020-06-16T10:39:26.690Z   Done running command

    Successfully started service vmware-stsd

    2020-06-16T10:39:26.698Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']

    2020-06-16T10:39:26.713Z   Done running command

    2020-06-16T10:39:26.722Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.722Z   Running command: ['/sbin/service', u'vmdnsd', 'status']

    2020-06-16T10:39:26.784Z   Done running command

    Successfully started service vmdnsd

    2020-06-16T10:39:26.794Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']

    2020-06-16T10:39:26.809Z   Done running command

    2020-06-16T10:39:26.818Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

    2020-06-16T10:39:26.818Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']

    2020-06-16T10:39:26.880Z   Done running command

    Successfully started service vmware-psc-client

    Service-control failed. Error Failed to start vmon services.vmon-cli RC=2, stderr=Failed to start sca, cm services. Error: Service crashed while starting

    PS) vCenter runs as a VM



  • 4.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 16, 2020 12:12 PM

    Have you searched these forums for past threads and possible resolutions? This question comes up weekly with many possible causes.



  • 5.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 17, 2020 03:56 PM

    I have tried some. But I was lost since there are many questions and solutions as you said!!



  • 6.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 16, 2020 08:46 PM

    can you check the cm logs /var/log/vmware/cm/cm.log and post it here so that we can review.. cm service failed to start and thus other dependent services as well

    thanks,

    MS



  • 7.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 17, 2020 05:00 PM

    I have attached the log file thank you!



  • 8.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 17, 2020 06:00 PM

    Please run the following commands from the vCSA's shell, to check whether one, or more certificates have expired.

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient

    André



  • 9.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 18, 2020 07:22 AM

    Hi Andres,

    How to run these commands from Window. Thanks !



  • 10.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 18, 2020 07:37 AM

    Hi Andres,

    How to run these commands from Vcenter Windows.
    If there is certificate expired, how to extend it.

    I am having the same issues but with Vcenter Windows. Thank you !



  • 11.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 18, 2020 08:11 AM

    toreshavn​ have you tried the article I have proposed, it has steps to follow for vCenter for windows also!



  • 12.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 18, 2020 09:20 AM

    I have tried to the article you have proposed and reset all certificates, but it stops at 85% for starting services.
    Please see image attached for the detail.

    ============================================
    Here is log:

    2020-06-18T09:10:21.262Z INFO certificate-manager Serial number before replacement: f1:e8:d2:42:ea:06:14:b6

    2020-06-18T09:10:21.262Z INFO certificate-manager Serial number after replacement: dc:17:ba:94:d5:9f:ef:7c

    2020-06-18T09:10:21.262Z INFO certificate-manager Thumbprint before replacement: 46:2E:AD:D9:76:89:4D:60:4E:7B:EE:34:A0:E3:05:97:34:C8:70:55

    2020-06-18T09:10:21.262Z INFO certificate-manager Thumbprint after replacement: B0:DA:23:46:E2:10:FB:6C:95:BA:30:DB:05:52:C9:55:8E:2C:41:C0

    2020-06-18T09:10:21.263Z INFO certificate-manager vpxd-extension certificate replaced successfully. SerialNumber and Thumbprint changed.

    2020-06-18T09:10:21.263Z INFO certificate-manager Running command :- "C:\Program Files\VMware\vCenter Server\bin\service-control.bat" --stop --ignore  --all

    2020-06-18T09:10:21.263Z INFO certificate-manager please see service-control.log for service status

    2020-06-18T09:11:22.345Z INFO certificate-manager Command executed successfully

    2020-06-18T09:11:22.345Z INFO certificate-manager all services stopped successfully.

    2020-06-18T09:11:22.345Z INFO certificate-manager None

    2020-06-18T09:11:32.348Z INFO certificate-manager Running command :- "C:\Program Files\VMware\vCenter Server\bin\service-control.bat" --start  --all

    2020-06-18T09:11:32.348Z INFO certificate-manager please see service-control.log for service status

    2020-06-18T09:19:16.961Z ERROR certificate-manager None

    2020-06-18T09:19:16.964Z ERROR certificate-manager Error while starting services, please see log for more details

    2020-06-18T09:19:16.966Z ERROR certificate-manager {

        "resolution": null,

        "detail": [

            {

                "args": [

                    "None"

                ],

                "id": "install.ciscommon.command.errinvoke",

                "localized": "An error occurred while invoking external command : 'None'",

                "translatable": "An error occurred while invoking external command : '%(0)s'"

            },

            "Error while starting services, please see log for more details"

        ],

        "componentKey": null,

        "problemId": null



  • 13.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 18, 2020 09:42 AM

    Actually I had the same problem... It stopped at 85% when starting services! Initially I thought it was a problem too! But it wasn't just go for a coffee and come back later :smileyhappy:

    I mean just let it work - hopefully after  a while it will complete 100%



  • 14.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Jun 18, 2020 09:56 AM

    @ssSFrankSss Unfortunately, It is not like this :smileyhappy:

    "

    Reset status : 85% Completed [starting services...]

    Error while starting services, please see log for more details

    Status : 0% Completed [Reset operation failed] "



  • 15.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Mar 21, 2021 05:16 PM

    Dear,

     

       How U solve this problem ,can you share with us .I met the same problem,thx



  • 16.  RE: Cannot start vCenter: 503 Service unavailable

    Posted May 12, 2022 06:25 AM

    I have got the same problem - how did you end up fixing it?



  • 17.  RE: Cannot start vCenter: 503 Service unavailable

    Posted May 12, 2022 06:28 AM

     

    I have got the exact issue you were facing, how did you end up fixing it? 

    stopped at 85% and rolled back after with error service-control failed. Error Failed to start vmon services.vmon-cli RD=2



  • 18.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Sep 05, 2022 11:40 AM

    I stucked in 85%. 

    About 30m then it displays this notification

     


    Rollback Status : 0% Completed [Rollback operation failed]

     

    pls help me



  • 19.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Apr 25, 2021 06:22 AM

    i had sam problem but it was stop at 0 % when i try to reset :

    2021-04-25T06:11:31.466Z INFO certificate-manager Thumbprint after replacement: 11:47:4F:BA:F1:32:36:17:A2:1C:EB:75:F5:13:B5:13:4D:E6:D9:DA
    2021-04-25T06:11:31.466Z INFO certificate-manager MACHINE_SSL_CERT certificate replaced successfully. SerialNumber and Thumbprint changed.
    2021-04-25T06:11:38.891Z ERROR certificate-manager 'lstool reregister' failed: 1
    2021-04-25T06:11:38.892Z ERROR certificate-manager please see /var/log/vmware/vmcad/certificate-manager.log for more information.

     

    pls help me with this.....



  • 20.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Dec 13, 2023 09:36 AM

    From This location
    C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli.exe entry list --store vsphere-webclient --text



  • 21.  RE: Cannot start vCenter: 503 Service unavailable

    Broadcom Employee
    Posted Jun 17, 2020 05:51 PM

    From the logs i see that the cert expired :

    Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed May 06 19:32:13 UTC 2020

            at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) ~[?:1.8.0_162]

            at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629) ~[?:1.8.0_162]

            at sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) ~[?:1.8.0_162]

            at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) ~[?:1.8.0_162]

            at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:1.8.0_162]

            at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223) ~[?:1.8.0_162]

    Seems we might have any expired certs in VECS store .

    You can follow below kb to verify:

    Kb2111411



  • 22.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Apr 25, 2021 06:43 AM

    pls help me, i had sent my log ,do i have same problem......

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x000055b7fe084690] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

    type : vCenter Server with an external Platform Services Controller

     



  • 23.  RE: Cannot start vCenter: 503 Service unavailable
    Best Answer

    Posted Jun 17, 2020 10:27 PM

    Thank you all it was a certificate issue this article helped me to return everything back to normal:

    VMware Knowledge Base



  • 24.  RE: Cannot start vCenter: 503 Service unavailable

    Posted Dec 07, 2023 05:29 AM

    Hi Enthusiast,

    I was following the below Article.. But when it prompts for Credentials.. It is not accepting. 

    In your case, what password was provided ?

    | 7. Revert last performed operation by re-publishing old |
    | certificates |
    | |
    | 8. Reset all Certificates |
    |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
    Note : Use Ctrl-D to exit.
    Option[1 to 8]: 8
    Do you wish to generate all certificates using configuration file : Option[Y/N] ? : Y

    Please provide valid SSO and VC priviledged user credential to perform certificate operations.
    Enter username [Administrator@vsphere.local]:
    Enter password:

    Incorrect Password! Try Again! (2 attempt/s left)
    Password:

    Incorrect Password! Try Again! (1 attempt/s left)
    Password:

    Reached max number of attempts, terminating operation.
    root@rc01vm02 [ ~ ]#

     

    Here i tried with administrator@vsphere.local and root

    Enter username [Administrator@vsphere.local]: