Cannot start vCenter: 503 Service unavailable

[ssSFrankSss]

Hi,

I get this message from web browser, while I try to connect to vCenter:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x0000557c588763f0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

And I have this image from shell that shows which services have started:

There are like many solutions online, but I do not know where to start...

I appreciate your time trying to help me,

Frank

[ssSFrankSss]

[ssSFrankSss]

I hope this helps also in order to find a solution

Command> service-control --status

Running:

applmgmt lwsmd vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-eam vmware-psc-client vmware-rhttpproxy vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vmon vmware-vpostgres vsphere-client vsphere-ui

Stopped:

pschealth vmcam vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sca vmware-sps vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm

Command> service-control --start --all

Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True

2020-06-16T10:39:26.127Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'lwsmd']

2020-06-16T10:39:26.136Z   Done running command

2020-06-16T10:39:26.144Z   Service lwsmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.144Z   Running command: ['/sbin/service', u'lwsmd', 'status']

2020-06-16T10:39:26.210Z   Done running command

Successfully started service lwsmd

2020-06-16T10:39:26.221Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmafdd']

2020-06-16T10:39:26.236Z   Done running command

2020-06-16T10:39:26.245Z   Service vmafdd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.245Z   Running command: ['/sbin/service', u'vmafdd', 'status']

2020-06-16T10:39:26.308Z   Done running command

Successfully started service vmafdd

2020-06-16T10:39:26.317Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdird']

2020-06-16T10:39:26.332Z   Done running command

2020-06-16T10:39:26.341Z   Service vmdird does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.341Z   Running command: ['/sbin/service', u'vmdird', 'status']

2020-06-16T10:39:26.404Z   Done running command

Successfully started service vmdird

2020-06-16T10:39:26.413Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmcad']

2020-06-16T10:39:26.428Z   Done running command

2020-06-16T10:39:26.436Z   Service vmcad does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.437Z   Running command: ['/sbin/service', u'vmcad', 'status']

2020-06-16T10:39:26.500Z   Done running command

Successfully started service vmcad

2020-06-16T10:39:26.509Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-sts-idmd']

2020-06-16T10:39:26.524Z   Done running command

2020-06-16T10:39:26.533Z   Service vmware-sts-idmd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.533Z   Running command: ['/sbin/service', u'vmware-sts-idmd', 'status']

2020-06-16T10:39:26.595Z   Done running command

Successfully started service vmware-sts-idmd

2020-06-16T10:39:26.604Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-stsd']

2020-06-16T10:39:26.618Z   Done running command

2020-06-16T10:39:26.626Z   Service vmware-stsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.626Z   Running command: ['/sbin/service', u'vmware-stsd', 'status']

2020-06-16T10:39:26.690Z   Done running command

Successfully started service vmware-stsd

2020-06-16T10:39:26.698Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmdnsd']

2020-06-16T10:39:26.713Z   Done running command

2020-06-16T10:39:26.722Z   Service vmdnsd does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.722Z   Running command: ['/sbin/service', u'vmdnsd', 'status']

2020-06-16T10:39:26.784Z   Done running command

Successfully started service vmdnsd

2020-06-16T10:39:26.794Z   Running command: ['/usr/bin/systemctl', 'is-enabled', u'vmware-psc-client']

2020-06-16T10:39:26.809Z   Done running command

2020-06-16T10:39:26.818Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2020-06-16T10:39:26.818Z   Running command: ['/sbin/service', u'vmware-psc-client', 'status']

2020-06-16T10:39:26.880Z   Done running command

Successfully started service vmware-psc-client

Service-control failed. Error Failed to start vmon services.vmon-cli RC=2, stderr=Failed to start sca, cm services. Error: Service crashed while starting

PS) vCenter runs as a VM

[daphnissov]

Have you searched these forums for past threads and possible resolutions? This question comes up weekly with many possible causes.

[ssSFrankSss]

I have tried some. But I was lost since there are many questions and solutions as you said!!

[msripada]

can you check the cm logs /var/log/vmware/cm/cm.log and post it here so that we can review.. cm service failed to start and thus other dependent services as well

thanks,

MS

[ssSFrankSss]

I have attached the log file thank you!

[a_p_]

Please run the following commands from the vCSA's shell, to check whether one, or more certificates have expired.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient

André

[toreshavn]

Hi Andres,

How to run these commands from Window. Thanks !

[toreshavn]

Hi Andres,

How to run these commands from Vcenter Windows.
If there is certificate expired, how to extend it.

I am having the same issues but with Vcenter Windows. Thank you !

[ssSFrankSss]

toreshavn​ have you tried the article I have proposed, it has steps to follow for vCenter for windows also!

[toreshavn]

I have tried to the article you have proposed and reset all certificates, but it stops at 85% for starting services.
Please see image attached for the detail.

============================================
Here is log:

2020-06-18T09:10:21.262Z INFO certificate-manager Serial number before replacement: f1:e8:d2:42:ea:06:14:b6

2020-06-18T09:10:21.262Z INFO certificate-manager Serial number after replacement: dc:17:ba:94:d5:9f:ef:7c

2020-06-18T09:10:21.262Z INFO certificate-manager Thumbprint before replacement: 46:2E:AD:D9:76:89:4D:60:4E:7B:EE:34:A0:E3:05:97:34:C8:70:55

2020-06-18T09:10:21.262Z INFO certificate-manager Thumbprint after replacement: B0:DA:23:46:E2:10:FB:6C:95:BA:30:DB:05:52:C9:55:8E:2C:41:C0

2020-06-18T09:10:21.263Z INFO certificate-manager vpxd-extension certificate replaced successfully. SerialNumber and Thumbprint changed.

2020-06-18T09:10:21.263Z INFO certificate-manager Running command :- "C:\Program Files\VMware\vCenter Server\bin\service-control.bat" --stop --ignore  --all

2020-06-18T09:10:21.263Z INFO certificate-manager please see service-control.log for service status

2020-06-18T09:11:22.345Z INFO certificate-manager Command executed successfully

2020-06-18T09:11:22.345Z INFO certificate-manager all services stopped successfully.

2020-06-18T09:11:22.345Z INFO certificate-manager None

2020-06-18T09:11:32.348Z INFO certificate-manager Running command :- "C:\Program Files\VMware\vCenter Server\bin\service-control.bat" --start  --all

2020-06-18T09:11:32.348Z INFO certificate-manager please see service-control.log for service status

2020-06-18T09:19:16.961Z ERROR certificate-manager None

2020-06-18T09:19:16.964Z ERROR certificate-manager Error while starting services, please see log for more details

2020-06-18T09:19:16.966Z ERROR certificate-manager {

    "resolution": null,

    "detail": [

        {

            "args": [

                "None"

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'None'",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        },

        "Error while starting services, please see log for more details"

    ],

    "componentKey": null,

    "problemId": null

[ssSFrankSss]

Actually I had the same problem... It stopped at 85% when starting services! Initially I thought it was a problem too! But it wasn't just go for a coffee and come back later :smileyhappy:

I mean just let it work - hopefully after  a while it will complete 100%

[toreshavn]

@ssSFrankSss Unfortunately, It is not like this :smileyhappy:

"

Reset status : 85% Completed [starting services...]

Error while starting services, please see log for more details

Status : 0% Completed [Reset operation failed] "

[Alan_Lau]

Dear,

 

   How U solve this problem ,can you share with us .I met the same problem,thx

[AE_Signedin]

I have got the same problem - how did you end up fixing it?

[AE_Signedin]

 

I have got the exact issue you were facing, how did you end up fixing it? 

stopped at 85% and rolled back after with error service-control failed. Error Failed to start vmon services.vmon-cli RD=2

[7867878787]

I stucked in 85%. 

About 30m then it displays this notification

 

Rollback Status : 0% Completed [Rollback operation failed]

 

pls help me

[adamefril]

i had sam problem but it was stop at 0 % when i try to reset :

2021-04-25T06:11:31.466Z INFO certificate-manager Thumbprint after replacement: 11:47:4F:BA:F1:32:36:17:A2:1C:EB:75:F5:13:B5:13:4D:E6:D9:DA
2021-04-25T06:11:31.466Z INFO certificate-manager MACHINE_SSL_CERT certificate replaced successfully. SerialNumber and Thumbprint changed.
2021-04-25T06:11:38.891Z ERROR certificate-manager 'lstool reregister' failed: 1
2021-04-25T06:11:38.892Z ERROR certificate-manager please see /var/log/vmware/vmcad/certificate-manager.log for more information.

 

pls help me with this.....

[Iskalafonger]

From This location
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli.exe entry list --store vsphere-webclient --text

[Sukanya D]

From the logs i see that the cert expired :

Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed May 06 19:32:13 UTC 2020

        at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274) ~[?:1.8.0_162]

        at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629) ~[?:1.8.0_162]

        at sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190) ~[?:1.8.0_162]

        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144) ~[?:1.8.0_162]

        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125) ~[?:1.8.0_162]

        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:223) ~[?:1.8.0_162]

Seems we might have any expired certs in VECS store .

You can follow below kb to verify:

Kb2111411

[adamefril]

pls help me, i had sent my log ,do i have same problem......

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x000055b7fe084690] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

type : vCenter Server with an external Platform Services Controller

 

[ssSFrankSss]

Thank you all it was a certificate issue this article helped me to return everything back to normal:

VMware Knowledge Base

[aruncarrer]

Hi Enthusiast,

I was following the below Article.. But when it prompts for Credentials.. It is not accepting. 

In your case, what password was provided ?

| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : Y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [[e-mail]]:
Enter password:

Incorrect Password! Try Again! (2 attempt/s left)
Password:

Incorrect Password! Try Again! (1 attempt/s left)
Password:

Reached max number of attempts, terminating operation.
root@rc01vm02 [ ~ ]#

 

Here i tried with [e-mail] and root

Enter username [[e-mail]]:

[Marco_Luvisi]

This issue don't recognize [e-mail] pwd because are stopped some services