503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0000000000FAD020] _serverNamespace = / _isRedirect = false _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

[Arie01]

Hello,

I inherited an existing environment that does not function properly.

I have vCenter 6.x that I can't access through the web client. When I try the url or IP of the server, I get

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0000000000FAD020] _serverNamespace = / _isRedirect = false _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe).

This is on a physical Windows 2012 server. I verified the services and some of them don't want to start. When I try to start them manually, it says something about dependencies failing.

I run service-control --stop --all then service-control --start --all and some services still not starting.

I was looking for some logs to view more details but they are either not in their default location or they simply don't exist on that server.

I wanted to install it from scratch, but I don't have the install files\iso of the server and I don't know what is the exact version and build of that vCenter server, so I can't download.

What should I do? How can I make the web client work or get it installed fresh? It is in Veeam and seems to be working fine, it's just that I can't do much with it without a working web client.

Thank you,

[msripada]

Hi,

Can you get the service-control --start --all  output please. I will guide you to check which service failed and how to check further?

Thanks,

MS

[Arie01]

Hi msripada,

Here is the output of service-control --start --all. I hope you can direct me to the solution.

Also, see all the VMware services, their status and startup type. This is after running the command above.

Thank you,

Service: vmware-cis-config, Action: start

2018-07-06T12:21:43.402Z   State for service vmware-cis-config: STOPPED

2018-07-06T12:21:44.433Z   Successfully started service vmware-cis-config

Service: VMWareAfdService, Action: start

2018-07-06T12:21:44.433Z   State for service VMWareAfdService: STOPPED

2018-07-06T12:21:46.465Z   Successfully started service VMWareAfdService

Service: rhttpproxy, Action: start

2018-07-06T12:21:46.465Z   State for service rhttpproxy: STOPPED

2018-07-06T12:21:52.543Z   Successfully started service rhttpproxy

Service: VMWareDirectoryService, Action: start

2018-07-06T12:21:52.543Z   State for service VMWareDirectoryService: STOPPED

2018-07-06T12:21:55.590Z   Successfully started service VMWareDirectoryService

Service: VMWareCertificateService, Action: start

2018-07-06T12:21:55.590Z   State for service VMWareCertificateService: STOPPED

2018-07-06T12:21:57.622Z   Successfully started service VMWareCertificateService

Service: VMwareIdentityMgmtService, Action: start

2018-07-06T12:21:57.622Z   State for service VMwareIdentityMgmtService: STOPPED

2018-07-06T12:21:59.887Z   Successfully started service VMwareIdentityMgmtService

Service: VMwareSTS, Action: start

2018-07-06T12:21:59.887Z   State for service VMwareSTS: STOPPED

2018-07-06T12:22:02.919Z   Successfully started service VMwareSTS

Service: VMwareComponentManager, Action: start

2018-07-06T12:22:02.919Z   State for service VMwareComponentManager: STOPPED

2018-07-06T12:22:23.093Z   Successfully started service VMwareComponentManager

Service: vmware-license, Action: start

2018-07-06T12:22:23.093Z   State for service vmware-license: STOPPED

2018-07-06T12:22:33.147Z   Successfully started service vmware-license

Service: VMWareNetworkCoredumpWebserver, Action: start

'VMWareNetworkCoredumpWebserver' startMode is Disabled, skipping to start:

Service: vmware-psc-client, Action: start

2018-07-06T12:22:33.148Z   State for service vmware-psc-client: STOPPED

2018-07-06T12:22:36.205Z   Successfully started service vmware-psc-client

Service: vmwareServiceControlAgent, Action: start

2018-07-06T12:22:36.206Z   State for service vmwareServiceControlAgent: STOPPED

2018-07-06T12:22:47.274Z   Successfully started service vmwareServiceControlAgent

Service: vmware-network-coredump, Action: start

'vmware-network-coredump' startMode is Disabled, skipping to start:

Service: vapiEndpoint, Action: start

2018-07-06T12:22:47.275Z   State for service vapiEndpoint: STOPPED

2018-07-06T12:22:58.388Z   Successfully started service vapiEndpoint

Service: vPostgres, Action: start

2018-07-06T12:22:58.389Z   State for service vPostgres: STOPPED

Unable to start service vPostgres, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vPostgres"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vPostgres'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

[msripada]

vPostgres failed to start. Can you check the logs for the postgres in programdata\vmware\vcenterserver\logs\vpostgres

and look for the last modified log when you started service

Most likely the postgres db has corruption but need to isolate which table is the problematic one.

Thanks,

MS

[Arie01]

Hi sm,

The last log that I found was this.

Database most likely corrupted. What do I do with it now? I don't know when the corruption happened and I'm not sure I have a good backup to restore from.

2018-07-05 11:22:31.261 EDT 5b3e37b7.2324 0 VCDB vc FATAL:  the database system is starting up

2018-07-05 14:38:53.077 EDT 5b3e65bd.25b4 0 VCDB vc FATAL:  the database system is starting up

2018-07-05 14:45:36.140 EDT 5b3e6750.1e70 0 VCDB vc FATAL:  the database system is starting up

2018-07-05 14:46:06.204 EDT 5b3e676e.1c84 0 VCDB vc FATAL:  the database system is starting up

2018-07-05 16:24:50.190 EDT 5b3bd056.106c 0   LOG:  received fast shutdown request

2018-07-05 16:29:29.623 EDT 5b3e7fa9.1238 0   LOG:  database system was interrupted while in recovery at 2018-07-03 15:37:02 EDT

2018-07-05 16:29:29.623 EDT 5b3e7fa9.1238 0   HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

2018-07-05 16:29:32.775 EDT 5b3e7fa9.1238 0   LOG:  database system was not properly shut down; automatic recovery in progress

2018-07-05 16:29:32.840 EDT 5b3e7fa9.1238 0   LOG:  redo starts at B/B55675C8

2018-07-05 16:29:37.298 EDT 5b3e7fa9.1238 0   LOG:  record with zero length at B/B7E5C080

2018-07-05 16:29:37.298 EDT 5b3e7fa9.1238 0   LOG:  redo done at B/B7E5C050

2018-07-05 16:29:37.314 EDT 5b3e7fa9.1238 0   LOG:  last completed transaction was at log time 2017-04-01 08:18:33.356-04

2018-07-05 16:29:42.358 EDT 5b3e7fb6.132c 0 VCDB vc FATAL:  the database system is starting up

2018-07-05 16:32:24.020 EDT 5b3e8058.19e0 0 VCDB vc FATAL:  the database system is starting up

[msripada]

Have you tried rebooting the vCenter?

thanks,

MS

[Arie01]

Hi MS,

Yes, that's the first thing I did.

Thank you,

[msripada]

Take snapshot of vcenter and try this

Stop all vcenter services using service-control --stop --all

Check "C:\ProgramData\VMware\vCenterServer\data\vpostgres\postmaster.pid" file - if exist delete the file or move it to another location

Reboot the vCenter

[msripada]

Also check if the vCenter has space issue where the postgres is residing like C drive nearing space .. It needs to have 10% space available minimum

[Arie01]

Hi MS,

My vCenter server is physical, I can't take a snapshot of it.

Also, there was postmaster.pid file in the vpostgres folder. I removed it and restarted the server and it's all the same.

No access to the web client.  Free space is also OK.

Thank you,

[msripada]

The possibly the postgres is corrupted.. IF you have backups, you might have to recover or contact support.

Thanks,

MS

[uldericozemacha]

Hi all, I have the same problem, vcenter server 6.7 on win server 2012 on esxi 6.5 platform.

I did:

C:\ProgramData\VMware\vCenterServer\bin

service-control --stop vsphere-client

service-control --start vsphere-client

then

service-control --start vsphere-ui

All works perfectly.

Thansk

Rico

[smw104IT]

I see the batch file and the notepad. Where in the batch file do I add "--stop". 

[regchoo]

I found the simple solution by delete the registered vcenter server DNS from your DNS server. Then IPconfig /renew from the vcenter server to let it re-register into DNS server again.

[anval76]

this actions work for me. only restart the services.

Thanks you