Vcenter appliance gives error 503 Service Unavailable

[Wojciech_Rodzik]

Hi,

I have vcenter appliance 6.5.

ll of the sudden it started giving Error 503 when attempting to access. We attempted multiple restarts, but they had no effect

VPXD service is not starting, I am getting an error occurred while starting service ' (0)s'  error message when attempting to start service manually. I tried workaround from:

VMware Knowledge Base

But hosts entry for that server is already there (and it is a correct one).

Any idea how to fix the problem?

[diegodco31]

Hi

You can post the log file (/var/log/vmware/vpxd/vpxd.log).

[Wojciech_Rodzik]

How can I export the logs from appliance?

vpxd log is too long for print screens

[daphnissov]

SCP it from the appliance, zip, and attach it here (do not copy-and-paste text)

[Wojciech_Rodzik]

Newest log file attached

[RajeevVCP4]

Is PSC external or embedded ?

If it is external check PSC and VC time should be same.

[Wojciech_Rodzik]

PSC is embedded.

[Kevin Hagopian]

Looking at the attached logs, the vpxd service is repeatedly trying to talk to Component Manager:

2018-09-22T20:54:19.051+02:00 warning vpxd[7FE8AEEA2800] [Originator@6876 sub=componentManagerUtil] Failed to logout from Component Manager: Connection refused

2018-09-22T20:54:19.051+02:00 warning vpxd[7FE863BF7700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fe8984819b0, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:18090'>>, e: system:111(Connection refused)

2018-09-22T20:54:19.051+02:00 error vpxd[7FE8AEEA2800] [Originator@6876 sub=licenseServiceDiscovery] Failed to discover license service endpoint on retry: 1, error: N7Vmacore15SystemExceptionE(Connection refused)

Eventually going through 60 attempts to AuthZ:

2018-09-22T20:55:47.256+02:00 error vpxd[7FE8AEEA2800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)

--> [context]zKq7AVECAAAAAHU4kAATdnB4ZAAAJGcrbGlidm1hY29yZS5zbwAA9B4bAD5vGAEY5lR2cHhkAAEGPWIB5O32AUY9owFRKKMCMBcBbGliYXV0aHpjbGllbnQuc28AAlZJAQJuTwECWNsBAtbQAQE0nKEBPAVVAaQOVQFqsVMD4AUCbGliYy5zby42AAF1qVM=[/context]>

2018-09-22T20:55:47.258+02:00 info vpxd[7FE8AEEA2800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 23

Eventually having vpxd stop:

2018-09-22T20:57:44.029+02:00 error vpxd[7FE8AEEA2800] [Originator@6876 sub=Authorize] Failed to initialize authorizeManager

2018-09-22T20:57:44.029+02:00 warning vpxd[7FE8AEEA2800] [Originator@6876 sub=VpxProfiler] Start [VpxdAuthorize::Start()] took 184467 ms

2018-09-22T20:57:44.029+02:00 warning vpxd[7FE8AEEA2800] [Originator@6876 sub=VpxProfiler] ServerApp::Start [TotalTime] took 184514 ms

2018-09-22T20:57:44.029+02:00 error vpxd[7FE8AEEA2800] [Originator@6876 sub=Default] Failed to start VMware VirtualCenter. Shutting down

My best guess would be to check AuthZ or CM logging.

cm.log -> /var/log/vmware/cm/

authz-event.log -> /var/log/vmware/vpxd-svcs/

On a side note, does this issue persist after a reboot? I've seen some cases where a reboot was able to restore the service(s) impact.

[Wojciech_Rodzik]

Thank you for the hint, I will check the logs.

As for reboot - yes it persists, I tried rebooting multiple times.

[Wojciech_Rodzik]

I accidentally fixed the problem by changing root password for appliance