503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000a63051bb30] _serverNamespace = / action = Allow _port = 8309)

[cattlerustler]

Help    

I have ESXi 6.7 running, after a funny issue with the webportal not working correctly, stuck just before the login screen, I reset the management agents, now all I presented with is

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000a63051bb30] _serverNamespace = / action = Allow _port = 8309)

Have tried resetting the management agents again and the network agent both via ssh and on the console, I'm now at a loss please help me.

Thanks

Adrian

[GayathriS]

Hi

Could you please run below command and confirm if there is any services which are at stake 

service-control --status --all

If vsphere-ui is stopped then use below command to start it

service-control --start vsphere-ui

Please mark this as "correct" or " Helpfull" if this answers your query.

regards

Gayathri

[cattlerustler]

Sadly no joy here at all

[root@ESX1:~] service-control --status --all

-sh: service-control: not found

[root@ESX1:~]

[root@ESX1:~] service-control --start vsphere-ui

-sh: service-control: not found

[root@ESX1:~]

Any other ideas?

[rlucky1]

Hi, I've got the same issue and after checking status, I do see that vsphere-ui is stopped. When trying to start it I get this error:

 

C:\Program Files\VMware\vCenter Server\bin>service-control --start vsphere-ui
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vsphere-ui...
Error executing start on service vsphere-ui. Details {
"componentKey": null,
"resolution": null,
"problemId": null,
"detail": [
{
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vsphere-ui'"
,
"args": [
"vsphere-ui"
]
}
]
}
Service-control failed. Error: {
"componentKey": null,
"resolution": null,
"problemId": null,
"detail": [
{
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vsphere-ui'"
,
"args": [
"vsphere-ui"
]
}
]
}

 

Do you have any ideas?

-Lucky

[MikeStoica]

You need to run those commands on vCenter or VAMI VMware Knowledge Base

[cattlerustler]

ok, trying to install this now, not the easiest when the ESXi server is knackered to start with (install failed), trying to install another ESXi box first to deploy it on

[cattlerustler]

ok, just installed vCenter and when I go to add the host I get this error

Cannot contact the specified host (10.1.0.15). The host may not be available on the network, a network configuration problem

Clearly the host is not in state it can be contacted other than via SSH, anyone help at all? There must be something I can do via SSH?

Oh I can add the other ESXi 6.7 host with no issues

[MikeStoica]

Is your network configured correctly on the esxi host?

[cattlerustler]

Yes, all networking is ok, was all ok for over a month, then the browser interface went funny only getting to the blue screen prior to login, then refreshed the management agents and got this error, all rather odd and can't find another reference to it online anywhere. This is the free version as the hosted servers are for a non profit organisation, so they basically have no money, I do all my work for free as well. Obviously, I can reinstall ESXi but that would mean downtime for them or out of hours working for me and getting a member of their staff their to look after me, hence hoping there is a fix for it. I'd never run 6.7 before and I'm wishing I had stuck with 6.5 native, using the old thick client is so much nicer and easier than this nasty web interface. 

[MikeStoica]

If you are using the free version, then you can't added it to vCenter.

[mikemast]

I randomly have same issue. I am working on one right now is how I came across this thread. only access I have is console and SSH. tried restarted all the service and still get the 503 message and show disconnected in vcenter.  VM's are still running and responding.

ESX host responds to pings and can SSH into it fine. Just stops reporting to vcenter and web client is not working. Try to connect back to vcenter and it says host is un-avaliable, as it cant connect to the host on https.

Only way I have fixed this is reboot the host and kill all the VM's. I have not found a fix to this as of yet.

This looks to have been a intermittent issue for a while now and I don't see any fix or workaround for it. not good for a production environment when I have to kill running VM's.

ESXi Embedded Host Client - Bugs

ESXi Embedded Host Client - Bugs

Along with other posts, and only resolution is reboot host, with running VM's.

Mike

[letutruongan]

I got the same problem (Esxi 6.7). When I login host with SSH, try to use esxcli command to control vm I got "Connection refused/error".

I have to shutdown each vm manually, and reboot the host. Everything back to normal.

[vmwaresucksatre]

Yeah got this today. I was setting up a new server, running dell emc vmware esxi release build 14320388. Was at the point where i configure iscsi targets. So i did that but forgot to authorize the server on the array. So did that, went back and the server was just not doing anything.

Looked down and saw about 10 "rescan HBA" tasks. that were all at 0%. waited about an hour for it to resolve and when it didnt, i attempted a server restart. Server restart was locked up as well it seemed, so i used console to powercycle it.

After power cycle this message was displayed when going to the machine directly and my vcenter can not talk to the host. It is pingable though.

The last thing i did before these problems occurred was trying to set up iscsi connections to our shared storage arrays as i have done before....

will post back if i find a solution because the machine is inaccessible now.

[arreddick]

The same exact thing happened to me. I was connecting to shared iSCSI. Did you ever find a solution?

[VDadkhah]

I exactly have this issue and find no solution for it . only rebooting server resolve it and in production environment it is not acceptable , if you find any solution please let me know .

[EBMgmbH]

You have to restart the Management Agents.

from the Terminal: F2 - connect as Root/Troubleshooting Options/Restart Management Agents/
Enter F11 Enter Escape
via Putty:
service mgmt-vmware restart

 

Usually you also have the message HA Master Agent was not found as an error message in Vsphere.

 

[Raj1988]

Check these and ESXi Embedded Host Client - Bugs and VMware Knowledge Base   .

This can also happen is hostd is not-responding. Have you tried ESXi reboot.

[KHmaiesTN01]

I had the same problem

I just set the DNS and it works

[mrwboilers]

I'm having this issue too, but DNS is already configured properly. Any other ideas?

[pcteck]

Same here, ESXi host 6.5 happened after I installed updates on the server so it was in maintenance mode.

The httpd service was stopped so I started it, but still no change on the host/ui

I rebooted the host and now the ui loads.

On this link - it seems there is a bug with the esxui,

I checked from SSH

/etc/init.d/esxui status

it was running, so possibly /etc/init.d/esxui start may help if it is stopped. Others reinstalled the VIB.

Bugs | VMware Flings

[zeeball]

Thanks - It saved my bacon after changing the hostname.

[meaguilarg]

[Mhtd]

I used "services.sh restart"  command with SSH to restart all management agents on the host,

It worked for me.

[ronnieferrell]

---

 wrote:

I used "services.sh restart"  command with SSH to restart all management agents on the host,

It worked for me.

---

This was the ticket for me. thanks you!

[gaseker]

Saved my day...

[TD99]

Having same issue (Host not responding in vCenter).

 

• VMware ESXi, 6.7.0, 18828794
  
• Model: HPE ProLiant DL360 Gen9

 

Here are the troubleshooting steps I performed:

1. Restarted Management agents in ESXi Using Direct Console User Interface (DCUI).
   
   2. Tried changing management IP and DNS name to something else.
   
   3. Restarted the ESXi host daemon and vCenter Agent services using these commands:
   /etc/init.d/hostd restart
   /etc/init.d/vpxa restart
   
   4. Confirmed the services were running:
   [root@esx-kp-02:~] /etc/init.d/vpxa status
   vpxa is running
   
   [root@esx-kp-02:~] /etc/init.d/hostd status
   hostd is running.
   
   5. Restarted all management agents on the host, using the command:
   services.sh restart
   
   6. Tried to reset the management network on VMkernel interface "vmk0", using the command (which failed):
   [root@esx-kp-02:~] esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0
   Connection failed
   Connection failed
   
   7. Attempts to access ESXi web interface at "https://esx-kp-02.hs.local/" results in error:
   503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00000093edeaac30] _serverNamespace = / action = Allow _port = 8309)

 

Tried everything above (except rebooting the host which fixes the issue 95% of the time). No luck.

Contacted HPE Support who says the internal microSD card (NAND flash) where ESXi is installed, is failing. SCSI write commands sent to it are not responding.

In the ESXi DCUI, I pressed Alt-F12 and saw highlighted rows referring to "mpx.vmhba32" which is the USB Storage Controller on the host. See attached screenshots.

Only solution is to reboot host and place into Maintenance Mode, then set-up for microSD card replacement.

[ssboy]

Hi

I'll tell you about my "adventure"

Homelab Version: 7.0 Update 3  without vCenter. A very simple network config: two homelab connected to a lan and a internet connection via router.

A few days ago, suddenly, I can't connect to the host and the error in question appears. I connected the host via SSH, restarted the services (services.sh restart), restarted the server ... I tried everything! I was able to contact the host but after few seconds the error in question appeared.

In the same network I have another homelab 6.0.0 (Build 3620759) which instead worked correctly. and was always reachable.

Everything in the network seemed to work correctly ... except one thing: due to a failure of the provider, the internet connection was absent! When the internet connection was restored, after restarting the host services, it was possible to connect to the host and the error did not reoccur.
I did some tests and the result is always the same: if I disconnect the internet for more than 5-10 minutes while I am connected to the host, the host becomes unreachable with error. By restoring the internet connection, after restarting the services, the host becomes reachable again and remains reachable as long as the internet connection remains stable.

.Does the host need to contact some external source? What causes the error?

Simon

[achandler85]

Guys,
I just fixed if for my scenario:
- ESXi was controlled by Vcenter
- I removed veeam backup of one VM on the ESXi
- Then ESXi required disk consolidation for this VM
- Then ESXi became unavailable
- Then I restarted this service via CLI
- Then I restarted this service via KVM (It wasn't available until I reset it via cli)
- Vcenter also wasn't able to communicate with this ESXi
- Then cat /etc/vmware/vpxa/vpxa.cfg
You should see ESXi and VCenter IP addresses.
I removed Vcenter IP address using vi /etc/vmware/vpxa/vpxa.cfg
and restarted vpxa service
- Then ESXi Web interface became available and I reconnected it in Vcenter
- Then cat /etc/vmware/vpxa/vpxa.cfg again - the Vcenter IP was added back

[sanchezmwhs]

Hey achandler85

 

I am running into this issue and I also have Veeam as a backup service on many vms I am wondering if veeam is causing this because a restart works but then it just breaks again.

[ilsava]

it's because the ssl certificate is invalid, try to check these two files rui.key and rui.crt in /etc/vmware/ssl, use the two file certificate that are either valid or invalid from vmware installer or you can get it from the installer esxi from vmware workstation installed esxi 6.7 then copy the two files, then restart using /etc/init.d/hostd restart then check the browser ui again, the crux of your problem is only in those two files.

[adamabel1]

I ran into this last week while swapping out a dedicated storage NIC for another that was using the iSCSI software adapter.  I rebooted my host because the datastores were not coming up as "not consumed"  when the system came back up after a very long boot up process I got the 503 unavailable error.  I tried all the fixes listed here to date and none of them work.  What I found out after I had reinstalled the OS and it came back up is that it was related to the iSCSI connection to my Netapp array.  After I reinstalled I setup the software iSCSI adapter again and when I went to reboot to fix another issue I got the same 503 unavailable error.  Once I took the iqn's out of the initiator group on the Netapp and rebooted everything worked again.  I'm not sure why this is since the OS is stored on a local SSD mirrored set of drive on the host.  Hopefully this help someone else running into this problem.