ESXi host disconnected from vCenter

[NathanF1]

Hey all,

I've got an issue with one of my ESXi 5 hosts. It had become disconnected from vCenter and cannot be reconnected. It is not pingable from the vCenter on it's management interface, but the rest of the cluster hosts are. But the un-pingable host can happily ping out, resolve it's gateway, the DNS servers and it's hostname. It can also ping itself. I've restarted the management agents, but this has had no effect. Nothing has physically changed on the network, so I'm a little stuck! I've followed some of the guides for when a host becomes totally disconnected from the network, but outbound seems to be OK on this host.

Can anyone offer advice?

Regards,

Nathan.

[admin]

Hi Nathan,

This is a tricky situation, lets start with checking if you have any firewalls on the vCenter server machine that are blocking the ports 902 and 443 which are required for the host to connect to the vCenter server

[NathanF1]

Hi there,

No, there are no active firewalls on the vCenter. Also, the ESX01 host is not pingable from the other hosts (ESX02 & ESX03), but it can ping them! Does that imply the issue lies with that ESX01 host?

Regards,

Nathan

[admin]

yep, seems to be the problem on that host alone. try re-configuring the network settings on the host ESX01 and see if that resolves it.

Thanks,
Avinash

[GaneshNetworks]

Check the VLAN that you set for mgmt interface in DCUI.

Verify that you have configured the correct gateway.

Try to restart all the services - /sbin/services.sh restart

[Gav0]

NathanF1 wrote:

Hi there,

No, there are no active firewalls on the vCenter. Also, the ESX01 host is not pingable from the other hosts (ESX02 & ESX03), but it can ping them! Does that imply the issue lies with that ESX01 host?

Regards,

Nathan

Have you managed to connect to the host from outside the vSphere environment to perform these tests?  eg, can you ping it from a physical server/workstation, can you connect to the host directly with the VI Client or command-line?

Just trying to narrow down the scope of the problem

[NathanF1]

Hey all,

The host was not accessible via ping or from the viclient or SSH from anywhere on the network, be it inside or outside the vSphere environment. But the host could ping everything without fail. Very strange.

All network settings, gateway, VLAN etc were checked and verified. Restarted the Management Network and the Management Agents, changed the IP, unpatched alternate NIC's, nothing helped.  Finally I just logged into each of the VM's via RDP and shut them down gracefully, then rebooted the host via the iDRAC. Everything has come back up fine, but I still have no idea why it happened, which is a shame.

I did speak to an ex-colleague that said he had this issue randomly on ESXi 5.1, but they have been trouble free since upgrading to 5.5. That may be the best way forward for me.

Thanks for all the advice though.

Regards,

Nathan.

[Gav0]

NIce work! Glads its working :smileyhappy:

[VMADMINAAA]

Try restarting the management console service mgmt-vmware restart. Also check the NICs on your service console vSwitch? Make sure all of them are working. For troubleshooting purposes leave the service console vSwitch with only one NIC and do the troubleshooting steps above. Do the same for the rest of the nics on the vSwitch.

Let me know if any of that worked. Thanks

[joviyach]

I saw something similar just recently myself, also with v5.1

In this case, I was able to SSH to the host, but it would lock up after a few moments, even when doing something trivial like changing directories. All but two of the VMs were moved via HA, and the VMs left over were not production machines, so we opted for reboot. We ended up having to do this using iLO to power cycle the host, because even issuing reboot command from the console would lock up.

The reboot brought the host back to life and we reconnected successfully to the VC. So far there haven't been any additional problems. We weren't able to recover the logs however, so I can't say what the root cause may have been.

[beckham007fifa]

If it is CISCO blade, try checking the Mac address of VMknic and ethernet adapters. If they are same then change VMKnic mac address for proper connectivity.

[beckham007fifa]

or whatever server,

use Esxcfg-vmknic -l command to confirm

esxcfg-nics -l