Cannot add host to vCenter

[pabloleiva]

hello! I know this same question has been asked multiple times; I've already read like 50 post and went over all google search pages.

I just installed vCenter VCSA 6.7 and trying to add some ESXi hosts we have (version 6.0). Getting this error:

Cannot contact the specified host (<server>). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

 

I've tried:

* Tested using both hostname and IP

* Verified DNS (I can ping hosts from vCenter using IP and hostname)

* Verified port 902 is not blocked on FW

* Verified hosts certs are valid (self-signed)

* Tried certmgmt mode thumbprint and vcsa

* Tried to restart the vpxa service

* Tried to regenerate the self-signed certs

* Installed VCSA 6.5 - same problem

* Installed vCenter on Windows - same problem

* I installed a new ESXi host (VM) to test and that one worked (same ESXi version). I've tried comparing all settings and seem to be the same as the non-working.

* Checked the vpxa logs on the host but seems nothing is written when I try to add them. I believe it could be an SSL or certificate issue but cannot find why the existing hosts cannot be added, but the new one I built yes.

* I cannot reboot hosts at the moment (actually trying to get vCenter to move VMs around and be able to reboot and upgrade ESXi)

 

Any ideas would be appreciated!

[a_p_]

How are these hosts licensed? Hosts with a free Hypervisor license cannot be managed by vCenter Server.

André

[pabloleiva]

Thanks, its Enterprise license

[Rob Bishop]

Have you checked reverse lookup on the DNS names and also have you tried adding with FQDN?

[pabloleiva]

Yes, tried adding by IP and FQDN. DNS seems ok; I means I checked the hosts are part of the enterprise DNS and from my PC I can ping by host, also ping -a the IP

[Rob Bishop]

The other big thing that comes to mind is time, are all showing the same time and timezone, standalone hosts have a big habit of having time drift.

[pabloleiva]

Good question, I checked all are pointed to NTP. And from vSphere they show the correct date/time. But when I was checking some logs, the time on the log was a bit off. How can I confirm? Whats the ‘show ntp status’ for the esxi?

I fear the only solution will be a reboot, which I cannot still do for weeks, as this is a 24/7 business.

[Rob Bishop]

Got to the web interface for the esxi host, there should be an option for time & date under the manage menu, sorry don't have a 6.0 host to hand to be able to give exact instructions or SSH to host and run ntpq -p.

 

 

 

[pabloleiva]

thanks, it seems to be synschronized with NTP. Correct time shows up in vsphere here:

 

 

[amohammadimir]

I am not sure about your network diagram but can you also check the MTU?

[pabloleiva]

I found MTU helped in many cases here in communities. I tried to set it to 1400 in the Host (management network) and in vcenter too, but not sure if it means I need ro change something on the routers/switches. But really there’s no WAN here. Well we do have 2 DCs, but I cannot even add hosts on the local DC. VCenter is a VM on one of the hosts i’m trying to add, so i’’ guessing MTU shouldn’t be an issue.

[Mark Edwards]

How did you verify port 902 is not blocked on the firewall? Was that verbally or by actually testing?

 

Also check that both UDP and TCP are configured for access.

 

You can test TCP connectivity by running curl <esx.-fqdn or IP>:902 from vCenter and see if it is able to connect to the ESXi host? For UDP you'll need to run a packet capture to verify connectivity.

 

HTH

[pabloleiva]

yeah, network team says there are no drops on port 902. so its allowed or maybe not even going thru the FW.

 

 

[Mark Edwards]

Ok that's TCP 902, but what about UDP 902? As above, you need to test for both 

 

[Yaniv Dinar]

Sounds like networking mismatch.

look at the DNS configuration of your ESXI under Networking\Default TCP/IP stack and make sure you can ping your DNS server from your ESXI host.

also try and do the same with your vCenter Appliance. 

login as root and check your networking configuration from the management: "vcenter address"\"host name":5480 and make sure you can ping your DNS server.

[JDMils_Interact]

You've mentioned that you have done quite a bit of testing however you have not mentioned from WHERE you have tested. let me know what happens when you:

• PING the host via IP from the vCenter.
• PING the host via DNS from the vCenter.
• Check port 443, 80 open from the vCenter to the host.
• Move the vCenter to another host and then try to connect again.
• Run the command "ssh root@<Host_IP>". Does the SSH connection connect to the host?

Do your hosts have separate PHYSICAL uplinks for each switch/dvSwitch? For example, I've seen the issue where the Standard and dvSwitches on a host were using the same PHYSICAL uplink connection to the upstream switch and what happens here is vCenter was on the Standard switch and the host was on the dvSwitch but both on the same PHYSICAL uplink and in this situation the upstream switch will not forward packets between the two and thus you get packet drop and thus disconnection.

Put the vCenter on another vlan/subnet and try again. Put the vCenter on the same vLAN/subnet and try again.

Ensure that the /etc/hosts file is using 127.0.0.1 pointing to localhost.localdomain localhost. Check the hosts file on both to make sure either no entries exist (because you are using DNS) or the correct IPs & names appear for both vCenter and ESXi host if not using DNS. Adding the vCenter and ESXI host names and IPs to your DNS server will help.

I assume your hosts can PING the vCenter (via CLI)? The hosts can PING each other? Also try restarting the management agents on the host and retry.

Is the version/build of vCenter equal to or greater then the hosts? Ensure your vCenter is patched to the latest version/build.

Change the IP of the vCenter and see what happens in case there is a duplicate IP on the network.

Check in DNS server that you have a reverse DNS/IP record for the host. You can test this with:

NSLOOKUP <Host_IP> and if you get an error of some sort then you need a reverse DNS record setup.

One of the logs will have the issue recorded, it's a matter of which one. When you try to add the host, do you see anything in these logs:

/var/log/vmware/vpx/vpxa.log
/var/log/vmware/hostd.log
vpxd.log

And this happens for ALL your hosts?