vCenter 7 Bug? (No Healthy Upstream)

[khatdumangeng]

Hi Guys

We've been Experiencing this error / bug / problem in vCenter 7 whenever we do the ff:

- Test High Availability (VMs restarts on the other server except for the vCenter Server)
    After powering on the esxi host where the vCenter is residing and the vCenter Server the error occurs

Tried the ff: re-installing, checking VM Hardware Version, checking the OS: no Photon OS in the list. restarting services but nothing works
in the end we need to restore our back-up.

Screen shot below

Thanks!

[NathanosBlightc]

Could you open the VAMI interface of the vCenter server? (https://vc_ip:5480) If it response, please check all VCSA services status ...

Don't try to restore a backup immediately or it it's not possible, please keep a original corrupted VM of the VCSA for investigation the root of problem

[aevirtual]

Could you please take a putty session to the vCSA and run the below command to confirm if you have any space issues,

hostname -f && date && uptime && vpxd -vl && chage -l root && df -h && service-control --status.

if there is no issues with the space then please use the below article to find if the sts certificate is expired,

VMware Knowledge Base.

Please feel free to raise a case with VMware support team to help you if you need an extra hand.

Thanks.

Please like and mark this as correct solution, if this helps you to solve your problem.

[faleko]

Hi,

Today I had the same incident, and many automatic vcsa services were stopped

services were starting slowly and the vcsa cpu was 100%

it has taken a long time to function normally, we will have to monitor this incidence

[carter910]

In my case, "no healthy upstream" was caused by expired certificates. After starting STS service, login issues persist - "user name and password required".  After rebooting, numerous services failed to start and i received: Exception in invoking authentication handler [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076).  I followed the first link to regenerate certs - option 8. Followed the second link to regenerate STS cert. Rebooted and everything works!

https://kb.vmware.com/s/article/2097936

https://kb.vmware.com/s/article/76719

[mrc247]

I have the same problems with a fresh installation of VCSA 7.0.0c.

I gave the VM more CPUs and waited 30 minutes after booting, but I still get the message "no healthy upstream".

The service "vAPI-Endpoint" reports the error "Failed to connect to f592d7af-0df2-4c7f-8e29-a9f03efd7a34 \ com.vmware.vcenter.nsxd.vapi vAPI provider."

A restart doesn't help either. What should I do?

[omoyanogoa]

Same problem I think. Expired certificate.

 

But in VCSA can't start vmware service etc. Error: Service crashed while starting 

 

Great bug vmware! 

[Blanla]

Link 1 with option 8 did the trick for me,   https://kb.vmware.com/s/article/2097936
Thanks steaks

[foxmr]

Hi,

Check your vCenter hosts file.

If it looks like this
10.200.111.25 vcenter.mydomain.local vcenter

10.200.111.25 vcenter.mydomain.local vcenter

then backup it first and after that change it to this

127.0.0.1 vcenter.mydomain.local vcenter localhost

::1 vcenter.mydomain.local vcenter localhost ipv6-localhost ipv6-loopback

Reboot the vCenter and check if you can access it

This is what support did when i raised a ticket.

Regards,

foxmr

[Jul3s]

It worked for me.

Thanks foxmr for sharing :smileyhappy:

[daviscj]

Thanks!! This worked for me.

[BochoveM]

Thanks a lot! I just wanted to reply that this worked for me as well.

I only added the 127.0.0.1 line to my /etc/hosts file and rebooted, that did the trick.

[vcloud3d]

no healthy upstream vCenter 7.0 error:

Issue fixed:

after 3.5 hrs of quick search on various blogs/vmware communities/mylab work/organization cook books. found the solution.

Steps tried out:

1. VAMI <IP add:5480> - Stop-Start all Services.

2. Stopped temp Antivirus solution on local host also stopped Windows Defender Firewall.

3. VAMI <https://IP add:5480> click on TIME  - change Time Zone to UTC also Time Sync - Mode - set it to Disabled.

4. upgraded VM hardware to version 18. <virtualhw.version = "18">

5. created a new host "A" record FLZ (forward lookup zone) on DNS Server for vcsa <192.168.40.110 vcsa.mylab.com>

6. created Pointer (PTR) record (Reverse lookup zone) on DNS Server for vcsa.<192.168.40.X vcsa.mylab.com>

7. restarted DNS server.

8. restarted vcsa 7.0.1 appliance (VM).

9. cleared web browser cache, tried multiple browsers thinking it can be a problem but, 

didn't work all above steps <1 to 9>. (wasted my time though but found a way to fix it by editing " host file"  on my local PC [DELL SERVER].

Added below entries on " host file" 

Path - C:\Windows\System32\drivers\etc

###########vcenter###################
192.168.40.110 vcsa.mylab.com
192.168.40.100 dc01.mylab.com
192.168.40.110 https://vcsa.mylab.com:5480
192.168.40.110 https://192.168.40.110:5480
127.0.0.1 vcsa.mylab.com

rebooted the vCenter Server and then try to access it and it " Worked fine"  for me. thanks.

regards,

vcloud3d

[Mohamed2233]

Hello, I have tested this in a lab environment; the error will be fixed when using an external NTP server during installation. And everything will run smoothly. Please check this:

https://www.systemadminslabs.com/2020/11/22/vmware-vcenter-7-server-deployment-failing-and-no-healthy-upstream-forever-errors/

 

[yangjie8366]

感谢foxmr的分享，按照你的方法，我的环境已经正常工作，多谢多谢

[yangjie8366]

感谢foxmr的分享，按照你的方法，我的环境已经正常工作，多谢多谢

[Ravi Shankar]

Usually, we see this message in the web client while all the services are starting. So wait for some time and check the web client.
If you still see the same message in the web client, try checking the status of the services from the command-line (service-control --status --all) or through VAMI (https://vc-ip:5480).

If all the services are not UP and running, the best way is to restart all the services (service-control --stop --all) & (service-control --start --all). Now when all the services are starting VC  should throw an error if there are any potential issues.

[Mohamed2233]

I faced the same problem I make new deployment i synchronize vCenter installation to public NTP servers, and synchronize DNS to the same NTP server, the deployment succeed and vCenter works in timely manner after restart.

vCenter 7 deployment failing and "no healthy upstream" forever error - System Admin Labs

[msabousian]

Thanks to foxmr, I confirm that editing hosts file did the trick for me. everything works fine now

[杨海艳]

注意：如果登录vCenterServer 是页面出现“no healthy upstream”字样，则是vCenterServer服务没有完全启动，稍等一段时间，再次登录即可，或登录管理页面：https://vc_ip:548，找到“服务”，手动将没有自动启动的服务启动即可。导致vCenterServer服务没有完全启动的原因可能是域名服务器不能连接，或NTP时间配置不当等。

[ceyhunkirmizita]

I had similar issue and this is what I have done to fix it.

Problem: When I try to access vCenter Server URL, It gives an error "no healthy upstream"

I remember it was accessible few days ago and no configuration changes has been done for vCenter

Check: When I checked certificate, I saw that it was expired 2 days ago. (All certs were self signed)

Fix: I regenerated all certs by doing steps written at How to regenerate vSphere 6.x and 7.x certificates using self-signed VMCA 2112283:   https://kb.vmware.com/s/article/2112283

• Launch the vSphere 6.x Certificate Manager
  • /usr/lib/vmware-vmca/bin/certificate-manager
• I selected option "8" for my case. Select the correct option at this stage. Menu items are pretty self explanatory. 
• Type the [e-mail] password when prompted.

Note: You may also have similar issue if your VMCA date/time is not correct. For this case, you can try to correct date/time, use NTP servers for VMCA etc.