vCenter 7 error - no healthy upstream

[Seb]

Hi All,

I'm working on brand new ESXi 7.0.0 deployment and have run into an issue when deploying vCenter 7.

When browsing to the vCenter UI I receive a "no healthy upstream" error.

I can access vCenter Server Management without issue.

I've done some googling and have been pointed towards certificate issues however my Leaf and Root certs are showing as valid.

All of the services which are set to Automatic start are running without any errors or warnings.

Is there a documented fix for this issue?

[Alex_Romeo]

Hi,

in this article there is a solution, try it:

https://geneve.com.au/2020/04/16/hello-world/

ARomeo

[Seb]

Hi Alessandra,  yes I've seen this article, it doesnt work unfortunately.

[IRIX201110141]

About the article.   The vHW Upgrade of the VCSA VM is not supportet.

Regards,
Joerg

[Seb]

So should I have not upgraded the HW Compatibility?  The VM was configured as Linux 3.3 64bit by default, I had to upgrade the HW to 7.0 to be able to select Photon OS.

[milindng]

I am  getting same error. I have checked and found my vcenter ip is already configure with static IP.

I think this not related to DHCP or static ip.

 

[cleston_tellis]

 , it is not related to DHCP or static ip as the vcenter vm will be given an A dns record on your dns server, so that is fine.

open putty and ssh to the vcenter vm.

than locate the host file /etc/hosts/

add the below record in the host file /etc/hosts/ to fix the issue...

127.0.0.1 (FQN name here vcsa.***.local) vcsa localhost

 

save and reboot the VM.

[Seb]

I've re-created my STS Certificates (using the script provided here: VMware Knowledge Base ) and rebooted, still the same issue.

So I don't think it's Cert related.  I can resolve the host with the FQDN so it's not DNS.

Any more ideas?

[virtual-power]

Can you clear all browser history.cache and do this again after restarting VCSA ? If still problem persists then might have to check the vsphere-ui logs

[Seb]

So I reployed a new VCSA, and it worked without issue, I then upgraded the vHW to 7, and set the OS to Photon, still working without issue.

Weird!  I'm afraid I can't offer a fix/solution.

[MTB75]

We see the same issue here on all new installations.

The trick seems to be waiting for the cpu usage of the vm goes down. It needs some time to start all the services.

On one installation we had to wait for about 30 minutes.

After the cpu usage is back to normal we were able to login to the vcsa.

It seems that vcsa7 needs a lot of ressources.

[mrc247]

I have the same problems with a fresh installation of VCSA 7.0.0c.

I gave the VM more CPUs and waited 30 minutes after booting, but I still get the message "no healthy upstream".

The service "vAPI-Endpoint" reports the error "Failed to connect to f592d7af-0df2-4c7f-8e29-a9f03efd7a34 \ com.vmware.vcenter.nsxd.vapi vAPI provider."

A restart doesn't help either. What should I do?

[Fanya249]

Spend 5 hours digging up today installing/reinstalling. So my problem with "no healthy upstream" was DNS:

It seems critical for VCSA 7 during setup stage to have access to the DNS. DNS should contain BOTH A record & PTR record.

My test environment was lacking PTR.

Hope that helps.

[MarcusZheng]

Great, that fixed my error too!

My DNS settings on the DNS server for the vCenter are correct, but the primary DNS was pointing to 127.0.0.1 on vCenter. I need to run /opt/vmware/share/vami/vami_config_net command on vcenter via SSH, select option 4, changed the primary DNS IP from 127.0.0.1 to a proper DNS server IP, then rebooted vCenter and Worked!!!!

0) Show Current Configuration (scroll with Shift-PgUp/PgDown)
1) Exit this program
2) Default Gateway
3) Hostname
4) DNS
5) Proxy Server
6) IP Address Allocation for eth0
Enter a menu number [0]: 4

[promaster]

Thanks got this error after the latest update, the dns whas changed to 127.0.0.1. now vcenter works.

[Gilbert XX]

Hello 

I cannot connect thru SSH. Is there another way to change the DNS value ?

[Gilbert XX]

Hello

I cannot ssh my VC. is there another way to check/change DNS?

[sean468]

Thanks mate, this is exactly what fixed mine. 

[dvbsolutions]

I had the same issue.  I deleted and installed vcenter twice before I read this solution.  The problem I had was that I was shutting down my other host (which had my DC and DNS) , so that I could add the host into Vcenter.  I had to put the host in maintenance mode prior to adding it into vcenter.  When I launched vcenter I got the upstream error.  I exited maintenance mode and started my DC, and Vcenter launched correctly. 

[MladenZG]

Hi,

I am trying to install vCenter last 3 weeks.  no matter what i do ,  I have same  error: no healthy upstream

I tried to deploy image via  UI, via manual OVA deployment,  no mater what method I do , same problem exists.

login into https://<vcenter_IP_ADDRESS>:5480  works. (Server Management)

I did deployment, then upgraded compatibility to esx7, changed to proton OS, instead 2 CPU / 12GB ram, I increased to 8 CPU / 24GB RAM and I got performace increase, but same error when trying to Launch vsphere client (HTML5)

I used version 7.0.0.10600 during installation / build version 16620007

tried before with  build 16386292, and had same thing.

Since I am using trial version, I am not sure do I have rights for official support?

I could  troubleshoot on my  own , but I am not sure where to seek logs?

anyone can pinpoint locations where logs are located if I access wia SSH ?

best regards

--

Mladen

[mrc247]

I redeployed the vcsa and now the "no healthy upstream" error is gone. Crazy, apparently there is a bug in the installer.

[MladenZG]

Personally tried  over 5 times to do reinstallation.

what build did you used ?

b,r,

Mladen

[mrc247]

the latest VCSA 7.0.0c (7.0.0.10600)

[oparnes]

What do you mean redeploy?

reinstall on the existing VM ?

or deploy a new one from fresh?

[Hua Gong]

1.log in https://VCFQDN:5480   check the service  is ok  ,   VMware vCenter Server , VMware vSphere Client

2. if VMware vSphere Client can not start , check the /var/log/vmware/vsphere-ui/logs/*runtime.log.stderr

I have same error , now working to fix it .

[Hua Gong]

In My case , vcenter vm use thin mode ,and  the datastore is full, after check the  /var/log/vmware/vsphere-ui/logs/*runtime.log.stderr  in vceter os

find /usr/lib/vmware-vsphere-ui/server/conf/server.xml  is empty  , I copy server.xml from another vc7.0 and it works .

[David Petzer]

I had this issue intermittently. It occurred when the configured DNS server for vCentre was not available.

As soon as the DNS server is available, happy days.

We have an internal DNS server that is a windows domain controller. I noticed this when changes were being made to that domain controller.

[aljbali]

same here, I made some changes to my DNS router ( added pfsense ) that broke my vCenter7

[wikusvanderwalt]

Hi, I came across the same issue when deploying the latest release (7.0.0c).  However, the first release (7.0.0) does not give the error! 

HTH!

[psstumlakia1]

• Check whether you have enough storage space left on the datastore that vcenter is running
• Check whether there's enough resources left on the host (CPU, Memory)

and then from the ssh shell start all the services (services-control --start --all)

[AmitGN]

I also faced the same problem.

In my case.

1. log directory was full, so increased the log partition ( after increasing the log partition also same error continued )

2.  after the recreation of the certificate problem is solved.

cmd:  /usr/lib/vmware-vmca/bin/certificate-manager

[ESXi_Since_3_5]

I just came across this issue. This happened to be an issue with Chrome. I powered off the vcsa upgraded it to run hw for esxi 7 and changed the os to Photo - I don't think this had anything to do with it but I recommend everyone do this because it runs probably about 10x smoother... boot time is about 4 minutes less. But I then tried it again in chrome no go loaded up IE 11 and it went in just fine. I cleared out my cache and history from chrome and restored to defaults and it then allowed me back in. I can't say for certain if loading it in IE assisted anything, but have to put it in there because it was the steps I took.

[welder314]

It's likely DNS. You need a forward A record and a PTR record before you deploy. I see lots of people missing the PTR.  If you were missing a PTR record, and you simply add one, it won't magically start working, even if you restart vCenter.

However if you need to add your PTR record after-the-fact, you can simply Edit the network settings in vCenter Management (no need to make changes) and that will force the network settings to refresh, and then it will work.

[nary4484]

However if you need to add your PTR record after-the-fact, you can simply Edit the network settings in vCenter Management (no need to make changes) and that will force the network settings to refresh, and then it will work.

This was the fix for me.  Thank you for saving me a bunch of time... I was a couple google searches away from redeploying another version or something like that :smileyhappy:

[billdossett]

 

bingo - forgot to add a new reverse zone for the new vlan subnet I'm building on so no PTR recs.  Created it and edited the network as you suggested which looked fine and then said can't connect to vcenter server - but it had finished and I can now login to my spanking new VCSA, thanks!

[Bill Hill]

I'll add to some of the recent commentary. I recently had a power outage in my homelab. Upon starting up the hosts, my vCenter auto-started. However, the DNS server did not. vCenter was not happy and I had the no healthy upstream error. After a little investigation, I identified the issue with the VM hosting DNS being offline and brought it back online. Then, I restarted the VCSA using the VAMI. I still got the no healthy upstream error for a couple of minutes... and then, after a page refresh, the UI was there and everything was working well.

As I was troubleshooting the error (prior to realizing the DNS server was not running), I noticed that the vAPI Endpoint was showing Healthy - With Errors (or something similar). I don't see that any longer.

For those that may find this page in the future, make sure your environment supports:

• FQDN resolution (ex: vcenter.domain.com)
• Short resolution (ex: vcenter)
• Reverse resolution (ex: if vCenter's IP is 192.168.86.221, then  221.86.168.192.in-addr.arpa should resolve to vcenter.domain.com)
• IP-based routing between nodes in the network

In my instance, without DNS existing, vCenter lost its mind and couldn't find what it needed to. Bringing DNS back online enabled FQDN, Short, and Reverse resolution options!

[ESXi_Since_3_5]

Yeah this is definitely a DNS issue. You can flush your DNS on your local box first. If that didn't work, verify check out to verify that you have forward and reverse records. A and PTR records are required.

[whc]

This problem may be caused by DNS，try to create a DNS server and resolve your vCenter FQDN.If you don't have a DNS server，set the DNS of vCenter VM to 0.0.0.0，keep hostname as localhost。Good luck！

[ipasipas]

Check the diskspace of the system. I my case the /storare/log volume was completely full. This cause the "no healthy upstream" error.

In particular my /storage/log/vmware/pod dir was full of pod-startup.log.xxxx files. I deleted them all, and the server started correctly.

[genevt]

Well, I ran into the same issue, but waited a bit and it started working.  I have done what others have stated by logging onto the vCenter management on port 5480 and checked on the performance.

CPU was pegged while I was getting the "no healthy upstream" message.  From the graph shown below, it took 5 minutes before the CPU settled down and I was able to log onto the vCenter on 443.  The error message could be more useful.  I don't know what a healthy upstream is LOL, but there you have it.  I didn't have to change the vHW version or change the OS to Photon. 

 

 

[ojw_nc]

For me it was necessary to fix the hosts file, as per: https://communities.vmware.com/t5/vCenter-Server-Discussions/vCenter-7-Bug-No-Healthy-Upstream/m-p/2286788/highlight/true#M74811

[cleston_tellis]

added this record in the host file /etc/hosts/ to fix the issue...

127.0.0.1 (FQN name here vcsa.***.local) vcsa localhost

[kronoz19]

thank you, this option solved mi issues 

 

Nombre DNS:localhost

DNS: 127.0.0.1

 

[Mohamed2233]

Hello, I have tested this in a lab environment; the error will be fixed when using an external NTP server during installation. And everything will run smoothly. Please check this:

https://www.systemadminslabs.com/2020/11/22/vmware-vcenter-7-server-deployment-failing-and-no-healthy-upstream-forever-errors/

 

[HostaHost]

Wow, I just caused a VCSA failover and this 'no healthy upstream' message by performing the incredibly dangerous operation of disconnecting the vcsa install DVD by switching from datastore ISO file to client device.  Seriously; 7.0 is so fragile that disconnecting a DVD causes a complete failure and HA failover... unreal.  Fortunately it did fail successfully so the interface came back a few minutes later.

[BinurajKR]

This issue can be happened if you attempted to rename the hostname of vCenter server from console. if this is the case, add host and PTR record in DNS and then change the hostname from VAMI interface (under networking). this will reboot the appliance and then the issue will be solved. Hopes this helps.

[jc0ke]

this work for me, when i had same issues

http://vcloud-lab.com/entries/vcenter-server/vmware-vcenter-server-error-no-healthy-upstream-

[grumpyboy]

I also had this problem, turned out it just takes quite a long time for vCenter server to finish booting up ( about 20mins! ). You can can login via root (ssh) and wait untill the CPU useage goes low again, thats when its finished.

After 20mins wait the CPU had gone to normal and I checked the vCenter URL again and this time all was normal and working!

 

[Long9499]

I got the same error, i found a thread on somewhere on google and they said it could be because the ipv4 was set to DHCP, so i checked it under Networking -> Networking Settings -> Edit, but it was Static since the beginning, so i leave it there and finish editing, Networking restart and kaboom the error gone. So i think if facing this you guys can do these:

- Check DNS server

- Check Client and Networking service

- Restart Networking service

[virtual-power]

This is not an issue with vCSA. Please wait for all services to be up. Monitor the VAMI page and validate all services are up and then only connect to VC using HTML. Otherwise that error is expected. I have see in my lab services taking 10-15 mins after vCSA reboot(It can vary even).

[Long9499]

Yes you are right, i did a check and wait for about 10 mins after vCSA start, the Web Cient working perfectly, also there is a trick that you guys could pay attention to how much memory vCSA consume, for example, my vCSA working normally with all services up take about more than 10 gbs, so when my vCSA use 10 gbs of memory, i know that the time i can use Web Client

[Marv1]

Hi All I had the same issue here after I reconfigured the DNS server

I tried logging in with just the ip address but never worked not matter what I tried, I decided to add the port at the end of the ip address...... ie..192.168.34.2:5480 and it worked perfectly fine

PS: it was working fine before I changed the ip address and the DNS

Hey i'm Marv by the... just too lazy to create an account

[vgarczynski]

I had the same issue when i just used https://<vCenter IP> , but when I added the /ui to the url it worked.

[vgarczynski]

After logging in to the VAMI i found that the /storage/seat disk was over 95% full Once that was fixed I no longer got the no healthy upstream message

[sasolwintelteam]

Only run this cmd after installation from a putty session with vcentre server , i got this problem 8 time and evrytime this command sorted . 
service-control --start --all

[ThereAreSomeWho]

Thank you, this appears to have resolved my issue after updating to 7.0 Update 3n

[TALAC]

Hello, Me for this same problem I just put the host and the VCSA on the same time zone and at the same time. Hop it worked

[Atallah92]

for me i just add the port 5480 then go to services and restart the vAPI endpoint, it works every time hope it helps.

[avenonet]

/usr/lib/vmware-vmca/bin/certificate-manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

|                                                                     |

|      *** Welcome to the vSphere 6.8 Certificate Manager  ***        |

|                                                                     |

|                   -- Select Operation --                            |

|                                                                     |

|      1. Replace Machine SSL certificate with Custom Certificate     |

|                                                                     |

|      2. Replace VMCA Root certificate with Custom Signing           |

|         Certificate and replace all Certificates                    |

|                                                                     |

|      3. Replace Machine SSL certificate with VMCA Certificate       |

|                                                                     |

|      4. Regenerate a new VMCA Root Certificate and                  |

|         replace all certificates                                    |

|                                                                     |

|      5. Replace Solution user certificates with                     |

|         Custom Certificate                                          |

|         NOTE: Solution user certs will be deprecated in a future    |

|         release of vCenter. Refer to release notes for more details.|

|                                                                     |

|      6. Replace Solution user certificates with VMCA certificates   |

|                                                                     |

|      7. Revert last performed operation by re-publishing old        |

|         certificates                                                |

|                                                                     |

|      8. Reset all Certificates                                      |

|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|

Note : Use Ctrl-D to exit.

Option[1 to 8]: 8

 

 

for me this is what I had to do then from here you are good to go and make sure to set your calendars to renew the certs! 

[Sofoniyas Teklegiorgis]

https://windowsreport.com/no-healthy-upstream-error/

this helps a bit

[Burti]

I was buy the dedicated server https://www.burtinet.com and I installed vCenter but its showing this error. How can I fix it?

[virtual-power]

Please start a new post. "no healthy upstream" is a very generic thing. Mostly means VC services are not up. 

[MarcelFernandez]

Recently encountered this error. logged on to host and upgraded vmware hardware compatibility - change os family to photon os. This caused no change, but might as well set.

what did work was that certs had expired and didn't auto renew. please look at the following link to check your certs and renew:

vCenter Server 7.0 HTML5 UI error “no healthy upstream” – vElements.net

[usmguy04]

For me, this was because I had changed the time zone on my vcsa from UTC to my local time zone. I changed it back to UTC, rebooted, and everything was fine.

[Sameer0619]

i restarted the service Services>VMware vCenter Server and it works .

hope this will help.

[romesds157]

I've done the same thing, restart vcenter server service from the Management Console. It looked like it was going to timeout because of the blue spinning wheel of death but it started, came online as "healthy" and i was able to access everything as usual! 

[Cukal42]

AmitGN response is correct, running certificate-manager fixes it. 

This kb details how to check for expired certificates: https://kb.vmware.com/s/article/82332

This kb details how to run certificate-manager: https://kb.vmware.com/s/article/2097936

 

[vmkfix-SSA]

[vmkfix-SSA]

I followed the below article to fix the issue.

 

https://vmkfix.blogspot.com/2023/05/vmware-vcenter-not-login-no-healthy.html 

[riscoto]

I have tried restarting all my services from vMware Server Management. And it worked on my end. I can now use vSphere Client and manage multiple Physical Server and VM Servers.

[AkmalAza]

Hello everyone, I encountered a similar issue with our vCenter UI, where the upstream status appeared unhealthy. We are utilizing linked mode for three vCenters. Numerous solutions were suggested, including DNS and NTP configurations, which we attempted but yielded no results. Additionally, we tried increasing the vCenter's memory and allocating a specific amount for the vpxd services, but the improvement lasted only for a period of 24 to 48 hours. To resolve this, we simply rebooted our SRM (Site Recovery Manager). It turned out that SRM had a bug causing the vpxd service to become busy. After rebooting both SRM and vCenter, the services were refreshed, and everything has been running smoothly for over two weeks without any interruptions even with reduced the memory allocation on the vCenter. my suggestion is to take note whatever connection connected to your vCenter and powered it off and bring it back on by one to eliminate and detect which appliance that causing it.   

[AkmalAza]

you also need to clear up your vpxd crash dump log as each time the vpxd stopped it will create 1 files which in my case around 10GB per file.

[ecerronj]

Soy Eduardo Cerrón, También me sale el error, "no healthy upstream", he intentado resolverlo viendo varios problemas similares, pero, sin éxito, alguien puede proporcionarme mas información, por favor.

[genemcsween]

This turned out to be an expired STS certificate chain in my environment keeping vmware-stsd from starting.  At the CLI, I ran service-control --status and saw the service was stopped.  The script in this article (Checking Expiration of STS Certificate on vCenter Servers (79248) (vmware.com)) threw a bunch of errors because it couldn't connect to STS.

I ran the script from this article Article Detail (vmware.com) then ran option 8 from this article How to use vSphere Certificate Manager to Replace SSL Certificates (2097936) (vmware.com), rebooted and I'm up and running again!!

I now have a reminder set in my email to renew this cert when it gets 60 days out from expiring.

[IK-2]

Check DNS, and all IP's that are set, and all physical network connections. Also, if you use a switch or switches, reboot the one that is passing thru the network to your datacenter on the same subnet. Most likely this is either a network issue or a big bug in a release.

[Sina_Rastgouyan]

You can solve this problem by restarting the VMware Vsphere Client service or starting it manually using your vcsa management panel.

[Brian Willmert]

I had this same problem and it was unfixable by any means I found on this forum.  I had several services that refused to start.  So none of the "just restart this service to fix it" was valid in my case. I tailed the logs that were produced after trying to start one of the services and the message was "failed to create folder blah\blah\blah" (sorry I didn't take a better note).  Well going down the rabbit hole of that error message and only finding results that weren't related to VMware at all, I finally gave up and just re-deployed the stupid thing.  Luckily, this was in my development environment with only 2 hosts so very minimal configuration needed.  But still a pain.  It was clear while troubleshooting this that it can be caused by any number of reasons.  And unless you happen to stumble on your specific fix within an hour, you're better off redeploying.

[Sravan_k]

In my case it happened recently after code upgrade, I found the log store/disk attached to VC is full and it resolved once I scaled the size and restarted the VC...

It may help someone