VMware vCenter Appliance vsphere-ui failing to start

[Allan McClure]

Afternoon,

I have had a number of issues with this server today its not allowing us to login and we get the error about unhealthy upstream..

After sometime we can get into the console and see that there are services that havent started main one vsphere.ui

Is there logs somewere I can check or recomendations from anyone?

Cheers 

[Alexandru Capras]

Do you see any alerts in vCenter, like the 'vSphere UI Health Alarm'? You might also want to check for warnings in VAMI.
My first guess would be a memory issue - maybe the VCSA machine needs more memory or a heap size increase.

You can also check this log: /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

[Allan McClure]

Thanks for the reply I only see this error message at the moment in the VAMI.

I could increase the memory from 18GB but its currently only using 5GB

VMWare Performance Chart Service error  health.statsReoptInitalizer.illegalStateEx

[Allan McClure]

from log file

[2025-05-22T01:53:25.260+12:00] [INFO ] wcp-plugins-notifier-1        c.v.v.plugin.notification.WcpServicePluginNotificationsListener   Updating WCP plugin subscriptions...
[2025-05-22T01:53:48.935+12:00] [INFO ] plugin-service-proxy-84       com.vmware.vise.plugin.proxyconfig.PluginServiceConfigurer        Syncing plugin service records with Lookup Service.
[2025-05-22T01:55:05.837+12:00] [INFO ] health-status-9               com.vmware.vise.vim.cm.healthstatus.AppServerHealthService        Memory usage: used=508,941,272; max=803,733,504; percentage=63.32214216119078%. Status: GREEN
[2025-05-22T01:55:05.837+12:00] [INFO ] health-status-9               c.v.v.v.cm.HealthStatusRequestHandler$HealthStatusCollectorTask   Determined health status 'GREEN' in 0 ms
[2025-05-22T01:58:05.837+12:00] [INFO ] health-status-9               com.vmware.vise.vim.cm.healthstatus.AppServerHealthService        Memory usage: used=510,406,560; max=803,733,504; percentage=63.504452341456705%. Status: GREEN
[2025-05-22T01:58:05.837+12:00] [INFO ] health-status-9               c.v.v.v.cm.HealthStatusRequestHandler$HealthStatusCollectorTask   Determined health status 'GREEN' in 0 ms
[2025-05-22T01:58:25.335+12:00] [INFO ] wcp-plugins-notifier-1        c.v.v.plugin.notification.WcpServicePluginNotificationsListener   Updating WCP plugin subscriptions...
[2025-05-22T01:58:48.964+12:00] [INFO ] plugin-service-proxy-84       com.vmware.vise.plugin.proxyconfig.PluginServiceConfigurer        Syncing plugin service records with Lookup Service.
[2025-05-22T02:01:05.837+12:00] [INFO ] health-status-9               com.vmware.vise.vim.cm.healthstatus.AppServerHealthService        Memory usage: used=514,810,864; max=803,733,504; percentage=64.05243298156698%. Status: GREEN
[2025-05-22T02:01:05.837+12:00] [INFO ] health-status-9               c.v.v.v.cm.HealthStatusRequestHandler$HealthStatusCollectorTask   Determined health status 'GREEN' in 0 ms

[dabrigo]

Also, have a look if there are any expired certificates:

https://knowledge.broadcom.com/external/article/344201/verify-and-resolve-expired-vcenter-serve.html

https://knowledge.broadcom.com/external/article/385107/vcert-scripted-vcenter-expired-certific.html

[Allan McClure]

Thanks for the feedback, I have checked the certs they currently all look good and dont expire till next year.

The only log that makes sense is the one aboue password expiring which we resolved, but still not starting these are the services not currently running might help please

[Alexandru Capras]

From the last screenshots, the manual services and vCenter HA appear to be fine... they seem to be in their expected state.

So, just to confirm: your main issue is that the vsphere-ui service fails to start, and even if you start it manually, it stops again?

I recommend checking the STS certificate, it might be expired: https://knowledge.broadcom.com/external/article/318968/checking-expiration-of-sts-certificate-o.html

[Allan McClure]

Thanks for the reply, I have 2 servers so the second one I can login too, and was able to see that sts certs are all ok.

The main issues are we cant see those services started, and such when you go to login to the host you get this error about "no healthy upstream"

As we have anther remote site with a second server, I can login to that one using that name instead. But the primary one is no longer working i.e the failed start up of those services

[hgdeloitte]

Hi.

I am presuming you have tried a cold boot.

Have you tried migrating it to another host?

When looking at the disk usage using df -h in an ssh session, does all the storage have plenty of free space?  Except for the archive storage which typically runs around 95%.

[Allan McClure]

Many thanks I have moved it around, and also tried restoring from backup. 

Is there anywhere I can read a log file etc ? 

[Allan McClure]

Ok so I ran through the tool to check the certs and I get a few errors is there any recomendations for resolving these please

Checking Auto Deploy CA certificate                       NO SKID
Checking BACKUP_STORE entries:
   bkp___MACHINE_CERT                                     EXPIRED
   bkp_machine                                            EXPIRED
   bkp_vsphere-webclient                                  EXPIRED
   bkp_vpxd                                               EXPIRED
   bkp_vpxd-extension                                     EXPIRED
   bkp_hvc                                                EXPIRED
   bkp_wcp                                                EXPIRED
Checking VMCA certificate                                   VALID

Checking STS Signing Certs & Signing Chains
-----------------------------------------------------------------
Checking TenantCredential-1:
   TenantCredential-1 signing certificate                   VALID
   TenantCredential-1 CA certificate                        VALID
Checking TrustedCertChain-1:
   TrustedCertChain-1 signing certificate                   VALID
   TrustedCertChain-1 CA certificate                        VALID

Checking CA certificates in VMDir [by CN(id)]
-----------------------------------------------------------------
D99203A1D962C0A6C324D0A13251DB85F6D836BC                    VALID
D86670439202366A32856CF71DE045F28BDA43C9                    VALID
0CBB1A726462E8E482C06486CC1011D56D020954                    VALID
9874D612C22C179548614F98786E61E679778A0E                    VALID

Checking CA certificates in VECS [by Alias]
-----------------------------------------------------------------
683daad1b559a6a7e7e6693b4b25a86fc34246bb                    VALID
9af949af47d66b39c1491585eacba72b2fddd018                    VALID
4c656f9a978b4ee4b5c9d712f689cff811ad9df0                    VALID
e43ea8963dac19c3230bf7687ed139557b16244b                    VALID

Checking VECS Stores
-----------------------------------------------------------------
Checking status and permissions for VECS stores:
   MACHINE_SSL_CERT                                            OK
   TRUSTED_ROOTS                                               OK
   TRUSTED_ROOT_CRLS                                           OK
   machine                                                     OK
   vsphere-webclient                                           OK
   vpxd                                                        OK
   vpxd-extension                                              OK
   SMS                                                         OK
   APPLMGMT_PASSWORD                                           OK
   data-encipherment                                           OK
   hvc                                                         OK
   wcp                                                         OK

Checking Service Principals
-----------------------------------------------------------------
Node 7725c736-b03d-4bf0-abbf-8ee2b1463150:
   machine                                                PRESENT
   vsphere-webclient                                      PRESENT
   vpxd                                                   PRESENT
   vpxd-extension                                         PRESENT
   hvc                                                    PRESENT
   wcp                                                    PRESENT

Checking Certificate Revocation Lists
-----------------------------------------------------------------
Number of CRLs in VECS                                          4

Checking SSL Trust Anchors
-----------------------------------------------------------------
                                   VALID
                              MISMATCH

Checking vCenter Extension Thumbprints
-----------------------------------------------------------------
com.vmware.rbd (vpxd-extension)                          MISMATCH
com.vmware.vcIntegrity (vpxd-extension)                   MATCHES
com.vmware.vim.eam (vpxd-extension)                      MISMATCH
com.vmware.vmcam (Authentication Proxy)                   MATCHES
com.vmware.vsan.health (Machine SSL)                      MATCHES

Checking VMCA Configurations in VCDB
-----------------------------------------------------------------
vpxd.certmgmt.certs.cn.country                  'US'
vpxd.certmgmt.certs.cn.email                    '[e-mail]'
vpxd.certmgmt.certs.cn.localityName             'Palo Alto'
vpxd.certmgmt.certs.cn.organizationalUnitName   'VMware Engineering'
vpxd.certmgmt.certs.cn.organizationName         'VMware'
vpxd.certmgmt.certs.cn.state                    'California'
vpxd.certmgmt.mode                              'vmca'

Checking STS Server Configuration
-----------------------------------------------------------------
Checking VECS store configuration                              OK
Checking STS ConnectionStrings                          MISCONFIG