vCenter

 View Only
  • 1.  VCSA no healthy upstream and services failing to start

    Posted Apr 26, 2021 03:15 AM

    I had tried adding VCSA to a domain, then I started getting these issues.  I'm not certain that is the cause, but that was what I was doing right around the time it happened.  When I tried service-control --start --all, I see several fail.  Any help would be appreciated, thank you!

    root@vcenter01 [ ~ ]# service-control --start --all
    Operation not cancellable. Please wait for it to finish...
    Performing start operation on service lwsmd...
    Successfully started service lwsmd
    Performing start operation on service vmafdd...
    Successfully started service vmafdd
    Performing start operation on service vmdird...
    Successfully started service vmdird
    Performing start operation on service vmcad...
    Successfully started service vmcad
    Performing start operation on profile: ALL...
    Successfully started service vmware-vmon
    Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sca, vapi-endpoint, vpxd-svcs services. Error: Operation timed out



  • 2.  RE: VCSA no healthy upstream and services failing to start

    Broadcom Employee
    Posted Apr 27, 2021 01:06 AM

    I suspect expired solution user certs. 

    Run    this command on SSH fo find : 

    for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

    If expired check   https://kb.vmware.com/s/article/2112283

     



  • 3.  RE: VCSA no healthy upstream and services failing to start

    Posted Apr 27, 2021 01:46 AM

    Here is the output, the look good.

    root@vcenter01 [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
    STORE MACHINE_SSL_CERT
    Alias : __MACHINE_CERT
    Not After : Apr 25 06:09:33 2023 GMT
    STORE TRUSTED_ROOTS
    Alias : 8e02ecb5feb5b25ad64345dc430fe629fa20bff1
    Not After : Apr 19 18:09:32 2031 GMT
    STORE TRUSTED_ROOT_CRLS
    Alias : e1eb875a4a5e9b69b64a73ec47ba4e459e3e2baf
    STORE machine
    Alias : machine
    Not After : Apr 19 18:09:32 2031 GMT
    STORE vsphere-webclient
    Alias : vsphere-webclient
    Not After : Apr 19 18:09:32 2031 GMT
    STORE vpxd
    Alias : vpxd
    Not After : Apr 19 18:09:32 2031 GMT
    STORE vpxd-extension
    Alias : vpxd-extension
    Not After : Apr 19 18:09:32 2031 GMT
    STORE hvc
    Alias : hvc
    Not After : Apr 19 18:09:32 2031 GMT
    STORE data-encipherment
    Alias : data-encipherment
    Not After : Apr 19 18:09:32 2031 GMT
    STORE APPLMGMT_PASSWORD
    STORE SMS
    Alias : sms_self_signed
    Not After : Apr 24 18:18:34 2031 GMT
    STORE wcp
    Alias : wcp
    Not After : Apr 24 18:11:47 2023 GMT



  • 4.  RE: VCSA no healthy upstream and services failing to start

    Broadcom Employee
    Posted Apr 27, 2021 02:23 AM

    cert seems fine. What version of vcsa is this ?

    Investigate the below logs for issues. 

    /var/log/vmware/vapi/endpoint/endpoint.log
    /var/log/vmware/vpxd-svcs/vpxd-svcs.log



  • 5.  RE: VCSA no healthy upstream and services failing to start

    Posted Apr 27, 2021 02:33 AM

    VMware vCenter Server 7.0.2.00000

     

    I attached the logs in a txt, in a zip.  I see some errors, but I'm not sure about them.



  • 6.  RE: VCSA no healthy upstream and services failing to start

    Broadcom Employee
    Posted Apr 27, 2021 03:01 AM

    Was this upgraded from an earlier version or new install of vcsa 7.0U2 ?

     

    Noticed "Could not find any solution users from SSO" 

    Download the attached script 'solutionusers_fixer.zip and extract  and upload the .sh file to /tmp folder on VCSA.
    Make the file executable: chmod +rx /tmp/solutionusers_fixer.sh
    Run the script : /tmp/solutionusers_fixer.sh

    Restart all services : service-control --stop --all && service-control --start --all



  • 7.  RE: VCSA no healthy upstream and services failing to start

    Posted Apr 27, 2021 03:30 AM

    Did not work.  I really appreciate your help.  This is a new install.

     

    root@vcenter01 [ /tmp ]# chmod +rx solutionusers_fixer.sh
    root@vcenter01 [ /tmp ]# ./solutionusers_fixer.sh
    Solution users health check started on Tue Apr 27 03:20:09 UTC 2021
    SSO domain: ironrising.com

    Enter SSO admin password:
    Looking for machine user in groups

    Looking in cn=administrators,cn=builtin
    Looking in cn=SystemConfiguration.Administrators
    Looking in cn=ComponentManager.Administrators

    Looking for web client user in groups

    Looking in cn=administrators,cn=builtin
    Looking in cn=actasusers
    Looking in cn=licenseservice.administrators

    Looking for vpxd user in groups

    Looking in cn=users,cn=builtin
    Looking in cn=LicenseService.Administrators
    Looking in cn=systemconfiguration.administrators
    Looking in cn=componentmanager.administrators

    Looking for vpxd-extension user in groups

    Looking in cn=users,cn=builtin
    Looking in cn=actasusers
    Looking in cn=systemconfiguration.administrators
    Looking in cn=componentmanager.administrators
    Done on Tue Apr 27 03:20:14 UTC 2021. You may need to restart services...
    root@vcenter01 [ /tmp ]# service-control --stop --all && service-control --start --all
    Operation not cancellable. Please wait for it to finish...
    Performing stop operation on service observability...
    Successfully stopped service observability
    Performing stop operation on service vmware-pod...
    Successfully stopped service vmware-pod
    Performing stop operation on service vmware-vdtc...
    Successfully stopped service vmware-vdtc
    Performing stop operation on profile: ALL...
    Successfully stopped service vmware-vmon
    Successfully stopped profile: ALL.
    Performing stop operation on service vmcad...
    Successfully stopped service vmcad
    Performing stop operation on service vmdird...
    Successfully stopped service vmdird
    Performing stop operation on service vmafdd...
    Successfully stopped service vmafdd
    Performing stop operation on service lwsmd...
    Successfully stopped service lwsmd
    Operation not cancellable. Please wait for it to finish...
    Performing start operation on service lwsmd...
    Successfully started service lwsmd
    Performing start operation on service vmafdd...
    Successfully started service vmafdd
    Performing start operation on service vmdird...
    Successfully started service vmdird
    Performing start operation on service vmcad...
    Successfully started service vmcad
    Performing start operation on profile: ALL...
    Successfully started service vmware-vmon
    Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sca, vapi-endpoint, vpxd-svcs services. Error: Operation timed out



  • 8.  RE: VCSA no healthy upstream and services failing to start

    Broadcom Employee
    Posted Apr 27, 2021 03:43 AM

     

    Seems to me as user missing in a sso group  .   

    2021-04-26T03:08:54.918Z [pool-6-thread-1 ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper opId=] Hit ServiceFaultException while fetching admin group for the SSO Admin user
    : Administrator@ironrising.com

     

    Check if above account(Administrator@ironrising.com) is present  in Administrators and SystemConfiguration.Administrators

    Ajay1988_0-1619494825383.png

     



  • 9.  RE: VCSA no healthy upstream and services failing to start

    Posted Apr 27, 2021 03:58 AM

    AJ,

    How do I check that if I can't access vcenter?   I checked vCenter Server Management and didn't see that sections (on port 5480).  



  • 10.  RE: VCSA no healthy upstream and services failing to start

    Broadcom Employee
    Posted Apr 27, 2021 04:01 AM

    try to start the   vsphere-ui service and see if u can login to web-client and check it  . If not this needs a SR with VMware Support