vCenter

 View Only

  • 1.  Vcenter error 503

    Posted Dec 26, 2018 11:08 AM

    Hi !

    I have a problem with my VCSA server (v 6.0.0.2)

    The server storage was full and I had to delete some *.log files to start the server.

    But when I restarted the VCSA, I try to connect to the web page of the vSphere Web Client, this message is displayed :

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f92c30900f0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

    I checked the running services with the command service-control --status and the result is :

    INFO:root:Running:

    vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)

    Running:

    vmware-cis-license (VMware License Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vsan-health (VMware VSAN Health Service) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager)

    INFO:root:Stopped:

    vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()

    Stopped:

    vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vsphere-client ()

    I don't know what to do next. Could you help me to recover my VCSA server ?



  • 2.  RE: Vcenter error 503

    Posted Dec 26, 2018 12:56 PM

    Are you using external PSC or Embedded ?

    Provide df -h output



  • 3.  RE: Vcenter error 503

    Posted Dec 26, 2018 02:15 PM

    Thanks for your reply :

    The primary service controller is embedded.

    The result of the command

    /usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost

    =>

    https://*********/lookupservice/sdk

    (******* is the FQDN of the VCSA)

    and the result of df -h :



  • 4.  RE: Vcenter error 503

    Posted Dec 26, 2018 03:34 PM

    Hello...

    Can you check the /var/log/vmware/invsvc/invsvc.log output and share the last ending part of the email.

    This service is down and is required for vcenter as it is dependency.

    Thanks,

    MS



  • 5.  RE: Vcenter error 503

    Posted Dec 26, 2018 04:23 PM

    Hello msripada and thank you for your reply

    The "inv-svc.log" file is very long, it contains several errors like this :

    Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

    LDAP error [code: 49]

            at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)

            at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)

            at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

            at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

            at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

            at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

            at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

            ... 75 more

    Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

    LDAP error [code: 49]

            at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:275)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)

            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)

            at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)

            at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)

            at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)

            at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

            at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:296)

            at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

            at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)

            ... 85 more

    Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

    LDAP error [code: 49]

            at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)

            at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:122)

            at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:267)

            ... 96 more

    Caused by: java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

    LDAP error [code: 49]

            at com.vmware.cis.lotus.LdapConnectionFactory.getConnection(LdapConnectionFactory.java:142)

            at com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer.<init>(LotusInitializer.java:26)

            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

            at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)

            at java.lang.reflect.Constructor.newInstance(Unknown Source)

            at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)

            ... 98 more

    Is the "Invalid credentials LDAP error" message the origin of the problem ?

    How to resolve this ?

    When I open the URL https://myhostname.mydomain.loc:7444/lookupservice/sdk , I have this message :

    Etat HTTP 404 - /lookupservice/sdk

    type status report

    message /lookupservice/sdk

    description The requested resource is not available

    Pivotal tc Runtime 3.1.5.RELEASE/8.0.36.A.RELEASE



  • 6.  RE: Vcenter error 503
    Best Answer

    Posted Dec 26, 2018 05:47 PM

    Power off/On does not help here. Please follow this KB VMware Knowledge Base

    Ensure to take backup of vCenter/psc . If they are in linked mode, ensure to take snapshots of all PSC's in linkedmode in shutdown state

    Post restarting inventory, you can restart the vpxd service and others.

    Thanks,

    MS



  • 7.  RE: Vcenter error 503

    Posted Dec 27, 2018 11:06 AM

    RajeevVCP4​ : if only it was so simple... I rebooted / shutdown many times, but the problem is still there...

    msripada​ : thanks for the link.

    I followed all the steps on the KB and... IT WORKS !!!! :heart:

    Congratulations msripada, you are the best.



  • 8.  RE: Vcenter error 503

    Posted Dec 26, 2018 05:33 PM

    Do the power off and power on , not restart

    once done wait for 10 minutes then try to logon