vCenter 6 Inventory Service Not Starting

[antonbizzari]

Hello, we ran into the issue of the root volume running out of disk space due to very large log files the other day. I manged to clear up the space but I had to hard crash and reboot the vCenter virtual appliance in the process.

The system is back up and working, except vCenter is not starting up.

The issue I am running into is the vmware-invsvc and vmware-vpxd service will not start.

Starting VMware Inventory Service...

Waiting for VMware Inventory Service................................................

WARNING: VMware Inventory Service may have failed to start.

error logs from /var/log/vmware/invsvc/wrapper.log

STATUS | wrapper  | 2017/09/25 19:30:51 |

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[1] : -J

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[2] : vmware-invsvc

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[3] : -O

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-invsvc_jvm.conf

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command launched (pid: 4360), blocking for up to 3600 seconds...

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[0] : ln

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[1] : -f

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[2] : /usr/java/jre-vmware/bin/java

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command line[3] : /usr/java/jre-vmware/bin/vmware-invsvc

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command launched (pid: 4363), blocking for up to 15 seconds...

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command completed with exit code: 0

INFO  | wrapper  | 2017/09/25 19:30:51 | Event Command 'jvm_prelaunch': Command completed with exit code: 0  Continuing.

STATUS | wrapper  | 2017/09/25 19:30:51 | Launching a JVM...

INFO  | jvm 1    | 2017/09/25 19:30:52 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO  | jvm 1    | 2017/09/25 19:30:52 | WrapperManager: Initializing...

STATUS | wrapper  | 2017/09/25 19:31:44 | <-- Wrapper Stopped

STATUS | wrapper  | 2017/09/25 20:39:14 | --> Wrapper Started as Daemon

STATUS | wrapper  | 2017/09/25 20:39:14 | Java Service Wrapper Professional Edition 64-bit 3.5.24

STATUS | wrapper  | 2017/09/25 20:39:14 |  Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

STATUS | wrapper  | 2017/09/25 20:39:14 |    http://wrapper.tanukisoftware.com

STATUS | wrapper  | 2017/09/25 20:39:14 |  Licensed to VMware Global, Inc. for VMware vCenter Inventory Service

STATUS | wrapper  | 2017/09/25 20:39:14 |

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[1] : -J

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[2] : vmware-invsvc

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[3] : -O

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-invsvc_jvm.conf

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command launched (pid: 26295), blocking for up to 3600 seconds...

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[0] : ln

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[1] : -f

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[2] : /usr/java/jre-vmware/bin/java

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command line[3] : /usr/java/jre-vmware/bin/vmware-invsvc

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command launched (pid: 26298), blocking for up to 15 seconds...

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command completed with exit code: 0

INFO  | wrapper  | 2017/09/25 20:39:14 | Event Command 'jvm_prelaunch': Command completed with exit code: 0  Continuing.

STATUS | wrapper  | 2017/09/25 20:39:14 | Launching a JVM...

INFO  | jvm 1    | 2017/09/25 20:39:14 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO  | jvm 1    | 2017/09/25 20:39:15 | WrapperManager: Initializing...

STATUS | wrapper  | 2017/09/25 20:39:59 | <-- Wrapper Stopped

vcenter:/var/log/vmware/invsvc # cat vmware-invsvc-gc.log.0

2017-09-25T20:39:17.314+0000: 2.794: [GC2017-09-25T20:39:17.330+0000: 2.810: [SoftReference, 0 refs, 0.0003020 secs]2017-09-25T20:39:17.330+0000: 2.810: [WeakReference, 1097 refs, 0.0002410 secs]2017-09-25T20:39:17.330+0000: 2.810: [FinalReference, 3318 refs, 0.0055650 secs]2017-09-25T20:39:17.336+0000: 2.816: [PhantomReference, 1 refs, 0.0001440 secs]2017-09-25T20:39:17.336+0000: 2.816: [JNI Weak Reference, 0.0000230 secs] [PSYoungGen: 132096K->11180K(153600K)] 132096K->11188K(503296K), 0.0236970 secs] [Times: user=0.04 sys=0.02, real=0.02 secs]

2017-09-25T20:39:18.798+0000: 4.278: [GC2017-09-25T20:39:18.815+0000: 4.295: [SoftReference, 0 refs, 0.0004490 secs]2017-09-25T20:39:18.815+0000: 4.295: [WeakReference, 1055 refs, 0.0003090 secs]2017-09-25T20:39:18.816+0000: 4.296: [FinalReference, 4585 refs, 0.0044610 secs]2017-09-25T20:39:18.820+0000: 4.300: [PhantomReference, 0 refs, 0.0001230 secs]2017-09-25T20:39:18.820+0000: 4.300: [JNI Weak Reference, 0.0000170 secs] [PSYoungGen: 143276K->12052K(153600K)] 143284K->12132K(503296K), 0.0246030 secs] [Times: user=0.05 sys=0.02, real=0.03 secs]

2017-09-25T20:39:20.235+0000: 5.715: [GC2017-09-25T20:39:20.259+0000: 5.739: [SoftReference, 0 refs, 0.0002660 secs]2017-09-25T20:39:20.259+0000: 5.739: [WeakReference, 1058 refs, 0.0002230 secs]2017-09-25T20:39:20.259+0000: 5.739: [FinalReference, 3120 refs, 0.0037950 secs]2017-09-25T20:39:20.263+0000: 5.743: [PhantomReference, 0 refs, 0.0002100 secs]2017-09-25T20:39:20.263+0000: 5.743: [JNI Weak Reference, 0.0000430 secs] [PSYoungGen: 144148K->17825K(153600K)] 144228K->17913K(503296K), 0.0303560 secs] [Times: user=0.07 sys=0.01, real=0.03 secs]

2017-09-25T20:39:21.371+0000: 6.851: [GC2017-09-25T20:39:21.426+0000: 6.906: [SoftReference, 0 refs, 0.0006670 secs]2017-09-25T20:39:21.427+0000: 6.907: [WeakReference, 1061 refs, 0.0004830 secs]2017-09-25T20:39:21.427+0000: 6.907: [FinalReference, 3052 refs, 0.0066890 secs]2017-09-25T20:39:21.434+0000: 6.914: [PhantomReference, 0 refs, 0.0001320 secs]2017-09-25T20:39:21.434+0000: 6.914: [JNI Weak Reference, 0.0000100 secs] [PSYoungGen: 149921K->21495K(285696K)] 150009K->30004K(635392K), 0.0658520 secs] [Times: user=0.16 sys=0.02, real=0.07 secs]

2017-09-25T20:39:23.395+0000: 8.875: [GC2017-09-25T20:39:23.498+0000: 8.978: [SoftReference, 0 refs, 0.0004150 secs]2017-09-25T20:39:23.499+0000: 8.979: [WeakReference, 0 refs, 0.0001070 secs]2017-09-25T20:39:23.499+0000: 8.979: [FinalReference, 0 refs, 0.0001230 secs]2017-09-25T20:39:23.499+0000: 8.979: [PhantomReference, 0 refs, 0.0001010 secs]2017-09-25T20:39:23.499+0000: 8.979: [JNI Weak Reference, 0.0000170 secs] [PSYoungGen: 285687K->21504K(285696K)] 294196K->53238K(635392K), 0.1084870 secs] [Times: user=0.21 sys=0.10, real=0.11 secs]

Heap

PSYoungGen      total 285696K, used 171249K [0x00000007b8800000, 0x00000007de700000, 0x0000000800000000)

  eden space 264192K, 56% used [0x00000007b8800000,0x00000007c1a3c790,0x00000007c8a00000)

  from space 21504K, 100% used [0x00000007c8a00000,0x00000007c9f00000,0x00000007c9f00000)

  to  space 46592K, 0% used [0x00000007db980000,0x00000007db980000,0x00000007de700000)

ParOldGen      total 349696K, used 31734K [0x0000000729800000, 0x000000073ed80000, 0x00000007b8800000)

  object space 349696K, 9% used [0x0000000729800000,0x000000072b6fdad8,0x000000073ed80000)

PSPermGen      total 58368K, used 58223K [0x0000000719800000, 0x000000071d100000, 0x0000000729800000)

  object space 58368K, 99% used [0x0000000719800000,0x000000071d0dbc08,0x000000071d100000)

error log from inv-svc.log indicated LDAP error.

; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.Lo                                                                              tusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException:                                                                                Invalid credentials

LDAP error [code: 49]

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1317)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1076)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)

        ... 65 more

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/secu                                                                              rity-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframe                                                                              work.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-confi                                                                              g.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vi                                                                              m.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.l                                                                              dap.InvalidCredentialsLdapException: Invalid credentials

LDAP error [code: 49]

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1317)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1076)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)

        ... 75 more

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/                                                                              config/authorization-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate                                                                                bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com                                                                              .vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

LDAP error [code: 49]

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:286)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:993)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:897)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:281)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)

        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)

        ... 85 more

Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitiali                                                                              zer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid cr                                                                              edentials

LDAP error [code: 49]

        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141)

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:105)

        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:278)

        ... 96 more

Caused by: java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

LDAP error [code: 49]

        at com.vmware.cis.lotus.LdapConnectionFactory.getConnection(LdapConnectionFactory.java:142)

        at com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer.<init>(LotusInitializer.java:26)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)

        at java.lang.reflect.Constructor.newInstance(Unknown Source)

        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)

        ... 98 more

I tried resetting the the password using /usr/lib/vmware-vmdir/bin/vdcadmintool

and that did not fix it.

any help or advise appreciated.

[daphnissov]

You may need to follow the steps outlined in KB 2119422.

[antonbizzari]

Thanks. Yes, I saw this article.

My concern is, this is a pretty complex setup. Will we lose all our settings in the environment? the vSwitches and connections to the backend storage if we do this?

[daphnissov]

The inventory service does not store anything having to do with host- or vCenter-level networking nor backend storage settings. It's used for things like tags and storage profiles. Open a case with GSS if you have specific concerns and need guidance.

[antonbizzari]

Thanks.

I just tried the script. I am getting

###

# Step 5: Getting the current vCenter Server Solution User Certificates

##

Error: Failed to open the store.

vecs-cli failed. Error 1021: Cannot connect to vmafd service.

FORCEFULLY ABORTING! Could not fetch vCenter Server Solution User private key.

[RajeevVCP4]

I assumed you have external PSC, try these step if your drive have enough space.

Shut down Vcenter

Shut down PSC

Power on in revers order

Power on psc

Power on VC.

[antonbizzari]

It does not look like we have one.

vcenter:~/bin # service-control --start vmware-eam

INFO:root:Service: vmware-eam, Action: start

Service: vmware-eam, Action: start

2017-09-26T14:29:20.115Z   Running command: ['/sbin/chkconfig', u'vmware-eam']

2017-09-26T14:29:21.073Z   Done running command

2017-09-26T14:29:21.074Z   Running command: ['/sbin/service', u'vmware-eam', 'status']

2017-09-26T14:29:21.396Z   Done running command

2017-09-26T14:29:21.397Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-eam', 'on']

2017-09-26T14:29:21.452Z   Done running command

2017-09-26T14:29:21.453Z   Running command: ['/sbin/service', u'vmware-eam', 'start']

2017-09-26T14:29:28.060Z   Done running command

2017-09-26T14:29:28.060Z   Invoked command: ['/sbin/service', u'vmware-eam', 'start']

2017-09-26T14:29:28.060Z   RC = 1

Stdout = Starting VMware ESX Agent Manager...

Waiting for VMware ESX Agent Manager........

WARNING: VMware ESX Agent Manager may have failed to start.

Last login: Tue Sep 26 14:23:24 UTC 2017

Stderr =

2017-09-26T14:29:28.061Z   {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-eam, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-eam"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-eam'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Unable to start service vmware-eam, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-eam"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-eam'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

[RajeevVCP4]

Logon that esxi where this virtual machine running, do the clean power off and power on.

Or  try

ESX Agent Manager solution user fails to log in after replacing the vCenter Server certificates in vSphere 6.0 (2112577)…

[antonbizzari]

Thank you. After working almost 16+ hours on this, I finally got the built in Apache Tomcat application to boot on vCenter 6.0

The link you have was helpful.

I fixed it by:

- fixing NTP to work properly. The previous admin seems to have never setup NTP

- reset the password

- removed a rouge entry from an xml file /usr/lib/vmware-invsvc/lib/server/config/ . it had the password in plain text just sitting on a line by itself. i managed to debug that by looking through logs.

- i also re-generated certificates and re-registered them.

After a clean shutdown and reboot, the server came back up normally.

Lesson learnt -- never hard crash vCenter appliance.

[miaojw]

Thank you.

By altering the password of Oracle, it works.