ESX Agent Manager - vmware-eam will not start

[Christopher Dooks]

First I noticed /etc/vmware-eam/eam.properties was empty, so I poulated it with the correct entries using another working vCSA (changed URLs and CM configuration URL with the correct hostid, but the service still does not start.

I see the following output if I try and start EAM via shell.

I have a VMware ticket open but they are being really slow at getting back to me.

Is it possible to remove the extension and add it again? I suspect that due to the disk running out of space a few weeks ago some files may have become corrupt.

service-control --start vmware-eam

INFO:root:Service: vmware-eam, Action: start

Service: vmware-eam, Action: start

2017-12-20T10:38:45.407Z   Running command: ['/sbin/chkconfig', u'vmware-eam']

2017-12-20T10:38:45.442Z   Done running command

2017-12-20T10:38:45.442Z   Running command: ['/sbin/service', u'vmware-eam', 'status']

2017-12-20T10:38:45.489Z   Done running command

2017-12-20T10:38:45.489Z   Running command: ['/sbin/chkconfig', '--force', u'vmware-eam', 'on']

2017-12-20T10:38:45.524Z   Done running command

2017-12-20T10:38:45.524Z   Running command: ['/sbin/service', u'vmware-eam', 'start']

2017-12-20T10:38:49.191Z   Done running command

2017-12-20T10:38:49.192Z   Invoked command: ['/sbin/service', u'vmware-eam', 'start']

2017-12-20T10:38:49.192Z   RC = 1

Stdout = Starting VMware ESX Agent Manager...

Waiting for VMware ESX Agent Manager......

WARNING: VMware ESX Agent Manager may have failed to start.

Last login: Wed Dec 20 10:31:58 UTC 2017 on pts/1

Stderr =

2017-12-20T10:38:49.192Z   {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-eam, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-eam"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-eam'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Unable to start service vmware-eam, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-eam"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-eam'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

This is the content of /storage/log/vmware/eam/wrapper.log when I try and start it.

I have looked at the eam-server-beans.xml file and it is identical to a server that is working.

STATUS | wrapper  | 2017/12/20 11:26:30 | --> Wrapper Started as Daemon

STATUS | wrapper  | 2017/12/20 11:26:30 | Java Service Wrapper Professional Edition 64-bit 3.5.24

STATUS | wrapper  | 2017/12/20 11:26:30 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.

STATUS | wrapper  | 2017/12/20 11:26:30 |     http://wrapper.tanukisoftware.com

STATUS | wrapper  | 2017/12/20 11:26:30 |   Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager

STATUS | wrapper  | 2017/12/20 11:26:30 |

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command line[1] : -J

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command line[2] : vmware-eam

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command line[3] : -O

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command launched (pid: 12435), blocking for up to 3600 seconds...

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command completed with exit code: 0

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command line[2] :

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command line[3] :

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command line[4] :

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command launched (pid: 12442), blocking for up to 15 seconds...

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command completed with exit code: 0

INFO   | wrapper  | 2017/12/20 11:26:30 | Event Command 'jvm_prelaunch': Command completed with exit code: 0  Continuing.

STATUS | wrapper  | 2017/12/20 11:26:30 | Launching a JVM...

INFO   | jvm 1    | 2017/12/20 11:26:30 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M

INFO   | jvm 1    | 2017/12/20 11:26:30 | WrapperManager: Initializing...

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp:

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error: Encountered an error running main:

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'vcInventory' defined in class path resource [eam-server-beans.xml]: Could not resolve placeholder 'drs.demandCapacityRatio'

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:554)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:528)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at com.vmware.eam.util.Main.main(Main.java:50)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at java.lang.reflect.Method.invoke(Unknown Source)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)

INFO   | jvm 1    | 2017/12/20 11:26:31 | WrapperSimpleApp Error:       at java.lang.Thread.run(Unknown Source)

STATUS | wrapper  | 2017/12/20 11:26:33 | <-- Wrapper Stopped

[Christopher Dooks]

I have managed to fix it.

Somehow I had missed drs.demandCapacityRatio=100 off the tail of eam.properties.

Service now started.

[Victorious08]

Hi,

im have trouble same with this case. the eam.properties is blank.

what should i do?

[Christopher Dooks]

You need to copy the contents of the file from a vCSA server which has contents in the file, but edit the URL and CM contents so they reflect the correct vlaues.

[Victorious08]

Hallo,

can you help me to send me email/upload. because my VCSA server is corrupt.

Many Thanks

----

Stdout = Starting VMware ESX Agent Manager...

Waiting for VMware ESX Agent Manager.......

WARNING: VMware ESX Agent Manager may have failed to start.

Stderr =

2018-01-12T10:02:07.579Z   {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-eam, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-eam"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-eam'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Unable to start service vmware-eam, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-eam"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-eam'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

localhost:~ #

[Andrew_Judge]

So you come to this forum asking for help, then leave people in the dark? Pay it forward man 'cmon.

For those of you wondering what he's writing, there's a file called eam.properties on the vcenter appliance. It's located in the /etc/vmware-eam/ folder. Its contents configure the startup parameters for eam when startup occurs

# service-control --start vmware-eam

It looks like this (don't just copy this - notice his error had specific notation to drs...)

#########################################################################

# Copyright 2013-2017 VMware, Inc.  All rights reserved. VMware Confidential

#########################################################################

vc.proxy.host=localhost

vc.proxy.port=80

# Hostname or IP of the EAM server

# Fill only if EAM is not running on the same host as VC

eam.host=

# EAM service port used to configure the HTTP connector of the application server.

eam.int.http.port=15005

# Port and scheme configuration which is used by the ESX 6.x hosts to reach EAM Vib

# file server.

eam.ext.port=443

eam.ext.scheme=https

# Port and scheme configuration which is used by the ESX 5.x hosts to reach EAM Vib

# file server.

eam.ext.port.deprecated=80

eam.ext.scheme.deprecated=http

eam.support_linked_clone=true

eam.clear_db_on_startup=false

eam.debug_ref_count=false

eam.recent_event_size=20

# Value is specified in minutes (set to 24h = 1440m)

eam.scan_for_unknown_agent_vms=1440

# The timeout to wait for hostd to restart on a host (set to 5m=300s)

eam.hostd_restart_timeout=300

# The following entries will be added verbatim to the advanced options

# of hosts on which EAM is enabled. All are optional.

Net.DVFilterBindIpAddress=169.254.0.1

Net.TrafficFilterIpAddress=

#The IP for the VSWIF NIC on the dvFilter switch (for ESX classic).

Net.DVFilterVswifIpAddress=169.254.0.2

# Resource bundle configuration

eam.resourcebundle.filename=eam-resourcebundle.jar

# VLSI embedded tcServer configuration

#

tcserver.tmp.dir=/var/tmp/vmware/eam/tomcat

eam.web.root=/usr/lib/vmware-eam/web

# EAM SSL configuration

#

eam.keystore.type=VKS

eam.key.alias=vpxd-extension

eam.keystore.storename=vpxd-extension

# CM configuration

#

cm.url=http://localhost:18090/cm/sdk/?hostid=16a75f54-16e1-4e97-a211-8f4762f8041c

cm.wait.attempts=360

cm.wait.intervalSeconds=5

# SSO configuration

#

sso.wait.attempts=360

sso.wait.intervalSeconds=5

# VC SSL configuration

#

vc.truststore.type=VKS

vc.truststore.storename=TRUSTED_ROOTS

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService

vc.tunnelSdkUri=https://YOUR.VC.FQDN:8089/sdk/vimService

drs.demandCapacityRatio=100

##### END FILE

you can

# tail -f /var/log/vmware/eam/eam.log

in a separate session while starting the service through

# service-control --start vmware-eam

Watch out for cert errors - that can cause issues as well. I still haven't got mine going, but hopefully this helps someone. I can't upgrade a vSphere environment because of this and I have to...

[Victorious08]

Hello,

there is the tail-f of log.

2018-01-29T03:01:28.361Z |  INFO | WrapperSimpleAppMain | Main.java | 47 | Initializing Spring from 'standalone.xml' and 'eam.properties'

2018-01-29T03:01:28.392Z |  INFO | WrapperSimpleAppMain | AbstractApplicationContext.java | 412 | Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@37f0c4d2: display name [org.springframework.context.support.ClassPathXmlApplicationContext@37f0c4d2]; startup date [Mon Jan 29 03:01:28 UTC 2018]; root of context hierarchy

2018-01-29T03:01:28.574Z |  INFO | WrapperSimpleAppMain | XmlBeanDefinitionReader.java | 323 | Loading XML bean definitions from class path resource [standalone.xml]

2018-01-29T03:01:28.717Z |  INFO | WrapperSimpleAppMain | XmlBeanDefinitionReader.java | 323 | Loading XML bean definitions from class path resource [eam-server-beans.xml]

2018-01-29T03:01:28.814Z |  INFO | WrapperSimpleAppMain | AbstractApplicationContext.java | 427 | Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@37f0c4d2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@2b27f9cb

2018-01-29T03:01:28.947Z |  INFO | WrapperSimpleAppMain | PropertiesLoaderSupport.java | 178 | Loading properties file from class path resource [eam.properties]

2018-01-29T03:01:28.952Z |  INFO | WrapperSimpleAppMain | DefaultSingletonBeanRegistry.java | 396 | Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2b27f9cb: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,commonMapStore,vibServices,esxAgentMgr,vcTrustStore,eamKeyStore,eamKeyAccess,vimAndVumConnectionExecutor,vpxdConnection,vumConnection,vcOptions,serverAddr,serverAddrDepr,serverUrl,serverUrlDepr,vcListener,l10n,httpNfcLeaseTimeoutHandler,nameManager,vcDatastoreManager,vcEventManager,vcKeyValueStore,messageSource,vibServer,vibServerDepr,vibPackageDepot,vibPackageDepotDepr,secureHostsStartVersion,vibPackageDepots,vcInventory,springContext,scanForUnknownAgentVms,cmClient,stsCertManager,scheduler,heathStatusTaskScheduler,vmodlContext,versionFinder,clientAuthConnectionExecutor,clientAuthConnection,clientAuthenticator,localizationFilter,firstMobStep,mobConnectionExecutor,mobConnection,mobAuthenticator,localClient,localCorrelator,localFilterStartStep,localCompletionStep,sessionManager,dispatcher,adapterServer,eamVersion,bodyHandler,eamVersion2,bodyHandlerVersion2,eamVersion2_5,bodyHandlerVersion2_5,eamVersion3,bodyHandlerVersion3,eamVersion6,bodyHandlerVersion6,internalEamVersion1,internalBodyHandlerVersion1,internalEamVersion2,internalBodyHandlerVersion2,internalEamVersion2_5,internalBodyHandlerVersion2_5,internalEamVersion3,internalBodyHandlerVersion3,internalEamVersion6,internalBodyHandlerVersion6,httpResponseCorrelator,firstStep,validationStartStep,completionStep,filterStartStep,invocationStep,continuerStep,rejectedHandler,executor,httpServerEndpoint,listener,mob,jspServlet,fileServlet,health,managementUi,cmHealthServlet,cmResourceBundleServlet,vmomiService,mobService,jspService,filesService,healthService,managementService,vibService,vibdService,cmHealthService,cmResourceBundleService,httpServer]; root of factory hierarchy

the services wont up.

[Christopher Dooks]

What's the contents of your /etc/vmware-eam/eam.properties file? From your 'ls' it looks small. I can't share my file details with you as it looks quite different in 6.5, I presume EAM has changed somewhat.

Use what Andrew Judge posted and change it to match your environment. I *think* the only lines you need to change are:

cm.url=http://localhost:18090/cm/sdk/?hostid=16a75f54-16e1-4e97-a211-8f4762f8041c

vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService

vc.tunnelSdkUri=https://YOUR.VC.FQDN:8089/sdk/vimService

Obviously the cm.url value should match your vCSA server. I can't quite remember where I got my value from.

Do you have more than one vCSA server? What is your setup? Do you have backups?

[munishpalmakhij]

Hey Andrew,

We also ran into the same issue. How do you obtain the hostid for cm.url ?

I tried the uuid of the VC VM however I am still unable to start the vmware-eam service

Regards

-Munish

[Christopher Dooks]

Do you have a vCSA where thee am service starts?

[dertz]

Munish:

Ever get a response on the hostid?  Trying to rebuild file as mine got clobbered somehow and just need this piece of info.

Thanks,

Doug

[dertz]

figure out how to get the hostid

run command:   cat /etc/vmware/install-defaults/sca.hostid

THis will get you hostid.  There is a  good article on this issue at VMware Knowledge Base

[Blagovest]

Hi guys, we have the same issue and I need also the content of eam.properties as ours is blank.

Thank you in advance

[Christopher Dooks]

I am on holiday and don’t have access to my lab or work hence I haven’t answered. I hope that’s okay?