HMS services failed to start in vsphere replication appliance 8.2

[Akide]

recently i made a upgrade for vsphere replication appliance from 8.1 to 8.2, after upgrade i went to web console ip:5480 saws that

VRM service is

stopped

As the same configuration like before the VRM services still not start. And when i went to web interface ip/dr/#/home, there shows that HMS services can not access.

After that i check the HMS services by shell in appliance got these information attached below:

* hms.service - vSphere Replication Management Server

   Loaded: loaded (/lib/systemd/system/hms.service; enabled; vendor preset: enabled)

   Active: failed (Result: exit-code) since Sun 2019-06-09 23:06:35 UTC; 19s ago

  Process: 3873 ExecStart=/usr/java/default/bin/java $JAVA_FILE_ENCODING_OPTION $JAVA_METASPACE_OPTIONS $JAVA_GC_OPTIONS $JAV

  Process: 3869 ExecStartPre=/opt/vmware/hms/bin/checkHmsConfigured.sh (code=exited, status=0/SUCCESS)

  Process: 3787 ExecStartPre=/opt/vmware/hms/bin/checkHmsAppliance.sh (code=exited, status=0/SUCCESS)

Main PID: 3873 (code=exited, status=1/FAILURE)

Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.monitor.host.HostInventoryManagerImpl.lambda$0(HostInventoryManagerI

Jun 09 23:06:34 vsr hms[3873]:         at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)

Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.monitor.host.HostInventoryManagerImpl.getHostVersion(HostInventoryMa

Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.getHostDatastoreMounts(HostKeySe

Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.init(HostKeyServiceImpl.java:95)

Jun 09 23:06:34 vsr hms[3873]:         at com.vmware.hms.HmsService.startupProcedureStage1(HmsService.java:232)

Jun 09 23:06:34 vsr hms[3873]:         ... 9 more

Jun 09 23:06:35 vsr systemd[1]: hms.service: Main process exited, code=exited, status=1/FAILURE

Jun 09 23:06:35 vsr systemd[1]: hms.service: Unit entered failed state.

Jun 09 23:06:35 vsr systemd[1]: hms.service: Failed with result 'exit-code'.

Do anyone have idea to fix this problem?

Thanks

[daphnissov]

What do the logs say?

[Akide]

Could i know which log file is related to this services.

I am not sure the location of log file.

Thanks

[daphnissov]

I bet Google knows where those log files are.

[Akide]

Log details from /opt/vmware/hms/logs/hms.log

root@vsr [ ~ ]# cat /opt/vmware/hms/logs/hms.log | grep -v -e 'INFO' -e 'DEBUG' -e 'TRACE'

2019-06-10 08:50:31.574 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

2019-06-10 08:50:31.574 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

2019-06-10 08:50:31.575 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

2019-06-10 08:50:31.575 WARN  org.hibernate.orm.deprecation [main] (..hibernate.id.SequenceGenerator)  | HHH90000014: Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead.  See Hibernate Domain Model Mapping Guide for details.

2019-06-10 08:50:40.952 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-vapp-monitor-polling-interval'. Using the default.

2019-06-10 08:50:40.953 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-vapp-status-timeout'. Using the default.

2019-06-10 08:50:40.954 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-task-polling-admin-query'. Using the default.

2019-06-10 08:50:40.954 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.name'. Using the default.

2019-06-10 08:50:40.966 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.namespace'. Using the default.

2019-06-10 08:50:40.966 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.vcloud-namespace'. Using the default.

2019-06-10 08:50:40.966 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.view-right-name'. Using the default.

2019-06-10 08:50:40.967 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.manage-right-name'. Using the default.

2019-06-10 08:50:40.967 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.admin-right-name'. Using the default.

2019-06-10 08:50:40.968 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.view-right-vcd-form'. Using the default.

2019-06-10 08:50:40.968 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'service.manage-right-vcd-form'. Using the default.

2019-06-10 08:50:40.970 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-client-connect-timeout'. Using the default.

2019-06-10 08:50:40.970 WARN  com.vmware.hcs.gateway.vcd.VcdGatewayProperties [main] (..gateway.vcd.VcdGatewayProperties)  | Couldn't find a property 'vcd-client-read-timeout'. Using the default.

[]

2019-06-10 08:50:52.496 WARN  com.vmware.jvsl.sessions.net.VcVimServiceVersions [main] (..sessions.net.VcVimServiceVersions)  | Latest VC VMODL version '6.7.2' is not supported.

2019-06-10 08:50:52.750 ERROR com.vmware.hms.HmsService [main] (..vmware.hms.HmsService)  | stage 1 starting...FAILED

2019-06-10 08:50:52.754 FATAL hms [main] (..vmware.hms.App)  |

HMS SERVER ERROR

java.lang.RuntimeException: HMS Server failed to start successfully:

at com.vmware.hms.HmsService.startupProcedureStage1(HmsService.java:249)

at com.vmware.hms.HmsService.onRefresh(HmsService.java:447)

at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:880)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545)

at com.vmware.jvsl.util.SpringService.init(SpringService.java:152)

at com.vmware.jvsl.util.SpringService.init(SpringService.java:125)

at com.vmware.jvsl.util.SpringService.<init>(SpringService.java:61)

at com.vmware.hms.HmsSingletonSpringService.<init>(HmsSingletonSpringService.java:124)

at com.vmware.hms.HmsSingletonSpringService.init(HmsSingletonSpringService.java:99)

at com.vmware.hms.App.main(App.java:118)

Caused by: java.lang.NullPointerException

at com.vmware.hms.monitor.host.HostInventoryManagerImpl.loadHostVersion(HostInventoryManagerImpl.java:650)

at com.vmware.hms.monitor.host.HostInventoryManagerImpl.lambda$0(HostInventoryManagerImpl.java:630)

at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)

at com.vmware.hms.monitor.host.HostInventoryManagerImpl.getHostVersion(HostInventoryManagerImpl.java:630)

at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.getHostDatastoreMounts(HostKeyServiceImpl.java:141)

at com.vmware.hms.vmcrypt.hks.impl.HostKeyServiceImpl.init(HostKeyServiceImpl.java:95)

at com.vmware.hms.HmsService.startupProcedureStage1(HmsService.java:232)

... 9 more

[daphnissov]

It looks like a generic message. I'd try to first unregister and re-register the extension. Failing that, attach a new log file, don't paste a snippet.

[Akide]

Thanks for follow up, there is the log file of HMS service.

And i already tired to unregister VRMS and delete vSphere replication appliance and reinstall it.

This log took from a brand new installed vSphere replication appliance but problem is still occurred.

[sajithru]

I'm having the same issue. Have you found a solution?

[Akide]

No，i didn't find any useful information or solution. Through the test on my end, 8.1 version works properly but can not support vshpere 6.7 u2. In vmware docs, only 8.2 version support 6.7 u2, but it can not connect to vcenter properly. I quickly scanned logs, it's may a vmware software issue related to java give bad tracks back. 

[sajithru]

Yes, me too think it may be due to a bug from VMware's end. I have a successfully configured VR 8.2 instance on one site but when I try to deploy it on the 2nd site it fails. Tried several fresh installments with no luck. :smileysad:

[Akide]

Currently i got a solution may works for you, i change the VRM version to 8.1.2 or later which it's officially support VMware vSphere 6.7 Update 2.

Through the test on my end VRM 8.1.2 works properly.

release note attached below hope it helps:

VMware vSphere Replication 8.1.2 Release Notes

[Oleh Blyzniuk]

Hi sajithru

I'm having the same issue.

I have successfully upgraded VR 8.1.2 to 8.2 instance on one site, but when I try to start it after upgrade on the 2nd site it fails to start with "Warning: Bad service status: Service not running"

Have you found a solution?

[dirch201110141]

5 cents from me.
I also experienced problems with getting Replication server up and running, VRM service was stopped and was in no way shape to get running.

I got a "Bad exit code: 1". If i tried a new certificate i was told the certificate was wrong with nice red letters.

Log files told

- the hms service was not running. "hms not configured"

- route not found (or something like that)

My setup: homelab with 6.7u3 in domain. I did the official rename feature after initial install as I got stuck getting the vCenter servername to work correct in the first place.

After reinstalling 8.2 several times and trying 8.1.2 also and looking through 10 miles of logs, the following KB led me to check yet another file

VMware Knowledge Base (KB 2106709)

This points to checking the following:

/opt/vmware/etc/vami/ovfEnv.xml file. The file should contain the following properties:

evs:GuestApi/evs:URL
evs:GuestApi/evs:Token
evs:GuestApi/evs:X509Thumbprint

evs:VCenterApi/evs:IP
evs:VCenterApi/evs:X509Thumbprint
evs:VCenterApi/evs:HttpPort
evs:VCenterApi/evs:HttpsPort
evs:VCenterApi/evs:SelfMoRef
evs:VCenterApi/evs:Address

Looking into the xml-file there is valuable background info on the Replication setup and at least I found that VR was expecting to connect using the ancient vCenter name "photon-machine". The lazy fix was to create a new DNS-record, the right fix ... well, new install some other day.

My issue was probably not the same as OP, but the KB may be useful anyways.