Replication Server Error:127.0.0.1:8123

[rbglhec]

When I install the replication server through SRM things work fine until I reboot the appliance. After that I get the error TCP connection to server at 127.0.0.1:8123 failed.

The detail is:

VRM Server generic error. Please check the documentation for any troubleshooting information. The detailed exception is: 'org.apache.http.conn.HttpHostConnectException: Connection to https://127.0.0.1:8123 refused'.

All the settings in the web UI look correct, networking and everything else works great until the appliance is rebooted. I've saved and verified all the configs. I can even login to the appliance as root and ping the vcenter. I read somewhere that adding the vcenter to the hosts file helps but that didn't help either.

Any ideas?

[mvalkanov]

Hi,

Seems like the hbrsrv process is not running. You can check with /etc/init.d/hbrsrv status.

Please take a look at hbrsrv logs at /var/log/vmware.

Regards,

Martin

[rbglhec]

Service is running. I see some connection failures on port 80 in the log to some of  our ESX hosts. Is that the problem?

[mvalkanov]

Hi,

Can you take a look at /opt/vmware/hms/logs/hms.log for the HttpHostConnectException ?

It is VRMS (hms process) that failed to connect to VR server (hbrsrv process).

I might want to file an SR and attach logs from /opt/vmware/hms/logs/ and also support bundle for VR server (run hbrsrv-support-bundle.sh and copy file from /tmp/hbrsrv-....tgz).

Regards,

Martin

[rbglhec]

Its inidicating ping failure for 127.0.0.1:8123 and getting an org.apache.http.conn.httphostconnectexecption to it's local https://<vmipaddress>:8123 refused. Is this an cert issue?

[mvalkanov]

Can you verify that hbrsrv listens on port 8123:

netstat --numeric-hosts --numeric-ports --listen

and look for 8123.

Seems like the hbrsrv process is stuck in some strange state.

The hbrsrv support bundle will include live core of the process, so that team inside VMware can further investigate.

[rbglhec]

I don't see any 8123 listening ports. I even tried to telnet to 127.0.0.1 8123 and it was refused. This shouldn't be this difficult.

[rbglhec]

Ok found the problem. This is not in the documentation or KB. Our vCenter server did not have a managed IP setting. As soon as we set the managed IP for the vCenter no 8123 errors and things worked perfect.

Thanks for the help!

[rbglhec]

Ok, so this error pops up randomly and I've tried everything to correct it. If you wait a few hours and restart the service it works. This seems like alpha software.

Error:

VRM Server generic error: exception org.apatche.http.conn.httphostconnectexception: connection to https://127.0.0.1:8123 refused.

What I see on the appliance is that sometimes the appliance successfully listens on port 8123. More often it's not listening on port 8123. What service runs on port 8123?

[mvalkanov]

Hi,

As already commented above:

VR server (hbrsrv process) listens on that port.

Please file a support request and attach logs from /opt/vmware/hms/logs/ and also support bundle for VR server (run hbrsrv-support-bundle.sh and copy file from /tmp/hbrsrv-....tgz).

The hbrsrv support bundle will include live core of the process, so that team inside VMware can further investigate.

Regards,

Martin

[blabarbera]

Can you find any evidence of the connection succeeding at some point? For instance, does it fail 4 or 5 times and then succeed, and only reappear during a restart? If so, check your host firewalls and make sure that port 80 is open and accepts traffic from the replication appliance/server. We had an issue where port 80 was blocked and it was causing a delay in the HBR http service starting up - thus the connection would fail (connection refused) a few times and then eventually succeed once the services started. We would only have the problem during startup. Double check other necessary ports on the host as well.

There is also an internal VMware KB article that apparently deals with an SSL issue. Apparently updating the SSL certificate on the appliance can cause a similar problem. Steps for resolution are as follows:

Open an SSH session to the VRMS server and run these commands:

1. Change directory to /opt/vmware/etc/lighttpd using this command:

    cd /opt/vmware/etc/lighttpd

2. Run this command to obtain the new SSL thumbprint:

    openssl x509 -in server.pem -noout -fingerprint

3. Manually update the HMS thumbprint in the HBR database using the command:

hbrsrv-guestinfo.sh set guestinfo.hbr.hms-thumbprint 'new thumbprint value'

4. Restart HBR server process using the command:

/etc/init.d/hbrsrv restart

5. Review the HMS logs to verify the errors cease

6. You may also need to restart the HMS server process on the ESXi hosts (/etc/init.d/hms restart) to update the new HBR SSL thumbprint.

Anyway - hope that helps someone as I could find very little else on the problem. Make sure you follow VMware SSL guidelines when creating the certs as that can cause a plethora of issues as well.

[lspfilho]

Hi,

I'm having the same problem mentioned in the forum. I performed the above procedure, but without success. The environment there is already a Replication Server, but I need to create another Replication to replicate data through another network interface. Anyone have any idea?

Tks

[AndrewKV]

Hi,

the same issue in my case. Any updates how to fix it?

I do confirm that hbrsrv is running and listens on the port 8123 (I even can telnet to that port from network neighborhoods).

Thanks!

BTW: incidentally found that it is possible to scroll the error, below the full content of the error:

========

Cannot establish a TCP connection to server at '127.0.0.1:8123'.

Cause:

A generic error occurred in the vSphere Replication Management Server. Exception details: 'java.net.ConnectException: Connection refused'.

========

[AndrewKV]

...so after several tries I finally was able to deploy Replication Sever without any errors showed in Tasks (at least).

But I got strange issue: looks like Replication Sever was not registered / deployed properly. The icon for replication server didn’t appear in the Inventories. I can see available extension VR Management in Admnistration -> Solutions -> vCenter Server Extensions, but opening it and going to the tab "Solution" ends with error “Error #1009”

Error Stack
 ---------------------
 TypeError: Error #1009
 at com.google.code.flexiframe::IFrame/hideDiv()
 at com.google.code.flexiframe::IFrame/updateFrameVisibility()
 at com.google.code.flexiframe::IFrame/set visible()
 at com.vmware.vsphere.client.views.vcextension.legacy::VcExtensionLegacyTabsViewMediator/updateView()
 at com.vmware.vsphere.client.views.vcextension.legacy::VcExtensionLegacyTabsViewMediator/onTabInfoDataRetrived()
 at EventFunctor/notifyTarget()
 at EventFunctor/onEvent()
 at flash.events::EventDispatcher/dispatchEvent()
 at com.vmware.frinje::EventBus/dispatchEvent()
 at com.vmware.frinje::EventBus/onEvent()
 at flash.events::EventDispatcher/dispatchEvent()
 at com.vmware.data.query.commands::DataRequestCommand/onSingleEntityDataReturn()
 at DataAccessItem/notify()
 at DataAccessItem/handleResult()
 at DataAccessItem/processResult()
 at com.vmware.data.query::DataAccessManager/onDataRetrieved()
 at com.vmware.flexutil.proxies::BaseProxy/notify()
 at com.vmware.flexutil.proxies::BaseProxy/result()
 at com.vmware.flexutil.proxies::BaseProxy/onInvocationComplete()
 at OperationInvoker/resultResponseForRequest()
 at OperationInvoker/result()
 at mx.rpc::AsyncToken/http://www.adobe.com/2006/flex/mx/internal::applyResult()
 at mx.rpc.events::ResultEvent/http://www.adobe.com/2006/flex/mx/internal::callTokenResponders()
 at mx.rpc::AbstractOperation/http://www.adobe.com/2006/flex/mx/internal::dispatchRpcEvent()
 at mx.rpc::AbstractInvoker/http://www.adobe.com/2006/flex/mx/internal::resultHandler()
 at mx.rpc::Responder/result()
 at mx.rpc::AsyncRequest/acknowledge()
 at NetConnectionMessageResponder/resultHandler()
 at mx.messaging::MessageResponder/result()

Any ideas what to check? Thanks in advance!