DX Application Performance Management

 View Only
Expand all | Collapse all

Caused by: java.net.ConnectException: Connection refused

  • 1.  Caused by: java.net.ConnectException: Connection refused

    Posted Oct 21, 2016 11:04 AM

    We are seeing below SEVERE Message on Daily basis on our four node Forked Molecule(Cluster of JVM's).

    We are using CA APM to monitor the Atom JVM's (where the CA APM Agent is Installed on each node and added properties of CA APM in atom.vmoptions(JVM Properties file where we add CA APM configuration Properties).

     

    Could you please brief us what is causing this error and Is there any timeout value which is causing to throw Connection Refused to Host

     

    <Month Date>, <Year> <Time> <Time Zone> SEVERE [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff
    java.rmi.ConnectException: Connection refused to host: <IP Addresss> of Head Node; nested exception is:
    java.net.ConnectException: Connection refused

     

    Stack Trace
    <Month Date>, <Year> <Time> <Time Zone> SEVERE [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff
    java.rmi.ConnectException: Connection refused to host: <IP Addresss> of Head Node; nested exception is:
    java.net.ConnectException: Connection refused
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
    at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
    at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:906)
    at com.sun.jmx.mbeanserver.MXBeanProxy$GetHandler.invoke(MXBeanProxy.java:122)
    at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)
    at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)
    at com.sun.proxy.$Proxy50.getId(Unknown Source)
    at com.boomi.execution.forker.SunForkedProcessHandler$3.executeAction(SunForkedProcessHandler.java:372)
    at com.boomi.execution.forker.SunForkedProcessHandler$3.executeAction(SunForkedProcessHandler.java:368)
    at com.boomi.execution.forker.SunForkedProcessHandler$ForkedProcRetryable.execute(SunForkedProcessHandler.java:572)
    at com.boomi.util.retry.Retryable.call(Retryable.java:92)
    at com.boomi.execution.forker.SunForkedProcessHandler.executeRunnerAction(SunForkedProcessHandler.java:389)
    at com.boomi.execution.forker.SunForkedProcessHandler.interrogateVm(SunForkedProcessHandler.java:368)
    at com.boomi.execution.forker.SunForkedProcessHandler.getKnownRunners(SunForkedProcessHandler.java:296)
    at com.boomi.execution.forker.SunForkedProcessHandler.getExecution(SunForkedProcessHandler.java:263)
    at com.boomi.execution.forker.SunForkedProcessHandler.isExecuting(SunForkedProcessHandler.java:148)
    at com.boomi.execution.forker.SunForkedProcessHandler.isExecuting(SunForkedProcessHandler.java:142)
    at com.boomi.execution.forker.MultiProcessExecutionForker.reapAbortedProcesses(MultiProcessExecutionForker.java:770)
    at com.boomi.execution.forker.MultiProcessExecutionForker.isLocalIdle(MultiProcessExecutionForker.java:668)
    at com.boomi.container.cloudlet.execution.ClusterExecutionForker.isLocalIdle(ClusterExecutionForker.java:489)
    at com.boomi.execution.ExecutionManager.isLocalIdle(ExecutionManager.java:736)
    at sun.reflect.GeneratedMethodAccessor458.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
    at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
    at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
    at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
    at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
    at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
    at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
    at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
    at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
    at javax.management.StandardMBean.getAttribute(StandardMBean.java:372)
    at com.boomi.util.management.CustomMBean.getAttribute(CustomMBean.java:138)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
    at com.wily.introscope.agent.jmx.AJMXDataView.IJMXDataView_update(AJMXDataView.java:115)
    at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.pollViews(IntroscopeProtocolAdaptor.java:659)
    at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.findServersAndPollViews(IntroscopeProtocolAdaptor.java:549)
    at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.setContextAndPoll(IntroscopeProtocolAdaptor.java:434)
    at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor.access$8(IntroscopeProtocolAdaptor.java:419)
    at com.wily.introscope.agent.jmx.IntroscopeProtocolAdaptor$IntroscopeProtocolAdaptorHeartbeatBehavior.ITimestampedRunnable_execute(IntroscopeProtocolAdaptor.java:1410)
    at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)
    at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)
    at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)
    at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
    at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
    ... 54 more



  • 2.  Re: Caused by: java.net.ConnectException: Connection refused
    Best Answer

    Broadcom Employee
    Posted Oct 21, 2016 03:49 PM

    It has to do more with connectivity than a timeout value.  Basically this excpetion means that there is either no service listening on the port or something is blocking the traffic.  Are there any firewalls enabled and blocking the traffic?  Is the port set up correctly?  Are you using any JMX authentication method?



  • 3.  Re: Caused by: java.net.ConnectException: Connection refused

    Broadcom Employee
    Posted Oct 24, 2016 07:51 AM

    Hi Venkata:

         Matt provided the likely cause of this exception. There were no follow-up questions. So I am marking as closed. You are free to post follow-up information and responses if you wish. Hoping that the answer help you resolve the issue.

    Thanks

    Hal German



  • 4.  Re: Caused by: java.net.ConnectException: Connection refused

    Posted Oct 24, 2016 01:39 PM

    we have a four node cluster of JVM's ..For all Four Nodes (5 to 6 times only we are seeing this connection refused errors.

     

    If the port is not listening at all or wrong firewall rules the issue should exist all the time in a day, but we are seeing issue 04 to 05 times a day with different timings (no specific pattern of time).

     

    When CA APM lost communication  does it post any errors in APM log why it is not able to access? 



  • 5.  Re: Caused by: java.net.ConnectException: Connection refused

    Broadcom Employee
    Posted Oct 24, 2016 02:30 PM

    By default it will log a lost connection, but to see what the reason is, set the logging to debug on the agent and collectors.

     

    If its that intermittent your best bet is to open a support case.



  • 6.  Re: Caused by: java.net.ConnectException: Connection refused

    Broadcom Employee
    Posted Oct 24, 2016 05:21 PM

    The intermittent nature of the problem may point to Linux/Unix file descriptors being occasionally exhausted for the  jvm/java process and "ulimit -n" may need to be increased for the java process owner id to allow for the extra requirements of the agent.  However as suggested by Matt musma03 a related error should be visible in the agent debug log e.g. "java.io.IOException: Too many open files"



  • 7.  Re: Caused by: java.net.ConnectException: Connection refused

    Posted Oct 24, 2016 06:30 PM

    I have Just checked from APM for  past 7 days OpenFileDescriptorCount on all four nodes. No host crossed 1200 open files also. we have a limit of 6000

     

    JMX - java.lang - type= OperatingSystem - OpenFileDescriptorCount

    First Node:

     

    JMX|
         java.lang|
          type=OperatingSystem:
           OpenFileDescriptorCount
    Value = 1,138.00
    Min = 292.00
    Max = 1,138.00
    Count = 672

    2016 Oct 19 4:00:00 PM

     

    Second Node:

     

    JMX|
         java.lang|
          type=OperatingSystem:
           OpenFileDescriptorCount
    Value = 261.00
    Min = 244.00
    Max = 261.00
    Count = 672

    2016 Oct 19 6:48:00 PM

     

     

    Third Node:

     

    JMX|
         java.lang|
          type=OperatingSystem:
           OpenFileDescriptorCount
    Value = 264.00
    Min = 252.00
    Max = 264.00
    Count = 672

    2016 Oct 19 7:36:00 AM

     

    Fourth Node:

    Molecule_Prod_EMB_Molecule|
        JMX|
         java.lang|
          type=OperatingSystem:
           OpenFileDescriptorCount
    Value = 273.00
    Min = 262.00
    Max = 273.00
    Count = 672

    2016 Oct 21 1:36:00 AM

     

     

    Oct 23, 2016 7:01:26 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host:  (Third Node)
    Oct 23, 2016 8:00:51 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host: (Second Node)
    Oct 23, 2016 9:21:54 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host: (Second Node)
    Oct 23, 2016 10:00:51 PM EDT SEVERE  [com.boomi.util.retry.Retryable call] ForkedProcRetryable failed, will retry after backoff: java.rmi.ConnectException: Connection refused to host: (Second Node)



  • 8.  Re: Caused by: java.net.ConnectException: Connection refused

    Broadcom Employee
    Posted Oct 24, 2016 06:53 PM

    It may have been asked already, but do you have any loadbalancing rules in place?

    Any other unusual messages in the agent or application logs?



  • 9.  Re: Caused by: java.net.ConnectException: Connection refused

    Broadcom Employee
    Posted Oct 24, 2016 07:04 PM

    If Hiko's suggesttion does not help & you are still blocked please raise a new Support Case so an APM Support Engineer can review the logs in detail: SAP Portal Services 

    Thank-you

    Lynn



  • 10.  Re: Caused by: java.net.ConnectException: Connection refused

    Posted Oct 26, 2016 11:29 AM

    Same issue I have posted to our product vendor Dell boomi, I got the below update

     

    https://community.boomi.com/message/11089

     

    most likely exacerbated by your use of the "isLocalIdle" method for monitoring.  As i mentioned in my previous comment, i wouldn't recommend this method for monitoring.  If you can tell me why you are calling it, maybe there is a better metric to monitor instead.

     

     

     



  • 11.  Re: Caused by: java.net.ConnectException: Connection refused

    Broadcom Employee
    Posted Oct 26, 2016 05:56 PM

    OOTB the agent should not be explicitly monitoring method com.boomi.execution.ExecutionManager.isLocalIdle aso that must be indirect.

    You could try to disable socket tracing to see if that helps i.e. comment the line "TurnOn: SocketTracing" in file toggles-typical.pbd or toggles-full.pbd whichever is being used. If no help please raise a support case for more detailed advice as suggested above.

    Hiko_Davis for other suggestions.

     

    Thanks

    Lynn