DX Infrastructure Manager

Expand all | Collapse all

Hub Tunnel Disconnects - Error code 110

Jump to Best Answer
  • 1.  Hub Tunnel Disconnects - Error code 110

    Posted 02-23-2017 06:45 PM

    Hi,

     

    Every now and again, our support team receives a flood of Nimsoft Tunnel Disconnect tickets which we thought were caused by a particular client that was flooding the tunnel connection with data, therefore, causing it to drop out / restart.  We thought this because the tunnel disconnects were happening on one particular hub.

     

    Here recently, we have been getting Hub Tunnel Disconnect tickets from all over (multiple hubs).  I just happened to catch the log file in time before they rolled off (log level 3), and found the following before the Hub Tunnel restarted (sensitive data removed):

     

    Feb 23 13:24:43:879 [47140332755264] hub: TSESS-P-823-0 sent error message response
    Feb 23 13:24:45:469 [47141801335104] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
    Feb 23 13:24:45:469 [47141801335104] hub: TSESS-P-724-8 [] could not connect locally to /Gladiator/Hub09/nimhub09/hub (x.x.x.x/48002)
    Feb 23 13:24:45:470 [47141801335104] hub: TSESS-P-724-8 sent error message response
    Feb 23 13:24:45:927 [47138969073984] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
    Feb 23 13:24:45:927 [47138969073984] hub: TSESS-P-694-10 [] could not connect locally to /Gladiator/Hub09/nimhub09/hub (x.x.x.x/48002)
    Feb 23 13:24:45:927 [47138969073984] hub: TSESS-P-694-10 sent error message response
    Feb 23 13:24:46:635 [47138011362176] hub: EXIT HUB: hub shutdown
    Feb 23 13:24:46:636 [47138011362176] hub: Waiting for tunnel to terminate ...
    Feb 23 13:24:46:882 [47140101978432] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
    Feb 23 13:24:46:882 [47140101978432] hub: TSESS-P-773-3 [] could not connect locally to /Gladiator/Hub09/nimhub09/hub (x.x.x.x/48002)
    Feb 23 13:24:46:882 [47140101978432] hub: TSESS-P-773-3 sent error message response
    Feb 23 13:24:47:498 [47139955120448] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
    Feb 23 13:24:47:498 [47139955120448] hub: TSESS-P-762-4 sent error message response
    Feb 23 13:24:49:751 [47929251947392] hub: ----------------------------------------------------------------------------------------------------
    Feb 23 13:24:49:751 [47929251947392] hub: ----- Hub 7.80-jh2 [Build 7.80, Nov 9 2015] initializing -----
    Feb 23 13:24:49:752 [47929251947392] hub: Tunnel has started in thread [ 47929318897984 ]
    Feb 23 13:24:49:788 [47929251947392] hub: ----- HUB started -----

     

    As you can see, error code 110 was very prevalent before the hub tunnel dropped out.  Any info to the meaning of error 110 would be greatly appreciated.

     

    Additional info:

    -We have 14 hub servers that are linux boxes (centos)

    -Tunnel Encryption set to Medium

    -Hub tunnel SSL Settings Mode set to Normal

    -UIM ver 7.5 ---- Will be upgrading this coming March to 8.47

    -Hub version 7.80

     

    Thanks,

    Chris A.



  • 2.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 02-23-2017 06:52 PM

    An error code 110 is a connection refused error.

    Please make sure the following
    1) firewall is disabled

    2) no AV scanning/blocking



  • 3.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 02-23-2017 07:17 PM

    Chris,

     

    Furthermore, for this scenario and any other scenario where you need to save off/dump specific log files when an alarm occurs, please refer to this Article I recently updated and tested:

     

    How to save off (dump) log files when an alarm occurs

     

    https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec000003963.html 

     

    So you can catch it red-handed. In your situation I would set the loglevel to 5 and logsize to 40000 for the hub, controller and data_engine to get a better idea of what's happening. Use a regex for the tunnel disconnect alarm - its explained in the KB Article.

     

    This Article contains a proven method of capturing essential log files used for analyzing customer problems that are very difficult to catch when they occur. This method includes a nas LUA script to automatically save off (dump) specified log files when an alarm occurs (based on an AO profile that filters on the given alarm message or any other filter you choose to define in the AO profile.) This script may be edited to suit your particular needs for capturing the appropriate log files when a problem occurs.

     

    It is especially useful in situations where the problem is infrequent or inconsistent, occurs randomly or it exhibits a pattern of behavior possibly occurring at a given time on the weekend or during evening hours. Read the article for more excruciating details...Please note that you have to adjust the file paths for Linux.

     

    Many thanks to Jim Christensen for the core script.



  • 4.  Re: Hub Tunnel Disconnects - Error code 110
    Best Answer

    Posted 02-23-2017 07:39 PM

    Hi Stephen,

     

    Thank you for your suggestions!  These hubs are well established, and most definitely do not have a FW turned on, nor is there any AV scanning.

     

    Absolutely love the tutorial on "How to save off (dump) log files when an alarm occurs".  I am definitely going to implement this.  Thanks for the share.

     

    I have also opened a ticket with CA Support and they recommended:

     

    If "ignore_ip = no" then set to "ignore_ip = yes"

    If "check_cn = yes" then set to "check_cn = no"

     

    These changes are still recommended latest hub versions

     

    See TEC1836758

    https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1836758.html?intcmp=searchresultclick&resultnum=2  

     

    After a bit of digging, I located the following article which explains how to properly setup tunnels.  I confirmed that our server side hub.cfg and security.cfg files both have the "ignore_ip" set to yes, and our client side hubs also have "ignore check_cn" set to no. 

     

    Tunnel setup link:  https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec000002642.html  

     

    Thanks,

    Chris A.



  • 5.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 05-20-2019 08:35 AM

    Hi!

     

    The link to the script when saves log files when errors are detected is dead, can anyone post the script here? It sounds great.

     

    I've executed the pu command and outputted the contents of the PU command to list subs:

     

       i = 1
       probename = "nexec"
       probeaddr1 = "/DOMAIN/UIMHUB/UIMHUB/nexec"
     
       args = pds.create()
       pds.putString(args,"profile","NEXEC_PROFILE_NAME_TO_EXECUTE")

       while i <= 4 do
          reply,rc = nimbus.request(probeaddr1,"run_profile",args)
          if rc == NIME_OK then break end
          i = i + 1
          sleep(15000)
       end
     
       pds.delete(args)

     

    Unix command contained in a shell script sitting in /opt/nimsoft/bin:

     

    ./pu -u admin_username -p admin_password /DOMAIN/HUB/HUB/hub list_subscribers > output.txt

     

    Also need to set up an AO rule to execute the script on receipt of the alarm you want to action on.

     

    Hope it helps someone!



  • 6.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 05-20-2019 08:53 AM


  • 7.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 05-20-2019 08:59 AM

    Thank you!



  • 8.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 02-23-2017 07:23 PM

    I would also highly recommend that you open a support case for this issue.



  • 9.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 02-24-2017 11:14 AM

    Hey Chris,

     

    Have you seen this article on hub and tunnel setting recommendations?

    https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec000004536.html 

     

    In addition to Steve's suggestions, I recommend looking through that article and applying those settings.

    Those are a great help in many environments - particularly in previous versions.



  • 10.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 03-03-2017 08:40 AM

    Hi Phillip,

     

    Thank you for the link sir!  Being that we are on a custom hub (7.80-jh2), which was to address hub tunnel disconnects, and worked beautifully for nearly a year...making changes to the hub config first needs to go through some red tape.

     

    Your suggestions per the link have been discussed with Rito and Jason (CA Support).  Being that it is a Friday, we will likely not implement the changes today being that we are on the eve of a weekend.  Once final approval is received, I will get this config's implemented...likely Monday.

     

    Email chain with CA Support that may be useful for some:

     

    I asked Rito the following:

    -----------------------------------------------------------------------------------------------

    Hi Rito,

     

    Out of those 6 hub cfg settings within " tec000004536", we have 5 of them in our hubs.  Here are the current settings within our hubs:

     

       postroute_interval = 30

       postroute_reply_timeout = 60

       postroute_passive_timeout = 30

       hub_request_timeout = 30

       tunnel_hang_timeout = 300

     

    Would you be able to explain why we would need to switch those settings and how each could possibly help?

     

    And if we need to add back in "tunnel_hang_retries = 3", why was it removed and why should we add it back?

    -----------------------------------------------------------------------------------------------

    Response:

    -----------------------------------------------------------------------------------------------

    These timeouts collectively control the hub's behavior when it encounters slow responses from queues and other hubs.  Specifically the postroute_* timeouts have to do with queues. hub_request_timeout has to do with other hub-to-hub requests.   Increasing these will help in a busy environment as the hubs will be allowed more time to respond to each other.

     

    tunnel_hang_retries is not present by default, and when not present, defaults to 1.  This controls how many times a tunnel connection will be re-tried when unresponsive before the hub simply restarts itself entirely to attempt to self-heal. Setting this higher will allow the hub to internally retry the connection a couple more times before restarting.

    -----------------------------------------------------------------------------------------------

     



  • 11.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 03-03-2017 06:08 AM

    What i don't get is, the reference to port 48002, shouldn't that be 48003?

     

    Feb 23 13:24:45:469 [47141801335104] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110

     



  • 12.  Re: Hub Tunnel Disconnects - Error code 110

    Posted 03-03-2017 08:28 AM

    I found the following in this link which may shed some light on why 48002 is being used:  Installation Guide Release 7.1 

    In the left pane, at the bottom under 'More information', click the "Required Ports for SSL Tunnels

     

    Snippet from link:

    • Multi-hub infrastructure that uses tunnels that are NOT SLL tunnels:
      • All ports used in a single-hub installation
      • 48003, the port to the tunnel server (this also can bet set to 443)
    • Multi-hub infrastructure that uses Nimsoft SSL tunnels:
      • 48000 (controller) and 48002 (hub)
      • 48003 to allow the tunnel client to access the tunnel server