Hub Tunnel Disconnects - Error code 110

[Christopher Armstrong]

Hi,

 

Every now and again, our support team receives a flood of Nimsoft Tunnel Disconnect tickets which we thought were caused by a particular client that was flooding the tunnel connection with data, therefore, causing it to drop out / restart.  We thought this because the tunnel disconnects were happening on one particular hub.

 

Here recently, we have been getting Hub Tunnel Disconnect tickets from all over (multiple hubs).  I just happened to catch the log file in time before they rolled off (log level 3), and found the following before the Hub Tunnel restarted (sensitive data removed):

 

Feb 23 13:24:43:879 [47140332755264] hub: TSESS-P-823-0 sent error message response
Feb 23 13:24:45:469 [47141801335104] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
Feb 23 13:24:45:469 [47141801335104] hub: TSESS-P-724-8 [] could not connect locally to /Gladiator/Hub09/nimhub09/hub (x.x.x.x/48002)
Feb 23 13:24:45:470 [47141801335104] hub: TSESS-P-724-8 sent error message response
Feb 23 13:24:45:927 [47138969073984] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
Feb 23 13:24:45:927 [47138969073984] hub: TSESS-P-694-10 [] could not connect locally to /Gladiator/Hub09/nimhub09/hub (x.x.x.x/48002)
Feb 23 13:24:45:927 [47138969073984] hub: TSESS-P-694-10 sent error message response
Feb 23 13:24:46:635 [47138011362176] hub: EXIT HUB: hub shutdown
Feb 23 13:24:46:636 [47138011362176] hub: Waiting for tunnel to terminate ...
Feb 23 13:24:46:882 [47140101978432] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
Feb 23 13:24:46:882 [47140101978432] hub: TSESS-P-773-3 [] could not connect locally to /Gladiator/Hub09/nimhub09/hub (x.x.x.x/48002)
Feb 23 13:24:46:882 [47140101978432] hub: TSESS-P-773-3 sent error message response
Feb 23 13:24:47:498 [47139955120448] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110
Feb 23 13:24:47:498 [47139955120448] hub: TSESS-P-762-4 sent error message response
Feb 23 13:24:49:751 [47929251947392] hub: ----------------------------------------------------------------------------------------------------
Feb 23 13:24:49:751 [47929251947392] hub: ----- Hub 7.80-jh2 [Build 7.80, Nov 9 2015] initializing -----
Feb 23 13:24:49:752 [47929251947392] hub: Tunnel has started in thread [ 47929318897984 ]
Feb 23 13:24:49:788 [47929251947392] hub: ----- HUB started -----

 

As you can see, error code 110 was very prevalent before the hub tunnel dropped out.  Any info to the meaning of error 110 would be greatly appreciated.

 

Additional info:

-We have 14 hub servers that are linux boxes (centos)

-Tunnel Encryption set to Medium

-Hub tunnel SSL Settings Mode set to Normal

-UIM ver 7.5 ---- Will be upgrading this coming March to 8.47

-Hub version 7.80

 

Thanks,

Chris A.

[Stephen Danseglio]

An error code 110 is a connection refused error.

Please make sure the following
1) firewall is disabled

2) no AV scanning/blocking

[Stephen Danseglio]

Chris,

 

Furthermore, for this scenario and any other scenario where you need to save off/dump specific log files when an alarm occurs, please refer to this Article I recently updated and tested:

 

How to save off (dump) log files when an alarm occurs

 

https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec000003963.html 

 

So you can catch it red-handed. In your situation I would set the loglevel to 5 and logsize to 40000 for the hub, controller and data_engine to get a better idea of what's happening. Use a regex for the tunnel disconnect alarm - its explained in the KB Article.

 

This Article contains a proven method of capturing essential log files used for analyzing customer problems that are very difficult to catch when they occur. This method includes a nas LUA script to automatically save off (dump) specified log files when an alarm occurs (based on an AO profile that filters on the given alarm message or any other filter you choose to define in the AO profile.) This script may be edited to suit your particular needs for capturing the appropriate log files when a problem occurs.

 

It is especially useful in situations where the problem is infrequent or inconsistent, occurs randomly or it exhibits a pattern of behavior possibly occurring at a given time on the weekend or during evening hours. Read the article for more excruciating details...Please note that you have to adjust the file paths for Linux.

 

Many thanks to Jim Christensen for the core script.

[Christopher Armstrong]

Hi Stephen,

 

Thank you for your suggestions!  These hubs are well established, and most definitely do not have a FW turned on, nor is there any AV scanning.

 

Absolutely love the tutorial on "How to save off (dump) log files when an alarm occurs".  I am definitely going to implement this.  Thanks for the share.

 

I have also opened a ticket with CA Support and they recommended:

 

If "ignore_ip = no" then set to "ignore_ip = yes"

If "check_cn = yes" then set to "check_cn = no"

 

These changes are still recommended latest hub versions

 

See TEC1836758

https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1836758.html?intcmp=searchresultclick&resultnum=2  

 

After a bit of digging, I located the following article which explains how to properly setup tunnels.  I confirmed that our server side hub.cfg and security.cfg files both have the "ignore_ip" set to yes, and our client side hubs also have "ignore check_cn" set to no. 

 

Tunnel setup link:  https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec000002642.html  

 

Thanks,

Chris A.

[Sam Green]

Hi!

 

The link to the script when saves log files when errors are detected is dead, can anyone post the script here? It sounds great.

 

I've executed the pu command and outputted the contents of the PU command to list subs:

 

   i = 1
   probename = "nexec"
   probeaddr1 = "/DOMAIN/UIMHUB/UIMHUB/nexec"
 
   args = pds.create()
   pds.putString(args,"profile","NEXEC_PROFILE_NAME_TO_EXECUTE")

   while i <= 4 do
      reply,rc = nimbus.request(probeaddr1,"run_profile",args)
      if rc == NIME_OK then break end
      i = i + 1
      sleep(15000)
   end
 
   pds.delete(args)

 

Unix command contained in a shell script sitting in /opt/nimsoft/bin:

 

./pu -u admin_username -p admin_password /DOMAIN/HUB/HUB/hub list_subscribers > output.txt

 

Also need to set up an AO rule to execute the script on receipt of the alarm you want to action on.

 

Hope it helps someone!

[Stephen Danseglio]

Hi Chris,

 

Sure, here is the url.

 

https://comm.support.ca.com/kb/how-to-save-off-dump-log-files-when-an-alarm-occurs/kb000034013

 

Steve

[Sam Green]

Thank you!

[Stephen Danseglio]

I would also highly recommend that you open a support case for this issue.

[Philip Cashion]

Hey Chris,

 

Have you seen this article on hub and tunnel setting recommendations?

https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec000004536.html 

 

In addition to Steve's suggestions, I recommend looking through that article and applying those settings.

Those are a great help in many environments - particularly in previous versions.

[Christopher Armstrong]

Hi Phillip,

 

Thank you for the link sir!  Being that we are on a custom hub (7.80-jh2), which was to address hub tunnel disconnects, and worked beautifully for nearly a year...making changes to the hub config first needs to go through some red tape.

 

Your suggestions per the link have been discussed with Rito and Jason (CA Support).  Being that it is a Friday, we will likely not implement the changes today being that we are on the eve of a weekend.  Once final approval is received, I will get this config's implemented...likely Monday.

 

Email chain with CA Support that may be useful for some:

 

I asked Rito the following:

-----------------------------------------------------------------------------------------------

Hi Rito,

 

Out of those 6 hub cfg settings within " tec000004536", we have 5 of them in our hubs.  Here are the current settings within our hubs:

 

   postroute_interval = 30

   postroute_reply_timeout = 60

   postroute_passive_timeout = 30

   hub_request_timeout = 30

   tunnel_hang_timeout = 300

 

Would you be able to explain why we would need to switch those settings and how each could possibly help?

 

And if we need to add back in "tunnel_hang_retries = 3", why was it removed and why should we add it back?

-----------------------------------------------------------------------------------------------

Response:

-----------------------------------------------------------------------------------------------

These timeouts collectively control the hub's behavior when it encounters slow responses from queues and other hubs.  Specifically the postroute_* timeouts have to do with queues. hub_request_timeout has to do with other hub-to-hub requests.   Increasing these will help in a busy environment as the hubs will be allowed more time to respond to each other.

 

tunnel_hang_retries is not present by default, and when not present, defaults to 1.  This controls how many times a tunnel connection will be re-tried when unresponsive before the hub simply restarts itself entirely to attempt to self-heal. Setting this higher will allow the hub to internally retry the connection a couple more times before restarting.

-----------------------------------------------------------------------------------------------

 

[Legacy User]

What i don't get is, the reference to port 48002, shouldn't that be 48003?

 

Feb 23 13:24:45:469 [47141801335104] hub: nimSession - failed to connect session to x.x.x.x:48002, error code 110

 

[Christopher Armstrong]

I found the following in this link which may shed some light on why 48002 is being used:  Installation Guide Release 7.1 

In the left pane, at the bottom under 'More information', click the "Required Ports for SSL Tunnels

 

Snippet from link:

• Multi-hub infrastructure that uses tunnels that are NOT SLL tunnels:
  • All ports used in a single-hub installation
  • 48003, the port to the tunnel server (this also can bet set to 443)
• Multi-hub infrastructure that uses Nimsoft SSL tunnels:
  • 48000 (controller) and 48002 (hub)
  • 48003 to allow the tunnel client to access the tunnel server