best practices on hub tunnel settings in large installations

[Anon Anon]

hi,

here's something 1&1 identified in the hub tunnel settings that had a huge effect on the performance of their hub/tunnel setup.

 

They were facing the fact that tunnel connections seemed to be quite instable and the message flow was fluctuating periodically.

 

First of all, they increased the Tunnel Session Pool from 5 (default) to 30 in order to increase the throughput of the tunnels. Something you will probably do as well I assume once the installation has to handle large volumes of messages.

This did increase the throughput but also caused the system to become more instable.

Now, the trick - is to also adapt the Server Cache Size of the hub (you find all those settings in the hub under the advanced-tab). The Cache Size is set to 1024 by default. Now this is fine if you do not have a lot of tunnels with a lot of connections - but if you do, you need to increase this value.

The formula to calculate the correct server cache size for the hub is:
<number of tunnels / known hubs> x <Tunnel Session Pool> x 2 = Server Cache Size

The reason why you have to multiply it with 2 is that you configure the session pool for one direction only, the hub creates twice the amount of connections though, the incoming and the outgoing ones.

Once these settings were made the message flow is totally even and we do no longer have weird disconnects.

 

Something else to watch out for is to not set the SSL_Reconnect-times and timeouts too short. By default those are quite high and 1&1 is using 180 seconds as the tunnel timeouts - setting those lower seemed to thrash the system once the network went down for a couple of minutes.

-chris

 

ps: if you think these settings are wrong - let me know. we figured those out by analyzing the tcp traffic and they work just fine for a couple of weeks now but they were not officially suggested by our R&D.

[Anon Anon]

This is useful information. I'm onsite at a large customer - MSP with 150+ customers. They have a hub at each of their customer sites and have set up 3 tunnel server hubs to handle the tunnel connections. We were seeing weird things like occasional queue disconnects and Intermediate states of hubs in the Inf Mgr, so I decided to put these settings that Chris mentions into play. But they did not really seem to help.

 

My question is that for windows hubs, I've heard and seen about 30-40 tunnel connections as a max before seeing weird things like queue disconnects. At this customer site, we have linux tunnel servers which I thought would help alleviate some of the performance hits and perhaps be able to handle more tunnel connections than windows servers. Before I ask the customer to spin up more VMs for tunnel servers and move the tunnel connections around, does anyone know if having 50-60 tunnel connections to a linux tunnel server is overkill? The CPU and Memory on the tunnel servers are constantly at 50% so we're not seeing a performance hit or having a need to put in more RAM or CPU for these current tunnel servers.

[Anon Anon]

At Jack Henry we had some CentOS tunnel servers with upwards of 60 customers connected to each one, we usually rolled over to a new tunnel hub server at that 50-60 mark just to be safe.I know early on we had some tunnel connectivity issues because the tunnel encryption was set too high and that was causing some timeouts, there were also some other issues in June. That case can be found here https://na4.salesforce.com/50060000007tjCP not sure if it will help, but there is a spreadsheet with some tunnel hub settings.

[Greg Polk]

Can we get the correct url for the document loaded into this document.

[Yu Ishitani]

It is also a retired link.

Please post a new thread for your question with hub version you are using because hub behavior have continuously changed over years.

[Daniel Blanco]

So what should a tunnel server hub's settings be set to for a tunnel server that handles ~30 tunnel clients each with 4 get queues to each TC hub?

 

Hub Probe \ Tunnels \ Advanced Tab

 

Tunnel is hanging timeout: 120s

SSL Session Cache: YES

Server Cache Timeout: 7200

Server Cache Size: 8192

Use Client Cache:  YES

 

Should this be increased?

And on the Tunnel Client side what should their #'s be? And can someone explain what these #'s and settings actually control? 

[Stephen Danseglio]

Tunnel is hanging timeout: 120s
The hub continuously checks if one or more of the active tunnels are hanging. No new connections can be established through tunnels that are hanging. If one or more tunnels are hanging, the hub attempts to restart the tunnel(s). If no success, the hub performs a restart after the specified number of seconds.

 

SSL Session Cache: YES
Enable caching of SSL Sessions (reuse previous session credentials) speeds up the connection time between the client and the server a lot, assuming the client has enabled 'Use Client Cache'. The tunnel server keeps a cache of recent SSL sessions and when a client attempts to "resume" a previous session, this cache is checked to see if the server has the session information cached. If not, then a new session must be initiated. The sessions will remain cached for the number of seconds indicated by the timeout value -- the number of sessions which will remain in this cache is dependent on the cache size, as new sessions will "bump" old sessions out of the cache. Increasing this value could improve tunnel performance especially when there are a large number of tunnel clients present, but increasing it 'too high' could create resource usage (memory) issues on the hub.

 

Server Cache Timeout: 7200
Defines how long the cached sessions are valid for reuse by the client.

 

Server Cache Size: 8192
Defines how many sessions can be stored in the cache. When the cache is full, the oldest sessions will be thrown out when new connections are established. The formula to calculate the correct server cache size for the hub is:

 

(Number of Hubs or Tunnels) x (Tunnel Session Pool) x 2 = Server Cache Size value

 

Use Client Cache:  YES
Enable caching of SSL Sessions (reuse previous session credentials if still valid).
When this is enabled we are caching the client-side information about the client's session(s) in a similar manner as Use Server Cache.

 

Are you having any tunnel problems?

[Daniel Blanco]

Thanks danst04 (Stephen) for that great info. What is "Tunnel Session Pool" ? Where is this # found?

Not having any issues but just experience slowness from time to time and if a few tweaks can make things smoother all the better..

 

[Stephen Danseglio]

Tunnel Performance – Advanced Settings

The hub Advanced tab allows you to configure the alarm settings, assign the first tunnel port, establish the hanging timeout, set the session pool size, and configure the SSL session cache.

Default Client Connection Pool Size

The size of the tunnel session pool. The session pool is symmetric, meaning inbound and outbound connections have separate pools of the same size.

Example: setting the client connection pool size value to 30 will create 60 connections between the client and the server.

The default is 5 which is normally sufficient unless you have a lot of traffic traversing the tunnel and find that tunnels are dropping often. It may make sense to have more than 5 configured in the case where the tunnel is configured and operating as both a Server and a client as well.

Note that in 7.x this parameter no longer exists.

The optimal session pool size value on Windows is about 30. If the value is set too high the hub tunnel performance will decrease.

 

Calculating Session Pool size

The formula to calculate the correct session pool size is: 

(session_pool_size * 3) * (the number of tunnel clients in the environment) 

So if session_pool_size is 5, 5 * 3 = 15, so 15 * the number of tunnel clients .... 

Ideally, you want to approach but not exceed a value of 900 - which means basically if you have 30 tunnel clients you can set session_pool_size on each client up to 10.  If you have 60 tunnel clients per hub then it should be set to 5, etc.

------------------------------
Support Engineer
Broadcom
US
------------------------------