The one concern with the above recommendation, which is fine if you are sure you aren't going to grow or you have full control over the remote hubs, is that when you get to the point of outgrowing your primary hub as a tunnel server, you will need to recreate all the client->server tunnels because the server CA will be changed. Not a big deal if you have 20 systems that are easily modified but if you have 100 and you have change control to go through for each, it might be easier to set up the tunnel server architecture now.
And one more piece of background information that drives this conversation is the term "subscriber".
On the server side of things, the hub software needs to know when something happens that it needs to attend to. Speaking to the two common platforms, Windows and Linux, they approach this two different ways.
When Windows needs to track something like this, it creates a "handle" for the object (in this case it would be a queue message arriving or a TCP/IP packet arriving). Then there's a windows system call that takes a list of handles to watch and puts the process to sleep until one of the handles is triggered. In Linux there's a similar concept but based on changes to a file descriptor using select/poll/epoll system calls.
In windows the number of handles that can be watched by this call is 64. If you look at any common Windows hub running 9.20 you'll have about 40 queues active (30 for probes and another 10-20 temp queues). That only leaves you space for 20ish tunnels. When you get into the >64 subscribers range the hub will continue to work but instead of using the Windows OS call it will start a round robin approach of checking where it will listen on a connection for a second or two, give up, move to the next and listen, give up, move to the next, etc. If you have enough subscribers, you'll start timing out on connections because the hub won't be checking them often enough. This also really impairs throughput because so much time is being wasted waiting on connections with nothing to do.
In the Linux world, older versions of the hub used the poll() call. This would take a bitmap of file descriptors that was 1024 bits in size and so could watch for changes in the first 1024 file descriptors created by the hub. Tunnels typically consume 6 to 10 descriptors and a queue consumes one so you're effective number of subscribers on an old Linux system would be around 100 to 150 - really dependent on the number of sessions maintained on active tunnels. If you exceeded that 1024 file descriptor limit, the hub code would crash eventually because of the buffer overrun that would happen i tries to put more than 1024 bits of information into space for 1024.
The 7.97 and newer hubs use epoll() which does not have this limit and personally I've pushed a version of this hub to over 1400 subscribers with satisfactory results. I don't do that now but it needed to be pushed to see what happened.
So it's a lot of words but there's a huge benefit to having that tunnel server be Linux on a very recent hub version (we use 7.97 on CentOS 7 for instance) but Windows for the central hub.
And when you hear about "small" environments or "large" environments, I'd make the following statement:
Small = <4 hubs with tunnels. (no need fora tunnel server)
Medium = 4 < hubs < 20 with tunnels. (think about needing a tunnel server)
Large = 20 < hubs < 50 with tunnels. (have to have a tunnel server)
Huge = 50 or more hubs with tunnels. (have to have many tunnel servers and another tier of tunnel concentrators)
Robots don't really matter - they just drive the demand on your network infrastructure and database.
In my environment I have 3000 remote hubs, 106 tunnel servers (32 HA pairs) , 8 tunnel concentrators (4 HA pairs) and two central hubs (1 HA pair). At around 50 hubs, I was forced to move to having tunnel concentrators. At around 2000 hubs, I had to introduce the tunnel concentrators.
And if you're thinking HA, remember that you need 2 of everything and that doubles your active tunnels so you reach that subscriber limit that much faster.
-Garin
Original Message:
Sent: 12-06-2019 12:40 PM
From: Keith Kruepke
Subject: Tunnel server and client
The architecture that @Daniel Blanco posted looks very robust. A great idea but probably not something you need right now if it hasn't come up separately from the documentation guidelines. I also think has a good point that the recommendation has to do squarely with load. If you aren't having any load issues, you can probably keep doing what you are today. But because of the potential for it to increase load, you should definitely keep an eye on the health of the primary hub and be "ready" for the possibility of changing things later as you grow. Should that day ever come, I suspect you'd probably prefer to add a layer between your primary hubs and your customer hubs so that you don't need to make each customer hub a tunnel server. I think that would look like the way @Daniel Blanco said they used to do it before they needed to add a 4th layer.
Original Message:
Sent: 12-05-2019 02:56 PM
From: Daniel Blanco
Subject: Tunnel server and client
Hi Keith,
If you are setting this up from scratch then Throw an extra layer between the primary hub and your tunnel server(s). Your client hubs are the tunnel clients that connect up to the tunnel server(s). You don't want many tunnel clients hitting up your primary server which in your case is also acting as the tunnel server.
We have this structure today. We used to have all the tunnel servers right under the primary but were getting "Over subscribed" issues. By adding the extra layer to the hub collector it now works much better.
Primary Hub (Windows)
^(gets all queues from V hub collector)
Hub Collector (uimcol) (CentOS)
^(gets all queues from V tunnel server hubs)
Tunnel Server (6 of these) - UIMHUB1|2|3|4|5|6 (CentOS)
^(gets all queue msgs from client tunnel client hubs)
Tunnel Client Hubs (100+) (Windows)
------------------------------
Daniel Blanco
Enterprise Tools Team Architect
DBlanco@alphaserveit.com
Original Message:
Sent: 12-05-2019 01:48 PM
From: Keith Clay
Subject: Tunnel server and client
Will this affect how queues are setup? Will GET/ATTACH have to reversed?
Original Message:
Sent: 12-05-2019 01:11 PM
From: Gene HOWARD
Subject: Tunnel server and client
it is not a BAD think it is just not recommended as this puts extra load on the primary hub.
Basically remove your tunnels and set them up in the other direction.
If you set up your ports between hubs as open bidirectionally this should not be an issue.
if you only opened ports in one direction or you are using nat in some way you may need to change your firewall settings to be able to operate in the other directions.
------------------------------
Gene Howard
Principal Support Engineer
Broadcom
Original Message:
Sent: 12-05-2019 01:09 PM
From: Keith Clay
Subject: Tunnel server and client
Our Primary hub is the tunnel server. The secure hub and robot doc says that is a bad thing. How do we fix that?
Original Message:
Sent: 12-05-2019 01:03 PM
From: Gene HOWARD
Subject: Tunnel server and client
the tunnel setup has two parts a tunnel server that generates the SSL certificate and listens in port 48003 by default for incoming requested.
And the tunnel client that stores the SSL cert and password and is responsible for making the connection to the tunnel server.
a Hub can be both a tunnel server and a tunnel client in a three-tiered system the relay or concentrator hub acts as both a tunnel client usually to the primary hub and then as a tunnel server to the client hubs who act as tunnel clients as well.
When a hub is acting as a tunnel client this tab is setup.
------------------------------
Gene Howard
Principal Support Engineer
Broadcom
Original Message:
Sent: 12-05-2019 12:53 PM
From: Keith Clay
Subject: Tunnel server and client
This is from the Secure Hub and Robot doc
If your primary hub is not a tunnel server or tunnel client, the 9.2.0 upgrade installer converts your primary hub into a tunnel server (when the secure option is used). It also converts other hubs that are pointing to that primary hub into tunnel clients. It is not ideal to have a primary hub as a tunnel server, but it might be useful in very small environments or for demonstrating proof of concepts (PoCs). However, in production environments, it is recommended that you always make your primary hub as a tunnel client.
We currently use tunnels. Which end is the server and which the client? If the primary hub shouldn't be a tunnel server, how do you do that?