Tunnel server and client

[Keith Clay]

This is from the Secure Hub and Robot doc
If your primary hub is not a tunnel server or tunnel client, the 9.2.0 upgrade installer converts your primary hub into a tunnel server (when the secure option is used). It also converts other hubs that are pointing to that primary hub into tunnel clients. It is not ideal to have a primary hub as a tunnel server, but it might be useful in very small environments or for demonstrating proof of concepts (PoCs). However, in production environments, it is recommended that you always make your primary hub as a tunnel client.

We currently use tunnels.  Which end is the server and which the client?  If the primary hub shouldn't be a tunnel server, how do you do that?

[Gene Howard]

the tunnel setup has two parts a tunnel server that generates the SSL certificate and listens in port 48003 by default for incoming requested.
And the tunnel client that stores the SSL cert and password and is responsible for making the connection to the tunnel server.

a Hub can be both a tunnel server and a tunnel client in a three-tiered system the relay or concentrator hub acts as both a tunnel client usually to the primary hub and then as a tunnel server to the client hubs who act as  tunnel clients as well.

When a hub is acting as a tunnel client this tab is setup.

When a hub is acting as a tunnel server this tab is used.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Keith Clay]

Our Primary hub is the tunnel server.  The secure hub and robot doc says that is a bad thing.  How do we fix that?

[Gene Howard]

it is not a BAD think it is just not recommended as this puts extra load on the primary hub.
Basically remove your tunnels and set them up in the other direction.
If you set up your ports between hubs as open bidirectionally this should not be an issue.
if you only opened ports in one direction or you are using nat in some way you may need to change your firewall settings to be able to operate in the other directions.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Keith Clay]

So make the customer hubs servers and the primary hub a client of each one? So all it means is that the customer hubs initiate the connections to the primary.

[Gene Howard]

your first statement is correct yes:
So make the customer hubs servers and the primary hub a client of each one?

your second statement is not correct
the hub being the tunnel client will initiate the contact to the tunnel servers.


------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Keith Clay]

So the primary (client) will initiate the connection to the customer (secondary)?

[Gene Howard]

yes the tunnel client is responsible for creating and maintaining the connection

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Daniel Blanco]

@Keith Clay just FYI and remember this that when your creating your Tunnel Certificate make sure you set the expiration to 10 yrs+ later. The default is 1yr and that means when it expires, you'd need to create a new one and then re-setup all your tunnel clients with the new tunnel server cert. Just and FYI... :P

------------------------------
Daniel Blanco
Enterprise Tools Team Architect
[e-mail]
------------------------------

[Keith Clay]

Will this affecting setting up HA/failover?

[Gene Howard]

the tunnel should be set up the same way on the HA as it is on the primary.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Garin Walsh]

And just to chime in here, the statements about "puts extra load on the primary hub" comes with a negative connotation but I've found that very often these statements are very outdated or based on marginal hardware. And it discounts the management side of things. And maybe that "extra load" is just something that you expect - like discovery_server. It does so little of value but it has to be there and putting it on the central hub is the best place for it even though it represents a very significant amount of resource consumption.

The tunnel server is the one that creates the certificates you use to set up the tunnels. So if you make your central hub the tunnel server and have the remote servers the clients, then every remote client uses that same tunnel server certificate to connect.

If you turn that around and make the remote hubs the servers, then you need a unique certificate for every tunnel server, which means that you need to manage many server certificates instead of the one.

Also, if you have firewalls and whatnot involved, if you make the remote systems the servers, each needs to allow inbound connections from the central hub. If you own all the locations that might not be an issue but it does mean that you have to manage multiple inbound rules as opposed to making your central hub a server, then you have the one inbound to manage to your central hub.

But either way will work - or a mix of the two ways.

The terms "tunnel server" and "tunnel client" only apply to the endpoint of one tunnel - a single hub can be many tunnel servers and many tunnel clients at the same time. It just depends on your configuration needs. 

-Garin 

[Keith Clay]

Will this affect how queues are setup? Will GET/ATTACH have to reversed?

[Gene Howard]

nope will not affect how anything else works.
everything else happens AFTER the tunnel is connected.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Daniel Blanco]

Hi Keith,
If you are setting this up from scratch then Throw an extra layer between the primary hub and your tunnel server(s). Your client hubs are the tunnel clients that connect up to the tunnel server(s). You don't want many tunnel clients hitting up your primary server which in your case is also acting as the tunnel server. 
We have this structure today. We used to have all the tunnel servers right under the primary but were getting "Over subscribed" issues. By adding the extra layer to the hub collector it now works much better. 

Primary Hub  (Windows)

          ^^{(gets all queues from V hub collector)}

Hub Collector (uimcol) (CentOS)

          ^^{(gets all queues from V tunnel server hubs)}

Tunnel Server (6 of these) - UIMHUB1|2|3|4|5|6  (CentOS)

          ^^{(gets all queue msgs from client tunnel client hubs)}

Tunnel Client Hubs (100+) (Windows)

------------------------------
Daniel Blanco
Enterprise Tools Team Architect
[e-mail]
------------------------------

[Keith Kruepke]

The architecture that @Daniel Blanco​​ posted looks very robust. A great idea but probably not something you need right now if it hasn't come up separately from the documentation guidelines. I also think @Garin Walsh​ has a good point that the recommendation has to do squarely with load. If you aren't having any load issues, you can probably keep doing what you are today. But because of the potential for it to increase load, you should definitely keep an eye on the health of the primary hub and be "ready" for the possibility of changing things later as you grow. Should that day ever come, I suspect you'd probably prefer to add a layer between your primary hubs and your customer hubs so that you don't need to make each customer hub a tunnel server. I think that would look like the way @Daniel Blanco said they used to do it before they needed to add a 4th layer.

[Garin Walsh]

The one concern with the above recommendation, which is fine if you are sure you aren't going to grow or you have full control over the remote hubs, is that when you get to the point of outgrowing your primary hub as a tunnel server, you will need to recreate all the client->server tunnels because the server CA will be changed. Not a big deal if you have 20 systems that are easily modified but if you have 100 and you have change control to go through for each, it might be easier to set up the tunnel server architecture now.

And one more piece of background information that drives this conversation is the term "subscriber".

On the server side of things, the hub software needs to know when something happens that it needs to attend to. Speaking to the two common platforms, Windows and Linux, they approach this two different ways. 

When Windows needs to track something like this, it creates a "handle" for the object (in this case it would be a queue message arriving or a TCP/IP packet arriving). Then there's a windows system call that takes a list of handles to watch and puts the process to sleep until one of the handles is triggered. In Linux there's a similar concept but based on changes to a file descriptor using select/poll/epoll system calls.

In windows the number of handles that can be watched by this call is 64. If you look at any common Windows hub running 9.20 you'll have about 40 queues active (30 for probes and another 10-20 temp queues). That only leaves you space for 20ish tunnels. When you get into the >64 subscribers range the hub will continue to work but instead of using the Windows OS call it will start a round robin approach of checking where it will listen on a connection for a second or two, give up, move to the next and listen, give up, move to the next, etc. If you have enough subscribers, you'll start timing out on connections because the hub won't be checking them often enough. This also really impairs throughput because so much time is being wasted waiting on connections with nothing to do.

In the Linux world, older versions of the hub used the poll() call. This would take a bitmap of file descriptors that was 1024 bits in size and so could watch for changes in the first 1024 file descriptors created by the hub. Tunnels typically consume 6 to 10 descriptors and a queue consumes one so you're effective number of subscribers on an old Linux system would be around 100 to 150 - really dependent on the number of sessions maintained on active tunnels. If you exceeded that 1024 file descriptor limit, the hub code would crash eventually because of the buffer overrun that would happen i tries to put more than 1024 bits of information into space for 1024.

The 7.97 and newer hubs use epoll() which does not have this limit and personally I've pushed a version of this hub to over 1400 subscribers with satisfactory results. I don't do that now but it needed to be pushed to see what happened.

So it's a lot of words but there's a huge benefit to having that tunnel server be Linux on a very recent hub version (we use 7.97 on CentOS 7 for instance) but Windows for the central hub.

And when you hear about "small" environments or "large" environments, I'd make the following statement:
Small = <4 hubs with tunnels. (no need fora tunnel server)
Medium = 4 < hubs < 20 with tunnels. (think about needing a tunnel server)
Large = 20 < hubs < 50 with tunnels. (have to have a tunnel server)
Huge  = 50 or more hubs with tunnels. (have to have many tunnel servers and another tier of tunnel concentrators)  

Robots don't really matter - they just drive the demand on your network infrastructure and database.

In my environment I have 3000 remote hubs, 106 tunnel servers (32 HA pairs) , 8 tunnel concentrators (4 HA pairs) and two central hubs (1 HA pair). At around 50 hubs, I was forced to move to having tunnel concentrators. At around 2000 hubs, I had to introduce the tunnel concentrators.

And if you're thinking HA, remember that you need 2 of everything and that doubles your active tunnels so you reach that subscriber limit that much faster.

-Garin

[Keith Clay]

Can you flesh this out much more?  The primary hub is either the server or the client. What you said doesn't make any sense with the tunnel server not on a client or primary.

[Keith Clay]

What is the diff between a tunnel concentrator and a tunnel server? Tunnel concentrators only talk to tunnel servers

[Garin Walsh]

Nothing really, it's a term that defines a general work responsibility.

In my specific deployment, I have 106 "tunnel servers" which still far exceeds the subscriber limits in Windows and so I have a fourth tier of hubs that I term "tunnel concentrators". Their sole responsibility is to run get queues against these tunnel servers and to host attach queues for the central hub to get from.

[Louis Savage]

Using Tunnel Concentrators are some people say "Relay Hubs" is a key to larger infrastructures to funnel things up.  Along with File Descriptor limitations and adding higher open file limits. 

There is also slight difference on the Linux vs Windows Hubs which cause some limitations on the Windows side. 

I am not sure if the 3 Tier Architecture designed is emphasized enough in the documentation and sizing of the database servers as we run ~30,000 node DXIM install. Giving a recommendation on this post as people should really read this.

[Keith Clay]

Daniel,

I was thinking of this setup.  The Secondary HUBs A and B handle all the queues from the client hubs (so if A is unavailable, B would take over)  Then the Primary and the Primary HA would only have the queues from the Secondaries rather than all the client hubs.  Suggestions?

Primary HUB                        Primary HUB HA

Secondary HUB A                Seconday HUB B

Client HUBS (10-20 at most)

[Keith Clay]

Daniel,

For tunnels servers,
1) are they/can they be load balance or how does that work?
2) Their only attach queues are those going to the primary/primary HA?

[Garin Walsh]

1) Yes, works the same as any other pair of hubs with regards to probes and queues. You have to create the two tunnels to be active all the time because there's no mechanism in the client to have an active/passive tunnel pair.

2) yes - though each tunnel server hub will have it's attach queue always active - just the get queues on the passive hub would be inactive and managed by the HA probe.

[Keith Clay]

I really need help.  I set up a secondary hub which is going to take info from the customer hubs and send it to the primary.  I setup the server tunnel setup (no customer hubs yet) and the client tunnel setup to the primary.  When I go to queues setup, there are no queues listed for setting up attach queues. 

What am I doing wrong?

[Louis Savage]

You need to setup an attach queue on the remote hub then setup a get queue on the receiving hub you want the data to be at.

[Keith Clay]

There are no queues to setup in the queue tab on the hub probe.

[Keith Clay]

And I really don't know what I'm doing and the docs don't seem to help me figure it out.

[Louis Savage]

The old CA Educate youtube is a great resource for just starting out ..

https://www.youtube.com/channel/UC6L0DBYWqAkmwfawTUMaR3g

How to create queues : 

Example : https://www.youtube.com/watch?v=6dCoidsR510

[Keith Clay]

On the secondary do I have to setup the server cert section for the hubs answering to it?

[Daniel Blanco]

Here you go... Hope this helps:
This is my lab but the hub at bottom imagine its your client hub. 

The GET queue you specify the hub you want to get the messages out from. 

Also the Tunnel Server is the one with the certificate. You then take that certificate and import it or specify it on the customers tunnel client that connects to the tunnel server. 
In your environment how are the tunnel \ customer hubs connecting to the tunnels on the tunnel hubs? VPN ? or public IP on internet?

------------------------------
Daniel Blanco
Enterprise Tools Team Architect
[e-mail]
------------------------------