DX Infrastructure Management

Expand all | Collapse all

Tunnel server and client

  • 1.  Tunnel server and client

    Posted 12-05-2019 12:53 PM
    This is from the Secure Hub and Robot doc
    If your primary hub is not a tunnel server or tunnel client, the 9.2.0 upgrade installer converts your primary hub into a tunnel server (when the secure option is used). It also converts other hubs that are pointing to that primary hub into tunnel clients. It is not ideal to have a primary hub as a tunnel server, but it might be useful in very small environments or for demonstrating proof of concepts (PoCs). However, in production environments, it is recommended that you always make your primary hub as a tunnel client.

    We currently use tunnels.  Which end is the server and which the client?  If the primary hub shouldn't be a tunnel server, how do you do that?


  • 2.  RE: Tunnel server and client

    Posted 12-05-2019 01:04 PM
    the tunnel setup has two parts a tunnel server that generates the SSL certificate and listens in port 48003 by default for incoming requested.
    And the tunnel client that stores the SSL cert and password and is responsible for making the connection to the tunnel server.

    a Hub can be both a tunnel server and a tunnel client in a three-tiered system the relay or concentrator hub acts as both a tunnel client usually to the primary hub and then as a tunnel server to the client hubs who act as  tunnel clients as well.

    When a hub is acting as a tunnel client this tab is setup.

    When a hub is acting as a tunnel server this tab is used.


    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 3.  RE: Tunnel server and client

    Posted 12-05-2019 01:09 PM
    Our Primary hub is the tunnel server.  The secure hub and robot doc says that is a bad thing.  How do we fix that?


  • 4.  RE: Tunnel server and client

    Posted 12-05-2019 01:11 PM
    it is not a BAD think it is just not recommended as this puts extra load on the primary hub.
    Basically remove your tunnels and set them up in the other direction.
    If you set up your ports between hubs as open bidirectionally this should not be an issue.
    if you only opened ports in one direction or you are using nat in some way you may need to change your firewall settings to be able to operate in the other directions.

    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 5.  RE: Tunnel server and client

    Posted 12-05-2019 01:16 PM
    So make the customer hubs servers and the primary hub a client of each one? So all it means is that the customer hubs initiate the connections to the primary.


  • 6.  RE: Tunnel server and client

    Posted 12-05-2019 01:33 PM
    your first statement is correct yes:
    So make the customer hubs servers and the primary hub a client of each one?

    your second statement is not correct
    the hub being the tunnel client will initiate the contact to the tunnel servers.


    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 7.  RE: Tunnel server and client

    Posted 12-05-2019 01:39 PM
    So the primary (client) will initiate the connection to the customer (secondary)?


  • 8.  RE: Tunnel server and client

    Posted 12-05-2019 01:43 PM
    yes the tunnel client is responsible for creating and maintaining the connection

    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 9.  RE: Tunnel server and client

    Posted 12-06-2019 02:20 PM
    @Keith Clay just FYI and remember this that when your creating your Tunnel Certificate make sure you set the expiration to 10 yrs+ later. The default is 1yr and that means when it expires, you'd need to create a new one and then re-setup all your tunnel clients with the new tunnel server cert. Just and FYI... :P

    ------------------------------
    Daniel Blanco
    Enterprise Tools Team Architect
    DBlanco@alphaserveit.com
    ------------------------------



  • 10.  RE: Tunnel server and client

    Posted 12-05-2019 01:29 PM
    Will this affecting setting up HA/failover?


  • 11.  RE: Tunnel server and client

    Posted 12-05-2019 01:33 PM
    the tunnel should be set up the same way on the HA as it is on the primary.

    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 12.  RE: Tunnel server and client

    Posted 12-05-2019 01:51 PM
    And just to chime in here, the statements about "puts extra load on the primary hub" comes with a negative connotation but I've found that very often these statements are very outdated or based on marginal hardware. And it discounts the management side of things. And maybe that "extra load" is just something that you expect - like discovery_server. It does so little of value but it has to be there and putting it on the central hub is the best place for it even though it represents a very significant amount of resource consumption.

    The tunnel server is the one that creates the certificates you use to set up the tunnels. So if you make your central hub the tunnel server and have the remote servers the clients, then every remote client uses that same tunnel server certificate to connect.

    If you turn that around and make the remote hubs the servers, then you need a unique certificate for every tunnel server, which means that you need to manage many server certificates instead of the one.

    Also, if you have firewalls and whatnot involved, if you make the remote systems the servers, each needs to allow inbound connections from the central hub. If you own all the locations that might not be an issue but it does mean that you have to manage multiple inbound rules as opposed to making your central hub a server, then you have the one inbound to manage to your central hub.

    But either way will work - or a mix of the two ways.

    The terms "tunnel server" and "tunnel client" only apply to the endpoint of one tunnel - a single hub can be many tunnel servers and many tunnel clients at the same time. It just depends on your configuration needs.

    -Garin 



  • 13.  RE: Tunnel server and client

    Posted 12-05-2019 01:48 PM
    Will this affect how queues are setup? Will GET/ATTACH have to reversed?


  • 14.  RE: Tunnel server and client

    Posted 12-05-2019 01:49 PM
    nope will not affect how anything else works.
    everything else happens AFTER the tunnel is connected.

    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 15.  RE: Tunnel server and client

    Posted 12-05-2019 02:56 PM
    Edited by Daniel Blanco 12-05-2019 03:26 PM

    Hi Keith,
    If you are setting this up from scratch then Throw an extra layer between the primary hub and your tunnel server(s). Your client hubs are the tunnel clients that connect up to the tunnel server(s). You don't want many tunnel clients hitting up your primary server which in your case is also acting as the tunnel server.
    We have this structure today. We used to have all the tunnel servers right under the primary but were getting "Over subscribed" issues. By adding the extra layer to the hub collector it now works much better. 


    Primary Hub  (Windows)

              ^(gets all queues from V hub collector)

    Hub Collector (uimcol) (CentOS)

              ^(gets all queues from V tunnel server hubs)

    Tunnel Server (6 of these) - UIMHUB1|2|3|4|5|6  (CentOS)

              ^(gets all queue msgs from client tunnel client hubs)

    Tunnel Client Hubs (100+) (Windows)



    ------------------------------
    Daniel Blanco
    Enterprise Tools Team Architect
    DBlanco@alphaserveit.com
    ------------------------------



  • 16.  RE: Tunnel server and client

    Posted 12-06-2019 12:41 PM
    The architecture that @Daniel Blanco​​ posted looks very robust. A great idea but probably not something you need right now if it hasn't come up separately from the documentation guidelines. I also think ​ has a good point that the recommendation has to do squarely with load. If you aren't having any load issues, you can probably keep doing what you are today. But because of the potential for it to increase load, you should definitely keep an eye on the health of the primary hub and be "ready" for the possibility of changing things later as you grow. Should that day ever come, I suspect you'd probably prefer to add a layer between your primary hubs and your customer hubs so that you don't need to make each customer hub a tunnel server. I think that would look like the way @Daniel Blanco said they used to do it before they needed to add a 4th layer.


  • 17.  RE: Tunnel server and client

    Posted 12-06-2019 02:03 PM
    The one concern with the above recommendation, which is fine if you are sure you aren't going to grow or you have full control over the remote hubs, is that when you get to the point of outgrowing your primary hub as a tunnel server, you will need to recreate all the client->server tunnels because the server CA will be changed. Not a big deal if you have 20 systems that are easily modified but if you have 100 and you have change control to go through for each, it might be easier to set up the tunnel server architecture now.

    And one more piece of background information that drives this conversation is the term "subscriber".

    On the server side of things, the hub software needs to know when something happens that it needs to attend to. Speaking to the two common platforms, Windows and Linux, they approach this two different ways.

    When Windows needs to track something like this, it creates a "handle" for the object (in this case it would be a queue message arriving or a TCP/IP packet arriving). Then there's a windows system call that takes a list of handles to watch and puts the process to sleep until one of the handles is triggered. In Linux there's a similar concept but based on changes to a file descriptor using select/poll/epoll system calls.

    In windows the number of handles that can be watched by this call is 64. If you look at any common Windows hub running 9.20 you'll have about 40 queues active (30 for probes and another 10-20 temp queues). That only leaves you space for 20ish tunnels. When you get into the >64 subscribers range the hub will continue to work but instead of using the Windows OS call it will start a round robin approach of checking where it will listen on a connection for a second or two, give up, move to the next and listen, give up, move to the next, etc. If you have enough subscribers, you'll start timing out on connections because the hub won't be checking them often enough. This also really impairs throughput because so much time is being wasted waiting on connections with nothing to do.

    In the Linux world, older versions of the hub used the poll() call. This would take a bitmap of file descriptors that was 1024 bits in size and so could watch for changes in the first 1024 file descriptors created by the hub. Tunnels typically consume 6 to 10 descriptors and a queue consumes one so you're effective number of subscribers on an old Linux system would be around 100 to 150 - really dependent on the number of sessions maintained on active tunnels. If you exceeded that 1024 file descriptor limit, the hub code would crash eventually because of the buffer overrun that would happen i tries to put more than 1024 bits of information into space for 1024.

    The 7.97 and newer hubs use epoll() which does not have this limit and personally I've pushed a version of this hub to over 1400 subscribers with satisfactory results. I don't do that now but it needed to be pushed to see what happened.

    So it's a lot of words but there's a huge benefit to having that tunnel server be Linux on a very recent hub version (we use 7.97 on CentOS 7 for instance) but Windows for the central hub.

    And when you hear about "small" environments or "large" environments, I'd make the following statement:
    Small = <4 hubs with tunnels. (no need fora tunnel server)
    Medium = 4 < hubs < 20 with tunnels. (think about needing a tunnel server)
    Large = 20 < hubs < 50 with tunnels. (have to have a tunnel server)
    Huge  = 50 or more hubs with tunnels. (have to have many tunnel servers and another tier of tunnel concentrators)

    Robots don't really matter - they just drive the demand on your network infrastructure and database.

    In my environment I have 3000 remote hubs, 106 tunnel servers (32 HA pairs) , 8 tunnel concentrators (4 HA pairs) and two central hubs (1 HA pair). At around 50 hubs, I was forced to move to having tunnel concentrators. At around 2000 hubs, I had to introduce the tunnel concentrators.

    And if you're thinking HA, remember that you need 2 of everything and that doubles your active tunnels so you reach that subscriber limit that much faster.

    -Garin


  • 18.  RE: Tunnel server and client

    Posted 12-09-2019 11:59 AM
    Edited by Keith Clay 12-09-2019 12:05 PM
    Can you flesh this out much more?  The primary hub is either the server or the client. What you said doesn't make any sense with the tunnel server not on a client or primary.