DX Infrastructure Manager

Expand all | Collapse all

Deleted Hubs keep coming back

  • 1.  Deleted Hubs keep coming back

    Posted 02-25-2020 10:27 AM
    Edited by Daniel Blanco 02-25-2020 10:33 AM
    So this has been more of a problem after upgrading to 9.2.0 I know its an issue with this version.
    I've noticed that we are constantly having this issue where deleted hubs, renamed hubs keep re-appearing in the HUB list over and over until we do a full Nimsoft Stop, delete the hubs.sds file then start.
    Yesterday I renamed a client hub, got it up and running under the new hub name. I then removed the GET queue references on its tunnel server to grab from the new hub name and also opened the primary hub, and all 6 tunnel servers and the hub collector hubs probe and in the HUBs tab did a remove on the old client hub name. Within 5mins the old client name re-appears in the Primary Hubs, Hub Probes Hub tab. It's still there today.

    Is there a way to fix this w/o having to do the hubs.sds delete procedure? Again this has been happening only after we upgraded to 9.20 from 8.51 back in Oct 2019.

    It's not the actual old hub with the robots under it. The robots are under the new hub that was renamed. It's more like a ghost reference to the old hub's name just re-appears in the HUB tab list and also in the IM and Admin console.

    ------------------------------
    Daniel Blanco
    Enterprise Tools Team Architect
    DBlanco@alphaserveit.com
    ------------------------------


  • 2.  RE: Deleted Hubs keep coming back

    Posted 02-25-2020 10:54 AM
    This behavior has always existed for us. Engineering has said that there's no way to fix it with the way it is currently designed. You have the behavior because every hub has a copy of what it's network looks like and every time there's a change, a hub detecting a change sends a copy of it's version of the network. You eventually reach a number of hubs or network delay or software responsiveness where you can no longer get one change propagated everywhere before another change happens to start a competing update. There have been some changes to control which servers get the messaging - I could be wrong but I think that one of the changes that might be in 9.20 is to correct the determination of "adjacent hub" which might be why you are seeing the issue now and not before.


  • 3.  RE: Deleted Hubs keep coming back

    Posted 02-26-2020 12:49 PM
    Thanks Garin for the reply. I know you've mentioned this issue many times before especially in your environment with 5000+ hubs (maybe more)

    Anyone from Broadcom who can comment on how to fix this w/o having to do the hubs.sds delete on the primary hub method.

    When I try to fix this I have 8 hub probe GUI's open (PriHub, HubCol, and the 6 Tunnel Server Hubs) and on the HUBs tab I hit F5 and then on the broken hub entry that keeps re-appearing I scroll over to the Source column to see where its coming from.
    Its a mess!
    PH, TSH4 say that the ghost hub's source is from TSH1
    TSH1 doesn't even list the ghost hub - ???
    TSH2 lists it but says its source is from TSH4

    I did the stop nimsoft, delete hubs.sds on TSH1, start and thought it would fix this. Nope...

    Aren't hubs supposed to be 'removed' from the tree after X # of hours if they don't 'checkin'?





    ------------------------------
    Daniel Blanco
    Enterprise Tools Team Architect
    DBlanco@alphaserveit.com
    ------------------------------



  • 4.  RE: Deleted Hubs keep coming back

    Posted 02-26-2020 01:05 PM
    I've just given up on getting old hubs out of my display - it just doesn't work.

    And to your "Aren't hubs supposed to be 'removed' from the tree after X # of hours if they don't 'checkin'?" question, from what I have learnt, the hubs don't pay attention to when the information created, only the time that they acquired it. So if you are in the situation where you have an old hub getting passed around to other hubs based on unrelated changes getting propagated, the "age" of that information is stamped when it is incorporated by the hub learning about it.

    The couple things that I've discussed with Broadcom about fixing this: Create a blacklist - that way when a hub is accepting a hublist update it could filter that based on the blacklist and not accept data about hubs you intentionally exclude (analogous to the "don't rediscover" option in discovery_server). Add a timestamp to the hub information that's passed along with it so that a receiving hub knows when the information came into being, not just when it was received. Move to a DNS style name resolution method to eliminate the need for every hub to know about every other hub all the time. Most hubs only need to know about the upstream tunnel server hub and nothing else. This is a huge change but it would improve so much long term.