DX Infrastructure Management

Expand all | Collapse all

Hub tunnels

  • 1.  Hub tunnels

    Posted 04-30-2014 08:11 PM

    Has anyone written anything that looks and verifies if nimsoft tunnels are up? If not I will start from scratch.   I don't like the way the alert only comes in once and if someone acks it you would never know the tunnel is still down doesn't happen often but has happened so I would rather be certain.



  • 2.  Re: Hub tunnels

    Posted 05-02-2014 12:08 PM

    Hi!

     

    No only some sort of indirect test, by accecssing a controler of a robot through the tunnel

     

    cheers

    Matthias



  • 3.  Re: Hub tunnels

    Posted 05-05-2014 11:35 AM

    We have something custom, but it isn't much code to be honest. Basicly,
    a cronjob on running on a hub that triggers nimalarm with a specific
    sid, and it gets logged to a logfile after having gone thru the tunnel
    and primary hub.

     

    A script on checks to see if it's logged, and clears the alarm.

     

    We're not running it on all hubs tho, as we mainly use it to thest the
    alarm flow thru the nas and our custom build alert system.



  • 4.  Re: Hub tunnels

    Posted 05-05-2014 11:42 AM

    I check ports for hdb, spooler and controller for every robot periodically, so that verifies the tunnels as well.

     

    Also working on a probe to do that, this and other health check stuff.. I hope I can get back to it soon..

     

    -jon



  • 5.  Re: Hub tunnels

    Posted 08-14-2014 06:22 PM

    We actually implemented it this way and works fairly well. If the hub can't get to an attach queue it generally throws an alarm.

    What we put in place is an Auto-Operator rule: if the hub cannot access say the "QoS" queue from Client_*** then it generates an alarm. If the count of that alarm reaches 5 we then create a new alarm via the nas that say, "Check HUB at: Client_***" If the NOC see's this message they are to check the Client_*** hub entry in the NMS tree. If they can access then ignore and if not then examine the issue further. 



  • 6.  Re: Hub tunnels

    Posted 08-15-2014 06:22 PM

    Yea that is kind of what we ended up doing it's more reliable than the tunnel alerts IMO,  I set the queue disconneted threshhold to 5 min as most tunnels seem to recover quickly unless there is another issue.  Are tickets for tunnels being donw has went way down which means I can sleep more at night since I don't get called LOL.