IM/AC not seeing a hub but tunnels and queues are connected

[Keith Clay]

We have 2 tunnel servers and each server has an HA box.  All the same version of linux, all secured.  Linux was patched and when things came up, all came up, but the two HA boxes show up in IM/AC as red even though the tunnels and queues are up and communicating

All are running 9.20HF9.  Any idea why the tunnels would be working but not showing up in IM/AC.  Before rebooting they were fine.

[David Michel]

Did you verify that robot and hub are running at those two systems? 
You can:
verify the Nimsoft Robot Watcher service is running there
check the hub and robot logs at the HA hub
check the hub log at the working hub to see if it is communicating to the HA hub
use IM manager to see if a login to either of those hub works

------------------------------
Support Engineer
Broadcom
------------------------------

[Keith Clay]

These are HA hubs. When I go to their tunnel server, I see the tunnels and queues up and receiving data from both.  It is only IM/AC that doesn't see them.    So TunnelBHA connects to Secondary, SecondaryHA and TunnelB.  TunnelBHA is HA for TunnelB and TunnelBHA's HA probe can see that TunnelB is up.


Here is the hub log from the TunnelBHA hub.  Do you see anything that is wrong?

Attachment(s)

bha_hub_log.log   1.06 MB 1 version

[James Christensen]

I am assuming that you are logging into the primary hub with IM and AC. Using IM and opening up the primary hub probe gui, on the Hubs tab, what do the entries look like for the two hubs in question? What does the primary hub see as the source? Is it the tunnel server? Or does it show that it "thinks" it is a direct connection?

------------------------------
[Designation]
[City]
------------------------------

[Keith Clay]

the primary is a tunnel server for the secondary which is a tunnel server for this HA box.  The secondary shows the HA tunnel and the queues are sending data it's just that IM can't see them.  Before the reboot they were fine. I've compared their config to the tunnelA box they are HA for and the settings look the same.

[James Christensen]

Hi Keith,

What I'm asking about is considered the hub routing table on the primary hub which is visible in the primary hub IM configuration GUI on the Hubs tab. When you log into the primary hub with IM, the IM GUI uses the routing information from the hub probe to attempt and maintain contact with all the other hubs and robots. You can also highlight the primary hub probe and type ctrl P to bring up the Probe Utility GUI, then execute the gethubs callback command and go through the returned list of hubs to see the information of the hubs in question. The interesting value will be what shows as proximity. I believe it should be a value of 1. If it is 0, then the primary hub "thinks" it has a direct connection to that hub. Here is the output from my lab. I have a Primary and an HA hub that are peer connection (no tunnel and proximity = 0) and a secondary hub that is a tunnel connection, but is still a direct connection to the primary hub, proximity = 0. Can you run this command on your primary hub probe and provide the output? Also, try using the secondary hub for the IM login hub and see if that makes a difference.

 

Probe  : /JMC_Lab/UIM-PRI-01/UIM-PRI-01/hub

Address: xxx.xxx.xxx.xxx

Command: gethubs

 

hublist         <TABLE START>                         

   0            -                                      

      name      UIM-PRI-01                            

      domain    JMC_Lab                               

      robotname UIM-PRI-01                            

      addr      /JMC_Lab/UIM-PRI-01/UIM-PRI-01/hub    

      ip        xxx.xxx.xxx.xxx                         

      port      48002                                 

      last      1585680894                            

      license   1                                     

      temp      0                                      

      sec_on    2                                     

      sec_ver   128                                   

      password_s2                                     

      status    0                                     

      version   9.30 [Build 9.30.1209, Mar 10 2020]   

      is_primary1                                     

      proximity 0                                     

      epoc_skew 0                                     

      synch     1                                     

      ssl_mode  0                                     

      origin    JMC_Lab                               

      req_tused 0                                     

      type      1                                     

   1            -                                     

      name      UIM-HA-01                             

      domain    JMC_Lab                               

      robotname UIM-HA-01                             

      addr      /JMC_Lab/UIM-HA-01/UIM-HA-01/hub      

      ip        xxx.xxx.xxx..xxx                         

      port      48002                                 

      last      1585759720                            

      license   1                                     

      temp      0                                     

      sec_on    2                                     

      sec_ver   128                                   

      password_s2                                     

      status    0                                      

      version   9.30 [Build 9.30.1209, Mar 10 2020]   

      is_primary0                                     

      proximity 0                                     

      epoc_skew 0                                     

      synch     1                                     

      ssl_mode  0                                     

      origin    JMC_Lab                               

      req_tused 0                                     

      type      4                                      

   2            -                                     

      name      UIM-Sec-Hub01                         

      domain    JMC_Lab                               

      robotname uimwinsechub01                        

      addr      /JMC_Lab/UIM-Sec-Hub01/uimwinsechub01/h

      ip        xxx.xxx.xxx.xxx                         

      port      48002                                 

      last      1585759742                            

      license   1                                     

      temp      0                                     

      sec_on    2                                     

      sec_ver   128                                   

      password_s2                                     

      status    0                                      

      version   9.30 [Build 9.30.1209, Mar 10 2020]   

      is_primary0                                     

      proximity 0                                     

      epoc_skew 0                                     

      synch     0                                     

      ssl_mode  0                                     

      origin    UIM-Sec-Hub01                         

      req_tused 0                                     

      type      8                                      

      tunnel_ip xxx.xxx.xxx.xxx                         

      tunnel_por48007                                 

 

James Christensen

Sr. Services Architect  |  Enterprise Studio

HCL Technologies Ltd.

Mobile: [phone]  |  eMail: [e-mail]

www.hcltech.com  | www.ca.com/services

 

 

Vacation Alert:

 

[Keith Clay]

Here is a pic of one of the HA hubs hat can't be seen in IM

[James Christensen]

Was this taken from the Primary Hub? If so, the proximity value should be a 1 as it is supposed to go through the secondary hub. Can you share a picture of one of the hubs that can be seen/conected to in IM?

------------------------------
[Designation]
[City]
------------------------------

[James Christensen]

Try this - on the primary hub probe configuration GUI. Go to the Hubs tab and locate this hub. I will assume it shows as red in the GUI. right click and remove it. It should get rediscovered with the source being the secondary hub. Once the hub route is correct, then IM and AC should be able to connect to it properly.

------------------------------
[Designation]
[City]
------------------------------

[Daniel Blanco]

Hi Keith,
You might be hitting the same problem we've been having ever since upgrading to 9.20HFx.
Our Primary Hub is constantly getting out of SYNC with tunnel client hubs going RED.

To only way we found to fix things is to stop the Primary hubs Nimsoft Robot Service.
Delete the hub.sds file under nimsoft\hubs\ then start it back up. 

I have a case now open for many, many months as L2 cannot reproduce it. Case #: 20306991

We can reproduce it whenever we use the UMP to deploy robots to a Tunnel Client Hub.​ We stopped using this feature b/c it constantly broke the primary hub instantly and we could no longer access that client hub.
We could only access the broken tunnel client hub if from w/in IM we r-click on the Tunnel Server that Tunnel client hooked up to and did a LOGIN directly to the Tunnel Hub. Then we were able to access it in IM. AC still broken though.

------------------------------
Daniel Blanco
Enterprise Tools Team Architect
[e-mail]
------------------------------

[Keith Clay]

We rebooted the primary hub and they magically came back online.  Let me know how it goes.  We can produce a number of times as we are doing a large number of new installs

[James Christensen]

Hi Keith,
Glad to hear the reboot resolved this for you. To me, this would indicate that it was the hub routing table info that the primary hub had that was the problem. If you run the gethubs callback command again, you'll likely see a difference in the entry for TunnelBHA.

------------------------------
[Designation]
[City]
------------------------------