DX Unified Infrastructure Management

 View Only
Expand all | Collapse all

monitoring heartbeat for tunnel server/client hubs

  • 1.  monitoring heartbeat for tunnel server/client hubs

    Posted Sep 15, 2020 06:52 AM
    We recently had 2 DC's (2 hubs) down for 30 mins but we did not receive any alert. robot inactive is set to 2hr , neither hub nor robot alert received .the hub and associated robots were down for 40 mins, 
    How can i set the hub heartbeat monitoring?

    ------------------------------
    HCL Technologies
    ------------------------------


  • 2.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 15, 2020 07:49 AM
    Edited by Luc Christiaens Sep 15, 2020 07:57 AM
    net_connect check on port 48002/48003 for all hubs is a first step


  • 3.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 15, 2020 12:47 PM
    Should 48003 connect from tunnel clients ?
    port 48003 inbound/outbound is usually open from tunnel server to tunnel client , should it be bidirectional?

    ------------------------------
    HCL Technologies
    ------------------------------



  • 4.  RE: monitoring heartbeat for tunnel server/client hubs

    Broadcom Employee
    Posted Sep 15, 2020 01:24 PM

    So as any hub can act as both a tunnel server. Which has 48002 as inbound.
    and as a tunnel client, which needs 48003 outbound, support always suggests that it be open bidirectionally.

    But based on your environment usually, the firewall only needs to be set up to allow 48003 inbound when a hub is acting as a tunnel server.



    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 5.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 17, 2020 02:48 PM
    Hi Gene,

    We have a tunnel server created through which we have 20 tunnel client hubs connecting.the port 48003 is allowed inbound on tunnel server.
    so from tunnel client the tunnel server wont resolve on port 48003 .
    also if the hub/controller is down on tunnel client hub, would it send alert from net_connect if i setup port 48002 for monitoring ?

    ------------------------------
    HCL Technologies
    ------------------------------



  • 6.  RE: monitoring heartbeat for tunnel server/client hubs

    Broadcom Employee
    Posted Sep 17, 2020 03:14 PM

    so usually net_connect would be installed on another hub usually upstream.

    If you are using tunnels you should not have both ports 48002 and 48003 open between the hubs, so something here does not sound like it is setup correctly.

    so if you can not telnet from the tunnel client to the tunnel server on port 48003 you need to have your network team determine why.
    most likely there is some firewall between the two blocking the communication especially if other client hubs are connecting to the tunnel server ok.

    if your tunnel server is a Windows Server you might be running into a subscriber issue and it can not accept any new connections.
    hub subscriber limits (Knowledge Base Articles - 33649)

    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=33649



    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 7.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 18, 2020 05:05 AM
    Edited by Sid Sep 18, 2020 05:06 AM

    Hi Gene, 
    I think you got me wrong, 
    We have installed hub and created that hub as tunnel client. On the local server hub probe would obviously work on 48002, and they are connecting to a tunnel server on port 48003 ..(tunnel server has inbound 48003 open for tunnel client server )
    Also none of these servers tunnel server or tunnel client or joined in DNS, should that affect any robot/hub/tunnel capability ?

    Microsoft Windows [Version 10.0.14393]
    (c) 2016 Microsoft Corporation. All rights reserved.
    C:\Users\Adm_TGITS>nslookup frhdstd-apdxm01
    Server: dn-ind-par-zic-zsi.corp.local
    Address: 10.27.XX.XX

    *** dn-ind-par-zic-zsi.corp.local can't find frhdstd-apdxm01: Query refused
    DNS not oinging with hostname

    C:\Users\Adm_TGITS>ping dn-ind-par-zic-zsi.corp.local
    Ping request could not find host dn-ind-par-zic-zsi.corp.local. Please check the name and try again.


    DNS pinging with IP
    C:\Users\Adm_TGITS>ping 10.27.XX.XX

    Pinging 10.27.XX.XX with 32 bytes of data:
    Reply from 10.27.XX.XX: bytes=32 time<1ms TTL=64
    Ping statistics for 10.27.XX.XX
    Packets: Sent = 3, Received = 3, Lost = 0 (0% loss),

    Also, can a hub work as tunnel server and tunnel client both as the same time for different servers ?



    ------------------------------
    HCL Technologies
    ------------------------------



  • 8.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 18, 2020 05:28 AM
    This is very frequent 

    Sep 16 13:18:42:634 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 16 13:18:43:642 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 16 13:18:43:652 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 13:19:04:032 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 13:19:04:033 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 16 13:19:06:044 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 13:19:06:056 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 13:19:16:619 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 13:31:07:377 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 13:31:17:626 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 13:39:09:974 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 13:39:09:985 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 13:39:18:276 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 13:47:11:403 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 13:47:18:802 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 13:55:12:384 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 13:55:13:758 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 14:03:15:009 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 14:03:15:020 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 14:03:24:203 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 14:11:16:460 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 14:11:23:176 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 14:25:17:794 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 14:25:27:070 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 14:33:19:403 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 14:33:29:577 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 14:41:20:301 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 14:41:26:699 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 14:49:21:110 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 14:49:28:914 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 14:57:22:113 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 14:57:29:162 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 15:05:23:014 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 15:05:29:399 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 15:13:25:628 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 15:13:25:628 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 15:13:34:061 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 15:25:27:116 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 15:25:33:437 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 15:33:29:731 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 15:33:29:741 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 15:33:38:276 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 15:41:31:115 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 15:41:37:485 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 15:49:32:317 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 15:49:40:239 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 15:57:33:100 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 15:57:41:569 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 16:05:34:540 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 16:05:45:998 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 16:13:34:069 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 16 16:13:35:421 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 16:13:45:159 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 16:21:36:187 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 16:21:43:142 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 16:29:37:252 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 16:29:43:740 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 16:37:38:649 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 16:37:44:186 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 16:45:40:097 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 16:45:46:486 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 16:53:42:624 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 16:53:42:634 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 16:53:51:462 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 17:01:44:044 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 17:01:51:005 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 17:09:45:175 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 17:09:54:645 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 17:17:47:830 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 17:17:47:845 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 17:17:56:978 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 17:31:49:325 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 17:31:51:564 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 17:39:50:657 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 16 17:39:51:115 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 17:40:01:361 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 17:47:52:532 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 17:47:59:114 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 17:55:55:085 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 17:55:55:095 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 17:56:04:685 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:03:56:374 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 18:04:04:065 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:11:58:010 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 18:11:58:019 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 18:12:07:338 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:19:59:398 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 18:20:05:724 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:28:00:273 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 18:28:08:422 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:36:03:021 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 18:36:03:034 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 18:36:12:033 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:43:19:889 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 18:43:48:457 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:43:49:488 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 16 18:44:04:452 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 18:44:15:009 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:52:05:087 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 18:52:15:066 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 18:59:51:546 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 16 19:00:06:091 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:00:14:079 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 19:08:08:706 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 19:08:08:717 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:08:17:009 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 19:16:10:175 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:16:18:569 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 19:24:11:074 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:24:21:025 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 19:32:12:525 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:32:19:177 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 19:40:14:097 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 19:40:14:107 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:40:23:549 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 19:48:15:031 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:48:21:441 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 19:56:16:102 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 19:56:22:530 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:04:17:096 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 20:04:22:316 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:12:18:739 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 20:12:18:751 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 20:12:28:057 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:19:35:474 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 20:20:05:189 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:20:06:210 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 16 20:20:20:227 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 20:20:31:426 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:28:21:700 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 20:28:30:193 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:36:23:098 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 20:36:33:606 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:44:24:072 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 20:44:31:473 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 20:52:27:604 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 20:52:27:617 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 20:52:35:952 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:00:29:138 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:00:35:246 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:08:30:687 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 21:08:30:697 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:08:40:943 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:16:32:184 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:16:38:607 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:24:34:762 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 21:24:34:762 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:24:43:919 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:32:36:253 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:32:42:189 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:40:38:745 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 21:40:38:756 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:40:47:438 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:48:40:180 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:48:46:572 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 21:56:41:179 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 21:56:47:128 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 22:04:42:641 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 22:04:46:041 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 22:12:43:174 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 16 22:12:44:311 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 22:12:54:766 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 22:20:45:358 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 22:20:52:306 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 22:28:47:065 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 22:28:55:790 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 22:36:48:022 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 22:36:58:274 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 22:44:49:043 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 22:44:51:477 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 22:52:50:658 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 22:52:50:672 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 22:53:00:239 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:00:52:135 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:00:59:155 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:08:53:779 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 23:08:53:790 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:09:03:016 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:16:55:246 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:17:01:147 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:24:56:214 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:25:05:062 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:32:57:000 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:33:07:082 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:40:58:459 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:41:04:873 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:49:00:037 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:49:05:444 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:56:01:877 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 16 23:56:36:568 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 16 23:56:37:595 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 16 23:57:01:107 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 16 23:57:11:167 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:05:02:526 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 00:05:04:949 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:13:35:742 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 00:13:41:266 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:21:38:251 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 00:21:38:262 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 00:21:47:021 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:29:39:120 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 00:29:45:280 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:37:16:547 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 00:37:16:547 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 00:37:17:548 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 00:37:25:894 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:37:40:297 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 00:37:46:208 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:44:26:107 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:44:27:133 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 00:45:41:201 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 00:45:45:592 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 00:53:42:702 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 00:53:42:712 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 00:53:52:136 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:01:44:273 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:01:50:307 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:09:21:882 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:09:21:882 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:09:37:886 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 01:09:45:892 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 01:09:45:907 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:09:49:349 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:17:47:519 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:17:56:382 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:25:47:096 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:25:47:096 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:25:48:100 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 01:25:48:100 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:25:49:521 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:33:49:174 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:33:58:686 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:41:18:549 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:41:18:549 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:41:34:554 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 01:41:49:731 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 01:41:49:732 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 01:41:50:077 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:42:01:210 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:49:51:079 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:49:57:529 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 01:57:28:289 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:57:28:289 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 01:57:44:337 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 01:57:53:678 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 01:57:53:688 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 01:58:02:292 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 02:05:55:202 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 02:06:01:030 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 02:13:55:816 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 02:13:56:050 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 02:14:06:907 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 02:21:57:470 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 02:22:04:214 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 02:29:59:098 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 02:29:59:108 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 02:30:08:631 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 02:38:00:165 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 02:38:06:536 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 02:46:01:885 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 02:46:07:075 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 02:54:03:263 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 02:54:10:037 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:02:05:813 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 03:02:05:823 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:02:14:731 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:10:07:222 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:10:13:415 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:18:08:328 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:18:14:082 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:26:09:004 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:26:15:050 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:34:11:572 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 03:34:11:581 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:34:20:502 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:42:12:149 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:42:18:447 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:50:14:802 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 03:50:14:812 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:50:23:086 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 03:58:16:238 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 03:58:23:344 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 04:06:18:903 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 04:06:18:913 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 04:06:27:501 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 04:14:20:037 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 04:14:27:432 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 04:26:22:172 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 04:26:22:183 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 04:26:31:497 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 04:34:23:380 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 04:34:30:036 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 04:42:25:938 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 04:42:25:948 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 04:42:34:280 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 04:50:27:178 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 04:50:33:097 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 04:58:28:154 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 04:58:34:519 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 05:06:29:114 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 05:06:37:267 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 05:14:30:013 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 05:14:36:421 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 05:22:31:645 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 05:22:31:659 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 05:22:31:839 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 05:22:41:607 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 05:30:33:088 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 05:30:39:477 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 05:38:34:496 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 05:38:41:015 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 05:46:36:025 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 05:46:44:400 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 05:54:37:074 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 05:54:40:516 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:02:38:680 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 06:02:38:689 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:02:48:544 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:10:40:131 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:10:41:504 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:18:43:691 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 06:18:43:700 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:18:51:570 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:25:58:080 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 06:26:21:701 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:26:22:717 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 06:26:45:134 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:26:54:430 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:34:46:066 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:34:56:051 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:42:47:099 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:42:55:569 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:50:47:601 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 06:50:48:062 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:50:58:010 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 06:58:49:493 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 06:58:50:460 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 07:06:50:232 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 07:06:50:242 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 07:07:00:059 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 07:14:51:228 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 07:14:55:471 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 07:22:52:862 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 07:22:52:873 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 07:23:02:029 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 07:30:54:134 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 07:31:00:016 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 07:38:55:147 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 07:39:01:586 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 07:46:56:047 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 07:47:03:650 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 07:54:57:077 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 07:55:04:606 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:02:59:603 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 08:02:59:603 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:03:08:821 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:11:00:203 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:11:07:117 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:19:01:018 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:19:04:900 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:27:03:623 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 08:27:03:633 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:27:12:812 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:35:05:047 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:35:06:105 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:43:08:671 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 08:43:08:681 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:43:16:233 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:51:10:083 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:51:17:212 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 08:59:11:115 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 08:59:19:697 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 09:07:13:757 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 09:07:13:767 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 09:07:22:365 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 09:15:15:172 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 09:15:24:023 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 09:23:15:605 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 09:23:16:078 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 09:23:26:251 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 09:31:17:227 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 09:31:24:388 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 09:39:19:730 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 09:39:19:730 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 09:39:28:093 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 09:47:21:023 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 09:47:27:480 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 09:55:22:115 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 09:55:28:159 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:03:23:567 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:03:25:019 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:11:25:162 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 10:11:25:173 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:11:35:095 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:18:42:036 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 10:19:11:300 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:19:12:318 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 10:19:26:300 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:19:27:600 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:27:27:012 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:27:37:056 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:35:28:496 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:35:29:177 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:43:31:106 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 10:43:31:116 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:43:39:499 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:51:32:090 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:51:39:113 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:59:10:828 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 10:59:10:828 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 10:59:11:849 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 10:59:17:757 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 10:59:33:164 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 10:59:34:586 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 11:07:34:062 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 11:07:42:404 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 11:15:35:132 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 11:15:43:025 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 11:23:36:679 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 11:23:36:679 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 11:23:46:108 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 11:31:38:118 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 11:31:44:959 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 11:39:40:681 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 11:39:40:690 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 11:39:49:688 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 11:47:42:112 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 11:47:49:103 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 11:55:20:267 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 11:55:20:267 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 11:55:21:269 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 11:55:44:565 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 11:55:44:575 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 11:55:53:004 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:03:45:068 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 12:03:46:487 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:11:28:203 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 12:11:28:203 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 12:11:44:205 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 12:11:46:682 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 12:11:46:682 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 12:11:56:065 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:19:48:122 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 12:19:54:109 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:27:49:574 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 12:27:57:034 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:35:51:056 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 12:35:52:037 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:43:34:962 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 12:43:34:962 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 12:43:50:998 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 12:43:53:015 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 12:43:53:017 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 12:44:02:113 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:51:54:011 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 12:52:00:536 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 12:59:30:063 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 12:59:30:063 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 12:59:46:110 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 12:59:56:594 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 12:59:56:594 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:00:05:879 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 13:07:58:030 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:08:06:092 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 13:15:58:691 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 13:15:59:036 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:16:09:547 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 13:24:00:505 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:24:07:092 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 13:31:39:620 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 13:31:39:620 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 13:31:55:623 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 13:32:02:090 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 13:32:02:102 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:32:11:805 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 13:40:04:711 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 13:40:04:711 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:40:13:632 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 13:48:06:105 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:48:07:111 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 13:55:49:453 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 13:55:49:453 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 13:55:50:455 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 13:56:08:784 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 13:56:08:794 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 13:56:17:601 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:04:10:203 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 14:04:16:004 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:12:10:736 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 14:12:11:081 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 14:12:21:008 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:20:12:178 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 14:20:19:557 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:27:50:241 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 14:27:50:241 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 14:27:51:242 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 14:27:53:358 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:28:13:065 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 14:28:14:806 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:36:15:550 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 14:36:15:560 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 14:36:24:814 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:44:17:014 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 14:44:23:435 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 14:52:19:567 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 14:52:19:567 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 14:52:28:897 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:00:20:312 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:00:30:618 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:08:21:100 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:08:28:215 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:15:24:467 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 15:16:06:557 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:16:07:574 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 15:16:22:032 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:16:30:212 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:24:23:151 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:24:32:030 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:32:24:573 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:32:30:246 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:40:01:292 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 15:40:01:292 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 15:40:17:339 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 15:40:25:183 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 15:40:25:183 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:40:35:009 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:48:26:156 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:48:36:034 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 15:56:26:765 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 15:56:26:765 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 15:56:27:766 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 15:56:27:778 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:56:48:071 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 15:56:48:071 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 15:56:50:075 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 15:56:50:085 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 15:57:00:383 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 16:04:51:010 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 16:04:53:588 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 16:12:53:615 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 16:12:53:624 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 16:13:02:737 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 16:20:54:746 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 16:21:01:127 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 16:28:57:260 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 16:28:57:270 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 16:29:06:308 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 16:36:58:220 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 16:37:05:154 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 16:44:59:092 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 16:45:05:143 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 16:53:00:000 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 16:53:08:234 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:01:01:035 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:01:07:255 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:09:02:713 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 17:09:02:724 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:09:12:591 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:17:04:219 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:17:12:891 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:25:05:629 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 17:25:06:879 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:25:16:020 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:33:08:320 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:33:14:299 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:41:10:925 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 17:41:10:935 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:41:19:052 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:49:12:399 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:49:20:014 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 17:57:13:010 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 17:57:14:390 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 17:57:24:014 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:05:15:351 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 18:05:23:033 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:13:17:872 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 18:13:17:883 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 18:13:26:894 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:21:04:496 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 18:21:19:287 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 18:21:27:133 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:29:21:825 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 18:29:21:835 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 18:29:30:515 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:36:59:626 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 18:37:23:212 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 18:37:33:238 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:45:23:831 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 18:45:24:064 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 18:45:34:107 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:52:41:074 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 18:53:04:448 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 18:53:05:464 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 18:53:25:454 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 18:53:26:835 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:01:25:064 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 19:01:26:186 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:01:37:000 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:09:27:188 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:09:35:103 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:17:29:809 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 19:17:29:825 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:17:38:930 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:25:31:235 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:25:32:242 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:33:34:746 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 19:33:34:759 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:33:42:204 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:41:36:215 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:41:44:888 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:49:36:817 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 19:49:37:049 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:49:47:460 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 19:57:38:164 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 19:57:39:440 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 20:05:39:574 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 20:05:43:965 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 20:13:40:117 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 20:13:40:128 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 20:13:50:397 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 20:21:41:475 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 20:21:50:238 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 20:29:43:081 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 20:29:43:091 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 20:29:52:483 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 20:37:44:013 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 20:37:51:415 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 20:45:22:354 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 20:45:22:354 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 20:45:38:357 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 20:45:46:574 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 20:45:46:586 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 20:45:55:559 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 20:53:48:106 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 20:53:48:116 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 20:53:57:823 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:01:49:130 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:01:55:506 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:09:26:493 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 21:09:26:493 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 21:09:27:502 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 21:09:51:661 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 21:09:51:678 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:10:00:961 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:17:53:133 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:17:54:142 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:25:55:724 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 21:25:55:724 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:26:04:017 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:33:57:177 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:34:04:015 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:41:35:988 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 21:41:35:988 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 21:41:51:991 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 21:41:57:801 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 21:41:57:802 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 21:41:58:035 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:42:03:395 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:49:59:412 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:50:05:038 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:57:36:788 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 21:57:36:788 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 21:57:37:809 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 21:57:45:256 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 21:58:00:291 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 21:58:11:826 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 22:06:01:091 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 22:06:04:728 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 22:11:19:421 [2488] 0 Controller: Port dropped: spooler 48001
    Sep 17 22:11:53:067 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 22:12:24:125 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 22:12:24:172 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 22:12:24:363 [2488] 0 Controller: stop failed (communication error)
    Sep 17 22:12:25:039 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 22:20:24:666 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 17 22:20:25:012 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 22:20:35:601 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 22:28:26:380 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 22:28:34:249 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 22:36:27:103 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 22:36:37:038 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 22:44:28:468 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 22:44:35:522 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 22:52:31:000 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 22:52:31:011 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 22:52:39:334 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 23:00:33:733 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 23:00:33:733 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 23:00:42:048 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 23:08:35:299 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 23:08:41:075 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 23:22:36:665 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 23:22:38:012 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 23:30:31:046 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 23:30:31:046 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 17 23:30:37:146 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 17 23:30:40:186 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 17 23:30:40:196 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 23:30:49:217 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 23:38:41:527 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 23:38:50:027 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 23:46:42:084 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 23:46:50:052 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 17 23:54:43:061 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 17 23:54:51:060 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:02:43:615 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 18 00:02:44:073 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:02:54:731 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:11:16:269 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 00:11:16:278 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:11:25:307 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:18:02:638 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:18:03:669 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 18 00:19:17:153 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:19:21:208 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:27:18:044 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:27:28:026 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:35:20:587 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:35:23:960 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:43:20:086 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 18 00:43:21:767 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:43:31:741 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:51:23:375 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:51:30:007 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 00:59:26:020 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 00:59:26:030 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 00:59:35:741 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 01:07:27:732 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 01:07:29:118 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 01:15:30:314 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 01:15:30:323 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 01:15:39:124 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 01:23:31:200 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 01:23:37:621 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 01:31:08:850 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 01:31:08:850 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 01:31:24:853 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 18 01:31:33:728 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 01:31:33:739 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 01:31:42:204 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 01:39:35:320 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 01:39:42:033 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 01:47:13:533 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 01:47:13:533 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 01:47:14:536 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 18 01:47:37:932 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 01:47:37:944 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 01:47:46:883 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 01:55:39:622 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 01:55:42:883 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:03:40:654 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 02:03:50:043 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:11:42:137 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 02:11:48:074 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:19:19:490 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 02:19:19:490 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 02:19:20:499 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 18 02:19:44:692 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 02:19:44:703 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 02:19:53:744 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:27:46:224 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 02:27:53:026 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:35:23:788 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 18 02:35:47:520 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 02:35:57:416 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:43:48:563 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 02:43:58:558 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:51:49:110 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 02:51:57:096 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 02:59:49:685 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 18 02:59:50:950 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:00:00:973 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 03:07:52:360 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:08:00:043 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 03:15:55:012 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 03:15:55:022 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:16:03:892 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 03:23:57:118 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:24:02:391 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 03:31:32:019 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 03:31:32:019 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 03:31:48:067 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 18 03:31:59:697 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 03:31:59:708 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:32:08:106 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 03:40:01:293 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:40:08:169 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 03:48:03:875 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 03:48:03:875 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:48:04:061 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 18 03:48:12:212 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 03:56:05:013 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 03:56:06:304 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 04:03:48:989 [2488] 0 Controller: failed to send alive to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 04:03:48:989 [2488] 0 Controller: failed to send alive (async) to hub SAFIISCENTRAL(10.27.106.55) - communication error
    Sep 18 04:03:49:990 [2488] 0 Controller: Failed to send set_hub to spooler (communication error)
    Sep 18 04:04:05:585 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 04:04:06:059 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 04:04:17:491 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 04:12:07:488 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 04:12:13:475 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 04:20:09:089 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 04:20:09:099 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 04:20:18:956 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 04:28:10:612 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 04:28:12:993 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 04:36:12:194 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 04:36:12:205 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 04:36:21:794 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 04:44:13:146 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 04:44:19:545 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 04:52:15:737 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 04:52:15:747 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 04:52:24:050 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:00:18:298 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 05:00:18:312 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:00:27:388 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:08:20:024 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:08:25:277 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:16:21:332 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:16:27:726 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:24:22:286 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:24:28:196 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:32:23:897 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 05:32:23:907 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:32:33:006 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:40:25:429 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:40:32:039 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:48:26:230 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:48:32:684 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 05:56:27:185 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 05:56:28:569 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 06:04:29:694 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 06:04:29:705 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 06:04:38:611 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 06:12:31:069 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 06:12:40:265 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 06:20:31:637 [2488] 0 Controller: cb_pacallback request failed cmd=_status to 10.27.106.55/48002
    Sep 18 06:20:32:987 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 06:20:42:988 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 06:28:34:419 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 06:28:42:545 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 06:36:36:102 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 06:36:36:113 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 06:36:45:704 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 06:44:37:025 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 06:44:41:431 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 06:52:39:531 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 06:52:39:542 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 06:52:48:482 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:00:40:149 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:00:46:058 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:08:41:752 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 07:08:41:762 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:08:51:286 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:16:43:105 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:16:49:019 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:24:45:749 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 07:24:45:759 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:24:54:172 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:32:48:412 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 07:32:48:422 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:32:57:473 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:40:50:034 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:40:56:028 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:48:51:479 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:49:00:006 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 07:56:53:012 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 07:56:59:565 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 08:04:54:260 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 08:05:00:628 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 08:12:56:810 [2488] 0 Controller: hub localhost(10.27.106.55) NO CONTACT (communication error)
    Sep 18 08:12:56:820 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 08:13:05:851 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 08:20:58:282 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 08:21:05:016 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 08:28:59:138 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 08:29:05:612 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established
    Sep 18 08:37:00:018 [2488] 0 Controller: _ProcStart - Probe 'hub' - starting
    Sep 18 08:37:06:956 [2488] 0 Controller: Hub localhost(10.27.106.55) contact established


    ------------------------------
    HCL Technologies
    ------------------------------



  • 9.  RE: monitoring heartbeat for tunnel server/client hubs

    Broadcom Employee
    Posted Sep 18, 2020 09:28 AM

    This would suggest maybe your hub is crashing.
    check windows event logs and or Linux for core files.

    You may want to uninstall the C++ run time libraries.
    uninstall the hub 
    Restart the server
    reinstall all of the c++ run time libraries
    the reinstall the hub and see if it stays up.



    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 10.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 18, 2020 11:08 AM
      |   view attached
    The issue is there for all the tunnel clients connecting to tunnel server

    ------------------------------
    HCL Technologies
    ------------------------------

    Attachment(s)

    docx
    SA.docx   39 KB 1 version


  • 11.  RE: monitoring heartbeat for tunnel server/client hubs

    Broadcom Employee
    Posted Sep 18, 2020 11:22 AM

    Ok so if I understand correctly.
    The tunnel server is going down repeatedly.
    This is causing all 30 tunnel clients to disconnect.
    The logs you keep showing are the controller logs that really do not help when dealing with the hub and tunnels.

    We can see that the controller is not talking to the hub but the controller logs can not tell us WHAT is happening with the hubs.
    Only the hub logs can do this.

    If the tunnel hub is restarting on its own then it is probably crashing.
    You should be able to determine this by watching the processes on the system to see if the PID for the hub is changing or not.

    if the hub process is crashing/restarting then yes I would Stop the robot service make a backup copy of the Nimsoft folder.
    uninstall the hub and the C++ runtime libraries, reboot then reinstall the c++ runtime libraries then reinstall the hub.

    If you left the nimfost folder in place and install to the same directory you should still have all of your settings.



    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 12.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 18, 2020 11:17 AM
    from tunnel server event logs 


    ------------------------------
    HCL Technologies
    ------------------------------



  • 13.  RE: monitoring heartbeat for tunnel server/client hubs

    Broadcom Employee
    Posted Sep 18, 2020 11:23 AM
    yup, your hub process is crashing uninstall and reinstall should fix it.

    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 14.  RE: monitoring heartbeat for tunnel server/client hubs

    Broadcom Employee
    Posted Sep 18, 2020 09:22 AM

    hi,
    So yes a hub can be both a tunnel SERVER and a tunnel Client. This is usually the case in a 3+ tier model for UIM.
    Most UIM setups for Tunnel and hub to hub communication do not rely on DNS as we store the IP address of the machines in our configuration files.
    So generally speaking not having DNS name resolution is not an issue.

    I see you tested using ping but do not see a test using telnet.
    from the tunnel client, you must be able to connect to the IP address and port of the tunnel server so for example the Tunnel server is on IP address 192.168.1.1 and using port 48003 for the tunnel.

    from the tunnel client, you should be able to do
    telnet 192.168.1.1 48003

    If you can not then there is a problem at the network level.
    This could be a firewall, NAT or something else but your network team would have to explain this.

    You can double-check the tunnel server by doing the same command on the tunnel server itself it should resolve.



    ------------------------------
    Gene Howard
    Principal Support Engineer
    Broadcom
    ------------------------------



  • 15.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 15, 2020 12:30 PM
    Why is your robot inactive set so high? 
    Open the controller probe : Setup Tab > Misc Tab and set the "Hub update interval (in minutes)" to say 7. 
    Robot down Alert is thrown after 1.5 the update value has passed w/o an update received. So if value is 7 alert is thrown after 10ish minutes of no reply. 

    In the robot.cfg this setting is:
    • hub_update_interval = 420



    ------------------------------
    Daniel Blanco
    Enterprise Tools Team Architect
    DBlanco@alphaserveit.com
    ------------------------------



  • 16.  RE: monitoring heartbeat for tunnel server/client hubs

    Posted Sep 17, 2020 10:31 PM
    You may wish to use dirscan probe with schedule.  
    And NAS profile to trigger the alerts. 
    Remember to set a clear msg for your nas profile.