The Hub *** has to many active subscribers (165).

[Anon Anon]

Keep getting the following alarm notification (blue):
"The Hub *** has to many active subscribers (165). You should consider to offload some of the subscribers to another Hub."

What's causing this? Has it something to do with the fact that we've added/connected all "satellite hubs" as Name Services (Static Hubs) on one central hub?

[Anon Anon]

There are 2 kinds of subscribers:

• Queues, which are configured explicitly on the hub or other hubs
• Temporary subscribers, which listen for certain messages but do not have queues

I believe the warning message refers to both of these combined.  If you look at the Status tab in the hub properties GUI, you should see a list of all subscribers there.

So this probably has something to do with the hubs you are connecting to your central hub, but not the name services.  This would be specific to the queues.

We also have an environment with a central hub connected to multiple satellite hubs.  When we ended up with too many subscribers on our central hub, we had to add a layer in our architecture.  Now our root hub connects to multiple intermediary hubs (also in the central location), which in turn connect to the satellite hubs.

Based on the information I have received, you want to keep the subscriber count limited to around 60 on a single hub.  (Apparently bad things can start to happen when you reach about 64; we had issues right around that number.)

Regards,
Keith

[Anon Anon]

Here's the reason for the 64 queue "limit" for anyone (can't be too many?) who wants to know:

Microsoft has a limit of 64 objects you can check on to see if they need attention. That's the WaitForMultipleObjects() function which we use to figure out which queue has messages it needs to send. Once you reach 64 objects you then have to split your list of objects in two, check the first list, then if nothing is ready check the second list. This is disgustingly inefficient since there is a timeout involved in the call to WaitForMultipleObjects, meaning that anything on the second list has to wait until there is nothing to send in the first list, the timout happens and then it gets some attention. The result is that your data throughput drops dramatically!

In the example shown (165 queues) it gets even worse...the list is split in three! What this means in effect is that a queue on the third list has to wait "forever" before it will get attention, and will not be able to efficiently send out messages.

There is code in the hub to space out where a queue is on the list (once it has had some attention it drops to the bottom of the list(s) and everyone else moves up) so data does flow, just at a trickle instead of a flood

-Stian

[Anon Anon]

Picture_15.png - It looks like it's well within the 64 object limit, so I guess I've misconfigured something else..

Picture_16.png - Maybe this is the root cause? Instead of "connecting" to the central hub like this, should I place one NAS in each satellite location (connecting to it's respective local HUB) and then use NAS "Forwarding & Replication" to send alarms/messages back to the central location?

Appreciate all your help so far!

[Anon Anon]

It seems like I've solved it by adding a NAS to each external location and then set up a unidirectional forwarding rule back to the central NAS. The "Hub *** has to many active subscribers (165)" message has disappeared.

However, we discovered another problem after doing this. There seem to be a longer delay in receiving/clearing the alarms. Is there a "polling interval" I can try changing?

Regards,
Fredrik Holmberg

[Anon Anon]

Now we're receiving several instances of this message "Auto-adjusted replication blocksize to 1000, this could indicate a slow transfer rate to 'hubs'." Multiple hubs seem to be complaining..

These entries show up in the central NAS log:
May  7 14:47:26:062  nas: replPostQueue 'Hub1' has been disconnected.
May  7 14:47:42:296  nas: replPostQueue 'Hub2' has been disconnected.
May  7 14:47:58:531  nas: replPostQueue 'Hub3' has been disconnected.
May  7 14:48:14:593  nas: replPostQueue 'Hub4' has been disconnected.

[Anon Anon]

Fredrik,

The screenshot in Picture_15.png was taken when you were getting the alarm about active subscribers?  That is very surprising; I expected to see a full list there!

How were alarm messages getting back to your central hub?  It looks like you were not using get/attach queues, so were you using post queues?  I have not used post queues much at all, so I am not sure if they might count as subscribers but not be listed in the hub GUI.

When it comes to forwarding between NAS probes, I am under the impression that updates are supposed to be forwarded more or less in real time, rather than being sent on a specific interval.  I could be wrong about that.  I do not see any configuration options in the GUI or in the config file that look like they might help.

It sounds like your NimBUS environment is somewhat similar to ours, and we have had a very good experience with the central NAS model.  (We do have 2 NAS probes installed for DR purposes, but they are both in central locations.)  We had to add a middle tier of hubs to support so many queues, but it has worked very well.  I was recently informed that the Linux hub should not have the same performance issue as the Windows hub, so we want to test that.  If it works out, we will convert our middle tier of hubs to Linux and support more queues with fewer servers.

If you can explain your architecture a bit more, such as how your hubs are setup and what kind of queues they use, maybe we can make more sense of what you were seeing in that alarm.

Regards,
Keith

[rtirak]

Hi Keith I was wondering if you had a visual diagram of this setup you are describing. We are currently starting just about maxed out with our subscribers. We have 1 primary hub and 7 secondary hubs. We are currently trying to come up with the most efficient way to free up subscribers? Any advice? Thanks.

[Sam Green]

We have also received the same error recently having never seen it before.

 

We run 1 Primary NMS Server and one HA with 17 external customer hubs (All linux) using get/attach queues.

 

We do have the added complication where we use "Remote App" (a Citrix style implementation) and use a thin client version of Infrastructure Manager.  So Users who just click the X don't log off and their session remains active.

 

I've been tempted to try a multi NAS environment many times but never had a real reason to try it.  I'm tempted to test it out by deploying a NAS to each customer Hub, this would take the strain off my central NAS by distributing the rules across each customer.  This would also make alarm_enrichment a little easier.

[Anon Anon]

Hi,

 

We thank you for your request "Re:  - Re: The Hub *** has tomany active subscribers (165).".

 

Your ticket id:"238031".

 

Please keep the subject line if you want to reply to this mail.

 

Best Regards,

 

IT Drift Marcello

 

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[Anon Anon]

Hi,

 

We thank you for your request "Reply marked as helpful on The Hub *** has to many activesubscribers (165).".

 

Your ticket id:"238033".

 

Please keep the subject line if you want to reply to this mail.

 

Best Regards,

 

IT Drift Marcello

 

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[Daniel Blanco]

We had this:

 

Pri-Hub (nas)

^^^^

4 Tunnel Servers

^^^^^^^

Client Tunnel Client Hubs

 

This was hitting the subscriber limit and we had to tell folks to log off the primary hub and use the tunnel servers when they needed to use IM. We have 4 get queues from the Primary to each Tunnel Server (Alarm, QoS, Audit, Discovery) but when you factor in Baseline & Prediction that's almost 6 queues to each tunnel so that's 24 links from Pri-Hub to each Tunnel + everyone logging into the Pri-Hub = Way over subscribed.

 

To quickly check select your Primary Hub probe and Hit Ctrl+P, then in drop down "list_subscribers" Scroll all the way to the bottom and that's the # of subscribers.

 

We recently re-structured the overall and threw in a middle hub between the Pri-Hub and the Tunnel Servers.

 

Pri-Hub (nas)

^

Hub Collector

^^^^

4 Tunnel Servers

^^^^^^^

Client Tunnel Client Hubs

 

Now our Primary only has the 4 get queues to the Middle Hub and then middle hub has all the gets to the tunnel servers. Its been running 100% better since. All middle hubs are CentOS and Primary is Windows.

We also only have the nas probe on the primary hub. Our subscriptions to the primary averages in the 40's when everyone is on.

 

We also hit a limitation on the tunnel servers. We expected that the Linux version of the hub would support many more client server hubs but no. We found that anything more than 30 would hit issues. Resource wise on the box was minimal bu the hub probe just could not keep up. We now limit each to 25 to be safe.

[Anon Anon]

Hi,

 

We thank you for your request "Re:  - Re: The Hub *** has tomany active subscribers (165).".

 

Your ticket id:"238108".

 

Please keep the subject line if you want to reply to this mail.

 

Best Regards,

 

IT Drift Marcello

 

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[Anon Anon]

Hi,

 

We thank you for your request "Reply marked as helpful on The Hub *** has to many activesubscribers (165).".

 

Your ticket id:"238138".

 

Please keep the subject line if you want to reply to this mail.

 

Best Regards,

 

IT Drift Marcello

 

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[Sam Green]

Very interesting read, thanks for sharing

 

If you get a moment, could you do a network diagram? I'd be interested to see it!

[Anon Anon]

Hi,

 

We thank you for your request "Re:  - Re: The Hub *** has tomany active subscribers (165).".

 

Your ticket id:"238139".

 

Please keep the subject line if you want to reply to this mail.

 

Best Regards,

 

IT Drift Marcello

 

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[Daniel Blanco]

Why am I not seeing any of the pictures mentioned in this thread?

Also fredrik1-d do you have an auto reply to every post here? All I'm seeing is:

Hi,

We thank you for your request "Reply marked as helpful on The Hub *** has to many activesubscribers (165).".

Your ticket id:"238138".

Please keep the subject line if you want to reply to this mail.

Best Regards,

IT Drift Marcello

This is auto added to every post here.

[Anon Anon]

Hi,

 

We thank you for your request "You have been mentioned by Daniel Blanco in Re: The Hub *** has tomany active subscribers (165). in CA Communities".

 

Your ticket id:"238142".

 

Please keep the subject line if you want to reply to this mail.

 

Best Regards,

 

IT Drift Marcello

 

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[Anon Anon]

Hi,

 

We thank you for your request "Re:  - Re: The Hub *** has tomany active subscribers (165).".

 

Your ticket id:"238143".

 

Please keep the subject line if you want to reply to this mail.

 

Best Regards,

 

IT Drift Marcello

 

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[B.Nikhil]

Hi

 

We have also hitting the max 64 limit and experiencing slow alert processing. Ours is 1 central hub and 23 remote hubs . I would like to know what all changes need to done for offloading the subscribers on to new secondary hub. We are planning to introduce linux machine as distributed secondary hub , but still searching for a document on how to offload some subscribers on to the new one . I’d love to hear your advice on this one. 

[Anon Anon]

Dear User,

 

Thank you for your e-mail, it has been registered with ticket id: "2132".

 

If you are a customer with a response contract and you have an urgent or critical request please

place a direct call to: [phone]

 

All request will be answered consecutively during our opening hours which is 24/7-365.

If you have any further information/questions, please reply to this e-mail and keep

the subject field unchanged.

 

Best Regards,

Marcello Technology

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no

[Sam Green]

As I mentioned earlier, we had the same "number of subcribers" alert so I introduced a new layer of remote hubs using distributed architecture.  

 

We have more than 3 customer hubs - this is just a representational image.

 

Linux can handle 100 connections (queues) and Windows can handle 64.  Our Users still log directly onto our Primary NMS Server, and so do our dashboards.  So by splitting these over 2 servers our connection count has more or less halfed.

 

However today I 've recieved an alarm stating that my new Primary Remote server hit 65 connections yesterday, looking at the list_subscribers we have 37, which is odd.  I'm not sure how it reached the limit..

 

I also set the QoS and Audit messages to use the same queue, so we only have Alarm & QoS/Audit queues coming in from the customer hubs.  All these connections are concentrated down to 3 queues as they pass through the Primary Remote.

 

[Anon Anon]

Dear User,

 

Thank you for your e-mail, it has been registered with ticket id: "113193".

 

If you are a customer with a response contract and you have an urgent or critical request please

place a direct call to: [phone]

 

All request will be answered consecutively during our opening hours which is 24/7-365.

If you have any further information/questions, please reply to this e-mail and keep

the subject field unchanged.

 

Best Regards,

Marcello Technology

 

Marcello Consulting | Gjerdrumsvei 4 | 0484 Oslo

Mail: [e-mail] | www.marcello.no