Collector Load-balancing

[Steve_Oak]

Over the last several releases I have noticed more of what I would call Collector-based load-balancing. For example when a Collector goes away the other collectors notify the agent of the available collectors:

 

2/06/17 07:09:06.116 AM EST [INFO] [PO:main Mailman 2] [Manager.LoadBalancer] Allowed collectors list {30.128.190.97@5001, 30.128.133.51@5001, 30.128.190.88@5001, 30.128.133.51@5001, 30.128.132.31@5001, 30.128.133.51@5001, 30.128.132.47@5001, 30.128.133.51@5001, 30.128.133.30@5001, 30.128.133.51@5001, 30.128.190.33@5001, 30.128.133.51@5001, 30.128.190.34@5001, 30.128.133.51@5001, 30.128.190.87@5001, 30.128.133.51@5001, }@1486382946140 sent to agent mom9plvwbs002|WebSphere|PROD_C_CELL/Service_Server_01A

 

What I have noticed is that even if I specify that agents should not go to a specific collector and the weight of the collector is set to 0 (others are set and either 1500 or 2000) I am still getting agents on the collector since it is still included in the above list (with no weighting).

 

I ultimately want to remove the collector 30.128.190.97@5001 from the cluster and want it to not server any new agent metrics and want it to just server history for a period of time before removal. Right now I have set in the loadbalaning.xml:

 

<agent-collector name="va10p5069x">
        <agent-specifier>.*\|.*\|.*</agent-specifier>
        <exclude>
            <collector host="VA10P50696" port="5001"/>
            <collector host="30.128.190.97" port="5001"/>

         
        </exclude>
    </agent-collector>

 

And my MoM has

 

introscope.enterprisemanager.clustering.login.p50696.host=30.128.190.97
introscope.enterprisemanager.clustering.login.p50696.port=5001
introscope.enterprisemanager.clustering.login.p50696.weight=0
introscope.enterprisemanager.clustering.login.p50696.publickey=internal/server/EM.public

 

 

Is there an property in the EM Properties that I can set in the Collector to say not to participate in the collect/agent level communication.

 

BTW this is a 10.2 cluster and this collector was serving agent and I thought that changing above would take the agent off this server and move then to other collectors in the cluster.

 

Over the weekend I may shut the server down for an hour so to see when it comes back up if it will go thru the MoM load-balancing and not take any new agents....

[Hallett German]

Dear Steve:

Not sure if this page will help . But is a good summary of APM EM-agent connection states. This is for 10.2

 

How Configuration Affects Agent to Enterprise Manager Connections - CA Application Performance Management - 10.2 - CA Te… 

 

Thanks

Hal German

[Hallett German]

Also submitting this recent KB on Load Balancing Behavior that may help with future questions

 

https://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.TEC1645768.html 

[Steve_Oak]

Yes, this is the way I understand the normal MoM based load balancing. But what I am seeing is that I went to the MoM and changed the weight to 0 (from 1599) while the other collectors that I want to use have either 1500 (virtual servers) or 2000 for our physical boxes. I then put a statement in the loadbalaning.xml, as above, excluding this collector and a final include for all the other boxes.

 

I thought, as documented, that normal load balancing would take the agents from this server and move them to one of the others. However, I am still seeing agents (I made this change over the last weekend) if fact am seeing more agents being added and I think is it because of it being included in the available collector list being sent to the agents.

 

Anyway, over the weekend I will shut the EM down for enough time for the MoM to get the agent completely redistributed and see when I bring it up if is will remain new agent free and just server historical metrics.

[Hallett German]

Dear Steve: 

    What did you find out over the weekend ? Marking as answered since responses were given and there were no action items at this time. We are glad to assist further if needed based on your findings. This may be through a case.

 

Thanks 

[Steve_Oak]

I found in the EM Properties:

# There is an optional weight property to control the relative load of the collector:
#     weight: the load balancing weight for this collector EM
#
# All other things being equal, the percentage of metric load assigned to the collector
# is the weight divided by the total weight for the cluster.
# For example, if the weight of a collector is 50 and the sum of the weights of all collectors is
# 250, the collector should get 20% of the cluster's metric load.
#
# The weight should be a positive integer.
# Default is the average weight for the cluster.
#

 

So,with:

introscope.enterprisemanager.clustering.login.p50696.weight=0

 

since zero not a "positive integer" I think that it is getting the average weight of the cluster. So restarting the collector did not help but changing to:

 

introscope.enterprisemanager.clustering.login.p50696.weight=1

 

Did make a major change. However, I still have a few agents reporting. I think if I raise the weight of the other collectors high enough I could get those out...

 

I do not want to start a case on this since I do not want to really spend too much time on 10.2 since my goal is to be at 10.5.1 (hotfix) as soon as possible.

 

Thanx for the help...

[Hallett German]

Thanks Steve. Please keep us informed on your progress and if you need any help.