How to avoid agent rebalancing after initial connection

[Anon Anon]

Is there a way to aviod agent rebalance after initial connection? As the snapshot attached, this was the issue happened. Agnet was disconnected from one of our EMs and re-connected to another EM. 2 duplicated JVM instances showing up after the agent rebalancing, which caused false JVM connection alert. Is there a better way to set up JVM connection status alert to aviod this issue? We cannot keep sending false alerts for PROD env when this issue happens.. Please advise how do you set up the jvm connection status alert in your env if possible. Thanks.
Zhang

[Haruhiko Davis]

[list][*]What version are the agents?
[*]What version is your APM cluster?[list]

[Anon Anon]

9.1.4

[Haruhiko Davis]

I assume you mean for both agent and EMs?

Are your agents running in new or legacy mode?

[Anon Anon]

EMs are on9.1.4.. Some of the agents are on 9.1.4 but some of the old agents are still on 9.0.5 we are still in process of migrating those to new version.
It's running on legacy mode I think.

[Haruhiko Davis]

Please verify your 9.1.x agents are running new or legacy mode.

Are you seeing this problem with both version of agents, or just 9.0.x?

[Anon Anon]

Where can I check the running mode?
This issue happened only once after our infra upgrade to 9.1.4. I can't tell it's only for old version agent or new or both. I checked the MOM's log but I didn't see collector disconnecting msg at that time when agent disconnecting happened.

[Viruvaibhav_apm]

To check whether Agent is running in legacy mode -
1.If "introscope.agent.configuration.old=true" is appended in the IntroscopeAgent.profile &
2."introscope.autoprobe.directivesFile" directs to some legacy.pbl or.pbd file ( for example tomcat-typical-legacy.pbl )

Also, this information from Readme is useful for this particular case -
######### Excerpt from ReadMe##########
" In CA APM 9.1.x, the following capabilities are not
available when running the agent in legacy mode:

- Agent CPU usage and response time optimizations in version 9.1
- Dynamic Instrumentation for the .NET agent
- SQL Agent properties
- introscope.agent.sqlagent.sql.artonly
- introscope.agent.sqlagent.sql.turnoffmetrics"
######### Excerpt from ReadMe###########

Regards,
Vaibhav

[Anon Anon]

Hi Davis,
The agent is running on new mode.
And can you please let me know the difference agent running on legacy/new mode?

[Viruvaibhav_apm]

Hope it helps you in differencing the modes -

##############################################################
CA APM/Introscope 9.1 introduced a new agent transaction structure with new
tracers that replaces the transaction blame stacks that the old tracers
were using in previous releases. This new mode implementation optimizes
computation and processing to improve overall agent performance.
With CA APM/Introscope 9.1 customers can configure and run their agents in
the traditional pre-APM 9.1 transactional blame stacks instead of the new
transaction structure. This legacy mode of the agent is fully supported
but agent enhancements in future releases will only be implemented in the
new agent architecture. In CA APM 9.1.x, the following capabilities are not
available when running the agent in legacy mode:

- Agent CPU usage and response time optimizations in version 9.1
- Dynamic Instrumentation for the .NET agent
- SQL Agent properties
- introscope.agent.sqlagent.sql.artonly
- introscope.agent.sqlagent.sql.turnoffmetrics

Running old or legacy tracers with the APM 9.1.x agent in new mode is
known as mixed mode configuration. Do not run in mixed mode as this can
lead to additional memory consumption and service interruption. Customers
who have legacy mode tracers must also run the agent in legacy mode
until their tracers have been upgraded to the new agent mode. The following
message is written to the agent log to inform customers when they have
legacy tracers that must be updated. Revert to run the agent in legacy
mode in the interim.
"An agent tracer using legacy API s has been detected. Running legacy
tracers with the agent in new mode is not recommended. Please contact
support who can refer you to documentation on how to upgrade your legacy
tracers. In the interim, please configure the agent to use legacy mode or
use the pre-configured version of the legacy agent package. For example,
the legacy package for the Oracle WebLogic agent on UNIX is
IntroscopeAgentFiles-Legacy-NoInstallerx.x.x.xweblogic.unix.tar"
In addition, customers using the following out-of-the-box extensions must
configure their agents to run in legacy mode.

- CA APM for IBM WebSphere Portal
- CA APM for IBM CICS Transaction Gateway
- CA APM for IBM z/OS
- CA APMfor CA SiteMinder Web Access Manager
- CA APM Integration for CA LISA
########################################################################################

Regrads,
Vaibhav

[Anon Anon]

The running mode is related to agent disconnecting issue??
Got the false alert again because of the same agent re-balancing issue. I didn't see EM disconnecting from MOM.. So what is the possible reason to cause agent rebalancing?



7/10/13 05:00:00.020 AM EDT [INFO] [DefaultQuartzScheduler_Worker-1] [Manager] Starting alert downtime with id: AESEID: type=AlertDowntimeSchedule serial=1372816403683 and name [com.wily.introscope
.xml.binding.TranslatableTextType@34a963]
7/10/13 05:10:00.038 AM EDT [INFO] [TimerBean] [Manager] Ending alert downtime with id: AESEID: type=AlertDowntimeSchedule serial=1372816403683 and name [com.wily.introscope.xml.binding.Translatabl
eTextType@34a963]
7/10/13 06:10:00.035 AM EDT [INFO] [TimerBean] [Manager] Ending alert downtime with id: AESEID: type=AlertDowntimeSchedule serial=1372816403680 and name [com.wily.introscope.xml.binding.Translatabl
eTextType@61c2ea]
7/10/13 06:35:11.322 AM EDT [INFO] [Alarm Pooled Worker] [Manager.Action] Action "ESMD_ADAPTER_PROD Extended Email Action JVM Status" successfully sent SMTP mail to
7/10/13 06:37:40.185 AM EDT [INFO] [Alarm Pooled Worker] [Manager.Action] Action "ESMD_ADAPTER_PROD Extended Email Action JVM Status" successfully sent SMTP mail to

[Viruvaibhav_apm]

As per my understanding, due to Agents Connection history, this mismatching is done.

As you said you had changed the EM in clustered Env, because of which your MOM directs your agents to two of the collectors. ( i might be wrong but it seems so )

So what you can do is -
# Open Loadbalancing.xml file
#Set <include> statements to allow agents to connect to a specific set of Collectors.
#Set <exclude> statements to disallow agents from connecting to specific set of Collectors. ( Here you can set your previous EM if its not used in loadbalancing)
#Set the latched property to true to connect agents to a specific preferred Collector within a set of allowed Collectors.

In this way, you are guiding your MOM that which collectors are been used for which agents.

Hope it works for you!

Regards,
Vaibhav

[Anon Anon]

I did use <include> and <exclude> in the loadbalancing.xml file. I have 4 collectors and I include 3 of collectors to connect java and ep agents the exclude one collector for TIM services only.. What I want to know is why the agent disconnecting from the collector? As the attached snapshot above, I didn't see any error from MOM's log and collector. The load is averaged across our collectors . The agent shouldn't have disconnected from the collector.

[Haruhiko Davis]

Let us know if we can close this discussion.

 

The messages you posted originally have nothing to do with agent load balancing. Those were messages about an alert downtime schedule and custom email alerts being triggered.