What is the best strategy to upgrade an APM production environment with minimal impact?

[Legacy User]

-We are plannig do upgrade from CA APM 10.0.0.12 to last versión.

-Is it possible to avoid loss of service? Or given the change there will be loss of service.

-All CA APM components over linux platform Red Hat Enterprise Linux Server release 6.4.

 

non clustered MOM

one postgres cemdb server

six EM collectors

two MTP servers (CA6300)

one CA ADA server 

over 300 server's with PowerPacks (IIS) installed

six server's with Java Agent installed

various EPAgents with plugins

 

Thanks

[Hallett German]

This is a great topic Fernando and I am converting to a discussion.I am hoping that your fellow users can add their thoughts on this important topic. Made minor changes to title to get more replies hopefully.

[Legacy User]

Thanks Hallet

[David Richards]

In our environments you can't avoid some downtime as we have a single MOM which will need to be down at some point.

 

We have a live cluster with 16 collectors connected  to the MOM. This is basically 8 primariy collectors and 8 fail overs (we don't use MOM load balancing)

 

Typically we install the new version in test and our test environment collector is set up is same as production. We then create a tar file from that collector and use that to do the upgrades of the collectors in production, rather than using the installer. This speeds things up for us, but does require some post extraction "seds" in various config files to switch hostnames where required. The benefit here is also that if anything goes wrong, the previous version is untouched so rollback is very quick, for the collectors anyway.

[Legacy User]

Thanks Dave

[nkarthik]

Hi,

 

 It's recommended to upgrade MOM and collectors first because agents running on lower version can always connect to higher version of MOM and collector. I would suggest you to follow this approach if you don't have any test environment to replicate production like dave mentioned. There will be minimal impact where you have to restart complete setup at least once. Please do compatibility checks between MTP/ADA with latest version of APM before upgrading if you want MTP to be reported properly after upgrade of MOM and collectors.

 

1) Install latest version of MOM on a different location(Other than where actual MOM is running now). Ignore Postgres DB install here when prompted

2) Install latest version of collectors on a different location for all collector servers

3) Copy/Update necessary configuration from current running location to new installed location for both MOM and Collectors. For an example: IntroscopeEnterpriseManger.properties, domains.xml, realms.xml, tess-db-cfg.xml and other required configurations

4) Stop MOM and collectors from old running location and Start MOM and collectors from new installed location

5) Upgrade postgres DB

6) Upgrade MTP and ADA

7) Upgrade powerpacks, java agents and EP agents. I don't see any plan required when upgrading agents. you can upgrade in a parallel manner. But upgrade agents as a last step.

 

Thanks,

Karthik

[Legacy User]

Thanks Karthik

[Hallett German]

Hi Fernando:

  Were the responses supplied by Karthik and Dave helpful and address your needs?

Please let us know

Thanks

Hal German 

 

P.S Thanks Dave and Karthik for replying. This is exactly more of the type of interaction that will make this community richer! 

[Legacy User]

We have MOM with load balancing available, and certainly this generate me a little noise, I dont known if this is really necessary or if this util for this environment. The agents constantly are moving into collectors

 

[Legacy User]

I have not made previous upgrades of this platform, you can tell me your experience regarding the approximately time that upgrades each component. Our client is a bank and they use the monitoring of their services 7x24, so that lowering monitoring services for us always involves using slots outside of business hours.

[nkarthik]

Hi,

 

The complete APM MOM and collectors upgrade can be finished in around 2 hours. But agent upgrade will take days to complete.

Having said that, you can always install MOM and collectors in different location and restart services outside of business hours.

 

Thanks,

Karthik

[Legacy User]

Thank you for sharing your experiences.

[Legacy User]

The best practice is to install and run APM Database (Postgres or Oracle) on a different machine than the MOM, for scalability purposes.  I just realize that, after reading this post, it would also help minimizing down time when upgrading EM and APM database.

 

Francis