What is your source control and deployment management for APM artifacts?
There is more than a handful of files that are modified or customized within Application Performance Management installation, use, and administration.
To name a few:
EM - Configuration
custom email shell script
EM - Java Script Calculators
EP Agent (variant depending on host/OS)
Application Agent (java)
With each of these artifacts, there, hopefully, should be a deployment plan along with some sort of test/verification plan to get the changes through some sort of software life cycle within multiple environments (test, pre-production, production).
Currently we have the majority of our artifacts on a network directory with no check-in / check-out control. Our deployments are manual and custom per server.
Certainly, you could take that the APM, monitoring as not being a production impact and an acceptable loss if APM were to shutdown for some reason. But if you did, once in the past, deploy a pbd that caused issues on the production environment, the first questions usually are, was the changes tested, what is your roll-back plan, how long will production have to be down, what changed?
So, what is your APM source control and deployment plans?
There was mention of two configuration tools on the community board as of late:
Looks like the APM Studio might be able to help with the agent pbd/pbl into source control and deployment since it is a Eclipse plugin, but what about the other agents: epagent, IBM mq, webserver, browser agent?
Our environment performance agents help provide for some infrastructure performance monitoring (cpu, disk, memory, processes) and with that we have added quite a few Perl script plugins. To make our scripts portable, we have moved all host specific configuration to the APM agent user home directory and pushed the IntroscopeEPAgent.properties to the APM agent user home also. With this, we changed the EPACtrl.sh script to look for the ~/apmuser/config/IntroscopeEPAgent.properties file that has the specific plugins for that host within it.
EM Config Utility - 10.5 - command line tool
There is plenty of elements within APM with the custom host metrics to be able to really have a full featured EM control GUI that would include a graphical smartstor management. Hopefully smartstor is on the short list of being replaced with something else, but till then, the direction of being able to administrate an APM cluster enterprise managers would be very useful.
Thanks Billy for mentioning the ConfigUtil Doc. It is also documented in Knowledge Base Articles . Your feedback on it is appreciated.
So, blazing a trail here...construction ahead...
In prep for a 10.0 to 10.5 upgrade started to look at the pain points at trying to keep the changed, custom, one off files in view. At one point we had nine different epagents which was driving our system admins a bit batty. So, I devised a way to combine all of the agents and have a more configuration driven epagent, so that we have a single epagent.
IntroscopeEPA.properties use of Environment Variables
This does help start to frame the source control a bit better, just starting the fire with the epagent.
<environment performance agent> This would be a full directory structure agent with all the required files.
With a single epagent directory and all the environment or custom configuration being located within the /syshome/wilyapm/config directory, we can mass deploy the single epagent to all servers and all environments.
We would tag the tree when there was a change and then when we winscp the contents (being aware of binary for jar/so and text for basically everything else) we create a zip/tar/gzip with the APM release and tag value
So this is the epagent structure that I am planning on using within source control to manage the sea of custom and changed files.
So far, this structure has cut down on the variants and explaining what goes where such as the AIX epagent goes under epagent/production/AIX/host and that one is RedHat so it goes into the epagnet/production/RHEL/Host. But the only real difference is the existence of different plugins and the introscopeEPAgent.properties file.
When I get around to the Application agents (websphere, weblogic, .NET) and have to deal with pbl/pbd and .profile I'll try to post what I am planning on doing.
My Java agent install script will handle any issues with configuring the agent based upon environment. You can do a search for ‘installWily' on the community to view it. I also have a version of this for EPA.
If you're running Windows server with the UNIX integration, you could use the same script to modify your .NET agent profile. If you need some native, like PowerShell, that could be easily done also.
Wished I had enough permission to run an installation script on a server, but sadly, the APM admin typically gets a sandbox directory that the agent is copied into and we can make changes to things inside the directory.
If we provide a single agent directory (epagent, websphere1.6, webphere1.7, weblogic1.7) and provide simple instructions to put the contents of the zip/tar/gzip into a specific directory, then a few root command like chown, chmod, adding the EPACtrl.sh stop/start to the server start and stop, things go rather well.
Currently we have to instruct how to change the MOM host name in the application agents, but might look at externalizing that also since we already have a server conf file (/syshome/wilyapm/config/caapm_env.conf) that has a single variable with value of "PROD, NON_PROD, TEST". Gotta go see if there is a way of injecting the environment on the JVM.
Once we get into the system admins or WebSphere admins running an install on each server (+1500) now we have more effort and time. With basically using the system admin's package deployment software, we can throw an epagent on a server, have it start and we are done.
For the application agents, typically the websphere admin will script all of the agent's elements so when a server is created everything is already in place ready to go.
Another issue we will be facing that I'm still noddling, is how to inject custom pbd within an agent dependent on which application server software and which applications are present. Was thinking on having all of the 1.6 and 1.7 custom pbd defined and included for all instances. If the specific class inclusion or exclusion is not within the JVM, what harm would it cause? I'll investigate that and see if I can come up with something.