DX Application Performance Management

Expand all | Collapse all

Source Control / Deployment - CVS, SVN, PVCS, other

  • 1.  Source Control / Deployment - CVS, SVN, PVCS, other

    Posted 11-14-2016 12:13 PM

    Discussion:

     

    What is your source control and deployment management for APM artifacts?

     

    Details:

    There is more than a handful of files that are modified or customized within Application Performance Management installation, use, and administration.

    To name a few:

       EM - Configuration

          IntroscopeEnterpriseManager.profile

          Domain.xml

          users.xml

          WebView/Introscope.lax files

          Management Modules

          custom email shell script

       EM - Java Script Calculators

       EP Agent (variant depending on host/OS)

          plugins

          configuration

          start/stop scripts

       Application Agent (java)

          .pbl

          .pbd

          IntroscopeAgent.profile

    Database (postgresql)

          configuration/pg_hba.conf

       

    With each of these artifacts, there, hopefully, should be a deployment plan along with some sort of test/verification plan to get the changes through some sort of software life cycle within multiple environments (test, pre-production, production).

     

    Currently we have the majority of our artifacts on a network directory with no check-in / check-out control.  Our deployments are manual and custom per server.

     

    Certainly, you could take that the APM, monitoring as not being a production impact and an acceptable loss if APM were to shutdown for some reason.  But if you did, once in the past, deploy a pbd that caused issues on the production environment, the first questions usually are, was the changes tested, what is your roll-back plan, how long will production have to be down, what changed?

     

    So, what is your APM source control and deployment plans?



  • 2.  Re: Source Control / Deployment - CVS, SVN, PVCS, other

    Posted 11-18-2016 08:10 AM

    There was mention of two configuration tools on the community board as of late:

     

    APM Studio

    https://communities.ca.com/docs/DOC-231152665

    Looks like the APM Studio might be able to help with the agent pbd/pbl into source control and deployment since it is a Eclipse plugin, but what about the other agents: epagent, IBM mq, webserver, browser agent?

     

    Our environment performance agents help provide for some infrastructure performance monitoring (cpu, disk, memory, processes) and with that we have added quite a few Perl script plugins.    To make our scripts portable, we have moved all host specific configuration to the APM agent user home directory and pushed the IntroscopeEPAgent.properties to the APM agent user home also.  With this, we changed the EPACtrl.sh script to look for the ~/apmuser/config/IntroscopeEPAgent.properties file that has the specific plugins for that host within it.

     

     

    EM Config Utility - 10.5 - command line tool

    https://communities.ca.com/docs/DOC-231171532

     

     

    There are a few options such as listing changes/customization that would be very helpful during an upgrade.  But to move this up a notch, would need an Eclipse plug-in in order to get the source control and hopefully deployment abilities.  I didn't see mention on the features of management of the non-configuration files such as JavaScript calculators, Management Modules, custom scripts (email),

    There is plenty of elements within APM with the custom host metrics to be able to really have a full featured EM control GUI that would include a graphical smartstor management.  Hopefully smartstor is on the short list of being replaced with something else, but till then, the direction of being able to administrate an APM cluster enterprise managers would be very useful.



  • 3.  Re: Source Control / Deployment - CVS, SVN, PVCS, other

    Posted 11-18-2016 08:27 AM

    Thanks Billy for mentioning the ConfigUtil Doc. It is also documented in Knowledge Base Articles . Your feedback on it is appreciated. 



  • 4.  Re: Source Control / Deployment - CVS, SVN, PVCS, other

    Posted 01-24-2017 01:40 PM

    So, blazing a trail here...construction ahead...

    In prep for a 10.0 to 10.5 upgrade started to look at the pain points at trying to keep the changed, custom, one off files in view.  At one point we had nine different epagents which was driving our system admins a bit batty.  So, I devised a way to combine all of the agents and have a more configuration driven epagent, so that we have a single epagent.

     

    IntroscopeEPA.properties use of Environment Variables 

     

    This does help start to frame the source control a bit better, just starting the fire with the epagent.

     

    epagent

        <environment performance agent> This would be a full directory structure agent with all the required files.  

                    lib/agent.jar

                    bin/EPACtrl.sh

                    logs

                    plugin_one.pl

                    plugin_two.sh

                    plugin_three.java

                   IntroscopeEPAgent.properties.aix

                   IntroscopeEPAgent.properties.sles

                   IntroscopeEPAgent.properties.rhel

                <Host>

                      syshome/wilyapm/config/IntroscopeEPAgent.properties

                      syshome/wily.apm/config/caapm.conf

                      syshome/wily.apm/config/plugin_one.properties

     

    With a single epagent directory and all the environment or custom configuration being located within the /syshome/wilyapm/config directory, we can mass deploy the single epagent to all servers and all environments.

     

    We would tag the tree when there was a change and then when we winscp the contents (being aware of binary for jar/so and text for basically everything else) we create a zip/tar/gzip with the APM release and tag value

         epagent_10.0.0.12_20170124.zip

     

    So this is the epagent structure that I am planning on using within source control to manage the sea of custom and changed files.

     

    So far, this structure has cut down on the variants and explaining what goes where such as the AIX epagent goes under epagent/production/AIX/host and that one is RedHat so it goes into the epagnet/production/RHEL/Host.  But the only real difference is the existence of different plugins and the introscopeEPAgent.properties file.

     

    When I get around to the Application agents (websphere, weblogic, .NET) and have to deal with pbl/pbd and .profile I'll try to post what I am planning on doing.

     

          Cheers,

     

    Billy



  • 5.  Re: Source Control / Deployment - CVS, SVN, PVCS, other

    Posted 01-24-2017 01:48 PM

    My Java agent install script will handle any issues with configuring the agent based upon environment. You can do a search for ‘installWily' on the community to view it. I also have a version of this for EPA.

     

    If you're running Windows server with the UNIX integration, you could use the same script to modify your .NET agent profile. If you need some native, like PowerShell, that could be easily done also.



  • 6.  Re: Source Control / Deployment - CVS, SVN, PVCS, other

    Posted 01-24-2017 02:33 PM

    Thanks Hiko,

     

    Wished I had enough permission to run an installation script on a server, but sadly, the APM admin typically gets a sandbox directory that the agent is copied into and we can make changes to things inside the directory.

     

    If we provide a single agent directory (epagent, websphere1.6, webphere1.7, weblogic1.7) and provide simple instructions to put the contents of the zip/tar/gzip into a specific directory, then a few root command like chown, chmod, adding the EPACtrl.sh stop/start to the server start and stop, things go rather well.

     

    Currently we have to instruct how to change the MOM host name in the application agents, but might look at externalizing that also since we already have a server conf file (/syshome/wilyapm/config/caapm_env.conf) that has a single variable with value of "PROD, NON_PROD, TEST".  Gotta go see if there is a way of injecting the environment on the JVM.   

     

    Once we get into the system admins or WebSphere admins running an install on each server (+1500) now we have more effort and time.  With basically using the system admin's package deployment software, we can throw an epagent on a server, have it start and we are done.

    For the application agents, typically the websphere admin will script all of the agent's elements so when a server is created everything is already in place ready to go.

     

    Another issue we will be facing that I'm still noddling, is how to inject custom pbd within an agent dependent on which application server software and which applications are present.  Was thinking on having all of the 1.6 and 1.7 custom pbd defined and included for all instances.  If the specific class inclusion or exclusion is not within the JVM, what harm would it cause?  I'll investigate that and see if I can come up with something.

     

    Thank you,

     

    Billy