DX Application Performance Management

 View Only
Expand all | Collapse all

IntroscopeEPA.properties use of Environment Variables

  • 1.  IntroscopeEPA.properties use of Environment Variables

    Posted Jan 04, 2017 12:25 PM

    Within the Environment Performance Agent (CA APM 10.0), within the configuration has the MOM host name.  This is the only real variance between our three APM environments and we typically have to either have three of each of our epagents (AIX, SLES, RHEL) so nine total that covers each of the OS platforms for the three environments.

     

    Is there a way to configure the 

    introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=

     

    So that it references an OS environment variable? 

     

    Like 

    introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM

     

    This would cut the number of variant epagents to three instead of nine.



  • 2.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 04, 2017 12:33 PM

    Couldn't you just create one EPA image and use a script to change that property based upon a commandline variable?

    I do this all the time with my installWily script for Java agents in *nix.



  • 3.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 04, 2017 01:12 PM

    We did try to do the script solution but got talked to about deploying what was tested and the script would be modifying something, thus would not be what was tested.  

     

    This attempt was a script that actually modified the contents of the property file for that one value.  Things were going along pretty well, till the script truncated the property file and the agent did not function as expected.  Again, got into a discussion on deploying what was tested with no modification.

     

    So thought, hey, if I could add an environment variable that I could then inject into the property file we would not need any agent/configuration modification during deploy and we could deploy what we tested.  

     

    The second issue is one of ownership of the deployment script.  The APM administrators are well versed enough in building deployment scripts on AIX, Suse and Redhat so we have to depend on the system admins to build the script.  Once the system admins build the script, the ownership and support for the script becomes fuzzy.  Then we get back into deploying what you tested.



  • 4.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 04, 2017 01:35 PM

    I have tried %CAAPM_MOM and %CAAPM_MOM%, %{CAAPM_MOM} within the introscopeepa.properties file but looking at the base Java properties file, the program would need to parse the environment value directive.

     

    So my second thought is can we inject the 

    introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM within the EPACtrl.sh file in the EpaCmd

     

    EpaCmd="java -Xms${MIN_HEAP_VAL_IN_MB}m -Xmx${MAX_HEAP_VAL_IN_MB}m -cp lib/EPAgent.jar:lib/IntroscopeServices.jar:lib/Agent.jar:epaplugins/epaMQMonitor/epaMQMonitor.jar:epaplugins/epaMQMonitor:epaplugins/epaMQMonitor/lib/com.ibm.mq.pcf.jar:epaplugins/epaMQMonitor/lib/com.ibm.mq.jar:epaplugins/epaMQMonitor/lib/connector.jar:epaplugins/epaMQMonitor/lib/com.ibm.mqjms.jar com.wily.introscope.api.IntroscopeEPAgent"

     

    EpaCmd="java -Xms${MIN_HEAP_VAL_IN_MB}m -Xmx${MAX_HEAP_VAL_IN_MB}m -cp lib/EPAgent.jar:lib/IntroscopeServices.jar:lib/Agent.jar:epaplugins/epaMQMonitor/epaMQMonitor.jar:epaplugins/epaMQMonitor:epaplugins/epaMQMonitor/lib/com.ibm.mq.pcf.jar:epaplugins/epaMQMonitor/lib/com.ibm.mq.jar:epaplugins/epaMQMonitor/lib/connector.jar:epaplugins/epaMQMonitor/lib/com.ibm.mqjms.jar -Dintroscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM 

    com.wily.introscope.api.IntroscopeEPAgent"

     

    Then add within the .profile of the user that starts the agent

     

    CAAPM_MOM=testMOM.mydomain.org

    export CAAPM_MOM

     

    Anyone have any experince in setting the transport tcp host from the epactrl.sh file?



  • 5.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 04, 2017 01:48 PM

    Ok, attempted to add

    -Dintroscope.agent.enterprisemanager.transport.tcp.host=${CAAPM_MOM} 

    and

    -Dintroscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=${CAAPM_MOM} 

     

    To the EPACtrl.sh EpaCmd line and then commented out the like line out of the IntroscopeEPA.properties, restarted the agent and in both cases localhost was used.

     

    On echo, of the EpaCmd, it appears with the right value in place of the CAAPM_MOM environment variable.

     

    Is there hidden magic mojo that I'm missing?



  • 6.  Re: IntroscopeEPA.properties use of Environment Variables
    Best Answer

    Broadcom Employee
    Posted Jan 05, 2017 09:04 AM

    Yes there is hidden magic mojo.  

     

    Place this

    -Dintroscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM

    before the -cp in the java line.  

     

    I tried this in my environment and couldn't get it to work.  Finally after staring at it, the Xms variable tipped me off.  I then realized that -cp is taking your parameter as part of the classpath and not an actual variable.  Its a new year and still Monday.



  • 7.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 04, 2017 12:38 PM

    I do have a variation script for installing/configuring EPA if you'd like to use it.



  • 8.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 04, 2017 12:41 PM

    Hi Billy:

     Are you already using/considering a script approach like Hiko suggested as a workaround?

     

    Thanks

    Hal German



  • 9.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 04, 2017 01:15 PM

    Thank you Hal.

     

    Yes, we had a script approach but ran into an issue with the script modifying a tested element (configuration) and when things go south, which team had to roll up their sleeves and figure out what happened and how to correct it.



  • 10.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 05, 2017 08:41 AM

    musma03  Hiko_Davis  included.

     

    Can we manipulate EPACtrl.sh to do as Billy wants?  I also asked internally as well...

     

    Thanks

    Hal German



  • 11.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 05, 2017 10:13 AM

    Thank you Matt,

     

    Found the following:

     

    If I set the environment variable and not comment out the line from the IntroscopeEPA.properties file.  Then the value within the property file will be used.

     

    If I comment out the

     

    introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=greywind.aessuccess.org

     

    from the IntroscopeEPA.properties file, then the addition of the

     

    EpaCmd="java -Xms${MIN_HEAP_VAL_IN_MB}m -Xmx${MAX_HEAP_VAL_IN_MB}m -Dintroscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM . . .

     

    in the EPACtrl.sh will inject the MOM environment variable.

     

    Is there a way to do the following?

     

    Leave the host.DEFAULT line in the IntroscopeEPA.properties file but override that value from the command line?

     

    So that, we could have the following:

     

    MOM="";

    if [ -z "$CAAPM_MOM" ]; then
            echo "CAAPM_MOM NOT set";

    else

             echo "CAAPM_MOM set to $CAAPM_MOM";

          MOM="-Dintroscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM "

    fi

    EpaCmd="java -Xms${MIN_HEAP_VAL_IN_MB}m -Xmx${MAX_HEAP_VAL_IN_MB}m $MOM -cp lib/

     

    And have the environment set value override the setting within the property file?  This would allow for the agent to at least connect to the default MOM, like test.

     

    If not, then I guess I could do something like:

     

    Comment out the host.DEFAULT=<MOM> from the IntroscopeEPA.properties file then add the following to the EPACtrl.sh file.

     

    if [ -z "$CAAPM_MOM" ]; then
            echo "Hey YOU forgot to set the CAAPM_MOM NOT set in the wilyapm .profile! Go fix that before trying to start the agent again.";

          exit 1;

    fi

    EpaCmd="java -Xms${MIN_HEAP_VAL_IN_MB}m -Xmx${MAX_HEAP_VAL_IN_MB}m -Dintroscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM -cp lib/ . . .



  • 12.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 05, 2017 10:50 AM

    Unfortunately commenting out introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT will result in the Agent using the default of localhost.  Basically the property must be enabled in the properties file.

     

    However, if you set that property to HOSTA in the properties file and in the EPACtrl.sh script you set that property to HOSTB, this will win out and take precedence over the setting in the properties file.  

     

    So for what you're trying to do, you could have the property set to blank in the properties file and control it from the startup script.



  • 13.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 05, 2017 12:03 PM

    I set the environment value to a non-existing server and left the introscopeeap.property line with the correct MOM.  The agent was able to connect to the MOM defined in the property file.

     

    I set the environment value to a different MOM than the one that is in the property file.  On starting the agent, the agent connected to the MOM within the property file.

     

    Only after I commented out the property file line did the agent use the environment value injected from the command line.

     

    So from my testing, you have to comment out the property in order to assign a MOM with the command line -D option.



  • 14.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 05, 2017 09:17 AM

    Hi Billy:

       Did Matt's last suggestion solve the issue? If not, what are you seeing

    Thanks

    Hal German



  • 15.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 05, 2017 09:47 AM

    Thank you Matt and Hal.

     

    We had a production event last night and we are still in the throws of that so it might be later today or tomorrow before I can go test this.

     

    I will report back as soon as I can.

     

    Billy



  • 16.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 05, 2017 10:01 AM

    Billy:
       Since Matt has tested it and it works, marking as answered. However, we do want to continue to work this thread and await eagerly your chance to test when your test cycles permit.

      Thanks for the detail status updates. Makes it easier to get to a solution quicker

    Hal German



  • 17.  Re: IntroscopeEPA.properties use of Environment Variables

    Posted Jan 19, 2017 07:38 AM

    This was a bit bumpy but I think I have the bases covered in order to have a single epagent deployment that I can deploy to all AIX, SLES and RHEL servers including servers that have custom configuration outside of the base epagent configuration.

     

    First is the agent configuration file.  Each of the IntroscopeEPAgent.properties files are specific to an OS since not all of the Perl plugins we run will even make sense on another platform.  An example of OS based configuration is the lparstat on AIX or free on SLES or RHEL.  

    Not all hosts are created equal.  Sometimes there is a plugin that requires a bit of environment information in order to connect, query or locate the metric target.  So, we have an IntroscopeEPAgent.properties file that will override the OS configuration located at /syshome/wilyapm/config

     

    With the below code, if we don't have a configuration file, we will exit the epactrl.sh 

     

    Yes, the code is pretty ugly, but don't want to go through and try to get the indention and such sorted out.  In my implementation, everything is four spaced per block intended, which makes it really easy to line up the if and fi  

     

    #Use a custom file instead of default
    agentconfig="unset";

    #Use a custom file instead of default
    FILE="/syshome/wilyapm/config/IntroscopeEPAgent.properties"

    if [ -f "$FILE" ];
    then
    agentconfig="-Dcom.wily.introscope.epagent.properties=/syshome/wilyapm/config/IntroscopeEPAgent.properties"
    echo "[INFO][EPACtrl.sh] agentconfig = $agentconfig";
    else
    #custom configuration not found, determine which platform configuration to use
    value=`uname`

    if [ $value = "AIX" ]; then
    echo "[INFO][EPACtrl.sh] This is AIX";
    agentconfig="-Dcom.wily.introscope.epagent.properties=/wilyagent/IntroscopeEPAgent.properties.aix"

    else
    if [ $value = "Linux" ]; then

    value=`cat /etc/os-release |grep -c SUSE`

    if [ $value != 0 ]; then
    echo "[INFO][EPACtrl.sh] This is SLES";
    agentconfig="-Dcom.wily.introscope.epagent.properties=/wilyagent/IntroscopeEPAgent.properties.sles"
    echo "[INFO][EPACtrl.sh] agentconfig = $agentconfig";

    fi

    value=`cat /etc/os-release |grep -c Red`

    if [ $value != 0 ]; then
    echo "[INFO][EPACtrl.sh] This is RHEL";
    agentconfig="-Dcom.wily.introscope.epagent.properties=/wilyagent/IntroscopeEPAgent.properties.rhel"
    echo "[INFO][EPACtrl.sh] agentconfig = $agentconfig";
    fi
    fi
    fi
    fi

    echo "[INFO][EPACtrl.sh] agentconfig = $agentconfig";

    if [ $agentconfig == "unset" ]; then

    echo "[ERROR][EPACtrl.sh] operating system unknown!";
    exit -1;
    fi

     

    The second issue is the APM environment assignment.  Which of the Enterprise Managers (MOM) and in which environment should the agent report.

    Thought about trying to have an environment variable with the EM host but then, what if something changed or was added.  Then you would need to locate the specific hosts that had the value and change them.  If we decouple from the actual EM host name and instead have logic within the EPACtrl.sh to determine which EM to report to, we can, if there is a change, just redeploy the consolidated agent across the board and the problem is fixed.

     

     

    #Determine which MOM to use
    if [ -f "/syshome/wilyapm/config/caapm.conf" ]; then

    #load environment value
    . /syshome/wilyapm/config/caapm.conf;

    if [ -z "$CAAPM_ENV" ]; then
    echo "{ERROR][EPACtrl.sh] CAAPM_MOM NOT set! Please configure the appropriate MOM in /syshome/wilyapm/config/wilymom.conf"
    exit 1;
    fi

    case "$CAAPM_ENV" in
    "PROD") CAAPM_MOM="production host"
    ;;
    "NON-PROD") CAAPM_MOM="non prod host"
    ;;
    "TEST") CAAPM_MOM="test host"
    ;;
    *) # default - if value is unknown
    echo "[ERROR][EPACtrl.sh] CAAPM_ENV = $CAAPM_ENV. Value is unknown. Check the EPACtrl.sh file."
    exit 1
    ;;
    esac
    echo "[INFO][EPACtrl.sh] CAAPM_ENV = $CAAPM_ENV. CAAPM_MOM = $CAAPM_MOM";
    fi

     

    We tried to comment out  introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=localhost

    from the IntroscopeEPAgent.properties file but found that on AIX, if the value is not present in the properties file, then it will go to default "localhost".  So we had to uncomment this line and to keep everything the same, applied the same on all of the platforms.

     

    So as you can see, to define an environment, there is a config file caapm.conf within the /syshome/wilyapm/config directory.  If the file isn't there, we don't guess, we log an error and exit so we never have an issue with a production server sending data to a non-production environment.

     

    Then to add both to the EpaCmd line

    EpaCmd="java -Xms${MIN_HEAP_VAL_IN_MB}m -Xmx${MAX_HEAP_VAL_IN_MB}m

    -Dintroscope.agent.enterprisemanager.transport.tcp.host.DEFAULT=$CAAPM_MOM -cp lib/EPAgent.jar:lib/IntroscopeServices.jar:lib/Agent.jar:epaplugins/epaMQMonitor/epaMQMonitor.jar:epaplugins/epaMQMonitor:epaplugins/epaMQMonitor/lib/com.ibm.mq.pcf.jar:epaplugins/epaMQMonitor/lib/com.ibm.mq.jar:epaplugins/epaMQMonitor/lib/connector.jar:epaplugins/epaMQMonitor/lib/com.ibm.mqjms.jar $agentconfig com.wily.introscope.api.IntroscopeEPAgent"

     

    With these additions, we can have a single epagent directory that we can deploy to all hosts (AIX, SLES, RHEL) regardless of which environment and permits custom configuration.

     

    On a side note, any of our Perl plugins that require environment, database, port, etc information.  Within the plugin we have the code read in a plugin specific configuration file from the /syshome/wilyapm/config directory so the plugin can be added to the consolidated agent and is only activated with the custom IntroscopeEPAgent.properties file and the presence of the plugin specific configuration file. 

     

    With this, we can test what we deploy and deploy what we have tested.



  • 18.  Re: IntroscopeEPA.properties use of Environment Variables

    Broadcom Employee
    Posted Jan 19, 2017 09:26 AM

    Thanks Billy for an invaluable update and your adventure with EPAgent and getting the scripts correct. I am sure this will help others that are thinking of doing the same thing.