DX Application Performance Management

  • 1.  Openshift Infra Agent - setting min/max heap ?

    Posted Apr 10, 2018 08:05 AM

    How can we modify the min/max heap settings for this agent ?We dont want to rebuiild the agent.

    Looking at the startup scripts we can see that the min/max values can be passed as params, but how to do this ?

     

    As we have a large estate the agent is crashing due to default heap of 16/256 being too small

     

    ./APMIAgent.sh startwithlogs: APM Infrastructure Agent started

    APM Infrastructure Agent Start is Completed

    Tue Apr 10 11:41:45 UTC 2018 [INFO] APM Infrastructure Agent Start is Completed

    4/10/18 11:41:51 AM UTC [INFO] [IntroscopeAgent.OSEMetadataInfo] Operation Name = null

    4/10/18 11:41:58 AM UTC [INFO] [IntroscopeAgent.IntelligentInstrumentationService] Intelligent Instrumentation Service activated

    4/10/18 11:42:43 AM UTC [ERROR] [IntroscopeAgent.InfrastructureAgent Agent] Error on getting last deployment status for heapster

    4/10/18 11:42:48 AM UTC [INFO] [IntroscopeAgent.InfrastructureAgent Agent] AutoProbe has reloaded all directive configuration.  This is because a change to a directive configuration file was detected.

    4/10/18 11:43:02 AM UTC [ERROR] [IntroscopeAgent.OutgoingRouteConnector] Closing outgoing route connector due to unrecoverable error.

    java.lang.OutOfMemoryError: GC overhead limit exceeded

    4/10/18 11:43:02 AM UTC [ERROR] [IntroscopeAgent.InfrastructureAgent Agent] The thread named "UnknownHub Hub Transmit 2" has unexpectedly terminated because of java.lang.OutOfMemoryError GC overhead limit exceeded

    4/10/18 11:43:09 AM UTC [ERROR] [IntroscopeAgent.InfrastructureAgent Agent] The thread named "Thread-3" has unexpectedly terminated because of java.lang.OutOfMemoryError GC overhead limit exceeded

     

     

     

     

    The pod remains Running. We have attempted a couple of restarts, all produce the same results.

    When reviewing the container, I can see the /usr/local/openshift/apmia/run.sh calls apmia-ca-installer.sh which in turn call bin/APMIAgent.sh

     

    APMIAgent.sh takes default min/max heap from within the script as it expects them to be passed as arguments when being executed ($2 and $3)

    Is it possible to set these as ENV variables?



  • 2.  Re: Openshift Infra Agent - setting min/max heap ?

    Broadcom Employee
    Posted Apr 10, 2018 09:21 AM

    Hi Dave,

     

    Funnily enough I was looking at the same thing for the collector agent. Yes you can is the answer

     

    If you define a couple of variables MIN and MAX than call teh script with them as follows

     

    # export MAX=512
    # export MIN=128
     ./*sh start $MIN $MAX
    Starting /collectors/CollectorAgent-10.5.2/jre/bin/java -server -Xms128m -Xmx512m -cp lib/CollectorAgent.jar:lib/gson.jar:lib/Agent.jar -XX:ErrorFile=/collectors/CollectorAgent-10.5.2/logs/jvm_error.%p.log com.wily.introscope.agent.collector.CollectorAgent
    ./CollectorAgent.sh start: CollectorAgent (pid 15426) started

     

    If they are not set and you still call it with those arguments it just uses the default as the $2 $3 are missing


    # unset MIN
    # unset MAX
    # ./*sh start $MIN $MAX
    Starting /collectors/CollectorAgent-10.5.2/jre/bin/java -server -Xms16m -Xmx256m -cp lib/CollectorAgent.jar:lib/gson.jar:lib/Agent.jar -XX:ErrorFile=/collectors/CollectorAgent-10.5.2/logs/jvm_error.%p.log com.wily.introscope.agent.collector.CollectorAgent
    ./CollectorAgent.sh start: CollectorAgent (pid 15478) started

     

    hope this helps

     

    cheers

    Mike



  • 3.  Re: Openshift Infra Agent - setting min/max heap ?

    Posted Apr 10, 2018 09:46 AM

    Hi Mike, its not what we want. As the supplied "run "script is already running inside the Docker image we cant pass any parameters to do it without rebuilding the Docker image locally, which we would rather not do. Ideally we thought the min/max values might be able to get picked up from the Openshift environment where we can set them, rather than as command line args that we can't set



  • 4.  Re: Openshift Infra Agent - setting min/max heap ?

    Broadcom Employee
    Posted Apr 10, 2018 09:54 AM

    Hi Dave,

     

    I wont pretend to be an Openshift/Docker expert but if you have a script in the docker image that references variable names like in my example MIN MAX then you can set these externally to values or not. I admit this would need a rebuild to get it to be able to read the variables in the first place, but only once to set the mechanism up, you could then change the variable values in Openshift (if I understand you correctly).

     

    If I'm still missing something - apologies for that.

     

    cheers

    Mike



  • 5.  Re: Openshift Infra Agent - setting min/max heap ?

    Posted Apr 10, 2018 10:06 AM

    Hi Mike, it looks to us like the actual agent startup script that is supplied can only take the values as command line arguments. It doesn't appear to try to get them from the environment. So if there is no way to get them from the environment, we are going to have to modify the supplied image and rebuild that. We think we have got a workaround, but just wanted to know from CA if we were missing something. 



  • 6.  Re: Openshift Infra Agent - setting min/max heap ?

    Broadcom Employee
    Posted Apr 10, 2018 10:00 PM

    Hi Dave,

    Mike is saying that you will need to edit the OOTB APMIAgent.sh script to be able to process the values of MIN & MAX as environment variables. I do agree that you would also need to disable the argument checking for the heap values. The following changes work for me:

     

    OLD:

    ====

    # changes for passing heap values in arguments
    MIN_HEAP_VAL_IN_MB=16
    MAX_HEAP_VAL_IN_MB=256

     

    MIN_ARG_PRESENT=true
    if [ -z "$2" ]
    then
    MIN_ARG_PRESENT=false
    fi

     

    MAX_ARG_PRESENT=true
    if [ -z "$3" ]
    then
    MAX_ARG_PRESENT=false
    fi

    ====

     

    NEW:

    ====

    # modified to pass in heap values as environment variables
    MIN_HEAP_VAL_IN_MB=16
    MAX_HEAP_VAL_IN_MB=256

     

    if [[ "$MIN" ]]; then
    echo "Setting MIN_HEAP_VAL_IN_MB to environment variable value for MIN"
    MIN_HEAP_VAL_IN_MB=$MIN
    fi

     

    if [[ "$MAX" ]]; then
    echo "Setting MAX_HEAP_VAL_IN_MB to environment variable value for MAX"
    MAX_HEAP_VAL_IN_MB=$MAX
    fi

     

    # disable heap argument checking

     

    MIN_ARG_PRESENT=false
    MAX_ARG_PRESENT=false

     

    #MIN_ARG_PRESENT=true
    #if [ -z "$2" ]
    # then
    # MIN_ARG_PRESENT=false
    #fi

     

    #MAX_ARG_PRESENT=true
    #if [ -z "$3" ]
    # then
    # MAX_ARG_PRESENT=false
    #fi

    ====

     

     

    Hope that helps

     

    Regards,

     

    Lynn



  • 7.  Re: Openshift Infra Agent - setting min/max heap ?

    Posted Apr 11, 2018 04:00 AM

    Thanks Lynn and Mike

     

    Ok we understand what is required now. We've spoken to Nishant also, and we understand that there will be a mod to the script released at some point so the heap size can be picked up from the environment without having to amend the supplied script.