DX Application Performance Management

  • Private Community
 View Only

  • 1.  APM Setup is unstable

    Posted Mar 09, 2015 06:51 AM

    Hi All,

    We have a cluster APM setup with total of 20 Agents customized/configured in loadbalancing.xml to connect one collector. Hope worked fine for 10 days after we did the changes in loadbalancing.xml file.

     

    Issues found:

    1) Frequent disconnecting all agents.

    2) In Management Module, Metric Groupings, Dashboards and Alerts are not populating with data however when Agents are in connected status.

     

    Configuration changes done:

    Increased Java Heap Size in MOM, as well in two collectors.

     

    Observation from MOM Log:

     

    3/08/15 10:01:59.332 PM CAT [ERROR] [PO:WatchedAgentPO Mailman 2] [Manager.MessageService]
    Exception: java.lang.OutOfMemoryError: GC overhead limit exceeded calling method:
    com.wily.isengard.messageprimitives.service.MessageServiceCallMessage:
    {com.wily.introscope.spec.server.beans.agent.IAgentBridgeService.recordTimesliceBindingList, v1,
    [com.wily.introscope.spec.metric.CompressingAgentMetricDataSet2@513e1711]}

     

    I request anybody please suggest few valuable inputs.

     

    PFA for Log files from MOM Enterprise Manager and Collector.

     

    Swift response are highly appreciated.

    Many Thanks in Advance.



  • 2.  Re: APM Setup is unstable

    Broadcom Employee
    Posted Mar 10, 2015 04:21 AM
    Your EM ran out of memory.

    What is your current min and max heap settings?

    Are your running 32 or 64 bit OS?


  • 3.  Re: APM Setup is unstable

    Posted Mar 11, 2015 01:25 AM

    Hi Davis,

    We have configured min and max heap size as 8GB i.e. 8192mb on MOM and 4GB on two collectors shows as below:

     

    MOM:

    lax.nl.java.option.additional=-Xms8192m -Xmx8192m -Djava.awt.headless=false -XX:MaxPermSize=256m -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi

     

    Collector:

    lax.nl.java.option.additional=-Xms4096m -Xmx4096m -Djava.awt.headless=false -XX:MaxPermSize=256m -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi

     

    My OS is 64 bit on MOM as well on two Collectors.

     

    Regards,

    ShashiKrishna



  • 4.  Re: APM Setup is unstable

    Broadcom Employee
    Posted Mar 11, 2015 01:57 AM

    What version of APM are you using?

     

    Add '-Xss512k' to each EM's JVM arguments.



  • 5.  Re: APM Setup is unstable

    Posted Mar 11, 2015 06:29 AM

    Hi Davis,

    We are using CA Introscope 9.6.0.0 version.

     

    I request you to please detail the process to add the EM's JVM arguments.

     

    Thanks in Advance.

     

    Regards,

    Shashikrishna

    +27 8140 83191



  • 6.  Re: APM Setup is unstable

    Posted Mar 13, 2015 07:26 AM

    Hi Davis,

    Good day.

    We are waiting for your response since we are new to APM Implementation.

     

    Could you please direct us on your inputs to resolve APM setp stability problem.



  • 7.  Re: APM Setup is unstable

    Broadcom Employee
    Posted Mar 13, 2015 09:17 AM
    Did adding the additional property help? This is a parameter that should have been added during the installation. 


  • 8.  Re: APM Setup is unstable

    Broadcom Employee
    Posted Mar 13, 2015 10:01 AM

    Dear ShashiKrishna:

     

    There are two ways to look at these issues.

     

    Approach one: Fix instability. Then wait for next instability to occur and fix that one. Repeat as often as needed

     

    Approach two: Fix instability; Have quarterly healthchecks to optimize system so can handle present AND future needs. There are a variety of Tech Notes, KBs, and Tech Tips to help you do just that. It is important to have  a stable, scalable infrastructure

     

    Best of luck

     

    Thanks

    Hal German



  • 9.  Re: APM Setup is unstable

    Broadcom Employee


  • 10.  Re: APM Setup is unstable

    Posted Mar 13, 2015 10:39 AM

    Hi

     

    I strongly suggest you submit a CA support ticket; a quick look at your logs reveal a very unstable environment;

    - NTP sync problems

    - networks problems

    -very likely the collector smartstor is OOO.

     

    that's too much to offer a chance for not falling in OOM. or load balancer to work correctly.

     

    BR Marc