DX Application Performance Management

 View Only
  • 1.  Custom Metrics Not Reporting Issue

    Posted Sep 26, 2018 09:21 AM

    Hi, 

     

    A team is currently experiencing issues seeing their custom metrics within the Investigator and thus not being able to use the dashboards they've created. They have nine environments and 3 of the 9 are reporting. All of the agents are configured the same way and the custom pbd is also the same on all of the host. The agents have been reporting fine until recently. I've confirmed that there is no clamping for the agents that are not reporting and the class path defined is correct.

     

    We tried testing if recycling the EM in which one of the agents was running on would resolve the issue and it only resolved it partially. Only one of the custom metric folders started reporting out of the three.

     

    Does anyone have any suggestions as to what else should be checked or tried? 

     

    Thanks!

    Sam



  • 2.  Re: Custom Metrics Not Reporting Issue

    Broadcom Employee
    Posted Sep 26, 2018 11:15 AM

    Dear Community:

    Can you offer suggestions to Samantha?

     

    My suggestions. 

     - Is it related to EM/agent performance/heap ? 

    - What does the  autoprobe and agent log (verbose) indicate?

     

    > recycling the EM in which one of the agents was running on

    Clarification: The agent was connecting to the EM, not running on it. Correct?

     

    Hal German



  • 3.  Re: Custom Metrics Not Reporting Issue

    Posted Sep 26, 2018 12:55 PM

    Hi Hal, 

     - Is it related to EM/agent performance/heap ? 

    > Not that I can tell

     

    - What does the  autoprobe and agent log (verbose) indicate?

    > Autoprobe log: I can see the custom pbds get loaded in the beginning, but do not see anything in the processing part of the log

    >Agent.log: Reviewing the logs in verbose, however with just INFO no warnings/errors indicating something wrong with the agent

     

    - recycling the EM in which one of the agents was running on

    Clarification: The agent was connecting to the EM, not running on it. Correct? 

    >Yes the collector in which the agent was connected to at the time

     

    Thanks!

    Sam



  • 4.  Re: Custom Metrics Not Reporting Issue

    Broadcom Employee
    Posted Sep 26, 2018 06:17 PM

    Have a case like this. Restart the Agent to get a fresh autoprobe logs. Try VERBOSE if allowed on your environment


    When the dashboard doesn't see metrics is it all metrics in that dashboard? Is the same metric not seen on more than one across agents? Sounds like no patterns so far. 

     

    Opening a case and doing a webex may help . But want others in the community to respond with their thoughts



  • 5.  Re: Custom Metrics Not Reporting Issue

    Broadcom Employee
    Posted Sep 26, 2018 06:47 PM

    Hi Samantha,

    You mentioned "I've confirmed that there is no clamping for the agents that are not reporting..."

    Have you checked for clamps on both EM and agent sides?

    - On agent side the IntroscopeAgent.log would show something like this 

    [WARN] [IntroscopeAgent.Agent] Absolute metric clamp of nnnnn reached, no more metrics can be created. Instrumentation on this application should be reduced. 

    - On EM side the APM Status Console should show any active clamps.

     

    Other than that you also mentioned:

    "> Autoprobe log: I can see the custom pbds get loaded in the beginning, but do not see anything in the processing part of the log"

    If you are not seeing your custom pbd classes in the "Processing class" section of the Autoprobe log then it would indicate they are not being instrumented for some reason which is strange if they are definitely being executed.

     

    Hope it helps

     

    Regards,

     

    Lynn



  • 6.  Re: Custom Metrics Not Reporting Issue

    Posted Sep 27, 2018 08:49 AM

    Hi Lynn and Hal

     

    I checked this morning and I do not see any clamping on the EM side through the APM Console and I do not see logging in the agent log as well. Other metrics are reporting for the agents just no the custom ones which are the ones that are of importance to the team experiencing the issue.

     

    I put the logs on one of the agents in verbose momentarily and see the following:

     

    [IntroscopeAgent.BizTrxDef] Business Parameter Set did not find a match
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains Post Parameter that needs to be excluded from matching.
    9/26/18 11:27:04 AM EDT [VERBOSE] [IntroscopeAgent.BizTrxDef] Business Pattern Rule Set Contains the Following Parameters: Pattern Name = Path,Pattern Value

     

     

    Thanks!

    Sam



  • 7.  Re: Custom Metrics Not Reporting Issue
    Best Answer

    Broadcom Employee
    Posted Sep 27, 2018 11:14 AM

    Dear Sam:

        The logging above is about agent recording and not related to this. At this point opening a case and doing a webex is probably needed . If you do this, please post the case number, let us know the result to help others that may have the same use case.



  • 8.  Re: Custom Metrics Not Reporting Issue

    Broadcom Employee
    Posted Sep 27, 2018 05:03 PM

    Working with Samantha on case 01203089 for this. 



  • 9.  Re: Custom Metrics Not Reporting Issue

    Broadcom Employee
    Posted Oct 02, 2018 11:18 AM

    Issue was caused due to the way the application was used. This was a lower environment and so only batch jobs were run against it. Periods of application inactivity caused the metrics to disappear. 



  • 10.  Re: Custom Metrics Not Reporting Issue

    Posted Sep 27, 2018 09:59 AM

    Is there any way to get a timestamp on the autoprobe log?