DX Application Performance Management

Expand all | Collapse all

Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

Haruhiko Davis12-29-2015 12:32 PM

  • 1.  Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 08:06 PM

    Hi,

    I'm having an issue trying to monitor certain PerfMon metrics.... I'm pretty certain it has to do with the Syntax.

    Basically, processor is one of the key PerfMon metrics we need. If I add the additonal "out of the box" .NET and ASP metrics, this sometimes causes the Processor metric to dissappear. Is there any trick to adding additional PerfMon metrics?

    Thanks all,

    Mike



  • 2.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 08:09 PM

    Try adding individual metrics one at a time to the filterpattern to see what may be causing the problem.



  • 3.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 08:32 PM

    I was doing that... its very clear what metric added (again - out of the box) takes away Processor, but I can't figure out why.

    Let me post the details.



  • 4.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 09:36 PM

    I'll check this post again when you've updated with your filterpattern.



  • 5.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:25 PM

    Thanks for the assistance.

    Here is some context around the problem. The server(s) are Windows 2003 Standard x64, running MS BizTalk 2008. Two servers in a load balanced cluster.

    Here is the PerfMon filter line:

    introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.uNET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*

    With that line and specifically on these two servers, I get these metrics:

    .NET CLR Exceptions

    .Net CLR Jit

    .NET CLR Loading

    .NET CLR Security

    .NET Data Provider for SqlServer

    I need to test a few more things.... but if I remove all except the |Processor|*|* line I do get the "Processor" metrics. If I start adding the other metrics i lose "processor" piece. I'll give details in a bit. I'm pretty sure its the expression in some way.

     

    Thanks again,

    Mike

     



  • 6.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:31 PM
      |   view attached

    Here is my filter for BizTalk 2009/2010, based upon the TechNet documentation related to monitoring bottlenecks in BizTalk tier:

    introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*,|Processor|_Total|% Processor Time,|Process|BTSNTSvc|Private Bytes,|Process|BTSNTSvc|Virtual Bytes,|Process|BTSNTSvc|*Count,|PhysicalDisk|*|% Idle Time,|PhysicalDisk|*|Avg. Disk Queue Length,|BizTalk:Message Agent|*|Database size,|BizTalk:Message Agent|*|Message *throttling state,|BizTalk:Message Box:General Counters|*|Spool Size,|BizTalk:Message Box:General Counters|*|Tracking Data Size,|BizTalk:Message Box:General Counters|*|Tracking Spool Cleanup,|BizTalk:Message Box:Host Counters|*|Host Queue \- Length,|BizTalk:Messaging|*|Documents processed\/Sec,|BizTalk:Messaging|*|Documents received\/Sec,|XLANG\/s Orchestrations|*|Orchestrations completed\/sec,|XLANG\/s Orchestrations|*|Orchestrations dehydrated\/sec,|XLANG\/s Orchestrations|*|Orchestrations rehydrated\/sec,|XLANG\/s Orchestrations|*|Persistence points\/sec,|XLANG\/s Orchestrations|*|Orchestrations resident in memory,|XLANG\/s Orchestrations|*|Orchestrations suspended\/sec,|XLANG\/s Orchestrations|*|Transactional scopes committed\/sec,|XLANG\/s Orchestrations|*|Transactional scopes aborted\/sec,|XLANG\/s Orchestrations|*|Transactional scopes compensated\/sec

    I'm going to attach the filter in a file since the forum may not show the entire filterpattern.

    Attachment(s)

    txt
    biztalk.txt   1K 1 version


  • 7.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:35 PM

    You can find more info about monitoring BizTalk in TechNet: http://msdn.microsoft.com/en-us/library/dd547397(v=bts.10).aspx



  • 8.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:37 PM

    Thank you. Even using your PerfMon Filters i do not get everything... im not sure why as it does not log any reasons that I can locate.

     



  • 9.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:40 PM

    I'm going to ask some silly questions:

    1. Did you enable verbose logging in the agent?
    2. Did you restart the service for PerfMon Collection Service?


  • 10.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:43 PM

    Also, did you mean BizTalk 2006 R2 or 2009? There is no 2008 in TechNet.



  • 11.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:48 PM

    Restart the APM PerfMon Service? Yes.

    Verbose logging - can you be more specific? I'm not aware of a logging option I can location which switches verbose on/off.

    I'm sorry you're right. BizTalk 2009 Sql 2008.

     

    Thanks,



  • 12.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 10:59 PM
    1. Can you see the BizTalk counters when going directly to perfmon?
    2. Does the agent have the proper permissions to access perfmon counters for BizTalk?


  • 13.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 11:21 PM

    Ensure the following properties are set to their defaults:

    introscope.agent.perfmon.metric.limit

    Specifies the maximum number of Performance Monitor metrics that can be reported in each interval.

    Default

    The default is 1000 metrics.

    Example

    introscope.agent.perfmon.metric.limit=1000
    

    introscope.agent.perfmon.metric.pollIntervalInSeconds

    Specifies how frequently the Performance Monitor Collection Agent check Performance Monitor objects, counters, and instances for new metric values. The default polling interval checks all metric values every 15 seconds.

    Default

    The default interval is 15 seconds.

    Example

    introscope.agent.perfmon.metric.pollIntervalInSeconds=15
    

    introscope.agent.perfmon.category.browseEnabled

    Enables or disables the discovery of new Performance Monitor counters.

    Property settings

    True or False

    Default

    True (enabled)

    Example

    introscope.agent.perfmon.category.browseEnabled=true
    

    introscope.agent.perfmon.category.browseIntervalInSeconds

    Determines how frequently the Performance Monitor Collection Agent checks for new Performance Monitor objects to be discovered.

    Property settings

    Positive integer representing the number of seconds in the interval.

    Default

    The default interval is 600 seconds (10 minutes).

    Example

    introscope.agent.perfmon.category.browseIntervalInSeconds=600

     



  • 14.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 04-01-2014 02:27 PM

    When you get a chance, please show me your filterpattern. The one I provided has worked successfully in a number of installations.



  • 15.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 04-01-2014 02:49 PM

    Hi,

    I had a full response typed out and the site went into maintenance....

    • I can see the BizTalk metrics in PerfMon itself so I know there are there.
    • I updated the logging to do Verbose and restarted the Agent service.
    • All default PerfMon configuration is set as default as you mentioned in your last post.
    • "Permissions" I am not sure about... I do not see any indication that is the problem though.

    With the PerfMon Filter set as you mentioned:

    introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*,|Processor|_Total|% Processor Time,|Process|BTSNTSvc|Private Bytes,|Process|BTSNTSvc|Virtual Bytes,|Process|BTSNTSvc|*Count,|PhysicalDisk|*|% Idle Time,|PhysicalDisk|*|Avg. Disk Queue Length,|BizTalk:Message Agent|*|Database size,|BizTalk:Message Agent|*|Message *throttling state,|BizTalk:Message Box:General Counters|*|Spool Size,|BizTalk:Message Box:General Counters|*|Tracking Data Size,|BizTalk:Message Box:General Counters|*|Tracking Spool Cleanup,|BizTalk:Message Box:Host Counters|*|Host Queue \- Length,|BizTalk:Messaging|*|Documents processed\/Sec,|BizTalk:Messaging|*|Documents received\/Sec,|XLANG\/s Orchestrations|*|Orchestrations completed\/sec,|XLANG\/s Orchestrations|*|Orchestrations dehydrated\/sec,|XLANG\/s Orchestrations|*|Orchestrations rehydrated\/sec,|XLANG\/s Orchestrations|*|Persistence points\/sec,|XLANG\/s Orchestrations|*|Orchestrations resident in memory,|XLANG\/s Orchestrations|*|Orchestrations suspended\/sec,|XLANG\/s Orchestrations|*|Transactional scopes committed\/sec,|XLANG\/s Orchestrations|*|Transactional scopes aborted\/sec,|XLANG\/s Orchestrations|*|Transactional scopes compensated\/sec

    The only PerfMon metrics that show up in investigator are:

    1. .NET CLR Jit
    2. .NET CLR Loading
    3. .NET CLR Security
    4. .NET Data Provider for SqlServer

    The errors from the Introscope Log:

    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] Starting PerfMon Service
    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon metric polling period is 15000ms
    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon metric limit is 1000
    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] Perfmon category browse is enabled
    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon category browse period is 600000ms
    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon agent matching expression is *|*
    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] Polling interval is 15000 ms.
    4/01/14 11:04:08 AM GMT-04 [INFO] [IntroscopeAgent.PerfMonService] PerfMon Service has started successfully

     

    Regards,

    Mike

     



  • 16.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 04-01-2014 07:39 PM

    Hi Solis,

    Increase the number of performance monitor metrics from 1000 to 1500 or 2000 by modifying the below property:-

    introscope.agent.perfmon.metric.limit=1500

    or

    introscope.agent.perfmon.metric.limit=2000

     

    Restart the perfmon agent after modifying the property.  Give it a try & let me know if that helps.

     

    Thanks,

    Sunil



  • 17.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 04-01-2014 07:53 PM

    Also switch back to the below default property along with the previously suggested property change:-

    introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*

     

    Thanks,

    Sunil

     



  • 18.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 04-01-2014 07:58 PM

    Hi Sunhil,

    No difference. Same metrics only appear and nothing more.

    What has been odd, is the reason I started investigating this.... sometimes after a monthly patching or reboot, the server and agent will come up and the expected perfmon metrics show up. Then with another reboot they dissappear again. Makes no sense and the logs point nowhere....



  • 19.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 04-02-2014 07:44 PM

    Hi Solis,

    Can you try pointing the perfmon agent to a different  Enterprise Manager & see the behavior ?

    Thanks,

    Sunil



  • 20.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 05-30-2014 10:04 AM
    sunilhere:

    Hi Solis,

    Can you try pointing the perfmon agent to a different  Enterprise Manager & see the behavior ?

    Thanks,

    Sunil


    Unfortunatey we only have one single Enterprise Manager setup....

     


     



  • 21.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 05-30-2014 10:00 AM

    HI,

    I have the same problem . have you decided the problem?

    Thanks



  • 22.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 05-30-2014 10:05 AM
    agcoliveira:

    HI,

    I have the same problem . have you decided the problem?

    Thanks



    Not certain what you're asking? I was not able to get all of the metrics I was trying to get yet.

    If I figure out what the issue is, I will certainly report back.

     

    Mike Solis



  • 23.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 06-03-2014 11:10 AM

    Mike,

    I have some time now to work on this with you.

    Please let me know if you still need assistance.



  • 24.  RE: [CA APM General Discussion] RE: Monitoring PerfMon Metrics (.NET Agent

    Posted 06-03-2014 11:15 AM
    Sure. I’m free for the rest of the day for the most part…

    This is setup only in a PROD environment. As long as I don’t have to restart IIS or the server, we can troubleshoot now.

    Let me know if you want to setup a specific time...or email me directly at Mike.Solis@xlgroup.com

    I appreciate the help!


  • 25.  RE: [CA APM General Discussion] RE: Monitoring PerfMon Metrics (.NET Agent

    Posted 06-03-2014 12:32 PM

    I already resolved my problem. I only wanted measure the CPU and memory,and my solution went:

    - Delete the other counters: introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*,|Web Service|*|*,|Memory|*

    and post only the counters that I needed: introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|Memory|*

    Thanks

     

     



  • 26.  Re: RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 08-21-2014 10:57 PM

    Hi Mike,

     

    Did your issue resolve. ?
    Please share your investigation inputs.

     

     

     

    Regards,

    Vaibhav



  • 27.  RE: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 03-31-2014 11:09 PM

    To enable verbose logging, look for the following file in your agent directory:

    introscope.agent.log.config.path=logging.config.xml

     

    Follow the directions to enable verbose logging, save, and restart the agent.



  • 28.  Re: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 10-02-2014 04:03 PM

    Hi everyone.... I'm back.

     

    I've had a little better luck with this, but not directly with the BizTalk perfmon metrics. I still have a strange issue where adding certain counter filters make other counter's stop reporting into Introscope.

     

    We are testing out a new problematic app, so I was asked to add a bunch of IIS/Windows and .NET counters to Introscope. I was having excellent luck adding one at a time, until I got to the last one.... ASP.NET v*

     

    Here is my filter string:

    introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET CLR Memory|*Global*|*,|System|*,|Memory|*,|Process|{osprocessname}|*,|Web Service|*|*,|Web Service Cache|*,|Network*|vmxnet3*|*,|PhysicalDisk|*|*,|Internet Information Services Global|*,|ASP.NET v*|*

     

    How it displays with the above filter string applied:

    PerfMon_APM.JPG

     

     

    As you can see, the last filter is for ASP.NET v*. When this is added, I get the ASP.NET v2.0.50727 metrics, however Internet Information Services Global (|Internet Information Services Global|*) and Network Interface (Network*|vmxnet3*|*) disappear gray out. When I remove the |ASP.NET v*|*, Network Interface and Internet Information Services Global come right back. Obviously something about the ASP.NET v* filter is causing a problem but I cannot figure out what!

     

    Anyone know what the issue is here?

     

    Is there a formal/informal guide to the regular expressions used by this filter someplace that someone is aware of?

     

    Thanks!



  • 29.  Re: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 12-29-2015 12:32 PM

    Marking as answered.



  • 30.  Re: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 12-29-2015 01:11 PM

    Just to add to this - it appears the issues I had here all related to a single server. I have no idea why some of the counters did not show up, nor why they would appear/disappear as they felt like it. On any other server I have done perfmon configuration for (CA APM) this has worked fine.

     

    Thanks Haruhiko.



  • 31.  Re: Monitoring PerfMon Metrics (.NET Agent - 9.1.6)

    Posted 01-05-2016 03:19 PM

    There is a known issue where sometimes Windows installation do not always build and load the counter DLLs, which then requires you to reload them.

     

    https://support.microsoft.com/en-us/kb/300956