DX Application Performance Management

CA Tuesday Tip: .NET Agent installation problems, OOM, High CPU - Checklist

  • 1.  CA Tuesday Tip: .NET Agent installation problems, OOM, High CPU - Checklist

    Broadcom Employee
    Posted 05-16-2013 10:39 AM
      |   view attached
    CA APM Tuesday Tip by Sergio Morales, Principal Support Engineer for 5/16/2013

    Below two .NET Agent common issues reported in support, I have tried to compiled all the main points to check, I hope this helps:

    [size=5]Installation problems[size=5][size][size]


    Make sure you are using the correct .NET agent installer package.

    For v9.1, you can use the 64bit installer to monitor both 32 and 64bits applications.
    For v8.x and 9.0.x: Officially, you cannot instrument both 32 and 64bits at the same time, make sure to use the correct installer or agent package.

    Check for possible errors in the install log:

    For v8.x and 9.0.x: AGENT_HOME\wily\install\Introscope_Agent_for_.NET_x.x.x.x_InstallLog.log
    For v9.1: install log is on the same location as the installer

    Make sure the correct version of wily.Agent.dll has been registered in the GAC (c:\windows\assembly). If not listed, you need to register it manually:
    Execute: explorer C:\Windows\assembly
    Execute: explorer AGENT_HOME\wily\bin
    Drag and drop wily.Agent.dll from AGENT_HOME\wily\bin into C:\Windows\assembly

    If CLR is v4, set introscope.nativeprofiler.clrv4.transparency.checks.disabled=true, restart IIS, and try again. The .NET 4 CLR has some additional checks on certain assemblies which may invalidate instrumented code, thus throwing VerificationException. This agent profile setting will suppress these checks when set to true.

    Make sure permissions to the AGENT_HOME have been set accordingly.
    First, access/launch the .NET application, then execute AGENT_HOME\wily\wilypermission.exe, for example:

    If you are trying to instrument a window service or standalone app:
    <AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily <mytestapp.exe>

    For IIS:
    <AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily

    the output will tell you if there is a problem with permissions.

    NOTE: For v9.1 agent, this step is not longer required.

    Verify that the required environment variables exit:

    Check for any error messages in the Windows event viewer when starting the application, below some common examples:

    “Failed To Co CreateProfiler”
    “The profiler was loaded successfully. Profiler CLSID: '{D6E0BA92-3BC3-45ff-B9CC-B4B5AB7190BC}

    The above messages indicate that there is other CLR profilers preventing the .NET Agent from probing the monitored .NET process.
    APM .NET Agent COR_PROFILER GUID is {5F048FC6-251C-4684-8CCA-76047B02AC98}
    Open the regedit, search for “COR_PROFILER” or the CLSID and disable or uninstall any other .NET monitoring tool.

    A very common situation we can see is conflicts with Avicode application. see attached "Avicode conflicts with the .NET agent.pdf". The solution is to uninstall Avicode.
    NOTE: AVIcode .NET Application Monitoring is a subsidiary of Microsoft. AVIcode technologies enable in-production, always-on root-cause monitoring and analysis of Microsoft .NET Framework-based applications.

    If instrumentation is not working, try to configure the agent to instrument all .NET application by commenting the introscope.agent.dotnet.monitorApplications agent property. This will help you confirm that the problem is application related.

    If you are trying to instrument an standalone .NET applicaiton, note that NET Agent v9.1 doesn't support anymore using full path in introscope.agent.dotnet.monitorApplications. You should be using instead the process name.
    In addition, please note that this property is case sensitive and wildcards are not supported.

    Verify that.NET agent is attached to the .NET process using “tasklist /m”. For example: If you are trying to monitor IIS: access to the application, then run tasklist /m
    if Autoprobe instrumentation: “wiily.AutoProbe.dll" and “wily.Agent.dll" will be attached to w3wp.exe.
    If NativeProfiler: “wily.Agent.dll" and “wily.NativeProfiler.dll" will be attached to the process.

    [size=5]Overhead, Hangs, Deadlocks[size]


    Stop the .NET application.
    Open the IntroscopeAgent profile and set introscope.autoprobe.enable=false
    Start the process again.
    If the problem does not persist, it will confirm that the problem is related to the Wily instrumentation. Enable back introscope.autoprobe.enable=true and disable the below components:

    Disable perfmon collection:

    For v8.0, 9.0: set introscope.agent.perfmon.enable=false in the IntroscopeAgent.profile

    For v9.1: stop the”CA APM Perfmon Collector Service”

    Disable any additional custom instrumentation and in addition turn off the following tracing options in the toggles-typical.pbd:


    If CLR is v4, set introscope.nativeprofiler.clrv4.transparency.checks.disabled=true.

    If CEM integration is enabled, try disabling appmap and bizrecording agent properties in introscopeagent.profile.

    Remember these are temporal changes, If the problem does not occur with the base agent, you need to introduce back each component one by one until you reproduce the problem, however If the problem persists, you should contact CA Support and provide the following information

    Enable DEBUG: In the logging.config.xml
    If v9.x, set in the Agent profile : introscope.nativeprofiler.logBytecode=true and introscope.nativeprofiler.logAllMethodsNoticed=true

    Reproduce the issue and collect the below information:
    Zipped content of the AGENT_HOME/install/*.logs
    Zipped content of the AGENT_HOME/wily/logs folder
    Screenshot of events from Windows Event viewer.
    In case of high CPU issue, capture consecutive dumps from a .Net process. If you don’t have a preferred tool and setup on their server for such purpose, then below is one suggested approach:

    a. Download procdump.exe from Microsoft. It is a free utility included with Sysinternal or other Windows admin toolkit.
    b. Identify the PID (process ID) of the .NET process, e.g. say 6789 for a w3wp.exe process, that is having unusual high CPU.
    c. Run the following command to capture 3 dumps with 5 seconds apart: e.g.
    Procdump.exe -ma -s 5 -n 3 6789
    d. If the .net app is a 64-bit process, then add -64 to the commandline: e.g.
    Procdump.exe -64 -ma -s 5 -n 3 6789 5. Analyze the dumps.

    NOTE: For any high memory, high CPU, hang problem, you can use Debug Diagnostics Tool from Microsoft to capture user dump, which contains both heap and thread snapshot.
    The following KB has both download link and usage instructions: