CA APM Tuesday Tip by Sergio Morales, Principal Support Engineer for 5/16/2013
Below two .NET Agent common issues reported in support, I have tried to compiled all the main points to check, I hope this helps:
[size=5]
Installation problems[size=5][size][size]
Checklist:
1.
Make sure you are using the correct .NET agent installer package.
For v9.1, you can use the 64bit installer to monitor both 32 and 64bits applications.
For v8.x and 9.0.x: Officially, you cannot instrument both 32 and 64bits at the same time, make sure to use the correct installer or agent package.
2.
Check for possible errors in the install log:
For v8.x and 9.0.x: AGENT_HOME\wily\install\Introscope_Agent_for_.NET_x.x.x.x_InstallLog.log
For v9.1: install log is on the same location as the installer
3.
Make sure the correct version of wily.Agent.dll has been registered in the GAC (c:\windows\assembly). If not listed, you need to register it manually:
a.
Execute: explorer C:\Windows\assembly
b.
Execute: explorer AGENT_HOME\wily\bin
c.
Drag and drop wily.Agent.dll from AGENT_HOME\wily\bin into C:\Windows\assembly
4.
If CLR is v4, set introscope.nativeprofiler.clrv4.transparency.checks.disabled=true, restart IIS, and try again. The .NET 4 CLR has some additional checks on certain assemblies which may invalidate instrumented code, thus throwing VerificationException. This agent profile setting will suppress these checks when set to true.
5.
Make sure permissions to the AGENT_HOME have been set accordingly.
First, access/launch the .NET application, then execute AGENT_HOME\wily\wilypermission.exe, for example:
If you are trying to instrument a window service or standalone app:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily <mytestapp.exe>
For IIS:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily
the output will tell you if there is a problem with permissions.
NOTE: For v9.1 agent, this step is not longer required.
6.
Verify that the required environment variables exit:
Cor_Enable_Profiling=0x1
COR_PROFILER={5F048FC6-251C-4684-8CCA-76047B02AC98}
com.wily.introscope.agentprofile=<install_dir>\wily\IntroscopeAgent.profile
7.
Check for any error messages in the Windows event viewer when starting the application, below some common examples:
“Failed To Co CreateProfiler”
“The profiler was loaded successfully. Profiler CLSID: '{D6E0BA92-3BC3-45ff-B9CC-B4B5AB7190BC}
The above messages indicate that there is other CLR profilers preventing the .NET Agent from probing the monitored .NET process.
APM .NET Agent COR_PROFILER GUID is {5F048FC6-251C-4684-8CCA-76047B02AC98}
Open the regedit, search for “COR_PROFILER” or the CLSID and disable or uninstall any other .NET monitoring tool.
A very common situation we can see is conflicts with Avicode application. see attached "Avicode conflicts with the .NET agent.pdf". The solution is to uninstall Avicode.
NOTE: AVIcode .NET Application Monitoring is a subsidiary of Microsoft. AVIcode technologies enable in-production, always-on root-cause monitoring and analysis of Microsoft .NET Framework-based applications.
8.
If instrumentation is not working, try to configure the agent to instrument all .NET application by commenting the introscope.agent.dotnet.monitorApplications agent property. This will help you confirm that the problem is application related.
9.
If you are trying to instrument an standalone .NET applicaiton, note that NET Agent v9.1 doesn't support anymore using full path in introscope.agent.dotnet.monitorApplications. You should be using instead the process name.
In addition, please note that this property is case sensitive and wildcards are not supported.
10.
Verify that.NET agent is attached to the .NET process using “tasklist /m”. For example: If you are trying to monitor IIS: access to the application, then run tasklist /m
if Autoprobe instrumentation: “wiily.AutoProbe.dll" and “wily.Agent.dll" will be attached to w3wp.exe.
If NativeProfiler: “wily.Agent.dll" and “wily.NativeProfiler.dll" will be attached to the process.
[size=5]
Overhead, Hangs, Deadlocks[size]
Checklist:
Stop the .NET application.
Open the IntroscopeAgent profile and set introscope.autoprobe.enable=false
Start the process again.
If the problem does not persist, it will confirm that the problem is related to the Wily instrumentation. Enable back introscope.autoprobe.enable=true and disable the below components:
1.
Disable perfmon collection:
For v8.0, 9.0: set introscope.agent.perfmon.enable=false in the IntroscopeAgent.profile
For v9.1: stop the”CA APM Perfmon Collector Service”
2.
Disable any additional custom instrumentation and in addition turn off the following tracing options in the toggles-typical.pbd:
ServicedComponentTracing
RemotingClientProxyTracing
DirectoryServicesTracing
MessagingTracing
MessagingTransactionTracing
SQLAgentDataReaders
SQLAgentTransactions
3.
If CLR is v4, set introscope.nativeprofiler.clrv4.transparency.checks.disabled=true.
4.
If CEM integration is enabled, try disabling appmap and bizrecording agent properties in introscopeagent.profile.
Remember these are temporal changes, If the problem does not occur with the base agent, you need to introduce back each component one by one until you reproduce the problem, however If the problem persists, you should contact CA Support and provide the following information
Enable DEBUG: In the logging.config.xml
If v9.x, set in the Agent profile : introscope.nativeprofiler.logBytecode=true and introscope.nativeprofiler.logAllMethodsNoticed=true
Reproduce the issue and collect the below information:
1.
Zipped content of the AGENT_HOME/install/*.logs
2.
Zipped content of the AGENT_HOME/wily/logs folder
3.
AGENT_HOME/wily/IntroscopeAgent.profile
4.
Screenshot of events from Windows Event viewer.
5.
In case of high CPU issue, capture consecutive dumps from a .Net process. If you don’t have a preferred tool and setup on their server for such purpose, then below is one suggested approach:
a. Download procdump.exe from Microsoft. It is a free utility included with Sysinternal or other Windows admin toolkit.
b. Identify the PID (process ID) of the .NET process, e.g. say 6789 for a w3wp.exe process, that is having unusual high CPU.
c. Run the following command to capture 3 dumps with 5 seconds apart: e.g.
Procdump.exe -ma -s 5 -n 3 6789
d. If the .net app is a 64-bit process, then add -64 to the commandline: e.g.
Procdump.exe -64 -ma -s 5 -n 3 6789 5. Analyze the dumps.
NOTE: For any high memory, high CPU, hang problem, you can use Debug Diagnostics Tool from Microsoft to capture user dump, which contains both heap and thread snapshot.
The following KB has both download link and usage instructions:
http://support.microsoft.com/kb/2580960
Regards,
Sergio