I have CEM and Introscope available and providing stats perfectly fine.
But what I am looking is - as soon as Defects are created on breach of condition, defect details should contain "trace" details as well.
When I am starting the transaction trace manually from Incident details tab, defects contains the transcation traces. But it would be great that as soon as defects are generated, trace should also be collected for the same without triggering or starting tracing manually.[for e.g. trace for slow time defect etc automatically]
Though many configs options are present wrt Introscope Transaction Trace, but it would be great if someone could list down the high level steps to achieve this scenario.
Appreciate your help and quick response.
As you are probably aware, there is integration between CEM and Introscope to trigger a transaction trace session when a CEM Incident is created.
The idea behind this is that in any environment, there's a nominal set of defects that are expected to occur normally for reasons outside of your control. By using an Incident as the basis for creating a trace session, the product is helping ensure that traces are actually warranted.
Since you mentioned Slow Time specifically, one capability you may choose to leverage is the ComponentTimeAutoTraceTriggerTracer. This tracer supports a responsethreshold parameter (in seconds). If the threshold is crossed, a trace will automatically be triggered. CEM will match defects to traces regardless of how they were triggered, so traces triggered from this threshold will be matched to defects if they exist.
See APM docs: Configure Transaction Trace Options
Thanks Tim and Guenter for prompt response.
Actually to be frank, going with tracer will not work with me with big application Infra.But will propose the same in our meetings.
Regarding the CEM-Incident Tracer settings, I am facing issue with that only.
So for e.g., I am trying to trigger the tracing manually for most of the transactions and I am getting 0 tracers.Though I am able to trace at Introscope using normal Trace session.
Also at the same time I am getting the Defects generated breaching the condition of Slow Time.
Please suggest what needs to be taken care for this kind of issue.
In the documentation to Configure Transaction Traces you can see how the CEM traces are structured a little differently. CEM passes a threshold to Introscope that is based on a percentage of the Slow Time threshold, and pertains to the specific Business Transaction. The transaction trace session you initiate from Introscope does not have those same constraints.
The documentation covers how you can adjust those CEM thresholds, and also how you can specify additional matching backends or whether to trace non-identifying components. These are all ways of casting a larger net and matching more activity on the agent side.
Ultimately however, the idea is to only capture traces for extraordinary circumstances, and not for every transaction. It is one of the concepts that allows you to take noise out of the environment and focus on the real problems.
The threshold in your screenshot is already quite low. If you're not catching Business Transactions in those traces but can with even lower thresholds, I would typically consider that a success.
How are your CEM transactions defined? Is there Url translation between CEM span and the app server? Check the agent debug log for "BT matching" messages.
An agent log in debug may help. I also created a KB on this topic.
Hal German APM Support