DX Operational Observability

 View Only

A Primer On Distributed Traces with DX Operational Observability: From Basics to Mastery Using Transaction Trace Viewer (Part 1)

By Srinivas Venkata Bevara posted 11 days ago

  

Modern microservices applications generate complex, distributed transactions that span multiple services, protocols, and runtimes. OpenTelemetry has emerged as the de facto standard for capturing these traces, but extracting value from them depends greatly on how they're visualized and analyzed. Understanding how these distributed requests flow through your system is crucial for maintaining performance and reliability, and the real challenge isn't collecting traces, it's making sense of them.

​​Every digital transaction leaves a trail. The challenge isn’t in collecting that trail, it’s in making sense of it. OpenTelemetry gives us spans and traces, but without the right lens, they remain raw data. This is where the DX O2 Transaction Trace Viewer comes in. In this first part of our series, we’ll walk you through how distributed traces are captured, sampled, and then visualized in DX O2, giving you a complete picture of application flow.

Key Takeaways

  • Understand the fundamentals of OpenTelemetry traces and spans, and how they appear in DX O2 Transaction Trace Viewer.

  • Learn how sampling impacts trace completeness and why it matters for observability.

  • Get a guided tour of the Transaction Trace Viewer with screenshots: Trace List, Components, Summary, Performance, and Properties.

This guide is designed for DevOps engineers, SREs, platform teams, and developers implementing tracing in distributed architectures.

From OpenTelemetry Spans to Transaction Traces

An OpenTelemetry trace represents an end-to-end request across distributed systems. Think of a trace as the story of a transaction, and spans as the individual chapters. Each span carries a unique identity and is correlated (trace_id, span_id, parent_span_id) and tells us what happened at a specific point in time. Resource attributes describe where the span originated, while span attributes describe what happened. When DX O2 processes these spans, it stitches them together into a clear, connected narrative, one you can use to troubleshoot and optimize.

Here are the important span attributes for DX O2 trace ingestion:

  • trace_id → groups all spans of a request

  • span_id → uniquely identifies each span

  • parent_span_id → maintains hierarchy across services

  • span.name → maintains name of the trace 

  • span.kind → identifies Frontend or Backend trace.

  • service.name → used for Frontend’s application name, this comes from the resource attribute.

Example parent-child span hierarchy:

Trace ID: 1234abcd

└── Span A (SERVER: checkout service, root span)

      ├── Span B (CLIENT: call to payment service)

      │     └── Span C (SERVER: payment service handles request)

      ├── Span D (CLIENT: call to shipping service)

      │     └── Span E (SERVER: shipping service handles request)

      └── Span F (INTERNAL: calculateDiscount logic)

For more details, refer to the https://opentelemetry.io/docs/specs/otel/overview/#tracing-signal 

DX O2 ingests these OpenTelemetry spans via its OTel ingestion service and transforms them into the DX O2 Transaction Trace model, enabling visualization in the UI.

DX Operational Observability: Unlocking Insights from Distributed Traces

DX Operational Observability (DX O2) provides deep visibility into your application performance and infrastructure health, with a unique strength: transforming complex distributed traces into actionable insights.

In today’s microservices landscape, understanding how a request flows across dozens of services is critical for maintaining reliability. DX Operational Observability seamlessly ingests OpenTelemetry spans, the industry standard for distributed tracing, via its DX OTel ingestion service. It then intelligently transforms these raw spans into a structured DX O2 Transaction Trace model, making them easy to visualize and analyze through an intuitive interface.

Unlike raw trace viewers that present technical timelines, DX O2 adds business context, making it easier to connect spans to service-level impact.

Key aspects of DX O2 include:

  • Service-Centric View: Instantly identify entry points, group component calls, and see which services participated in a transaction.

  • Intelligent Correlation: Reconstruct trace hierarchies using trace_id, span_id, and parent_span_id to reveal how operations relate across services.

  • Actionable Performance Analysis: Spot performance issues quickly with span-level duration and timing insights.

  • Simplified Debugging: Present complex traces in a business-oriented way that operations teams, architects, and even non-technical stakeholders can understand.

DX O2 empowers organizations to move beyond raw trace data and into a clear, business-focused view of their distributed systems.

Before we explore the DX O2 UI, let's ground ourselves in some fundamental OpenTelemetry concepts that are crucial for generating meaningful transaction traces.

Transaction Trace Sampling Basics

Not every span makes it to the finish line. That’s where sampling comes in. Head-based sampling decides early which spans to keep, while tail-based sampling waits until the full story unfolds. For transaction visibility, complete traces matter, because missing chapters mean missing context. DX O2 ensures that what you see is more than data; it’s the whole picture.

Effective trace collection requires smart sampling decisions. The OpenTelemetry sampling documentation provides clear guidance on when and how to sample. Refer to the Head-Based Sampling and Tail-Based Sampling. 

A critical decision in any tracing setup is your sampling strategy.

For the purpose of getting a complete picture in a distributed transaction trace, tail-based sampling is the recommended approach. It ensures that all spans for a given request flow are collected and processed together, which is essential for accurate visualization in tools like DX O2.

For DX O2's trace ingestion, it is a best practice that all correlated spans arrive together in a single request. This allows for proper processing and mapping. The best way to achieve this is to use a single OpenTelemetry Collector with tail-based sampling configured. This guarantees that complete traces across multiple microservices are captured and sent for analysis.

The Anatomy of a Span: Kinds and Attributes

In OpenTelemetry, a span represents a single operation within a trace. Two of the most important properties of a span are its kind,name and its attributes (or tags).

  • Span Kind: This defines the relationship of the span to its parent. The primary kinds are:

    • SERVER: Indicates a span that covers a request received by a service (e.g., an incoming HTTP request, a message being consumed from a queue).

    • CLIENT: Indicates a span that covers a request made to another service (e.g., an outgoing HTTP request, a database call).

    • PRODUCER: Represents a message being sent to a messaging system.

    • CONSUMER: Represents a message being received from a messaging system.

    • INTERNAL: Represents an operation within an application that doesn't cross a process boundary.

  • Span Name: A human-readable identifier for the operation (e.g., checkoutService/PlaceOrder, db.query). In DX O2, the span name becomes the entry point resource name in the Transaction Trace Viewer, helping you quickly recognize what the span represents in business and technical terms.

  • Attributes: These are key-value pairs that provide context about the operation the span represents (e.g., http.request.method, db.system, service.name).

These concepts are the raw materials that DX O2 uses to build its rich transaction trace view.

The DX O2 Transaction Trace Viewer: A Five-Section Experience

Once OpenTelemetry spans arrive, DX O2 builds a structured transaction trace and displays it in five main sections:

1. Trace List (Entry Points)

​​Quickly identify which service initiated the transaction.

Shows "frontend" spans — typically incoming requests such as:

  • HTTP server handlers

  • gRPC service calls

  • Message consumers

Span kinds usually mapped here: SERVER and CONSUMER

Use this button (highlighted in red box) to maximize the Trace List to view all traces when you see scrolling. 

Note: Internal or backend traces might appear in the Trace List if they are received later through different span/trace requests from the source.

2. Components (Internal and Outbound Calls)

Displays calls made within or from the service, called “backends” / “internal” spans. These are mostly internal calls or outgoing requests/exit calls:

  • Internal spans

  • HTTP/gRPC client calls

  • Database queries

  • Messaging producers

Span kinds typically shown here: CLIENT, PRODUCER, INTERNAL

​​See exactly which services the root request called and how they interacted. 

Components show traces into three views, Summary, Timeline and Tree views. 

Summary View:

Provides a tabular breakdown of all components in the selected transaction trace, showing duration, invocation time, and call counts,  ideal for quickly spotting which services were invoked and how often. Selecting each component will show trace context on component attributes.

 

Timeline View:

Display spans in a chronological “wedding cake” format, letting you see how calls overlap in time and where latency is introduced across the transaction flow.

 

Tree View:

Shows the full parent–child hierarchy of spans with percentage of time spent per call, making it easy to trace dependencies, identify bottlenecks, and understand contribution to overall transaction duration.

 

These three views (Summary, Timeline and Tree) let you switch perspectives between breadth, sequence , and depth  for  a complete understanding of how a request flows across services.

3. Component Summary / Root Component Details

​​Anchor your analysis by understanding the entry point and root context of the trace.

Includes:

  • Trace/Span names

  • Application and agent details

  • OpenTelemetry span_id and parent_span_id

DX O2 generates its own internal trace IDs but still preserves the OTel spans for trace navigation.

4. Component Performance

Displays operational insights for each span:

  • Duration

  • Duration percent

  • Timestamp

(CPU metrics may not apply to OpenTelemetry-based traces.)

5. Component Properties / Attributes

This section shows span attributes such as:

  • HTTP URLs

  • Database details

  • Messaging topics

  • Resource attributes

Attributes are displayed with an internal prefix (e.g., ATTR_), which can be ignored when interpreting the tags.

Leverage metadata like URLs or DB queries to speed up root-cause investigation.

 

Three are important properties of Span: 

These component properties combine raw OpenTelemetry data with DX O2’s enriched correlation fields. Resource attributes (ATTR_) identify where the span originated, span attributes describe what happened, and DX O2 adds fields like Component ID, Caller Component ID, and CorCrossProcessData to rebuild full transaction flows. DX O2 also surfaces error messages, stack traces, and correlation details, making it faster to triage issues and share precise evidence with remediation teams.

Closing of Part 1

We’ve seen how OpenTelemetry spans become meaningful narratives in the DX O2 Transaction Trace Viewer. From high-level entry points to the details of attributes, Part 1 showed us how to explore traces in practice. In Part 2, we’ll dig into the mechanics: span-to-component mappings, correlation rules, ingestion strategies, and a side-by-side look at DX O2 and Jaeger.


Try this today! Open a real trace in your DX O2 Transaction Trace Viewer. Start at the Trace List to find the entry service, follow its Components to see all internal and outbound calls, and use the Performance tab to identify where most time was spent. You’ll not just view a trace, you’ll uncover the flow of a real transaction.

0 comments
10 views

Permalink