Distributed Tracing

What it is

A technique for tracking a single request as it propagates through a distributed system. A trace is the whole request's journey; it's made of spans, one per unit of work (a service call, a DB query), each with a start/end time, parent link, and metadata. A trace ID is propagated across every hop so the pieces can be reassembled into one end-to-end timeline.

The problem it solves

In microservices, one user action triggers a cascade of internal calls. When it's slow or failing, per-service metrics tell you each service's health but not how they compose for this request. Distributed tracing reconstructs the exact path and timing across all services, so you can see that the 800ms latency was 700ms waiting on one downstream call, turning 'somewhere it's slow' into 'here it's slow.'

How it works

When a request enters the system, a trace ID and root span are created. Each service that handles it creates a child span (with a span ID and parent ID) and propagates the trace context (typically via HTTP headers like W3C traceparent, or message metadata) to every downstream call. Each service reports its spans to a tracing backend (Jaeger, Tempo, Zipkin), which joins them by trace ID into a waterfall. Sampling decides which traces to keep; OpenTelemetry is the standard instrumentation layer.

Why use it

• Pinpoints exactly which service/hop owns the latency or error in a multi-service request
• Reveals the real call graph and dependencies, including surprise N+1 fan-outs
• OpenTelemetry standardizes instrumentation across languages and backends

What it costs you

• Context must be propagated through every hop, and one un-instrumented service breaks the trace into disconnected pieces
• Full-fidelity tracing is expensive at scale; sampling is required and can miss the rare bad trace if done naively
• Async/queued hops (a message sits in a queue for minutes) make spans awkward, so you must propagate context through the message

Where it shows up in our architectures

• Netflix →

  Tracing across hundreds of services is the only practical way to localize a slow API call to the responsible downstream

• Uber →

  A trace follows a ride request through dispatch, geo-index, and trip services to show which tier added latency during a surge

• Payment Gateway →

  A charge's trace spans the gateway, fraud check, ledger write, and async webhook so a stuck payment can be located precisely

Gotchas

• A single un-instrumented service (or one that drops the trace headers) breaks the chain, and downstream spans become orphans. Propagation must be end-to-end to be useful.
• Tail-based sampling beats head-based for debugging: decide whether to keep a trace after you know it was slow or errored, not at the start when every request looks the same.
• Tracing across async boundaries (queues, batch jobs) requires explicitly carrying the trace context inside the message; it won't flow automatically like it does over HTTP.
• Clock skew between hosts can make a child span look like it started before its parent. Tracing backends correct for this, but don't trust raw cross-host timestamps to the microsecond.