Tracing LLM Requests
Follow one request as a trace: gateway decision, prefill and decode spans, and which one owns your p99 — plus what traces cannot show.
An interactive AI Infrastructure lesson: 18 steps, about 30 minutes, on a live simulation in your browser.
The support bot lives behind edge again: chat on Llama 3.1 8B for the easy questions, code on 70B for the hard ones. Dinner time, the pager fires: TTFT p99 over its 1 s SLO. Averages are fine. The tail is not.
Push 60 requests a second for 20 s and read the service gauges with me. Throughput is healthy, TPOT is flat — and TTFT p99 is seconds, with requests refused at the edge of it. A single number like that starts an investigation; it never finishes one. For the ending you need to see inside one request.
What you will learn
The alert names a number
- TTFT p99 pages at dinner: A p99 alert says the tail is sick. Finding the organ means opening one slow request, not averaging slower ones.
- Three gauges, one story: Waiting plus a p99–p50 gap with flat TPOT means queueing. No waiting means the serving path itself is slow.
- Open one request up: A trace is a request itemized: each span names one phase, and the phases add up to the latency the user felt.
Anatomy of one trace
- The widest bar wins: Optimize the widest span first. Prefill versus decode dominance is traffic shape, not model character.
- A 4,000-token prompt: Prompt length moves prefill; answer length moves decode. The trace shows which one your traffic pays.
- A 5 ms trace: Two spans and 5 ms total means the gateway answered alone. Shape first, then numbers: the shape already tells the story.
Every outcome has a shape
- The fallback leaves a mark: Fallbacks record themselves as spans. A silent incident with no fallback spans never took the escape route.
- The door says no: Short traces mean short paths: two spans is the gateway deciding alone. Slowness with only short traces is a limit, not a bottleneck.
- Five shapes, no new ideas: Every gateway outcome has one trace shape. Read shape first: it routes your attention before any number can.
Names travels share
- Span names are a contract: Shared span names and attributes let traces join across systems. The convention is the feature, not the collector.
- Sampled in production: Trace everything here; sample in production. Keep all of the rare outcomes and a thin slice of the routine.
- Drill: name the owner: Name the widest span before touching anything. Latency has an owner, and the trace is its address.
- Traces find it, metrics watch it: Metrics watch populations; traces explain individuals. Alerts fire on the first, investigations run on the second.
Spend the budget on spans
- A 1 s budget in spans: Budget latency per span, and name which spans your traces cannot show. Queueing is read off gauges, joined by time.
- The batch doubles: Batching spends the decode span to buy throughput. TPOT is the batching SLO because decode is the batching span.
- Drill: the decode share: Shares turn traces into priorities: 99% says optimize decode or go home.
Recap & playground
- Cheat sheet
- Playground