Observability

The intuition

Classical APM is a security camera at the front door: it tells you someone entered, how long they stayed, and whether the door jammed. It does not tell you whether the conversation inside was useful.

LLM observability needs a camera inside the room. A request that returns HTTP 200 with a fluent paragraph can still be wrong, ungrounded, or off-policy. You need the full transcript of what the model saw (the rendered prompt), what it said, which chunks retrieval returned, which tools ran, how many tokens each step burned, and whether a user later hit "regenerate."

Two layers fall out of that need: tracing (what happened, in order, with payloads) and quality monitoring (was it any good, over samples and trends). Without both, you debug by folklore and discover quality cliffs only when support volume spikes.

Why it exists

LLM products fail in ways that do not throw exceptions. The hard constraints:

1. Success ≠ quality. A 200 with a confabulated answer is a silent product failure. Classical error rates stay green.
2. Pipelines are multi-stage. Embed, retrieve, rerank, generate, tools — each can be the culprit. Flat logs of "the LLM call took 6 s" do not disambiguate.
3. You cannot human-label everything. Quality must come from cheap proxies on 100% of traffic, judges on a sample, and user signals joined to traces.
4. The world moves when your code does not. Input mix shifts, corpora go stale, providers repoint model aliases — behavior drifts with zero deploys.
5. Cost and latency regressions hide in aggregates. Without per-feature, per-model, per-prompt-version metrics, you learn from the invoice and from angry users.

The alternatives lose. Generic APM only sees an opaque HTTPS call. Logs without structure cannot reconstruct an agent tree. Judging every request inline adds latency and cost and still needs calibration. No feedback join collects thumbs that cannot teach the system anything.

Observability exists so a bad answer becomes a diagnosable trace in minutes, quality trends page humans before Twitter does, and the worst traces flow back into eval datasets (evals and testing).

The core idea

Classical observability tells you a request was slow or failed. LLM observability has to answer a harder question: the request succeeded — was the answer any good? So it's two layers. Layer one is tracing: every request becomes a trace with a span per stage — query embedding, retrieval, rerank, each LLM generation, each tool call — and LLM spans carry payloads normal APM never stores: the exact rendered prompt, the completion, model and parameters, token counts, cost, and TTFT. Tools like Langfuse and LangSmith exist because that payload capture, plus cost accounting and prompt-version linkage, is what generic APM lacks. When a user reports a bad answer, the trace tells you in minutes whether retrieval fetched the wrong chunks, the prompt was rendered wrong, or the model hallucinated over correct context — three different fixes.

Layer two is quality monitoring, because nothing throws an exception when the model confabulates. You can't judge every response, so you sample: run an LLM-as-judge over 1–10% of production traces scoring faithfulness, relevance, tone; collect explicit user feedback (thumbs, ratings) and — more reliable — implicit signals (regenerations, edits to generated text, abandonment, escalation to human). Trend those scores by prompt version and model; alert on drops. Add drift detection — input distributions and retrieval quality shift under you even when your code doesn't change — and you have the full loop: trace everything, score a sample, alert on trends, and feed the worst traces back into your eval datasets.

How it actually works

The data model (Langfuse terms; LangSmith is analogous): a trace is one end-to-end request, holding user ID, session ID, metadata, and tags. Inside it, observations: spans (any timed unit — retrieval, rerank, tool execution), generations (a span specialized for LLM calls: model, parameters, prompt, completion, usage, cost), and events. Instrumentation is decorator- or callback-based (@observe, OpenTelemetry-backed SDKs); both platforms ingest OTel, and OTel's GenAI semantic conventions are standardizing span attributes so instrumentation isn't vendor-locked. Nesting matters for agents: a 15-step loop is a tree of generations and tool spans — a graph/tree view of it is how you debug "why did the agent loop forever," which is unreadable in flat logs. See agent foundations and orchestration.

Metrics derived from traces — the dashboard that matters:

Sampled LLM-as-judge in production: async, off the request path (often via batch API at 50% cost). Judge sees input, retrieved context, and output; scores narrow binary criteria — "is the answer supported by the context?" — not a vague 1–10 "quality." Sample smartly: random baseline percent, plus oversample the interesting: negative feedback, long latencies, retries, low retrieval similarity. Judge scores attach to traces as scores, so you can slice: "faithfulness by prompt version over the last two weeks." Calibrate the judge against human labels before trusting its trend (see evals and testing for judge biases). RAG-specific metrics: RAG evaluation.

Feedback signals, in increasing reliability: thumbs up/down (sparse, biased toward anger); regeneration and retry rates; user edits to generated output (an edit diff is a free label of what was wrong); task abandonment mid-flow; escalation to human support. Wire them to the trace ID at capture time — feedback you can't join to the trace that produced it is nearly worthless.

Drift detection. Three kinds. Input drift: the query distribution shifts (new user segment, new product surface, seasonality) — detect via embedding-distribution monitoring (centroid distance, cluster shifts) or topic classification over time; your prompts and eval sets were tuned for the old distribution. Retrieval drift: mean top-k similarity scores decline, retrieved-chunk age skews stale, or coverage drops — the index no longer matches the questions (stale corpus, embedding-model change, chunking regression). Behavior drift: the provider updates the model (aliases like -latest repoint; even pinned snapshots get infra changes) — output length, refusal rate, format-failure rate, and judge scores move with zero deploys on your side. Baseline all three so "nothing changed but everything changed" is diagnosable.

The flows

A worked example

An employee on the governed enterprise platform reports: "It told me the travel pre-approval threshold is $1,000; our policy says $2,500."

Debug with a complete trace (minutes):

Root cause class: retrieval / corpus hygiene (stale draft still indexed and visible), not model confabulation. Fix owners: ingestion filters and entitlement metadata (filtering and metadata, production RAG) — not a prompt tweak.

Same user report without tracing: days of arguing about temperature, system-prompt wording, and "the model is dumb," while the draft document remains in the index.

Quality monitoring around the incident:

Common drill-downs

A user reports a wrong answer from your RAG system. Walk me through debugging. Pull the trace by user/session. Check spans in order: was the query embedded/rewritten sensibly; what chunks did retrieval return and with what scores (wrong chunks → retrieval problem: chunking, index freshness, query mismatch); was the rendered prompt correct (template bug); did the model answer against the supplied context or hallucinate (faithfulness problem → prompt constraints, model choice)? Three different failure classes, three different owners — the trace disambiguates in minutes.

What do Langfuse/LangSmith give you over Datadog? LLM-native capture: full prompt/completion payloads, token usage and per-call cost, model parameters, prompt-version linkage, nested agent/tool trees, plus quality tooling — attach judge scores and human annotations to traces, build eval datasets from production traces, prompt management. Generic APM sees an opaque 6-second HTTPS call; it has no notion of "this span cost $0.04 and scored 0.6 on faithfulness."

How do you monitor quality without labeling everything? Cheap computed proxies on 100% (schema failures, refusal rate, output length, retrieval similarity); LLM-as-judge on a sample (random % + oversampled suspicious traces), async and batched; user feedback joined to traces; all trended by prompt/model version with alerts on trend breaks — not on single outputs.

What is drift for an LLM app that never redeploys? Input drift (query distribution shifts away from what prompts/evals were tuned on), retrieval drift (corpus staleness or falling similarity degrades grounding), and provider behavior drift (model updates change refusal rates, formats, style). Detect via embedding-distribution stats, retrieval-score trends, and behavioral metrics against a baseline.

Which alerts would you actually page on? Hard failures: error/timeout/fallback rate, p95 TTFT. Money: cost per request, cache hit rate. Quality: judge-score trend break, refusal or schema-failure spike. Everything else — individual bad generations, single thumbs-down — goes to dashboards and weekly review, not pages.

How do you attribute a quality drop to a cause? Only via versioned traces: every trace carries prompt version, model snapshot, retrieval config. Slice judge scores by those dimensions — drop isolated to prompt v42 → rollback; drop across all versions on one model date → provider update; drop only on queries with low retrieval scores → corpus/index issue.

What's the privacy story for storing prompts? Prompts/completions are user data: redact PII pre-persistence, encrypt, retention-limit, restrict UI access, scope by tenant; self-host the tracing stack when data can't leave your infra. Sampling payloads (full metadata, partial payload capture) is a middle ground.

Production concerns

• Alert on aggregates and trends, not single bad outputs. Individual hallucinations are noise; a 10-point weekly drop in judge faithfulness, a refusal-rate spike after a model update, or cost-per-request doubling are signals. Page on: error/timeout rate, p95 latency, cost per request, cache hit rate collapse, judge-score trend breaks, schema-failure spikes.
• Payload storage is a liability. Prompts and completions contain user data — PII redaction before persistence, retention limits, access controls on the tracing UI, and tenant scoping. Self-hosting (Langfuse is open source) is often the answer to data-residency objections. See security.
• Overhead: capture must be async/batched; never put trace writes on the request path. Cost of judge sampling scales with rate — 1–5% random plus targeted oversampling is a common operating point (cost).
• Trace completeness beats trace volume. One fully-instrumented pipeline (every prompt render, every tool result, linked feedback) is worth more than 100% capture of only the final LLM call. The debugging killer is the rendered prompt — log what the model actually saw, not the template.
• Version everything into the trace: prompt version, model snapshot, retrieval config, feature flags. Quality regressions are only attributable if the trace says which combination produced the output.
• Close the loop: the worst production traces (judge-flagged, user-flagged) should flow into curated eval datasets weekly — observability feeds evaluation, evaluation gates the next deploy (evals and testing).

Test yourself

Go deeper

• LLM Observability & Application Tracing — Langfuse docs — traces, observations, generations, scores; OTel-based SDKs.
• Observability — LangSmith docs — tracing, dashboards, alerts, production monitoring workflow.
• Your AI Product Needs Evals — Hamel Husain — the loop connecting production traces, error analysis, and eval datasets.
• Langfuse Intro — Observability & Tracing Deep Dive — Langfuse — official walkthrough of the tracing data model in practice.

Where this connects

• Evals & testing — production traces become golden cases; online metrics gate canaries that offline suites cannot fully predict.
• Latency and Cost — the metrics dashboards are fed by the same spans that debug quality.
• Reliability — timeout, error, and fallback rates are operational signals, not only quality ones.
• RAG evaluation — component metrics for retrieval vs generation when the product is grounded Q&A.