Observability

The intuition

Classical APM is a camera at the front door. It tells you someone entered, how long they stayed, and whether the door jammed. It does not tell you whether the conversation inside was useful.

LLM observability needs a camera inside the room. A request can return HTTP 200 with a fluent paragraph and still be wrong, ungrounded, or off-policy. You need the full transcript: the rendered prompt the model actually saw, the completion, which chunks retrieval returned, which tools ran, token burn per step, and whether a user later hit regenerate. That need splits into two jobs: tracing (what happened, in order, with payloads) and quality monitoring (was it any good, over samples and trends). Without both, you debug by folklore and discover quality cliffs only when support spikes.

Why it exists

LLM products fail without exceptions:

1. Success ≠ quality. A 200 with a confabulated answer leaves classical error rates green.
2. Pipelines are multi-stage. Embed, retrieve, rerank, generate, tools — flat logs of "the LLM took 6 s" do not name the culprit.
3. You cannot human-label everything. Quality needs cheap proxies on 100% of traffic, judges on a sample, and user signals joined to traces.
4. The world moves when your code does not. Input mix shifts, corpora go stale, providers repoint model aliases.
5. Cost and latency regressions hide in aggregates. Without per-feature, per-model, per-prompt-version metrics, you learn from the invoice and angry users.

The alternatives lose. Generic APM sees an opaque HTTPS call. Unstructured logs cannot reconstruct an agent tree. Inline judges add latency and cost. Unjoined feedback cannot teach the system.

Observability exists so a bad answer becomes a diagnosable trace in minutes, quality trends page humans before Twitter does, and the worst traces flow into eval datasets (evals and testing).

The core idea

Classical observability answers "was it slow or broken?" LLM observability has to answer a harder question after a 200: was the answer any good?

Tracing turns every request into a tree of timed units. Each stage — embed, retrieve, rerank, generation, tool call — is a span. LLM spans carry payloads normal APM never stores: the exact rendered prompt, the completion, model and parameters, token counts, cost, and TTFT. Langfuse and LangSmith exist because that capture, plus cost accounting and prompt-version linkage, is what generic APM lacks. A bad-answer report becomes a walk of the tree: wrong chunks, wrong prompt render, or hallucination over correct context — three different fixes.

Quality monitoring exists because nothing throws when the model confabulates. Sample an LLM-as-judge, collect user signals, trend scores by version, alert on drops. Add drift detection for silent shifts in inputs, retrieval, and provider behavior. Loop: trace what matters, score a sample, alert on trends, feed the worst cases into evals.

How it actually works

The data model (Langfuse terms; LangSmith is analogous). A trace is one end-to-end request (user ID, session ID, metadata, tags, environment). Inside it sit observations — nested steps: spans (any timed unit: retrieval, rerank, tool execution), generations (LLM-specialized spans: model, parameters, prompt, completion, usage, cost), and events. Scores attach later — judge output, user thumbs, custom metrics — so you can slice "faithfulness by prompt version." Multi-turn products group traces into sessions. Instrumentation is decorator- or callback-based (@observe, framework callbacks, OTel-backed SDKs). The industry is converging on OpenTelemetry GenAI semantic conventions: standard gen_ai.* attributes for provider, model, operation name (chat, execute_tool, embeddings), and token usage — portable even while parts of the spec remain in Development status as of mid-2026. Nesting matters for agents: a 15-step loop is a tree that flat logs cannot debug (agent foundations, orchestration).

Metrics from traces: tokens and cost per request (by feature/model/prompt-version); TTFT and per-stage latency (latency); error/timeout/fallback rates (reliability); cache hit rates (cost); refusal and schema-failure rates; judge scores and user feedback trended by version.

Sampled LLM-as-judge. Run async, off the request path — often via a queue into the batch API at ~50% token price. Score narrow binary criteria ("is the answer supported by the context?"), not a vague 1–10. Sample a random baseline percent, plus oversample negative feedback, long latencies, retries, and low retrieval similarity. Attach results as scores; calibrate against human labels first (evals and testing; RAG metrics: RAG evaluation).

Feedback, ordered by reliability: thumbs (sparse, anger-biased); regenerations; user edits (an edit diff is a free wrongness label); abandonment; escalation. Wire every signal to the trace ID at capture time — unjoined feedback is nearly worthless.

Drift detection — three kinds, independent of deploys. Input drift: query distribution shifts (embedding-distribution stats or topic labels). Retrieval drift: mean top-k similarity falls, chunks age, or coverage drops. Behavior drift: the provider updates the model (aliases like -latest repoint) — refusal rate, format failures, and judge scores move with zero deploys on your side. Baseline all three.

The flows

A worked example

An employee on the governed enterprise platform reports: "It told me the travel pre-approval threshold is $1,000; our policy says $2,500."

With a complete trace, the walk takes minutes. Metadata shows prompt_version=v42, model=…-latest, feature=policy-qa. The embed rewrite is fine. Retrieval ranked a 2024 draft chunk #1 (score 0.84); the 2026 effective policy is #4. Rerank left the draft on top. The generation's rendered prompt shows the model correctly quoted that draft; offline judge scores faithfulness 1 against retrieved context. Judge green, user red.

Root cause class: retrieval / corpus hygiene (stale draft still indexed), not model confabulation. Fix owners: ingestion filters and entitlement metadata (filtering and metadata, production RAG) — not a prompt tweak. Without tracing, the same report burns days arguing about temperature while the draft stays in the index.

Quality monitoring tells a different story from faithfulness alone. Random 5% judge sample: faithfulness stable. Retrieval mean top-1 similarity: down ~8% week-over-week on policy intents. Thumbs-down: up on feature=policy-qa only. Cost flat. Action: page on retrieval-score drift plus thumbs by feature; open a corpus-freshness incident. Without per-stage spans, rendered prompts, or version tags, you cannot assign the failure class or the owner. Judge-on-100%-inline blows up latency; feedback without trace IDs cannot teach the system; raw payload retention forever turns the stack into a PII liability (security).

Production concerns

Alert on trends, not single bad generations. Page on error/timeout/fallback rate, p95 TTFT, cost per request, cache-hit collapse, judge-score trend breaks, and schema-failure spikes. Individual hallucinations and single thumbs go to dashboards and weekly review.

Payload storage is a liability and a cost dial. Prompts and completions are user data: redact PII before persistence, encrypt at rest, set retention limits, restrict tracing-UI access, and scope by tenant. Self-hosting (Langfuse is open source) often answers data-residency objections (security). At high volume, store full metadata everywhere and full payloads on a sample — or put bodies in object storage with a span reference. Oversized traces hit vendor size limits and can be silently dropped.

Capture must be async; completeness beats volume. SDKs batch and flush in the background; short-lived workers must flush() before exit. Judge sampling at 1–5% random plus targeted oversample is a common operating point (cost); keep error paths and feedback-linked traces at 100%. One fully instrumented pipeline beats 100% capture of only the final LLM call. Version everything — prompt version, model snapshot (not only the alias), retrieval config, feature flags — and feed worst traces into eval datasets weekly (evals and testing).

Common drill-downs

A user reports a wrong RAG answer. Walk me through debugging. Pull the trace by user/session. Walk spans: query rewrite → retrieval chunks and scores (wrong chunks → chunking, freshness, query mismatch) → rendered prompt (template bug) → model answer vs context (faithfulness → prompt, model). Three failure classes, three owners — the trace disambiguates in minutes.

What do Langfuse/LangSmith give you over Datadog APM alone? LLM-native capture: prompt/completion payloads, token usage and per-call cost, model parameters, prompt-version linkage, nested agent/tool trees, plus quality tooling (judge scores, annotations, eval datasets). Generic APM sees an opaque multi-second HTTPS call. Teams often run both: OTel GenAI spans into an LLM platform for quality, correlated spans into APM for infra.

How do you attribute a quality drop? Only via versioned traces. Slice judge scores by prompt version, model snapshot, and retrieval config. Drop isolated to prompt v42 → rollback; across all versions on one model date → provider update; only on low-retrieval-score queries → corpus/index issue.

Test yourself

Go deeper

• LLM Observability & Application Tracing — Langfuse docs — traces, observations, generations, scores; OTel-based SDKs.
• Core Concepts (data model) — Langfuse docs — traces, sessions, observation types.
• Observability — LangSmith docs — tracing, dashboards, alerts, feedback.
• OpenTelemetry for Generative AI — OpenTelemetry blog — GenAI semantic conventions as portable instrumentation.
• Your AI Product Needs Evals — Hamel Husain — production traces → error analysis → eval datasets.

Where this connects

• Evals & testing — production traces become golden cases; online metrics gate canaries that offline suites cannot fully predict.
• Latency and Cost — the metrics dashboards are fed by the same spans that debug quality.
• Reliability — timeout, error, and fallback rates are operational signals, not only quality ones.
• RAG evaluation — component metrics for retrieval vs generation when the product is grounded Q&A.