LLM Observability: What to Measure Before Your Users Do

Traditional observability answers "is the service up and fast?" LLM observability has to answer a harder question: "is the service right?" A model can return 200 OK in 900ms while confidently giving your customer the wrong cancellation policy. Uptime dashboards will show a perfectly healthy system losing your users' trust one plausible-sounding error at a time.

This post describes the observability stack we deploy with every LLM system, organized around three timescales: per-request traces, online quality signals, and offline evaluation.

Traces: the unit of debugging is the request, not the log line#

Every LLM request should produce a structured trace capturing the full causal chain of the response:

{
  "trace_id": "tr_9f2c",
  "model": "provider/model-id@2026-04-11",
  "prompt_template": "support_answer@v14",
  "prompt_tokens": 2210,
  "retrieved_chunks": [
    { "doc": "refund-policy.md#s3", "score": 0.82 }
  ],
  "completion_tokens": 187,
  "latency_ms": { "retrieval": 240, "ttft": 620, "total": 2340 },
  "output_parsed": true,
  "guardrail_flags": [],
  "feedback": null
}

Three fields here do disproportionate work. Versioned identifiers (model@date, template@v14) turn "quality dropped last Tuesday" from an archaeology project into a GROUP BY. Retrieved chunks with scores let you distinguish retrieval failures from generation failures — in our experience the split is roughly 60/40 in favor of retrieval being the culprit, which surprises teams who spend all their tuning effort on prompts. output_parsed is the cheapest quality metric that exists: if you expect JSON and got prose, you know without any LLM judge that the request failed.

Store traces in your existing telemetry pipeline (OpenTelemetry spans work fine — model calls are just spans with attributes) but keep payloads long enough to replay. Replay is the point: a trace you can't re-run against a candidate fix is a screenshot, not a test case.

Online signals: proxy metrics that correlate with quality#

You cannot LLM-judge every production response at scale economics; you can compute cheap proxies on all of them and expensive judgments on a sample. The proxies that have earned their place on our dashboards:

• Schema validity rate — parse failures per template. A drop of even 2 points almost always precedes user complaints.
• Refusal rate — responses matching refusal patterns. A spike means either a prompt regression or a genuine input-distribution shift; both are worth knowing within the hour.
• Groundedness sampling — for RAG systems, an LLM judge scores a 2–5% sample for "is every claim supported by the retrieved context?" We alert on 7-day moving average, not point values.
• Retrieval health — mean top-k score and the rate of queries where no chunk clears the relevance threshold. The latter is your "we don't have content for this" signal and doubles as a content-gap report for the documentation team.
• User feedback, weighted honestly — thumbs-down rates are noisy and biased (angry users click more), but their trend within a fixed cohort is meaningful. Regeneration rate ("user asked again, rephrased") is a stronger dissatisfaction signal than explicit feedback and requires no UI.
• Token and cost per resolved task — not per request. Agentic systems can hide regressions in loops: cost per request stays flat while requests per task doubles.

The alerting philosophy: alert on distribution shifts, not individual bad outputs. One hallucination is a sample; a 3-sigma move in groundedness is an incident.

Drift: the model changes even when you don't#

Enterprise LLM systems drift from three directions. Input drift — your users start asking about a product you launched yesterday; retrieval has no content; quality craters for that slice while aggregate metrics barely move. Slice your metrics by intent cluster, not just globally. Provider drift — hosted models get updated. Pin model versions where the provider allows it, and when they don't, run a daily canary suite (50–100 golden prompts with known-good outputs, compared via judge) against the live endpoint. We've caught two silent provider-side behavior changes this way, both before users noticed. Data drift — the corpus your RAG system indexes changes underneath you. Track index freshness and re-run retrieval evals after bulk content updates.

Offline evals: the regression suite for behavior#

Every prompt change, model upgrade, and retrieval tweak should pass an evaluation suite before deployment — the same discipline as a test suite, adapted for non-determinism.

A workable enterprise eval suite has three tiers:

1. Assertions (hundreds, milliseconds each): deterministic checks — output parses, required fields present, no forbidden phrases, citations resolve to real documents. These run in CI on every commit.
2. Golden sets (100–500 cases, minutes): real (sanitized) production queries with graded reference answers, scored by exact-match where possible and LLM judge where not. Judges are calibrated quarterly against human raters; we require judge–human agreement above 85% before trusting a judge on a new task, and we always judge pairwise (candidate vs. current) rather than absolute scores, because pairwise judgments are markedly more stable.
3. Failure library (grows forever): every production incident becomes a permanent eval case. This is the highest-value tier per case — it encodes exactly the failures your system is prone to. Our longest-running deployment has 340 cases in its failure library; new model versions fail 5–15 of them on first try, every time, and that's the system working.

Gate deployments on tiers 1 and 3 strictly; treat tier 2 as a scored report requiring sign-off when it moves more than a threshold.

Where to start#

If you're instrumenting an existing LLM system this quarter, sequence it this way: structured traces with versioned identifiers first (one sprint, mostly plumbing), schema-validity and refusal dashboards second (days), a 100-case golden set third (the real work is labeling, not tooling), and judge-sampled groundedness last.

The mindset shift matters more than the tooling: an LLM feature without evals is not "working" — it is unobserved. Those are different states, and the difference is usually discovered by a customer, a regulator, or a screenshot on social media. Instrument first; the model will give you plenty to find.