AI observability helps organizations keep production assistants reliable after launch by tracking traces, evaluating real-world answers, and escalating high-risk failures to the right people. Learn how to connect health and quality monitoring, offline evals, human review, and cost tracking into a practical feedback loop that keeps AI systems accurate, safe, and operational.
Most of the work that makes an AI system trustworthy happens after the demo. Retrieval drifts when a page is rewritten. A model provider updates a version you thought you had pinned. A seasonal program starts and visitors ask questions the corpus does not cover. A skill change that improved the eval suite makes a small class of real answers worse.
If you only watch infrastructure uptime, you will learn about those failures from a staff member or a customer. Observability for a production assistant is the practice of seeing the answer being formed, judging it on a schedule, and routing the cases that need a person before they spread.
We already write about feedback loops with content owners and about cutting hallucinations without switching models. This post is the operational layer those practices sit on.
Trace the path, not only the final sentence
A usable trace records the sequence that produced an answer: the sanitized question, the skill and prompt versions, the model identifier, the retrieval query, the documents and chunks returned, the tool calls, the verifier result, latency and token counts, and the text that was shown.
That list looks long until you try to debug without it. "The assistant gave the wrong fee" can mean the index still held a retired page, the retriever ranked a brochure above the policy, the skill allowed paraphrase of a number, or a tool ran against an outdated view. Each of those fixes belongs to a different owner.
We attach a correlation identifier to the interface so a staff member can report "answer 7f3c" instead of forwarding a screenshot with the time zone cropped off. The trace is stored according to the same retention and redaction rules as other application logs. Public site questions can contain personal details even when you asked them not to. Treat the log as sensitive.
Separate health from quality
A 200 response from the model endpoint means the system spoke. It does not mean the system helped. We keep two dashboards.
Health covers availability, latency, tool error rates, index freshness, and budget burn. If retrieval is stale or a tool is failing closed, quality review will waste time.
Quality covers refusal rate, citation validity, human override rate, sampled accuracy, and the topics that produce repeated follow-up questions. Those numbers need a denominator. A rising refusal rate can mean the model got more careful, or it can mean a program launched and you did not publish the page.
Do not average quality into a single score that leadership watches like uptime. A 94 percent "good answer" rate can hide a 40 percent failure rate on eligibility questions, which may be the only questions that create institutional risk.
Offline evals catch regressions. Online sampling catches reality.
Before a skill or model change ships, we run the versioned eval suite. That suite is necessary and incomplete. It will not contain the wording a visitor uses at 10 p.m. after reading a news story about your program.
Online, we sample production traces on a cadence the team can actually review. A daily sample during the first weeks after launch, then a weekly sample once the error classes stabilize, has been enough for the systems we operate, with extra sampling when a content publish or model pin changes.
Reviewers score a small number of fields: grounded in the cited source, complete enough for the question, safe refusal when required, and no invented next step. Disagreements go to the content owner. The point of sampling is not to grade the model. It is to find broken sources, missing pages, and skill rules that do not match how people ask.
When a sampled miss repeats, it becomes an eval case. That is how the offline suite stays honest.
Decide in advance which failures page a human
Not every bad answer is an incident. If you page people for tone, they will stop reading the queue. We define escalation classes with the client before launch.
Immediate attention goes to answers that give a wrong price, deadline, eligibility rule, safety instruction, or legal condition, and to any case where a tool executed a side effect the verifier should have blocked. Those traces alert the operating engineer and the content owner for that source.
Same-day review goes to clusters: the same topic failing several times, a spike in refusals on a previously stable intent, or citation links that 404. Those are usually publish or index problems.
Backlog review goes to awkward phrasing, incomplete but not dangerous answers, and single misses that do not repeat. They feed the next skill or content change rather than an on-call rotation.
The routing has to name people who are available. An alert that lands in a shared inbox nobody staffs is decoration.
Close the loop without building a second records system
When a reviewer marks an answer wrong, the default repair is not a new prompt. It is a check of the source page and the index. If the page is wrong, the page gets fixed and republished. If the page is right and retrieval missed it, the chunking or metadata gets fixed. If retrieval was right and the skill overreached, the skill version changes through the same review path as any other release.
The reviewer note is stored on the trace and, when the miss is durable, copied into the eval suite. We do not maintain a parallel FAQ that only the assistant reads. That path is how organizations create a second source of truth and then wonder why the site and the copilot disagree.
Cost and latency belong on the same board
Quality work dies when nobody can see what it costs. Token use, cache hit rate, and a breakdown by skill tell you whether a verification step is worth what it adds. We have kept extra review passes that were cheap relative to the risk, and we have removed a second model call that added latency without changing sampled accuracy.
Observability that ignores spend will be turned off in the first budget cycle. Observability that only shows spend will train the organization to prefer a cheaper model that fails on the questions that matter.
What "good enough to operate" looks like
A production assistant is operable when a new engineer can answer three questions from the tools you already have: which skill version said this, which documents it used, and who gets notified if that class of answer is wrong again. If those questions require archaeology in chat logs, you launched a prototype and left it running.
Traces show how an answer was built. Evals and samples show whether that method still works. Paging rules decide which failures are another person's afternoon. Put those three in place and the feedback loop with content owners has something concrete to work from. Leave them out and you will keep arguing about the model when the problem is sitting in last month's unpublished page.