The AI Metrics That Matter After the Prototype

A prototype that answers sample questions correctly is easy to celebrate. The same system, once it is handling real questions from staff or from the public, needs a different set of numbers. The numbers that matter are the ones that show whether the system is reducing work, whether it is creating new work, and whether the cost of running it is justified by the outcomes the organization already measures.

We track a small set of metrics on the AI systems we operate. They are chosen because they can be collected from the existing application logs and content systems, and because a content owner or a project sponsor can interpret them without a separate analytics project.

Review time and correction volume

The first pair of numbers is how long a person spends reviewing the system's output and how often that review results in a correction. For a reporting assistant, this is the time between the generated answer and the moment a reviewer marks it accepted or edited. For a content drafting workflow, it is the time from draft creation to publish or rejection.

A system that saves drafting time but doubles the review time has not reduced total effort. We compare the review time against the time the same task took before the system existed, using the team's existing estimates or ticket data where those are available. We also count corrections by type: factual error, missing citation, wrong tone, or incomplete answer. A rise in factual corrections after a content publish usually points to a retrieval or indexing lag rather than a model problem.

Hallucination and refusal incidents

We count incidents where the system produced a statement that was not supported by the retrieved sources, and incidents where it refused a question it should have been able to answer from the published content. Both numbers are taken from the sampled reviews and from any staff reports that include a trace identifier.

These counts are kept separate from general quality scores. A low overall error rate can hide a higher rate on the questions that involve fees, eligibility, or deadlines. We break the counts out by topic when the volume is high enough to make the breakout meaningful. The breakout is what tells the content team whether a particular page or policy needs attention.

Task completion and downstream actions

For public-facing assistants, we measure whether the session that used the assistant ended in a form start, a document download, or a call to a published number more often than sessions that used only search or navigation. The comparison is against the same pages and the same time period. An increase in assisted sessions that do not lead to any next step is treated as a signal to review the answers, not as engagement.

For internal assistants, the equivalent measure is whether the person who asked the question completed the task without opening a ticket or escalating to another team. We do not count the number of questions asked as success. A system that answers many questions but does not reduce the number of follow-up tickets has shifted work rather than removed it.

Cost per successful outcome

Token and infrastructure costs are recorded per skill or per assistant. We divide that cost by the number of reviewed answers that were accepted without correction, or by the number of downstream actions that the measurement above attributes to the assistant. The resulting figure is compared with the cost of the manual process it replaced, using the loaded cost of the staff time that process required.

This comparison is approximate. It is still more useful than a total monthly AI spend figure that has no denominator. A higher cost per answer can be acceptable when the answers concern regulated content and the alternative is a longer review cycle. A low cost per answer is not acceptable when a large share of the answers require correction.

Avoided work

Some value shows up as work that did not have to be done. Security review assistants that correlate new scan results with previous findings can reduce the number of duplicate tickets. Integration discovery that produces a reviewed mapping can reduce the number of clarification cycles during development. We track these as counts of items that were closed or de-duplicated because of the system's output, with a note on who confirmed the de-duplication.

These counts are easy to overstate. We only include an item when a person has marked it as resolved by the assistant's output and has not reopened it within a defined window. The window keeps the metric from counting temporary closures.

What we report and what we do not

The monthly summary for a sponsor contains the review-time comparison, the correction and incident counts, the task-completion or downstream-action rate, and the cost per successful outcome. It does not contain the number of prompts written, the number of model calls, or a composite "AI maturity" score. Those figures do not answer whether the system should continue, be narrowed, or be turned off.

The systems that remain in production are the ones where these numbers stay stable or improve after the first month, and where the content or process owners continue to review the samples. When the numbers worsen and the owners stop reviewing, the system is paused until the retrieval sources or the skill are updated. The metrics are the mechanism that makes that decision visible instead of leaving it to whoever notices the problem first.