"The AI said it" is not a defense. We learned that the hard way during an internal review when someone asked, quite reasonably, "Can you show me exactly why the system gave that answer on that date?"
We could not.
The original RAG implementation logged the final answer and a rough list of source documents. That was enough for debugging. It was nowhere near enough for audit or legal scrutiny.
So we rebuilt the tracing layer.
Every production answer now carries a durable decision record that includes:
- The exact user question (and any rewritten version the system used)
- The full set of retrieved chunks with their document IDs, section anchors, and the ranking scores at retrieval time
- The model version, temperature, and system prompt that were active
- The final generated answer
- A cryptographic hash of the above so we can detect later tampering
- Timestamps and the identity of the calling application or user (where appropriate)
We also surface a human-readable citation block under every answer. It doesn't just say "Source: Product Guide."" It links to the specific section that was used and shows a short excerpt so the reader can verify the claim without hunting.
For high-stakes domains we added a second layer: the system can be configured to refuse to answer if it cannot produce citations that meet a minimum confidence threshold, or to route the question to a human with the full decision record attached.
The storage cost is real but manageable. We keep the full decision records for a configurable retention period and then move older ones to cheaper cold storage. The citation links themselves are generated at answer time from the same chunk metadata the retriever already has, so the extra latency is small.
The biggest change was cultural. Once people could click through from an AI answer to the exact paragraph that justified it, trust went up and the "the AI is making things up" complaints dropped sharply. When something was wrong, we could point to the precise source that was outdated or misinterpreted instead of arguing about model behavior in the abstract.
We also discovered that good citations improve the content itself. Authors started writing tighter, more self-contained sections once they realized those sections would be quoted back to users by the AI.
Legal and compliance teams still review the overall system, but they no longer treat every AI answer as an untraceable black box. That single shift moved us from "interesting prototype" to "system we are willing to put in front of customers and auditors."