Most RAG systems die the same quiet death. You launch, the answers look decent in demos, and then six months later the knowledge base is stale, the content team has no idea what's being retrieved, and the AI is confidently wrong about things that changed last quarter.
We decided that wasn't acceptable.
Our production assistant answers questions against a large corpus of product documentation, release notes, support articles, and internal process pages. Early on we treated the content team as the source and the AI as a pure consumer. That lasted about eight weeks. After that the complaints started: "Why is it still saying X when we updated that page in March?" and "I never even knew the AI was using this article."
So we built a closed loop.
Every answer the system returns now carries a small, non-intrusive "Was this helpful?" control plus a one-click "Flag this answer" button. When someone flags it, three things happen automatically:
- The full prompt, retrieved chunks, and final answer are logged with a unique correlation ID.
- The content owners of the source pages that were actually used receive a short notification in their existing work queue (we use the same system they already live in for content requests).
- A lightweight review form appears that lets them mark the answer as correct, partially correct, or wrong, and optionally leave a note or create a content ticket directly from the flag.
We deliberately kept the content owner experience minimal. They don't have to log into a separate AI dashboard. They just see a familiar-looking task: "The AI used your page on [title] and produced this answer. Does it still reflect reality?"
On the engineering side we store the feedback against the exact chunk IDs and the embedding version that was current when the answer was generated. That lets us do two useful things. First, we can surface "stale high-traffic chunks" in a weekly report. Second, when a content owner updates a page, we can automatically re-index only the affected sections and then re-run the most frequently flagged questions that previously pulled those chunks.
The biggest cultural win wasn't technical. Once content owners could see exactly how their pages were being used (and misused) by the AI, they started writing differently. Headings became more precise. Ambiguous pronouns disappeared. Edge cases that used to live only in people's heads started showing up as explicit call-outs. The content got better because the authors could finally see the consequences of their choices.
We still have humans in the loop for anything that looks high-stakes or legally sensitive. The feedback system doesn't auto-publish corrections. It just makes the distance between "AI said something wrong" and "the right person knows about it" measured in hours instead of weeks.
That single change turned the AI from a black box that occasionally embarrassed us into a system that actively improves the underlying knowledge base.