We did not change the model. We changed everything around it.
Early versions of our RAG assistant hallucinated at a rate that made internal users distrust it and external users complain. The classic symptoms appeared: confident answers that mixed two different product versions, invented configuration steps, and occasional claims that simply did not exist in the source material.
Throwing a larger model at the problem felt like the obvious move. We resisted. Instead we treated hallucination as a retrieval and grounding problem first.
The biggest single improvement came from tightening the retrieval stage. We moved from pure vector search to a hybrid approach that combines vector similarity with traditional keyword filters and metadata constraints (product, version, content type, last-updated date). We also added a lightweight re-ranker that scores chunks on both semantic relevance and "answerability"—whether the chunk actually contains the kind of information the question is asking for.
Next we changed how we construct the context window. Instead of stuffing the top-k chunks in rank order, we now do a small clustering pass and preferentially include chunks that cover different aspects of the question. We also inject explicit "do not invent" instructions that reference the provided sources by ID, and we force the model to quote or paraphrase from those sources when making factual claims.
We added a post-generation verification step for answers that look factual. A second, cheaper pass checks whether the key claims in the generated answer can be found (via embedding similarity plus exact phrase matching) in the retrieved chunks. If the overlap is too low, the system either regenerates with stricter instructions or falls back to a "I don't have enough grounded information" response.
Finally we instrumented everything. Every answer is scored offline against a growing evaluation set of real user questions with known-good answers. We track groundedness, relevance, and hallucination rate as first-class metrics and review the worst offenders weekly.
None of these changes required a new model. They required better data hygiene, tighter retrieval, clearer prompting, and a verification step that treats the model's output as untrusted until proven otherwise.
The measured hallucination rate on our production evaluation set dropped by roughly 60%. More importantly, the remaining errors became more predictable and usually pointed back to missing or ambiguous source content rather than pure model invention.
The lesson we keep repeating internally is simple: in a retrieval-augmented system, most "model" problems are actually retrieval, chunking, or prompting problems in disguise. Fix those first. The model is often the least interesting part of the stack.