RAG explained: how to build an AI that doesn't lie

RAG stands for retrieval-augmented generation: a language model that, before answering, first runs a search over a collection of documents you control — your knowledge base, your contracts, your case law — and feeds the passages it finds to the model as context. The model no longer makes things up out of thin air; it reads along with what's actually there first.

The short answer to the title: you build an AI that doesn't lie with four ingredients — solid retrieval (finding the right passages), a context that forces the model to stick to the source, a generation step that explicitly cites, and independent verification that checks every claim against the source text. Skip that last step and you've built RAG that reduces hallucination — not a system that eliminates it. The rest of this article explains why, and how to close that gap.

What RAG actually is

A language model on its own answers from two sources: what it memorized during training (parametric knowledge, frozen at a cutoff date) and whatever you provide in the conversation. For many questions that's enough. But once an answer needs to rest on current, specific, or large volumes of documents — case law, internal procedures, a product catalog — parametric knowledge falls short: the model never saw it, half remembers it, or knows an outdated version.

RAG solves that by putting a retrieval step in front. For every question, the system first searches its own document collection, selects the most relevant passages, and drops them into the prompt as context. The model then generates an answer conditioned on that context — instead of freely associating from training. That's the whole idea: not making the model bigger, but handing it the right briefing before it answers.

Why "just a bigger model" isn't enough

It's tempting to see hallucination as a model problem the next generation will fix. That's only partly true. Even the best language model has no built-in mechanism for saying "I'm not sure" — it always generates the most probable continuation, whether that's correct or not. And without retrieval, the model also has no access to your current or internal information, no matter how large it is. A language model trains on the public internet up to a certain date; it doesn't know your contract from last week, nor yesterday's ruling.

RAG partly addresses both problems: it gives access to current, proprietary data, and it reduces the odds of fabrication because the model has something to refer to. "Partly", because a model can still misrepresent a detail from the context, conflate two passages, or claim something the text doesn't quite say. Why that's so persistent, and what actually helps, is laid out in Why RAG without citations is worthless.

The architecture in five steps

1. A question in plain language. The user asks a question the way they'd ask a colleague — no keywords, no special syntax.

2. Retrieval over your own data. A search layer — vector search, full-text search, or a combination — searches the document collection and selects the passages closest to the question.

3. Context assembly. The retrieved passages, together with their provenance (document name, article number, case reference, page number), are combined into the context sent to the model.

4. Generation with a citation requirement. The model is instructed to answer only from the supplied context, and to tie every claim to a source.

5. Verification of every claim. A separate check step verifies that every citation in the answer actually appears in the source text it references. If it doesn't, the claim is flagged or removed rather than presented as fact.

Here's what that looks like at a scale beyond a demo: in our own Neuralex Legal showcase, this exact architecture searches 3.6 million Dutch court rulings in 70 to 300 milliseconds, with a source card (case reference + exact quote) attached to every claim in the answer.

Where RAG usually breaks in production

Building a RAG demo takes an afternoon; building a RAG system that survives a day with real users is a different story. The most common failure points: retrieval that ranks the wrong passage first, chunking that cuts a sentence in half and loses context, a generation step that ends up associating freely instead of sticking to the source, and simply lacking a benchmark to tell whether a change made things better or worse. A detailed breakdown of those five failure points is in From RAG demo to production: the five things that break.

How you actually know it works

"The answer sounds good" is not a benchmark. A RAG system you want to rely on needs an evaluation harness: a fixed set of test questions with a known-good answer (a golden set), against which you automatically measure every change to retrieval, chunking, or the prompt. Without that benchmark you're steering on gut feeling, and you'll discover a regression only when a client or colleague runs into it — too late.

Frequently asked questions

Is RAG the same as fine-tuning a model? No. Fine-tuning changes the model's weights based on examples; RAG leaves the model untouched and adds relevant documents to the context at query time. RAG is usually cheaper, updates instantly as new documents arrive, and enables citations — fine-tuning doesn't.

Does RAG fully prevent hallucination? Not on its own. The model can still claim something that isn't in the supplied context. Only a separate verification step that checks every claim against the source text closes that gap.

Do you need a vector database? Not necessarily. Vector search is one way to find relevant passages, but full-text search or a hybrid approach works at least as well in many domains — especially for exact terms, numbers and legal references.

Conclusion

RAG isn't a trick, it's an architectural choice: give a language model something to point to, force it to cite, and independently verify those citations. Skip that last part and you have a demo that lies less often. Do it, and you have a system you can actually base a decision on — which is exactly the difference between "AI that impresses" and "AI you can trust".