From RAG demo to production: the five things that break
You can build a RAG demo in an afternoon. In production it breaks in five predictable places: retrieval, chunking, fabricated citations, the missing benchmark, and scale. What we ran into on 3.6 million court rulings.
A RAG demo isn't hard. Ingest documents, cut them into chunks, generate embeddings, add a vector database, put a model in front — and on the ten questions you thought up yourself it works. That's exactly the problem: a demo proves the pipeline runs, not that the answers are correct.
We run a RAG system on 3.6 million Dutch court rulings. Retrieval runs on SQLite FTS5 (an index of roughly 19 GB) combined with pgvector for vector search. Along the way, the same five things broke that break in nearly every RAG project. Here they are, with what helps against them.
1. Why does my RAG system retrieve the wrong documents?
Because semantic search on its own isn't built for exactness. An embedding captures meaning, and meaning is precisely what you do not want when it comes to a case number, an article number, a proper name or a date. Search on a specific number and a pure vector search returns documents that are "about" it instead of the document that contains it. For legal, financial and technical domains — where identifiers are the core of the question — that's a fundamental problem, not a rounding error.
The practical solution is hybrid retrieval: full-text search alongside vector search, with the results merged. Full-text (FTS5 in our case) catches exact terms, numbers and proper names; the vector side (pgvector) catches paraphrases and related concepts. On top of that runs a reranker that reorders the combined candidate list by actual relevance to the question.
Hybrid retrieval plus reranking isn't a luxury optimization for later. It's the difference between a system that finds the right document and a system that finds something that looks like it.
2. What goes wrong with chunking?
Chunking is splitting documents into blocks that fit inside the model's context. Cut at a fixed length and sooner or later the cut lands in the middle of a line of reasoning: the consideration starts in block 7 and the conclusion sits in block 8. The retrieved block then looks perfect — relevant terms, right tone — but is missing half the meaning.
Worse still: a half block like that is not recognizably broken. The model gets a grammatically correct passage that appears to cover the question, and answers on the basis of half a thought. With rulings, that's the difference between "the court considered that X" and "the court considered that X, but ruled otherwise".
What helps: cut along the structure of the document instead of by character count — headings, numbered considerations, paragraphs. Let blocks overlap slightly so a thought doesn't die exactly on the boundary. And give every block metadata (source, date, position in the document), so a block stays interpretable on its own.
3. How do you stop the model from fabricating a source?
This is the most dangerous failure mode, for one reason: it looks good. A language model that has learned the format of an ECLI number can flawlessly produce one that doesn't exist. Correct format, plausible year, credible court — and entirely made up. A reader who doesn't check the reference sees no difference. A reader who does check it loses trust in the whole system.
So citations alone don't solve this; they only make the fabrications more convincing. What does help is citation verification as a separate step after generation. Our system checks whether a cited ECLI actually exists, and whether the cited passage really does support the claim. If a claim fails that check, it's flagged — not quietly passed along.
Worth being honest about: RAG with verification pushes hallucination back and makes it measurable, but doesn't rule it out entirely. Anyone promising guarantees is selling something other than a language model. The realistic goal is that unproven claims are visible instead of invisible. More on that in why RAG without citations is worthless.
4. How do you know your RAG system is getting better?
Usually: you don't. The typical improvement path is a series of isolated interventions — a different chunk size, a different embedding model, an adjusted prompt — after which someone asks five questions, nods approvingly and moves on. That's not a measurement, that's a sample of one. Meanwhile that same change may have made an entire category of questions worse without anyone noticing.
What you need is an evaluation harness with a golden set: a fixed set of questions with verified answers that reruns on every change. In our case that harness measures both quality and hallucination rate, so regression becomes visible before a change reaches production. The golden set doesn't have to be large to be useful — above all it needs to contain the edge cases the system tripped over before.
Added benefit: a golden set makes discussions concrete. "Feels better" is replaced by a difference you can point at — and sometimes by the finding that the elegant new approach delivers nothing.
5. What changes when you scale up?
Everything that was irrelevant at 500 documents. At that scale the index fits in memory, every query is fast enough, and the costs are negligible. At millions of documents those exact three things become the determining factors: index size, latency and cost per query. Our FTS5 index is roughly 19 GB — a size that forces you to think about where it lives, how it gets rebuilt and what happens during a rebuild.
Latency is the second. Hybrid retrieval plus reranking plus generation plus verification is a chain, and every link adds up. Cost is the third. A prototype where nobody looks at the bill becomes, in production, a line item that has to be visible per query. That's why we log model, tokens and cost per call — without that logging you can't explain after the fact where a bill came from, let alone steer it.
At scale, vendor risk counts too. That's why our answer layer is set up to be model-agnostic: any API can be plugged in, and if one goes down, the system moves on to the next provider. That's not theoretical tidiness — it's the difference between an outage at your supplier and an outage at you.
What separates a demo from a production system?
In short: a demo shows that the pipeline runs. A production system shows how often it gets it right, and what that costs. These four differences do most of the work:
- Hybrid retrieval with reranking instead of vector search alone.
- Structure-aware chunking instead of cutting by character count.
- Citation verification that checks whether the source exists and whether it covers the claim.
- A golden set that reruns on every change.
What that looks like in practice — including the choices around index, retrieval and verification — is worked out in our showcase on the case-law search engine.
Conclusion
The five breaking points are predictable, and that's good news: you can address them before you build instead of repairing them afterwards. Make retrieval hybrid, chunk along the structure, verify citations, build a golden set, and from day one measure per call what an answer costs. None of those five makes hallucination impossible — together they make it small enough to work with, and visible enough to decide on.