Five requirements for a serious RAG system

Giant context windows were supposed to make search pointless, but a model reads a long context unevenly: RAG is back, in a more demanding form. What “more demanding” means remains to be said. Since the ChatGPT wave of 2023, the word RAG has meant any assembly of the same recipe: split PDFs into fixed-size chunks, compute vectors, retrieve the five chunks closest to the question and hand them to the model.

The word guarantees nothing

The study Hallucination-Free?, carried out by researchers from Stanford and Yale and published in 2025 in the Journal of Empirical Legal Studies, assessed in 2024, on more than 200 legal questions, three legal research tools from the market’s two largest publishers, LexisNexis and Thomson Reuters, all built on a RAG architecture: each hallucinated between 17% and 33% of the time. These tools hallucinated less than GPT-4 used on its own, which confirms the value of the architecture. But writing RAG on a sales brochure guarantees very little: quality depends on how the system is built, then evaluated.

Intelligence professionals describe work on information as a cycle. You direct the search, you collect, you analyse what you found, you share the result, and that result redirects the next search. In this cycle, RAG covers the collection and analysis stages. Its quality is built in five places.

The five places where quality is built

1. Reproducible ingestion. The same document produces the same index every time, and results become reproducible. Structure matters as much as text: headings, tables, page numbers, appendices, images. A contract clause taken out of its section loses half its meaning.
2. Enriched passages. Every passage carries its metadata: date, author, document type, validity period, version. This is what lets you answer “according to the contract in force” rather than “according to a contract”.
3. Hybrid search. It combines semantic proximity with exact term matching, because an article number, a proper noun or a standard reference is best found word for word. A second pass then re-ranks results by actual relevance. Meaning finds ideas; the letter finds references.
4. Passage-level anchoring. Every claim in the answer points to the exact passage behind it, opened in one click. This is the precondition for any verification.
5. Structured data as sources. Part of an organisation’s knowledge lives in tables and databases. You query them, and their results carry the same traceability as text passages. A sourced figure is worth a sourced sentence.

Set against the 2023 recipe, these requirements show what it lacked. A fixed-size chunk ignores the section it came from; the first two requirements give the passage back its structure and context, its date and its version. Vectors alone find what is close in meaning; the third adds the letter, which finds references. The last two keep RAG’s original promise, to show the passage behind each answer, and extend it to tables and databases.

A more compact model, well served

These five requirements have a strategic consequence, which concerns the size of models.

When quality is built into search, a more compact, well-served model can be enough for many uses.

It is our conviction, and it is tested use case by use case. Its reach, however, goes beyond performance alone. When knowledge lives in the sources, the model becomes a replaceable component. An organisation that owns its corpus, its index and its evaluation sets can switch models in a few days, test the new one on its own questions, and compare the results before switching. Working with more compact models, well served by search, reduces dependence on a handful of frontier models and their publishers.

Everything starts with the source, and that is also what keeps the organisation free.