RAG is not dead: what giant context windows really show

In February 2024, Google unveiled Gemini 1.5, able to process a million tokens in a single request, and other vendors followed. A narrative took hold: since you can give the model everything, searching becomes pointless, and RAG is dead. LightOn, a French vendor specialising in document search, looks back on this period when articles announcing the death of RAG multiplied.

The argument had its logic. Yet it rested on a fragile assumption, already challenged by work published in 2023: that a model reads its whole context with the same attention.

A sound idea, poorly implemented

In 2020, a team at Facebook AI Research published a paper with a dry title, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The idea fits in one sentence. Before generating an answer, the model retrieves the relevant passages from an indexed corpus, then writes from what it found.

The authors had already identified the two weaknesses this architecture was meant to fix in language models: the difficulty of showing where an answer comes from, and the difficulty of updating their knowledge. With RAG, knowledge lives in the corpus. You update it by adding a document, and you can show the passage behind each answer.

In 2023, the ChatGPT wave turned the idea into a weekend tutorial. Split PDFs into fixed-size chunks, compute vectors, retrieve the five chunks closest to the question and hand them to the model. Hundreds of “chat with your documents” products appeared. The word RAG ended up meaning any assembly of this recipe, and disappointment followed.

The study Hallucination-Free?, carried out by researchers from Stanford and Yale and published in 2025 in the Journal of Empirical Legal Studies, is one of the few independent, preregistered evaluations on the subject. In 2024, on more than 200 legal questions, it assessed three legal research tools from the market’s two largest publishers, LexisNexis and Thomson Reuters, all built on a RAG architecture. Each hallucinated between 17% and 33% of the time. Lexis+ AI answered 65% of questions correctly, Westlaw AI-Assisted Research 42%.

These tools hallucinated less than GPT-4 used on its own, which confirms the value of the architecture. Their results also show that writing RAG on a sales brochure guarantees very little: quality depends on how the system is built, then evaluated.

What a model does with a long context

Three bodies of work show the limits of the assumption, on the models and tasks they evaluated.

The first signal dates from 2023. The study Lost in the Middle, led by Liu and co-authors, shows that a model makes good use of information placed at the start or end of its context, and far less when it sits in the middle. In other words, what you pour into the context is read unevenly.

In 2025, the NoLiMa benchmark, presented at ICML by researchers from Adobe Research and LMU Munich, measured the scale of the effect. It tests thirteen models that all claim at least 128,000 tokens of context. At 32,000 tokens, eleven of them lose more than half of their short-context performance. GPT-4o, one of the most robust, drops from 99.3% to 69.7%.

The same year, Chroma’s Context Rot report extended the observation to eighteen recent models across several experiments: performance degrades as context grows, including on simple tasks.

This last source calls for a caveat. Chroma sells a vector database, and analyst Cobus Greyling noted that the finding serves its ecosystem. That is why it comes last, after two academic studies pointing the same way.

Search before calling the model

NoLiMa adds a valuable nuance. Models collapse mainly when the question and the target passage share few words. Literal overlap helps them a great deal; inference across a long context, much less. This is precisely the role of document search: retrieve the right passage before calling the model. How well it works is measured use case by use case.

Business arguments complete the picture. An organisation’s corpus runs to thousands or millions of documents and far exceeds any window. Every token sent to the model is paid for on every request, so filling the window is expensive. The corpus changes daily and must stay current. Finally, an answer drawn from a massive context is hard to tie to a precise passage, whereas traceability is exactly what we are after.

The intuition of 2020 still holds. Everything that has happened since comes down to how it was implemented.

So RAG is back, in a more demanding form. Give the model less, but give it exactly what it needs.