Is RAG dead? What the measurements say about long context and retrieval

Long context got 73.1% of the questions right. Semantic RAG got 65.4%.

And long context cost 26 times more per query.

That is the short answer to the title's question, and it is neither yes nor no. A boundary exists between accuracy and cost, and what you should do depends on where you sit along it.

The good part is that a measured way out exists. On 63% of queries, both methods give the same answer, which means most of them can go down the cheap path with nothing lost.

The question turned reasonable once the windows passed a million tokens. If everything fits, retrieval becomes a step you might get to skip.

And it gets one thing right. The standard 2023 architecture, meaning slice every document, generate embeddings, store them in a vector database and pull the k most similar, was heavy for a good share of the problems it solved. Plenty of people built that stack for a corpus of two hundred documents that would have fit in a text search.

The question gets one thing wrong: it assumes the result is binary. Researchers measured both methods side by side, and a trade-off is what shows up, not a winner.

In June 2026, a group published a direct comparison between loading the whole collection into context and retrieving only the relevant passages.

The work is "The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications", by Austin Hamilton, Ryan Singh, Michael Wise, Ibrahim Yousif, Arthur Carvalho, Zhe Shan, Mohammad Mayyas, Lora A. Cavuoto and Fadel M. Megahed (arXiv 2606.20898, accessed 28 August 2026). The case study is industrial safety training, with 972 answers evaluated against an expert-validated benchmark, across three machines and two small models.

The authors' framing is worth reusing. They treat the two methods as regimes of epistemic access: broader access raises the chance the model has the right evidence in hand, and it charges a token tax for that.

Seven point seven points of accuracy, at the price of twenty-six times the cost. Whether that pays off depends on nothing but what a wrong answer costs in your case.

The result above earns confidence because it is not new.

In July 2024, Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei and Michael Bendersky published "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach" (arXiv 2407.16833, revised October 2024; accessed 28 August 2026). They compared both methods across several public datasets, with three models.

Their two conclusions, in the authors' own terms: given sufficient resources, long context outperforms RAG on average performance without exception; and RAG's far lower cost remains a distinct advantage.

Two groups, two years apart, different domains and methods, and the same shape of result. That convergence is this post's strongest argument, and it is what lets us treat the boundary as fact rather than as the opinion of whoever measured.

<!-- [UNIQUE INSIGHT] -->

Two measurements that look like they are fighting need reconciling here.

The post on the four pillars of context carried Chroma's work, where 18 models degrade as the input grows, before the window fills. Now a measurement appears where loading everything produces more correct answers.

Both coexist, and the explanation sits in the scale.

Long context's accuracy advantage exists while the collection fits in the window's reliable zone. In that regime, having all the material available beats the risk of retrieval bringing back the wrong passage. As the corpus grows, two effects add up against it: length degradation eats the accuracy gain, and the cost keeps climbing in a straight line.

The practical consequence is that the right answer stays unstable across the project's life. A system starting with two hundred documents and going to twenty thousand crosses the boundary at some point, and nobody sends a notice when that happens.

The way out the 2024 study proposes has a name: Self-Route. The model assesses its own ability to answer with the retrieved passage and, once it concludes it cannot, the query gets resent with the full context.

The numbers explain why that works so well:

In almost two thirds of cases, both methods reach the same answer. Paying 26 times more for them is pure waste, and routing exists to avoid paying.

A simple version of routing fits any project, before you sophisticate it. A heuristic by question type solves a good share: questions about a specific document go to retrieval; questions requiring you to compare or synthesize the whole collection go to full context. Measuring which route wins on your cases is eval work, with a small, stable set.

One concrete case, decided

Abstraction helps little here. An example makes the rule usable.

Picture an internal assistant answering questions about a company's documentation. Four thousand documents, updated every week, with areas not every employee can see.

By the size criterion, four thousand documents fail to fit a window with room to spare, so full context leaves the table on cost. By the freshness criterion, reloading everything on each update every week would be waste. And by the permission criterion, the decision is made before any arithmetic: filtering has to happen before generation.

Retrieval wins all three, and comparing accuracy never even came up.

Now change one detail. The same assistant, but each question's scope is a single thirty-page manual, chosen by the user before asking.

There the answer inverts. Thirty pages fit with room to spare, the cost per query is low, and permission got resolved in the choice of manual. Loading the whole document delivers the 7.7 points of accuracy without paying the tax that only shows up on large collections.

Same product, two different right answers. The change was in the framing of the question, not in the technology available. That is why routing beats a single choice: inside one system, different queries sit at different points on the boundary.

The retriever got simpler, and that is the most underrated change in the story.

The argument is well made by Fabio Akita, in "RAG Está Morto? Contexto Longo, Grep e o Fim do Vector DB Obrigatório", from April 2026: the bottleneck of 2023 was retrieval, and the one of 2026 is reasoning over messy context. With a large, fast, cheap reader, a simple high-coverage retriever makes more sense than a sophisticated semantic search stack.

I agree, and the token tax measurement explains why. If the model handles extra material well, retrieval has no need to be surgical. It needs to avoid losing what matters, which is an easier and cheaper problem to solve.

In practice, text search solves more cases than the standard architecture suggested, above all when you wrote the documents and the query uses the same vocabulary they do.

Naming the thing that died is worth doing, because retrieval is not it. It is the set of decisions that became standard with nobody ever measuring them. Slicing every document into fixed-size pieces, ignoring where ideas begin and end. Generating embeddings for everything, including what nobody queries. Always bringing back the same k results, regardless of whether the question needs one or nine. And treating a vector database as an entry requirement, before enough volume exists to justify it.

None of those four choices was necessary. All of them got copied together, as though they were one thing, and that package is what is coming apart.

The justification for vector search is specific:

• A corpus too large for a text sweep to be viable.
• Divergent vocabulary between question and document, when the user asks one way and the text answers another.
• A real need for semantic similarity, rather than term matching.

Worth noting that retrieving a tool is the same problem under another name. In the post on tool design, selection accuracy collapsed with the number of tools exposed, and the measured solution was retrieving instead of listing. The question "which of my options does this task need to see?" is always the same one.

The criterion is not the window size. It is five characteristics of the problem.

The permission row deserves the spotlight because it is no matter of preference. Once each user can see a different subset of the documents, loading the whole collection into context stops being an expensive option and becomes a wrong one. Filtering has to happen before generation, and that is what retrieval does by construction.

The boundary moves while the project runs, and nobody sends a notice. Three signals show the chosen architecture fell behind.

Cost per query grows without quality following. It is the classic symptom of full context with a growing corpus. The invoice climbs in a straight line with collection size, and accuracy does not. Watching it follows the same reasoning as what each usage pattern costs.

Retrieval starts missing questions it used to get right. It points to a corpus growing faster than the index quality, or user vocabulary drifting away from document vocabulary. That is the moment text search starts asking for semantic search.

Answers turn good on specific questions and bad on broad ones. It is the signal that a class of queries needs the expensive route, and that a single choice is hurting half the product.

None of the three appears without instrumentation. With the trace described in agent observability, all three are simple queries: tokens per query over time, success rate by question type, and the comparison between the two routes on the same cases.

The common error is deciding the architecture once, at the start, when the corpus was small and the question was simple, and never revisiting it.

<!-- [PERSONAL EXPERIENCE] -->

No vector database exists in this operation. Retrieval is text search in the repository and reading excerpts from files.

It is Akita's simple retriever, and it got here through productive laziness: no moment ever came where text search failed enough to justify building something else.

It works for two reasons worth naming, because they are conditions and not virtues. The corpus is small, at a few dozen files. And I wrote the documents, so the query uses the same words the text does, which means the divergent vocabulary problem does not exist here.

Where that choice would fail is predictable: a third-party corpus, user vocabulary different from document vocabulary, or volume beyond what a sweep can handle.

And it already carries an observable cost. Yesterday's post described memory's third failure mode, which is information existing and the agent never consulting it. With text retrieval, that risk is bigger than it would be with semantic search: if I wrote "cover" in the file and the task talks about "featured image", the search finds nothing. It has happened.

The honest conclusion is that the right choice here is the simplest one, and that this is a property of my corpus, not a general law.

If the blog grows to a few hundred posts, or if someone besides me starts writing, the two signals from the previous section appear together and the decision changes. Writing that down now is cheaper than discovering it later.

Is RAG dead?

No. In a June 2026 study with 972 evaluated answers, long context scored higher, at 73.1% against 65.4%, at a cost 26 times greater per query. The standard 2023 architecture went obsolete, with every document sliced and vector search mandatory at any volume. Retrieval keeps winning on cost, on a large corpus, on content that changes and in scenarios with permission per document.

Does long context replace RAG?

It replaces it when the whole collection fits the window with room to spare, the budget per query absorbs the cost and no access control per document exists. Outside that, no. Long context's accuracy advantage shrinks as the corpus grows, because model performance degrades with input length even before the window fills, and the cost keeps climbing.

Do I need a vector database?

Not in every case. Text search solves many cases, above all when the corpus is small and the query vocabulary resembles the documents'. A vector database justifies itself when the corpus is too large for a sweep, when user and document use different words for the same thing, or when you need semantic similarity for real. Start with the simple option and migrate when search fails in a measurable way.

Which is cheaper?

Retrieval, by an order of magnitude. The peer-reviewed measurement points to 26 times fewer tokens per query against long context. Be suspicious of numbers much larger than that: the claim of "1,250 times cheaper" circulates out there, with no primary source holding it up.

Can both be used on the same query?

They can, and it is the hybrid pattern that became common: retrieve to narrow the material and then use the large window to reason over what is left. The difference from routing is that here both techniques act on the same question, instead of one choosing between them. It works well when the collection is large and the answer requires comparing several sources: retrieval guarantees the right pieces came in, and broad context gives the model room to relate them. The cost lands between the two extremes, and the main adjustment is how many passages to bring, since too few lose the necessary piece and too many recreate the token tax.

How do I decide between the two?

Route instead of choosing. On 63% of queries, both methods produce an identical answer, so sending most of them down the cheap path costs no performance. A simple heuristic already delivers a good share of the gain: a question about a specific document goes to retrieval, a question requiring synthesis of the collection goes to full context. Then measure, with a small set of your own cases, which queries need the expensive route for real.

• Long context is more accurate and 26 times more expensive. 73.1% against 65.4%, across 972 evaluated answers.
• Two independent studies, two years apart, reach the same boundary.
• On 63% of queries both give the same answer. Routing cuts 39% to 61% of the tokens.
• Long context's advantage has a ceiling, and the ceiling is the size of the corpus.
• The "R" got simpler. Text search solves more than the 2023 stack suggested.
• Permission per document decides on its own: with it, loading everything leaks, and the price stops mattering.

That closes the week on context. The next one covers how agents talk to each other and to the world: the map of the protocols. The five layers continue in the pillar on harness engineering.