Long context didn't kill RAG — it changed what RAG is for

Every time a model ships a bigger context window, the same post goes around: RAG is dead, just put everything in the prompt. The argument has been made at 128K, at 200K, at a million tokens, and it will be made again at ten million.

It’s a reasonable-sounding argument, and I’ve watched several teams act on it. What happens next is consistent enough to be a pattern: the prototype is beautiful, the first invoice is not, and by month two they’ve quietly rebuilt a retrieval layer with worse ergonomics than the one they skipped.

Long context is genuinely useful. It just doesn’t do the job people expect it to.

Your corpus doesn’t fit, and that’s not close

Start with the boring constraint. A million tokens sounds enormous until you measure something real against it.

The demo corpus behind /demo on this site is about forty times a million-token window, and it’s four thousand PDFs — a rounding error next to what a 200-person company has accumulated in a decade of file shares.

So “just put everything in the prompt” is already off the table for most businesses at the first step. The interesting question isn’t whether you select a subset; it’s how. And selecting a relevant subset of a corpus is the definition of retrieval. You do not get to skip it. You only get to choose whether you do it well or by accident.

The cost per question is the part that ends the argument

Suppose your corpus does fit. Say 500K tokens, and you stuff all of it into every query.

At mid-tier frontier pricing — call it $3 per million input tokens, and check current rates because they move — that’s $1.50 per question. A retrieval pipeline that sends eight good chunks sends maybe 8K tokens: about 2.4 cents.

That’s a sixty-fold difference, and it lands on the exact same model.

The usual rebuttal is prompt caching, and it’s a fair one — caching a static prefix cuts the repeat cost substantially. But it only helps when the corpus is stable, the cache is warm, and every user is asking against the same documents. Change one file and you re-pay. Have per-user document permissions and you can’t share a cache at all. Go quiet for an hour and the TTL takes it. Caching turns a brutal number into a merely bad one under favorable conditions.

Accuracy degrades before the window does

The “needle in a haystack” tests that ship with new models are real, and models pass them convincingly. They are also the easiest possible version of the task: one verbatim fact, planted in filler text, retrieved by an exact match.

Real questions look nothing like that. They need four facts from three documents, one of which contradicts another because it’s from 2023, and the right answer depends on noticing which is current. Performance on that kind of multi-fact reasoning starts falling off long before you hit the context limit — and it falls off in the middle of the window more than at the edges.

Here’s the part that gets underweighted: irrelevant context is not free. Every document in the prompt that plausibly relates to the question is a distractor the model has to rule out. Filling the window with everything you own doesn’t give the model more to work with; it gives it more chances to answer from the wrong page. I’ve seen retrieval-fed answers beat full-corpus-fed answers on the same model, on the same questions, because the retrieved version simply contained fewer wrong options.

Latency you can feel

Time-to-first-token scales with input. A 500K-token prompt takes seconds to prefill before a single word comes back. An 8K-token prompt is effectively instant.

For an internal tool that people use forty times a day, that difference decides whether the tool becomes a habit or a thing they open when they have to. Nobody files a ticket saying “the assistant is six seconds slow.” They just stop using it.

The one that actually matters: citations

Ask a long-context model a question about a 400-page document dump and you get a paragraph. Ask a retrieval system and you get a paragraph plus these four passages, from these documents, at these pages.

If you work in a regulated field — legal, medical, tax, finance — that difference is the entire product. An answer nobody can verify is not usable output; it’s a suggestion. And retrieval gives you the surrounding infrastructure almost for free: per-user document permissions enforced at the retrieval step, a new file searchable the moment it’s indexed instead of on the next prompt rebuild, and a log of exactly which sources produced which answer.

You can ask a long-context model to cite its sources. It will. Some of those citations will be to page numbers that don’t say what it claims they say, and you won’t know which ones without checking all of them.

What I actually build now

Not a choice between the two. Long context changed the shape of retrieval, and the change is real:

Retrieve wider, chunk bigger. When 8K tokens of context was the budget, you passed three small chunks and prayed. With a large window you can pass 8–15 generously sized chunks with their surrounding sections intact. Chunk boundaries stop being the fragile part of the system. This is a genuine quality win, and it comes directly from the bigger window.

Still rerank, still cut. Wider recall, then a cross-encoder to sort it, then send the top handful. The window lets you be generous; it doesn’t mean you should be indiscriminate.

Skip retrieval when the unit of work is one document. “Summarize this contract,” “compare these two filings,” “review this PR” — the whole document fits, it’s obviously relevant, and chunking it would only lose structure. Retrieval here is pure overhead. This is what long context is genuinely, unambiguously good at.

The rule I’d give a team in one line: if the question is about one known document, use the window; if the question is about your corpus, use retrieval. Most business questions are the second kind, which is why the retrieval layer keeps not dying.

The short version

Long context is a better pipe, not a replacement for knowing what to put in it. It made retrieval systems better — bigger chunks, more of them, less brittle boundaries — and it made single-document work dramatically easier. What it did not do is make the question “which of my 40 million tokens matter for this query?” go away.

That question is retrieval. It was always retrieval. The window just got wide enough that answering it well is now cheap.

If you want to see the difference rather than read about it, the demo on this site runs hybrid retrieval over 2,885 IRS forms and answers only from what it retrieved — with the passages it used attached to every claim.