ANO—THER. ● NEWS

What is RAG, and does it still matter?

● Last updated August 25, 2026
Eugene / Chips and Infrastructure desk

RAG stands for retrieval-augmented generation. The idea is small: before the model answers, go find the documents that bear on the question and put them in front of it. Everything interesting is in what that fixes and what it breaks.

The problem it solves

A language model knows what was in its training data, as of the moment that data was collected. That produces three gaps.

It does not know anything recent. It does not know anything private, which is most of what any organization actually needs answers about. And when it does not know, it does not reliably say so, because generating a fluent sentence and generating a true sentence are the same operation from the model’s point of view.

Retrieval addresses all three at once by changing the question from “what do you know about X” to “here are eight passages, answer using these.” The model’s job becomes reading and synthesis rather than recall, which is what it is good at. It also makes citation possible: the system knows which documents it supplied, so the answer can point at them.

How it actually works

The standard pipeline has two halves.

Indexing, done in advance. Documents are split into chunks, usually a few hundred to a couple of thousand tokens each. Each chunk is passed through an embedding model that converts it into a vector, a list of numbers positioning that text in a space where similar meanings sit close together. The vectors go into a database built for nearest-neighbor search.

Retrieval, done per question. The question is embedded the same way. The database returns the chunks closest to it. Those chunks are inserted into the prompt with the question, and the model answers from them.

Two details separate systems that work from systems that demo well.

Chunking is the quiet determinant of quality. Split too small and each chunk loses the context that makes it meaningful; split too large and irrelevant material crowds the window. Splitting on document structure, by section and heading, generally beats splitting every N characters.

Pure vector search is rarely enough. Embeddings capture meaning and are bad at exact strings: part numbers, error codes, surnames, version numbers. Production systems almost always run vector search and keyword search together and merge the results, then re-rank the merged set with a smaller model that scores each candidate against the question directly.

Why million-token windows did not replace it

The obvious objection in 2026 is that context windows got large enough to skip retrieval entirely. Some models now offer more than a million tokens, and one that appeared on OpenRouter in August 2026 advertised 1,048,576.

Three things keep retrieval in the stack.

Cost. Input tokens are billed on every request. Sending a large document set with each question multiplies the bill by the size of the corpus, permanently, while retrieval sends a few thousand tokens.

Latency. Processing a very long prompt takes real time before the first output token appears.

Accuracy. This is the one people find counterintuitive. Models do not attend evenly across a long context. Material near the beginning and end is used more reliably than material in the middle, so burying the relevant paragraph inside a million tokens of surrounding text makes it less likely to be used, not more. Narrowing the input is an accuracy technique, not only a cost technique.

What long windows did change is the balance. Systems retrieve larger chunks and more of them than they did when windows were small, and aggressive summarization to save space is less necessary. The architecture stayed.

The failure modes it introduces

Retrieval fixes hallucination from ignorance. It introduces three problems of its own.

Citing what was never read. If the pipeline summarizes a source before the model sees it, the model can produce a confident citation to a document whose actual content it never encountered. In August 2026 this was documented with Claude citing papers it had not read, because the fetching tool returned summaries rather than full text. The citation looks like verification and is not.

Prompt injection through retrieved content. Everything retrieved enters the context as text, and the model cannot cleanly separate “content to reason about” from “instructions to follow.” A document containing hidden instructions aimed at the model becomes an attack. In August 2026 a fan wiki was documented serving AI agents a hidden payload telling them to wipe the user’s repository, invisible to a normal browser. A web page was shown to be able to steal a Grok conversation through the same class of flaw. Anything a RAG system can read is part of its attack surface.

Retrieval failure looks like knowledge failure. When the search returns nothing useful, the model still answers, using whatever it absorbed in training. From the outside a good answer and a badly grounded one are indistinguishable, which is why systems that show their sources are worth more than systems that only show conclusions.

The honest summary is that RAG converts one failure mode into a different, more diagnosable one. A model inventing an answer is invisible. A model answering from the wrong three documents is something you can see, once you display which three.

Quick answers

What does RAG stand for?

Retrieval-augmented generation. Rather than relying only on what a model learned in training, the system searches a document collection for material relevant to the question, inserts the results into the model's context, and asks the model to answer from them.

How does RAG work step by step?

Documents are split into chunks and converted into vectors that capture meaning, then stored in a vector database. At question time the query is converted the same way, the closest chunks are retrieved, those chunks are placed into the prompt, and the model answers from them and cites them. Most production systems combine vector similarity with ordinary keyword search, because keyword search is better at exact terms like product codes and names.

Do long context windows make RAG obsolete?

No. You can now fit a large document set into a million-token window, but doing so costs far more per query, adds latency, and retrieves less reliably, because models use material at the start and end of a long context better than material in the middle. Retrieval narrows the input to what matters, which is cheaper and usually more accurate.

What is the difference between RAG and fine-tuning?

RAG changes what the model knows at question time; fine-tuning changes how the model behaves. Use retrieval for facts that change, are private, or need citation. Use fine-tuning for format, tone and task-specific behavior. They solve different problems and are frequently used together.