Where RAG came from: REALM, RAG, RETRO, and FLARE

Retrieval-augmented generation is treated as a recent, almost off-the-shelf pattern: chunk your docs, embed them, retrieve top-k, stuff the prompt. But the core idea — pair a parametric model (knowledge baked into weights) with a non-parametric memory it can look things up in — has a clear research lineage running back to 2020. Four papers mark the path, and knowing them makes today's design decisions feel obvious rather than arbitrary. This is the backstory to the rag-fundamentals post.

REALM (2020): retrieval you can train

REALM, out of Google, made retrieval part of language-model pre-training rather than a bolt-on at inference. It learned a retriever over a Wikipedia index and backpropagated through the retrieval step — so the model didn't just consult a memory, it learned which documents were worth consulting. That's the foundational move: knowledge doesn't all have to live in the weights, and the act of looking things up can itself be learned.

RAG (2020): the pattern, and the name

A few months later, a Facebook AI Research paper — 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' — coined the term and defined the shape every RAG app still uses: a dense retriever (DPR) fetches passages from a Wikipedia index, and a pretrained seq2seq generator (BART) conditions its output on them. The paper offered two flavours: RAG-Sequence uses one set of retrieved documents for the whole answer, while RAG-Token can attend to different documents for different tokens. Strip away the training details and you're left with retrieve-then-generate — the ancestor of essentially every production RAG system, including the retrieval stack in the advanced-rag-retrieval post.

RETRO (2021): retrieval at trillion-token scale

DeepMind's RETRO ('Improving language models by retrieving from trillions of tokens') pushed on scale and architecture. Instead of retrieving into the prompt, it wired retrieval into the model itself via chunked cross-attention over nearest-neighbour passages drawn from a database of up to two trillion tokens. The striking result: a comparatively small model with retrieval could match one many times larger without it. The lesson that still holds — giving a model a big external memory can substitute for cramming everything into parameters.

FLARE (2023): retrieve when you're unsure

FLARE ('Active Retrieval Augmented Generation') changed when retrieval fires. Classic RAG retrieves once, up front, from the initial question. FLARE retrieves during generation: the model drafts the next sentence, and if that draft contains low-probability (low-confidence) tokens, it uses the draft as a fresh query to retrieve, then regenerates. Retrieval becomes iterative and driven by the model's own uncertainty — the seed of today's agentic and adaptive retrieval, and a close cousin of the confidence signals in the llm-confidence-scoring post.

What the lineage tells you

Read together, these four map two design axes you're implicitly choosing on every RAG project. Where does retrieval live — in the prompt (RAG) or in the architecture (REALM, RETRO)? And when does it fire — once up front (RAG) or iteratively, on demand (FLARE)? Almost all production RAG sits in one corner: retrieve-once, in the prompt. It's model-agnostic, works with any API model, and is simple to operate — which is exactly why the Lewis-2020 pattern won in practice even though the others are more powerful in principle.

And the frontier is quietly rediscovering the rest. Corrective and adaptive RAG (the rag-variants post) decide whether and how to retrieve based on the query — REALM's 'learn what to consult'. Agentic retrieval, where a model retrieves in a loop until it has enough (the rag-as-an-agent post), is FLARE's 'retrieve when unsure' wearing a tool-calling costume. None of it is truly new; it's the 2020–2023 research finally becoming practical engineering.

Today's RAG is the simplest corner of a design space mapped out years ago. The 'advanced' techniques aren't new inventions — they're the rest of that space, finally cheap enough to ship.