The Transformer: the architecture behind every modern LLM
The 2017 paper "Attention Is All You Need" replaced sequential processing with attention — and made today's LLMs possible. Here's the architecture, in plain terms.
Nearly every large language model in use today — GPT, Claude, Llama, Gemini — is built on one architecture introduced in a 2017 paper with an audacious title: "Attention Is All You Need." Understanding the transformer is understanding what makes modern AI tick.
The problem it solved
Earlier models processed text one token at a time, in sequence, which made them slow to train and forgetful over long distances. The transformer threw out the sequential bottleneck. Its key move: let every token look at every other token directly, through a mechanism called attention — no matter how far apart they are.
The pieces
- Embeddings + positional encoding — turn tokens into vectors, and tag them with their position, since attention itself is order-blind.
- Multi-head self-attention — every token weighs how much to attend to every other, several times in parallel (the "heads").
- Feed-forward network — a small network applied to each position to transform what attention gathered.
- Add & Norm — residual connections and normalisation that keep training stable as blocks stack up.
Why it changed everything
Because attention is parallelisable, transformers train efficiently on enormous datasets and scale gracefully with more compute. That scalability is the whole game — stack more blocks, feed more data, and capability keeps climbing. Most LLMs are just deep stacks of these blocks, decoder-only, trained to predict the next token.
The paper's title wasn't hype. Attention really was all we needed — and the eight years since have been the world catching up to it.