Self-attention and RLHF: how models focus, and how they're aligned
Two ideas do a lot of the heavy lifting in modern LLMs: self-attention, which lets a model decide what to focus on, and RLHF, which aligns it to what humans actually want.
If the transformer is the engine of modern AI, two ideas explain how it runs well: self-attention, the mechanism that lets a model decide what matters in its input, and RLHF, the process that turns a raw model into a helpful assistant. They operate at opposite ends — one is architecture, the other is training — and both are worth understanding.
Self-attention: deciding what to focus on
For each token, self-attention creates three vectors: a query (what am I looking for?), a key (what do I offer?), and a value (what do I carry?). A token's query is compared against every other token's key to produce attention weights — how much to focus on each — and the output is a weighted blend of the values. Do this several times in parallel (multi-head attention) and the model can track many kinds of relationship at once.
Self-attention is the model asking, for every word, "which other words should I be paying attention to right now?" — and answering it billions of times.
RLHF: aligning to what humans want
A pretrained model predicts plausible text, but plausible isn't the same as helpful, honest, or safe. Reinforcement Learning from Human Feedback closes that gap. It's how a raw next-token predictor becomes a model you'd actually put in front of a user.
- Supervised fine-tuning — teach the model to respond helpfully from human-written examples.
- Reward model — have humans rank model outputs, and train a model to predict those preferences.
- Reinforcement learning — optimise the model to score well against the reward model (often with PPO), nudging it toward responses people prefer.
The result is the difference between a model that can produce good answers and one that reliably chooses to. Self-attention gives a model its capability; RLHF points that capability at being useful.
Attention decides what the model sees. RLHF decides what it does with it. You need both to get something worth deploying.