Memory and state in multi-turn agents: context without blowing the window

A model is stateless — it remembers nothing between calls. 'Memory' is just whatever you put back into the context window each turn (see the context-window and anthropomorphize posts). Do that naively and a long conversation overflows the window and inflates the bill; do it well and the agent stays coherent for cheap.

The problem

Every turn you resend the conversation so the model has context. That history grows without bound until it blows the window and the cost with it. Naively truncating to the last few messages loses important earlier context — the user's name, the decision three turns ago.

Three tiers of memory

• Sliding window — keep the last N turns verbatim. Cheap, and covers immediate context.
• Rolling summary — periodically compress older turns into a running summary, so distant context survives at a fraction of the tokens.
• Long-term memory — store durable facts (preferences, entities, decisions) in a store and retrieve only the relevant ones per turn, using embeddings and vector search (see the RAG and embeddings posts).

Assemble the context deliberately

Each turn, compose the prompt on purpose: the rolling summary, plus the recent window, plus the handful of retrieved long-term facts, plus the new message. That's curating the window as a scarce resource rather than dumping everything into it — which is the whole discipline of working with context.

Agents don't have memory; they have whatever you choose to put back in the window. Memory design is deciding what's worth the tokens.