When summarization fails: structured facts, hierarchical summaries, and token budgets
The go-to fix for a growing conversation is to summarise it. But a single summary string quietly drops details or bloats over time. Here's the upgrade path when 'just summarise it' stops working.
The standard answer to a conversation that outgrows the context window is a rolling summary: periodically compress older turns into a running summary and keep only that plus the recent window (the memory-and-state post). It works — until it doesn't. When the interviewer asks 'and when does summarisation fail?', here's the honest answer and the upgrade path.
How a single summary fails
- It drops details — a summary is lossy by design, and the specific fact you need later (an account number, a decision's exact wording) is often the first casualty.
- It bloats — to stop dropping things, the summary grows, until it's no longer saving you tokens at all.
- It compounds errors — each summarisation summarises the last summary, so small omissions and distortions accumulate over a long conversation.
The upgrade path
- Structured fact extraction — pull the details that must not be lost (names, ids, decisions) into a key-value store instead of trusting prose to retain them (the persistent-memory post). Facts as fields don't get 'summarised away'.
- Hierarchical / map-reduce summarisation — summarise chunks of the conversation independently, then summarise those summaries. This scales to very long histories and avoids the compounding error of repeatedly re-summarising one growing string.
- RAG over raw history — keep the full transcript, embed it, and retrieve only the passages relevant to the current turn. Now nothing is lost — you just fetch the old detail on demand instead of trying to hold it all.
Cap the memory block's token budget
Whatever combination you use, give memory a fixed share of the context window and enforce it. The whole point of the window is that it's finite and contested (the context-engineering post); if the memory block is allowed to grow unbounded, it crowds out the retrieved documents, the tools, and the user's actual message. Decide 'memory gets at most N tokens', and make the summary and the retrieved facts fit that budget — truncating or re-summarising to stay under it. Memory that eats the whole window has stopped being memory and started being the problem.
'Just summarise it' is the right first answer and the wrong final one. Extract the facts that can't be lost, summarise hierarchically so errors don't compound, retrieve the rest on demand — and cap the whole thing to a token budget.