Post-retrieval processing: contextual compression, noise, and lost-in-the-middle
Retrieval and reranking get you the right chunks — but they arrive full of boilerplate, redundancy, and irrelevant surrounding sentences. What you do between reranking and the model is a stage of its own.
Most RAG discussions stop at reranking, as if the best chunks are ready to hand to the model. They're not. Between reranking and generation sits a stage tutorials skip: cleaning up the retrieved context. Even the right chunks carry noise, and both the model's attention and your token budget pay for it.
Contextual compression: after reranking, before the model
Contextual compression sits exactly where its name suggests in the pipeline — after reranking (so you compress the best handful, not the whole corpus) and before generation. Its job is to strip the parts of each retrieved chunk that aren't relevant to the query, so the model sees dense signal instead of padded passages. Less noise to distract it, fewer tokens to pay for.
What kind of noise?
- Structural boilerplate — headers, footers, disclaimers, navigation, and leftover markup that rode along with the real content.
- Out-of-context sentences — the one relevant sentence sits inside a chunk whose neighbours aren't relevant, because the chunk is bigger than the answer.
- Cross-chunk redundancy — several retrieved chunks repeat the same fact, spending context budget on saying it three times.
Compression techniques
- Perplexity-based token pruning (LLMLingua / LongLLMLingua) — a small language model scores how much information each token carries and prunes the low-information ones, shrinking the prompt while preserving meaning.
- LLM extractive summarization — an LLM pulls only the query-relevant sentences out of each chunk, dropping the rest.
- Sentence-level cosine filtering — embed each sentence in a chunk and drop the ones whose similarity to the query falls below a threshold — cheap, and no extra model call.
Lost in the middle: order matters
There's one more move after compression: reorder. Models attend most to the start and the end of a long context and can genuinely miss what's buried in the middle — the 'lost in the middle' effect. So place the highest-relevance chunks at the head and the tail of the context window, not the center. It's a nearly-free reordering that recovers accuracy a naive concatenation would have thrown away — and it's why reranking, which decides what's most relevant, and this reordering, which decides where it goes, work together.
Retrieval finds the right chunks; post-retrieval makes them usable — compressed so the signal is dense, and reordered so the model looks where the answer actually is.