Bi-encoders vs cross-encoders: the retrieval trade-off

Two kinds of model score relevance in a retrieval system, and knowing the difference is what lets you reason about why retrieval works the way it does. The distinction is simple once you see it: it's about when the query and the document are combined.

Bi-encoder: encode separately, compare vectors

A bi-encoder encodes the query and each document independently into vectors, then measures similarity — usually cosine — between them. The query and document never meet inside the model; they're each turned into a point in space and compared afterwards. The huge advantage: you can embed every document ahead of time and store the vectors, so at query time you only encode the query and do a fast nearest-neighbour search over millions of precomputed vectors. This is what powers the retrieval (first) stage. The cost is precision — because the two texts are encoded in isolation, the model can't reason about how they relate.

Cross-encoder: encode jointly, score the pair

A cross-encoder takes the query and the document together, as a single joint input, and runs them through the model with full cross-attention between every query token and every document token — then outputs one relevance score. Because the two texts interact inside the model, it judges relevance far more accurately. But there's no shortcut: you can't precompute anything, because the score depends on the specific pair, so you must run the model once per (query, document) pair at query time. That's far too expensive to run over a whole corpus.

Retrieve, then rerank

So you use both, in stages. A bi-encoder retrieves a wide shortlist cheaply from the full corpus (fast, approximate). Then a cross-encoder reranks just that shortlist precisely (slow, accurate, but only over a bounded set). You get the bi-encoder's scale and the cross-encoder's precision — which is exactly the two-stage retrieval from the advanced-RAG post, now with the reason each stage uses a different kind of model made explicit.

Which reranker: cross-encoder, hosted, or an LLM?

There are three ways to run the rerank stage, and picking is a real question. A cross-encoder is the default — Cohere Rerank as a managed service, or a self-hosted model like BAAI's bge-reranker-large when data can't leave your VPC or you want to cap cost (the trade there is GPU infrastructure and a network round-trip). The third option is to use an LLM as the reranker, and knowing when it's worth it is the nuance: a cross-encoder wins on latency and cost, and for pure relevance ranking it's usually just as accurate — that's exactly the task it was trained for. An LLM reranker earns its far higher cost and latency only when the ranking criteria are instruction-based or need reasoning ('rank by relevance, but prefer the most recent and down-rank anything speculative') — a nuance a cross-encoder's single relevance score simply can't express.

A bi-encoder asks 'where do these two land in space?' and can precompute the answer for a million documents. A cross-encoder asks 'how do these two actually relate?' and has to look every time. Retrieve with one, rerank with the other.