AnswersHybrid Search & Ranking

Do You Still Need a Reranker?

Where should a cross-encoder reranker sit in a modern retrieval pipeline, and when does better first-stage retrieval remove the need for one?

2 min readJuly 2026

The short answer

Add a reranker when your first-stage retrieval reliably gets the right documents into the top 50–100 but not the top 5, and you can afford a model pass per candidate at query time.

Skip it, or at least measure first, when your first stage already scores at token level (multi-vector / late interaction): a reranker exploits largely the same signal and may add latency for little gain.

What does a reranker actually add?

A cross-encoder reads the query and a candidate document together and judges relevance jointly (Nogueira & Cho, 2019), catching interactions that independently computed embeddings miss. First-stage retrieval optimizes recall: get the relevant documents somewhere into the candidate set. The reranker optimizes precision: get the best ones to the very top. It runs second because a model pass over the whole corpus would be far too slow.

Two-stage retrieval: cheap recall first, expensive precision on a shortlist.

Corpus

millions of documents

First stage

optimizes recall, top-100

Cross-encoder rerank

joint query-doc scoring

Top-5

precision where it counts

What does it cost?

Reranking costs one model forward pass per candidate, at query time, in series with retrieval. Reranking 200 candidates means 200 model calls of latency budget: acceptable for some search products, disqualifying for agent loops that fire dozens of queries in parallel.

When does multi-vector make it redundant?

The ColBERT paper introduced late interaction precisely as an efficient alternative to cross-encoder reranking: MaxSim does token-level matching in the retrieval stage itself, from precomputed embeddings. If your first stage is multi-vector, much of what a reranker would recover is already in the ranking.

The honest caveat: cross-encoders still hold the accuracy ceiling. If top-5 precision is everything and latency is nothing, rerank anyway, but the gap over a multi-vector first stage is far smaller than over a single-vector one, so benchmark before paying for both.

The decision in one line

If you run a single-vector first stage and hear quality complaints, add a reranker. If you run a multi-vector first stage, measure the reranker's lift before you ship it.

TopK's multi-vector retrieval and custom ranking run in a single query, which often removes the separate reranking hop entirely. It is worth testing against your current two-stage stack.

Keep reading

What Is Multi-Vector Retrieval?Multi-vector retrieval stores one embedding per token and scores documents with MaxSim, so precise queries match exact passages instead of a lossy document summary.What Is MaxSim?MaxSim sums each query token's best match against a document's tokens. It is the scoring operator behind multi-vector (late-interaction) retrieval.RRF vs Score Fusion vs True Hybrid: Which Should You Use?RRF merges lists by position, score fusion tunes weighted scores, and true hybrid scores every signal in one query. Start with RRF; move up as you measure.