AnswersMulti-Vector Retrieval

Is Multi-Vector Retrieval Too Expensive?

Can late-interaction retrieval run as a first-stage retriever at production scale, or do storage and MaxSim compute costs still rule it out?

2 min readJuly 2026

The short answer

It was. For years the cost was disqualifying, which is why late interaction stayed a reranking trick. One embedding per token means roughly 10–100× more storage than one embedding per document, and MaxSim scoring does orders of magnitude more work than a single dot product.

That has changed: modern engines approximate MaxSim cheaply to select candidates, then score only the shortlist exactly. If you ruled multi-vector out on cost a year ago, the math has moved.

Where does the cost come from?

Storage. A 500-token document stores 500 embeddings instead of one. Quantization and token pooling shrink this substantially, but token-level indexes remain larger than single-vector ones.

Compute. MaxSim evaluates a query-token × document-token similarity matrix per candidate. Run naively across a large corpus, that is thousands of times the scoring work of single-vector search.

How do engines make it affordable?

The answer is two stages: a cheap approximation narrows millions of documents to a small candidate set, then exact MaxSim ranks only the survivors. PLAID prunes candidates using quantized centroids. MUVERA builds fixed-dimensional encodings whose inner product approximates MaxSim, and it comes with theoretical guarantees, but the descriptors must be very large and dense to earn them.

TopK's SMVE (Sparse Multi-Vector Encoding) starts from the observation that expressive descriptors may need to be large, but they don't need to be dense. It projects each token embedding onto a large set of random spherical anchors, keeps only the top-k strongest projections per token, and pools the result into one sparse vector whose dot product approximates MaxSim. Cost therefore scales with the non-zero count rather than the dimensionality, and candidate selection runs on ordinary sparse-retrieval infrastructure. Exact MaxSim then reranks only the shortlist.

Two-stage multi-vector retrieval: pay the approximation everywhere, pay exact MaxSim almost nowhere.

Query

token embeddings

SMVE sparse retrieval

cheap, whole corpus

Candidate shortlist

small overfetch

Exact MaxSim

expensive, shortlist only

Top-k results

The measured result (March 2026, ColBERTv2 on BEIR): roughly 5–8× lower end-to-end latency than PLAID and MUVERA at competitive recall. On MS MARCO's 8.8M documents, SMVE averages 39.9ms for k=100, versus 221–318ms for PLAID and 310–444ms for MUVERA.

What does this change in practice?

Multi-vector stops being a reranker bolted onto some other retriever and becomes the first stage itself: token-level precision over the whole corpus at a cost curve close to conventional search. Affordability is settled. The remaining question is whether your queries need it.

TopK runs this stack in production as semantic_index, one schema annotation that embeds, indexes, and serves late interaction end to end.

Its measured system numbers (June 2026): 52.88% nDCG@10 across all 15 BEIR datasets (within about 1% of exact exhaustive MaxSim) at 295 QPS with ~75ms p99 latency, while ingesting over 1.5B tokens per hour with sub-second index lag.

Keep reading

What Is Multi-Vector Retrieval?Multi-vector retrieval stores one embedding per token and scores documents with MaxSim, so precise queries match exact passages instead of a lossy document summary.What Is MaxSim?MaxSim sums each query token's best match against a document's tokens. It is the scoring operator behind multi-vector (late-interaction) retrieval.When to Use Multi-Vector Embedding ModelsUse multi-vector models when queries target specific details in long or complex documents; use single-vector when queries are broad or cost dominates.