How is MaxSim computed?
For a query with token embeddings q₁…qₙ and a document with token embeddings d₁…dₘ:
MaxSim(Q, D) = Σᵢ maxⱼ sim(qᵢ, dⱼ)
Read it inside-out: for each query token qᵢ, compute its similarity (usually cosine or dot product) against all document tokens and keep only the maximum, which is that token's best match in the document. Then sum those maxima across every query token. Each query term contributes the evidence of its single strongest match, so a document is rewarded when it contains a good match for each part of the query.
Query token
one of n
Similarity vs every document token
Keep only the max
Sum over all query tokens
Document score
Khattab & Zaharia introduced MaxSim in ColBERT (2020) as the core of "late interaction."
Why "late" interaction?
The name marks the timing: query and document interact at scoring time, not encoding time. A standard single-vector model interacts early. It collapses each side into one vector before comparison, so the two are only ever compared as whole summaries.
MaxSim defers the comparison until both are represented as sets of token embeddings, keeping fine-grained detail available. That is what recovers precise, long-tail matches a summary would average away.
Why does the "max" matter?
The maximum keeps each query token's score independent and noise-resistant. A query token only needs one strong match somewhere in the document, so a relevant term buried in a long document still contributes fully; it isn't diluted by all the unrelated tokens around it. Summing across query tokens then rewards documents that satisfy the whole query, one term at a time.
What does MaxSim cost?
MaxSim costs a similarity matrix per candidate document. Instead of one dot product, it computes an n × m grid of query-token × document-token similarities, takes a row-wise max, and sums. That is on the order of thousands of times more scoring work than a single-vector comparison, plus the storage for one embedding per token. Those costs are why late interaction ran second, as a reranker, for years. If you have wondered why your engine only offers late interaction as a reranking step, this is the reason.
How is MaxSim made affordable at scale?
Engines approximate MaxSim in a cheap first stage and score only the survivors exactly. Techniques like PLAID and MUVERA, and TopK's SMVE, which turns multi-vector representations into sparse vectors whose dot product approximates MaxSim, let the first stage run at sparse-retrieval speed while final ordering comes from exact MaxSim on a small candidate set.
TopK measured this design at roughly 5–8× lower end-to-end latency than PLAID and MUVERA at competitive recall (March 2026, ColBERTv2 on BEIR). That is what lets multi-vector work as a first-stage retrieval primitive rather than just a reranker. The full decision framework is in when to use multi-vector embeddings.