# TopK > The retrieval engine beyond vector search, keyword search, and rank fusion. Hybrid search, multi-vector retrieval, custom ranking, and managed inference in one API. Built on object storage for 10x lower cost and massive scale. TopK is a managed retrieval engine for AI applications. It lets you build search, RAG, and agent context systems using semantic search, BM25, sparse vectors, multi-vector retrieval, metadata filters, and custom hybrid scoring in one query. Instead of stitching together a vector database, keyword index, embedding pipeline, reranker, and application-side ranking logic, TopK gives you one engine for high-quality retrieval at production speed. TopK integrates into existing stacks through Python, TypeScript, and Rust SDKs, a CLI, and an MCP server. Private deployment is available in your VPC with full data residency, auditability, and access control guarantees. ## Overview ### Home URL: https://www.topk.io TopK is a managed search engine for AI applications. It lets you build search, RAG, and agent context systems using semantic search, BM25, sparse vectors, multi-vector retrieval, metadata filters, and custom hybrid scoring in one query. Instead of stitching together a vector database, keyword index, embedding pipeline, reranker, and application-side ranking logic, TopK gives you one engine for high-quality retrieval at production speed. Built on object storage for 10x lower cost and massive scale. ### Documentation URL: https://docs.topk.io The documentation is the primary product reference for implementation details. It covers Python and JavaScript/TypeScript SDK references, CLI usage, MCP server setup, concepts, and operational guidance for integrating TopK into agent and retrieval workflows. - [Quickstart](https://docs.topk.io/quickstart): Get started with TopK in minutes and run your first hybrid search. - [Docs llms.txt](https://docs.topk.io/llms.txt): Machine-readable index of TopK documentation pages. ### Console URL: https://console.topk.io The TopK console is where users manage their account and billing, create and manage API keys, and monitor usage. ### Pricing URL: https://www.topk.io/pricing The pricing page describes TopK's usage-based pricing model. It includes a pricing calculator and pricing table for estimating costs based on ingestion and query workload. ### Benchmarks URL: https://www.topk.io/benchmarks The benchmarks page publishes TopK's measured performance: dense and sparse vector search latency percentiles across collection sizes (1M to 1B vectors) and filter selectivities, throughput under concurrent clients, and ingest and indexing times. Results are reproducible via the open-source topk-io/bench harness on GitHub. ### FAQ URL: https://www.topk.io/faq The FAQ page answers common questions about TopK: what it is, how it differs from vector databases and keyword search engines, when to use TopK vs alternatives, pricing, data privacy, and how to get started. ### Blog URL: https://www.topk.io/blog The blog is the primary long-form content area on the site. It includes technical writing about search infrastructure, hybrid retrieval, multi-vector retrieval, benchmark results, fundraising news, and company updates. ### Answers URL: https://www.topk.io/learn TopK Answers is a portal of focused questions about advanced retrieval with a definite, extractable answer up front: when to use multi-vector embeddings, what MaxSim is, whether multi-vector retrieval is too expensive, RRF vs score fusion vs true hybrid, when hybrid search is needed, whether a reranker is still necessary, and why RAG fails for agents. Each page states tradeoffs honestly and links primary sources. ### Team URL: https://www.topk.io/team The team page introduces the TopK team and links to careers. It highlights the people behind the company and positions TopK as a team building infrastructure for the AI era. ### Contact URL: https://www.topk.io/contact The contact page provides a direct contact form and support email address for reaching the TopK team. ## Blog Authors ### Jergus Lejko URL: https://www.topk.io/blog/author/jergus-lejko Blog posts by Jergus Lejko. - [TopK SQL: A Search Query Language](https://www.topk.io/blog/20260614-topk-sql): TopK now implements the Postgres wire protocol, so any Postgres client can run semantic search, hybrid search, and filtered retrieval as ordinary SQL. - [TopK Bench: Benchmarking Real-World Vector Search](https://www.topk.io/blog/20251201-topk-bench): This post evaluates how managed vector databases handle large-scale, production-like workloads—including millions of vectors, concurrent querying, real-world filtering, and continuous ingestion—using reproducible benchmarks across several leading providers. - [TopK at Data Council AI Launchpad 2025](https://www.topk.io/blog/topk-at-data-council-2025): At Data Council 2025, we introduced TopK, a unified cloud-native query engine that combines vector search, keyword matching, and flexible scoring in a single system. We demonstrated how TopK overcomes the limitations of traditional vector databases by supporting hybrid queries with custom ranking logic while maintaining high performance. ### Marek Galovic URL: https://www.topk.io/blog/author/marek-galovic Blog posts by Marek Galovic. - [The Fastest Regex Is the One You Don’t Run](https://www.topk.io/blog/20260814-fast-regex-search-sparse-grams): Why AI agents reach for regex search, and how TopK uses sparse n-grams to make those queries fast. - [RAG Is Broken for Agents. Here's How We Fixed It.](https://www.topk.io/blog/20260628-rag-is-broken-for-agents): Context is a search problem. Without the right context, even the best models fail. This post describes why dense embedding based RAG is broken for agents and how multi-vector (late interaction) retrieval fixes it. - [SMVE: Multi-Vector Retrieval That Just Works](https://www.topk.io/blog/20260311-smve-multi-vector-retrieval): TopK introduces SMVE, a fast and scalable approach to late-interaction retrieval based on sparse random projections. - [[Podcast] Building a Search Database from First Principles](https://www.topk.io/blog/20250907-podcast-building-search-database-first-principles): Listen to our founder Marek Galovic discuss the challenges and solutions in building a search database from the ground up, covering everything from vector search to hybrid retrieval and the architectural decisions that shaped TopK. - [Beyond RRF: How TopK Improves Hybrid Search by up to 7.8%](https://www.topk.io/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality): A case study on how TopK hybrid retrieval outperforms traditional RRF-based methods, improving nDCG@10 by an average of 4.5% (and up to 7.8% on some datasets) by leveraging score-aware ranking, dynamic weighting, and efficient result merging. - [Binary Vector Search at 350GB/s using ARM NEON](https://www.topk.io/blog/binary-vector-search-arm-neon): Optimizing binary vector search using ARM NEON instructions to achieve 350GB/s throughput. - [TopK at Data Council AI Launchpad 2025](https://www.topk.io/blog/topk-at-data-council-2025): At Data Council 2025, we introduced TopK, a unified cloud-native query engine that combines vector search, keyword matching, and flexible scoring in a single system. We demonstrated how TopK overcomes the limitations of traditional vector databases by supporting hybrid queries with custom ranking logic while maintaining high performance. - [Why Vector DBs Are the Wrong Abstraction – And What We Built Instead](https://www.topk.io/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch): We’ve spent the last three years building the most popular vector database on the market. In that time we realized that a database built around vectors as a primary key is simply the wrong abstraction, creating an unnecessary obstacle for users in production. ### Martin Spisak URL: https://www.topk.io/blog/author/martin-spisak Blog posts by Martin Spisak. - [SMVE: Multi-Vector Retrieval That Just Works](https://www.topk.io/blog/20260311-smve-multi-vector-retrieval): TopK introduces SMVE, a fast and scalable approach to late-interaction retrieval based on sparse random projections. ### TopK Team URL: https://www.topk.io/blog/author/topk-team Blog posts by TopK Team. - [High-Quality Search, Out of the Box](https://www.topk.io/blog/20260611-semantic-index-multi-vector-retrieval): TopK's semantic_index annotation brings state-of-the-art multi-vector retrieval to production — no embedding pipeline, no separate vector store, no reranking service. - [Scaling Without Complexity: Billion-Scale Hybrid Search with TopK](https://www.topk.io/blog/20250722-billion-scale-hybrid-search): TopK enables billion-scale hybrid search with <100ms latency, fast indexing, and high-quality results. - [We Raised $5.5 Million to build an AI-Native Search Engine for Enterprises](https://www.topk.io/blog/seed-round): TopK has raised $5.5 million in seed funding to accelerate the development of its unified, AI-native search platform, enabling organizations to seamlessly combine vector, keyword, and custom ranking in a single system. The round was led by top-tier investors, fueling our mission to redefine search for the AI era. ## Blog Posts ### The Fastest Regex Is the One You Don’t Run URL: https://www.topk.io/blog/20260814-fast-regex-search-sparse-grams Published: 2026-08-14 Summary: Why AI agents reach for regex search, and how TopK uses sparse n-grams to make those queries fast. When we give an AI agent a regular expression filter, it tends to use it for search. Not only for patterns that need regex, but also for ordinary keyword and phrase queries. That choice makes sense from the agent's point of view. A literal phrase is already a valid regex. Case insensitivity, word boundaries, optional spelling, and alternatives are small changes to the same expression. The agent can use one tool for all of these requests: ```text invoice (?i)nobel prize colou?r error (reading|writing) file ``` This is a good interface for an agent. However, it can be a very expensive operator inside a query execution plan even when using an automaton-based regex engine without exponential backtracking (see [ReDoS](https://en.wikipedia.org/wiki/ReDoS)). Regular expressions are fast when they run against one string. They are much slower when they run against every string in a large collection. An occasional scan may be acceptable for a person typing a query but not for agents that issue many concurrent queries while exploring, verifying, and refining an answer. That was the problem we had in TopK. Our regex engine was already efficient, but every regex filter still had to read the target field and evaluate the pattern for every document. A better regex kernel would improve it by a constant factor, but it couldn't address the fundamental problem. The useful observation was that agent-generated regexes often contain many literals. Even a pattern with optional sections or alternatives usually says that some characters must occur which can be exploited to speed up the queries using an index. So we changed the question. Instead of asking how to evaluate a regex faster, we asked how many documents needed the full regex at all. The result is a two-stage approach. A sparse n-gram index finds a set of candidate documents, then the full regex checks only the candidates instead of the full corpus. The index is allowed to return false positives, but it must never lose a real match. ```text regex → boolean expression over n-grams → candidate documents → exact regex ``` ## Just index trigrams? The usual way to accelerate substring search is a trigram index. Split each document into every overlapping sequence of three characters and store a posting list for each trigram. For example, `alan turing` produces: ```text ["ala", "lan", "an ", "n t", " tu", "tur", "uri", "rin", "ing"] ``` A document that contains `alan turing` must contain all nine trigrams. Intersecting their posting lists gives a candidate set that can be checked with the full regex. This approach works, and it has an important property: it cannot create false negatives (i.e. documents that don't match the index but match the full regex). However, trigrams are a compromise. Short grams are common. A trigram such as `the`, `ing`, or `tur` may occur in a large fraction of a corpus, so its posting list is expensive to read and does little filtering. Trigrams also forget adjacency beyond three characters. A document may contain every trigram from a query in unrelated places and still become a candidate. Using longer fixed n-grams improves selectivity, but creates another problem. An index over one width cannot help with required fragments shorter than that width which might force us to a full scan (remember, no false negatives). Indexing every width repeats much of the same text and makes the number of tokens per document grow with the maximum width which explodes the index size. Sparse-grams offer a solution. Guarantee that all minimum length n-grams are present, and probabilistically create longer n-grams to improve index selectivity. ## Content-defined n-grams Sparse-gram tokenization starts by assigning a deterministic weight to every bigram in the input string. We currently use CRC32 to compute bigram weights, though any deterministic function with a sufficiently even output can play the same role. Consider an n-gram spanning `n - 1` bigrams. We select it when its two boundary bigrams have greater weights than every bigram in its interior. Put another way, the two largest weights in the span must sit at its edges. The rule depends only on the contents of the span, not on its position in the surrounding document which makes tokenization context-free. That locality gives sparse grams their most important property. If a string `q` occurs inside a document `d`, every gram selected from `q` is also selected from the same interval in `d`. Adding text before or after the match cannot change a decision made entirely inside it. TopK uses two views of the tokenizer: - At index time, `all(d)` stores every eligible sparse gram selected from document `d`. - At query time, `covering(q)` keeps a smaller set of eligible grams used to constrain the query. The property we need is: ```text covering(q) ⊆ all(q) ⊆ all(d), when q is a substring of d ``` This is the correctness contract behind the index. A matching document contains every required query gram. Hash collisions or an imprecise regex plan can add candidates, but the exact regex removes them later. Try the parser below. `build_all` shows the grams stored at index time. `build_covering` shows the smaller set needed for a query. Open the boundary weights to see why each interval was selected. ## Why the index stays sparse Assume the bigram weights are independent and have no ties. An n-character span contains `n - 1` bigrams. Every ordering of their weights is equally likely. The span is selected only when the two largest weights land on its boundaries, in either order. Its selection probability is: ```text P(select an n-gram) = 2 / ((n - 1)(n - 2)) ``` Long intervals become progressively rarer. Their selection probability falls in proportion to `1 / n²`. If the minimum length is three, every trigram is selected, since `2 / ((3 - 1)(3 - 2)) = 1`. Longer grams are then added less frequently. For a long string of `L` characters, the expected number of selected grams with lengths from `a` through `N` is approximately: ```text E[grams] ≈ 2L × (1 / (a - 2) - 1 / (N - 1)) ``` The sum telescopes. With a minimum of three, the expectation approaches `2L` even as the maximum grows. A dense index over every width grows on the order of `LN`. This is the useful asymmetry: sparse grams can admit longer terms without indexing every longer interval. ## N-Gram information and selectivity Length is a useful proxy for selectivity, but it's not the whole picture. The quantity that matters for a particular gram `g` is its information, where `P(g)` is the probability that the gram begins at a given position: ```text I(g) = -log₂ P(g) ``` A rare gram carries more information than a common one. Two grams of the same length can therefore have very different posting lists. A familiar phrase may remain common even when it is long, while a name or unusual spelling can become selective after only a few characters. If a gram has information `I(g)` and a document offers roughly `L` positions where it could begin, a simple occurrence model gives: ```text P(document contains g) = 1 - (1 - 2^(-I(g)))^L ``` When the gram is rare, this is approximately `L × 2^(-I(g))`. The posting list starts to leave the saturated region when `I(g)` reaches about `log₂(L)` bits. In a corpus of `M` similar documents, its expected document frequency reaches one near: ```text I(g) ≈ log₂(M L) ``` Natural language does not spread probability evenly across all strings. It produces a small head of common grams and a long tail of names, numbers, phrases, and accidental combinations. Longer grams still help because they can accumulate more information. Extending a gram `x` by a character `c` adds the conditional information of that continuation: ```text I(xc) = I(x) - log₂ P(c | x) ``` A predictable continuation adds little. A surprising one adds a lot. Maximum n-gram length therefore controls how much evidence one index term can carry, without assuming that every extra character is equally valuable. The chart below lets you vary a gram's information directly to see how it affects its selectivity. The orange curve estimates its posting-list fraction. The teal line marks one document in the selected corpus. The two vertical guides show where posting lists begin to narrow and where the expected document frequency reaches one. Raising the maximum n-gram length from `N` to `N + 1` adds only about `2L / (N(N - 1))` index terms under the sparse selection model. It also admits grams that may carry more information so the query planner can then prefer the terms with the lowest measured document frequency, regardless of their length. ## Turning a regex into a boolean expression Literal strings are the easy case. Real regexes contain optional sections, alternations, character classes, repetitions, and wildcards. We parse the regex into the same high-level representation used by the exact matcher. For each node, the planner tracks what it can prove about every possible match: - exact strings, when the set is small enough; - required prefixes and suffixes; - whether the expression can match an empty string; - a boolean expression over grams that every match must satisfy. The boolean operators follow the regex structure. Concatenation usually combines constraints with `AND`. Alternation combines them with `OR`. A wildcard contributes no constraint, but known text on either side can still be useful. A subexpression that can match the empty string cannot be required, so its constraint is weakened or dropped. The planner feeds required literal runs through `covering()` and uses the resulting grams as leaves in the boolean expression. Those terms are guaranteed to exist in the index for every matching document. Consider: ```regex colou?r of the (sky|sea) ``` A conservative index query could look like this: ```text " the" AND "r of t" AND ("e se" OR "e sk") ``` The plan does not encode the full regex. It does not need to. It states only facts that must be true for a match. A document that fails this expression cannot match the regex and can be safely skipped. A document that passes is only a candidate for full regex verification. This one-way guarantee lets the planner control its own cost. It can drop a weak gram, cap a large alternation, or give up on an unhelpful subexpression. Each choice admits more candidates, but none can remove a true match. Patterns such as `.*`, or patterns made mostly from broad character classes, safely fall back to a scan. At execution time, TopK resolves the gram posting lists, evaluates the `AND` and `OR` tree, and gets sorted list of candidate document IDs. It then uses the candidate IDs to prune blocks that contain no candidates (saves I/O) and evaluates the full regex only on matching rows inside the remaining blocks. The regex engine is still the source of truth, it just nees to do far less work. ## What's the performance? Literal-heavy patterns benefit most because they yield long, selective grams. Optional text and small alternations can still produce useful constraints. Patterns dominated by wildcards, broad classes, or short fragments produce weak plans and may fall back to scanning. We measured the query path end-to-end on one of our clusters. The index path uses sparse grams to generate candidates, then verifies them with the exact regex. The scan path evaluates the regex through an unindexed full scan. For this workload at concurrency sixteen, the sparse index provides 55.3 times the throughput, 61 times lower average latency, and 51.2 times lower p99 latency. From concurrency one to sixteen, indexed throughput grows by almost thirteen times. Full scans saturate much earlier because concurrent queries compete to read and test the same body of text. This is especially relevant for agentic workloads. One agent may issue several parallel searches while refining an answer, and multiple agents may search at once. Candidate pruning significantly improves query efficiency to handle concurrent queries at scale. ## Conclusion The key idea was not a new regex engine. It was splitting the problem into two parts: fast candidate generation and exact verification. Similar two-stage approaches are already used in our query engine for dense and [multi-vector](/blog/20260311-smve-multi-vector-retrieval) search and probabilistic block filtering with bloom filters. The agent still gets the search language it naturally prefers. TopK turns whatever literal information the pattern contains into a selective plan, then runs the regex only where certainty is needed. ## References - [Regular Expression Matching with a Trigram Index, or, How Google Code Search Worked](https://swtch.com/~rsc/regexp/regexp4.html), Russ Cox - [The technology behind GitHub’s new code search](https://github.blog/engineering/the-technology-behind-githubs-new-code-search/), GitHub Engineering - [Fast regex search: indexing text for agent tools](https://cursor.com/blog/fast-regex-search), Cursor ### RAG Is Broken for Agents. Here's How We Fixed It. URL: https://www.topk.io/blog/20260628-rag-is-broken-for-agents Published: 2026-06-28 Summary: Context is a search problem. Without the right context, even the best models fail. This post describes why dense embedding based RAG is broken for agents and how multi-vector (late interaction) retrieval fixes it. ## Context is a search problem Even frontier models fail without the right context. As models have gotten better at reasoning, the constraint on agent quality has shifted. It's no longer _"can the model reason about this?"_ but _"can it find the evidence it needs to reason at all?"_. Getting noisy results can derail the whole trajectory and produce wrong answers with high confidence, making search a critical component of agentic systems. ## How agents retrieve information In practice, there are two main approaches: - **Naive RAG.** Embed the query, run a vector or hybrid search, maybe rerank, and put the results into context. This is fast, cheap, and scales well through various vector databases. **The problem is that it misses important details.** A single embedding is a summary, and summaries lose exactly the low-level details that agent queries rely on. - **Agentic / CLI harness.** Give the agent filesystem access, `ls`, `glob`, `grep` tools, and the ability to read documents directly. This is what a lot of agent harnesses do today, as it produces higher-quality answers than naive RAG. However, the quality comes at a **huge latency and cost premium**. The agent spends a lot of tokens on serial tool calls and reading potentially unhelpful documents to find the information it needs. The cost also scales linearly with the number of documents, making this approach impractical for large datasets. ## Why vector search fails for agents Current vector search approaches are predominantly based on dense embeddings. The problem is that these embeddings are **lossy summaries** of the input document compressed into O(1k)-dimensional vector space. They capture broad, semantic information about the document, but lose the low-level details required to answer precise, long-tail queries. Capturing high-level semantic information is perfectly fine for humans as they tend to issue broad, topical queries. Agents, on the other hand, tend to issue many precise queries in parallel, often trying different angles on the same topic. This produces a lot of overlapping results, leading to context bloat, without properly reflecting the agent's intent for each query. Unfortunately, the lack of representational capacity is not an engineering challenge, but a fundamental limitation of dense vector embeddings as proved in [this paper](https://arxiv.org/abs/2508.21038). ## How multi-vector fixes it Multi-vector search, also known as late interaction, changes the representation from one vector per document to one vector per token in the document. This allows it to keep the low-level details uncompressed, while also capturing the high-level semantic information of the document. Document-query scores are computed at token granularity using the **MaxSim** (Chamfer similarity) operator - for each query token, it finds the most similar document token, and then sums the maximum similarity scores for all query tokens to produce a scalar score for the entire document. ### Free lunch? The representational capacity of multi-vector embeddings and scoring expressiveness of MaxSim comes at a cost: - Token-level embeddings require **10-100x more storage** per document compared to dense embeddings. - MaxSim scoring is computationally expensive, requiring **~2000x more compute** than a single dot product. The storage and compute overhead makes exhaustive multi-vector search impractical at even relatively small scale. Similar to single-vector search, approximate methods to scale multi-vector retrieval exist, for example [PLAID](https://arxiv.org/abs/2205.09707) or [WARP](https://arxiv.org/abs/2501.17788). Generally, they rely on pre-computed indexes and heavy compression making online updates, deletes, filtering, and other production features difficult or impossible without costly index rebuilds. This is largely the reason why multi-vector search is not widely adopted outside the IR research community. ## Multi-vector search in production Okay, so now that we know why multi-vector search is the way to go, how do we actually run it in production? ### SMVE: The algorithm Our answer is Sparse Multi-Vector Encoding (SMVE). The core idea is to convert multi-vector document/query embeddings into sparse vectors so that their sparse dot product approximates MaxSim. Because storage and compute scale with the number of non-zero elements rather than the ambient dimensionality, SMVE produces expressive representations while remaining storage efficient and fast to query. If you want to learn more, you can read our [blog](https://www.topk.io/blog/20260311-smve-multi-vector-retrieval) where we describe the algorithm in more detail. ### TopK: Scalable retrieval engine SMVE is a fast first stage retriever. It allows us to prune billions of documents down to a handful of promising candidates with sub-100ms p99 latency. This ensures recall, but we still need to refine the candidates with full MaxSim to produce the final results with correct ranking. Combined, you get the recall and ranking of exhaustive MaxSim at orders of magnitude lower cost. As we've seen with prior approaches, having fast scoring is not enough for production use cases. Efficient embedding inference, document quantization, filtering, online updates and deletes, and multi-tenancy are all required for real deployments. TopK handles all of these out of the box and abstracts the underlying complexity through [multi_vector_index](https://docs.topk.io/guides/multi-vector-search) and [semantic_index](https://docs.topk.io/guides/semantic-search). ## Results ### BrowseComp-Plus (deep research) BrowseComp-Plus evaluates deep research agents against a fixed, human-verified corpus, which lets you isolate the retriever's contribution instead of hiding it behind a black-box web search. Swapping the retriever from BM25 to a dense model to late interaction lifts answer accuracy at every model size. Multi-vector search also enables smaller models, for example gpt-oss-120b, to achieve similar accuracy as much larger, proprietary, models with a weaker BM25 or single-vector dense retriever. ### OfficeQA Pro (enterprise research) OfficeQA Pro measures grounded reasoning over a messy, real-world enterprise corpus of complex documents with nested tables and long context dependencies. The baseline agent with CLI harness achieves only ~18% accuracy while consuming roughly $6.21 worth of tokens per query. Using the same model with late interaction retriever improves its accuracy about 2.3x to ~42% while cutting query costs by more than 10x to $0.47 per query. ## Takeaways Context is a search problem, and agent accuracy is bounded by retrieval quality. Single-vector embeddings have a theoretical ceiling that fine-tuning cannot lift, and the filesystem + tools fallback that teams often reach for is slow, expensive, and brittle at scale. Multi-vector (late interaction) is the retrieval primitive built to handle the long-tail, precise queries agents issue. SMVE makes it scale to billions of documents efficiently, and TopK makes it production-ready with online updates, filtering, multi-tenancy, and inference supported out of the box. ### TopK SQL: A Search Query Language URL: https://www.topk.io/blog/20260614-topk-sql Published: 2026-06-14 Summary: TopK now implements the Postgres wire protocol, so any Postgres client can run semantic search, hybrid search, and filtered retrieval as ordinary SQL. Postgres is the lingua franca of databases. Decades of tools, drivers, and workflows are built around its wire protocol: psql, psycopg2, node-postgres, tokio-postgres, ORMs, dashboards, and more. They all expect a Postgres-shaped server on the other end. Until now, TopK was available only through our Python, JavaScript, and Rust SDKs. **Now it speaks Postgres too.** Any client that talks to Postgres can connect to TopK and get state-of-the-art search quality as ordinary SQL: ```sql SELECT title, semantic_similarity(bio, 'an epic fantasy quest') AS score FROM books WHERE match_any(bio, 'dragon wizard') ORDER BY boost(score, published_year > 2010, 1.5) DESC LIMIT 10 ``` Semantic search, keyword filtering, and metadata-aware ranking in one query, over a standard Postgres connection. ## TopK SQL Specification TopK SQL is a search-oriented dialect: Postgres-shaped where that helps, extended where search needs more. It supports schemaless tables with vector, sparse, and multi-vector types; `SELECT` with semantic, keyword, vector, and hybrid scoring; standard `WHERE` predicates plus search filters; `INSERT` / `UPDATE` / `DELETE`; and a Postgres-compatible wire protocol. For the full language reference, see the [TopK SQL overview](https://docs.topk.io/sdk/topk-sql/overview). ### 1.1 Schema(less) **Schemaless by default.** A table has no fixed schema: rows can contain undeclared fields, a column can hold values of different types, and undeclared fields remain queryable and filterable. Declare a field when you want to index it or constrain its type. `CREATE TABLE` defines the table and declared columns — indexes are declared inline on each column. `DROP TABLE` removes the table. ```sql CREATE TABLE books ( title TEXT, published_year INTEGER, bio TEXT INDEX semantic_index(), embedding f32_vector(768) INDEX vector_index(metric = 'cosine') ); ``` Declared columns are not the full document. Rows can include fields that never appeared in `CREATE TABLE`: ```sql INSERT INTO books (_id, title, rating, tags) VALUES ('earthsea', 'A Wizard of Earthsea', 4.8, ARRAY['fantasy', 'magic']); SELECT title, rating FROM books WHERE contains(tags, 'magic'); ``` `rating` and `tags` were never declared; they are still stored, returned, and filterable. ### 1.2 Types Standard Postgres scalar types (`BOOLEAN`, `INT`, `FLOAT`, `TEXT`, `BYTEA`, and `JSONB`) are supported, along with typed arrays such as `BOOLEAN[]`, `INT[]`, `FLOAT[]`, and `TEXT[]`. We extend the type system with native support for the following vector and matrix shapes: | Shape | Type | Precisions | |---|---|---| | Dense | `*_vector(dim)` | `f32` `f16` `f8` `u8` `i8` | | Sparse | `*_sparse_vector` | `f32` `f16` `f8` `u8` `i8` | | Multi-vector | `*_matrix(dim)` | `f32` `f16` `f8` `u8` `i8` | | Binary | `binary_vector(dim)` | 1-bit | Vector values can be constructed with a JSON-string cast (`'[...]'::f32_vector`) or a constructor (`f32_vector(ARRAY[...])`). ### 1.3 `SELECT` queries Search queries in TopK SQL are `SELECT` statements built around scores. Semantic search, vector search, BM25, and multi-vector retrieval each produce scores that can be selected, aliased, combined, boosted, and sorted. ```sql SELECT _id, title, semantic_similarity(bio, 'tales of magic and adventure') AS score FROM books ORDER BY score DESC LIMIT 10; ``` The basic shape is: compute a relevance score, sort by it, and return the top results. From there, search and ranking can be tuned through composition: filter in `WHERE`, expose multiple scores in `SELECT`, and combine them in `ORDER BY`: ```sql SELECT _id, title, bm25_score() AS keyword_score, semantic_similarity(bio, 'an epic fantasy quest') AS semantic_score, vector_distance(embedding, '[...]'::f32_vector) AS vector_score FROM books WHERE match_any(bio, 'dragon wizard') AND published_year > 1950 ORDER BY 0.2 * keyword_score + 0.5 * semantic_score + 0.3 * vector_score DESC LIMIT 10; ``` This is hybrid search without multiple queries, client-side fusion, or reciprocal-rank fusion. Keyword matching contributes lexical relevance, semantic similarity contributes meaning, the vector score contributes similarity against your own embeddings, and the final ranking is a single SQL expression. #### 1.3.1 Search functions TopK SQL exposes retrieval modes as scoring functions. Each function targets an index and returns a score. See the [TopK SQL overview](https://docs.topk.io/sdk/topk-sql/overview#search-functions) for the complete list of search functions and index types. | Function | Index | Use it for | |---|---|---| | `semantic_similarity(field, query)` | `semantic_index` | query embedding, candidate generation, and reranking with [Iso-ModernColBERT](https://huggingface.co/topk-io/Iso-ModernColBERT) | | `vector_distance(field, vector)` | `vector_index` | dense or sparse ANN against client-supplied vectors | | `multi_vector_distance(field, matrix)` | `multi_vector_index` | late-interaction MaxSim retrieval | | `bm25_score()` | `keyword_index` | keyword relevance from `match_any(...)` / `match_all(...)` predicates | #### 1.3.2 Filtering Filters narrow the candidate set before ranking. TopK SQL supports standard predicates — comparisons, membership, text predicates, and regex — plus search-specific predicates. ```sql WHERE published_year > 2000 AND in_print = true AND genre IN ('fantasy', 'fiction') AND match_any(bio, 'dragon wizard') ``` Those predicates can be ordinary metadata filters, text search predicates, regexes, list checks, or keyword predicates (such as `match_any()`). #### 1.3.3 Scoring Scores are ordinary values. Alias them in `SELECT`, then combine them with arithmetic or ranking functions in `ORDER BY`. The composition example above adds keyword, semantic, and vector scores directly. You can also fold metadata into the ranking expression: ```sql ORDER BY boost(semantic_score, published_year > 2010, 1.5) DESC ``` Ranking stays inside the query: retrieval scores and metadata signals combine in one ordered expression instead of being merged in application code. ### 1.4 `INSERT` / `UPDATE` / `DELETE` TopK SQL supports the same write operations you expect from Postgres. `INSERT` writes a full document, including undeclared fields, and has upsert semantics: inserting an existing `_id` replaces the document. ```sql INSERT INTO books (_id, title, published_year, tags) VALUES ('hobbit', 'The Hobbit', 1937, ARRAY['fantasy', 'adventure']); ``` `UPDATE` changes fields on documents identified by `_id` — either `_id = '...'` or `_id IN (...)`: ```sql UPDATE books SET in_print = true WHERE _id = 'hobbit'; ``` `DELETE` removes the documents matched by a filter: ```sql DELETE FROM books WHERE published_year < 1900; ``` Unlike `UPDATE`, `DELETE` accepts the same filter expressions as `SELECT`, so you can delete by ID or by any predicate. ### 1.5 Protocol The SQL layer speaks the Postgres wire protocol, so standard clients can connect without custom adapters. `psql`, application drivers, prepared statements, and dashboard tools can all use the same endpoint. TopK implements both simple and extended query modes. Connecting to TopK SQL requires only an API key in the connection password field — see the [TopK SQL overview](https://docs.topk.io/sdk/topk-sql/overview#setup) for connection setup. ```bash psql "host=elastica.sql.topk.io password=" ``` The `host` must be set to your desired region, in the format `.sql.topk.io`. See the full list of supported regions at [docs.topk.io/regions](https://docs.topk.io/regions). #### 1.5.1 Type resolution One part of the Postgres protocol does matter for a schemaless database: column types are sent before rows. In Postgres this is natural because every selected column has a known type. In TopK, undeclared fields may not. `topk-sql` tries to infer the type of each `SELECT` column in order: ```text SELECT column ├─ explicit cast? ──► Postgres OID (::int4, ::float8, ::text) ├─ declared column? ──► type inferred from schema └─ unknown/mixed type? ──► JSON ``` Standard Postgres drivers deserialize JSON values into native maps and lists. Use an explicit `::` cast when you want a concrete wire type. #### 1.5.2 Table catalog Existing tables and their schemas can be inspected through `information_schema.tables` and `information_schema.columns` virtual tables. `EXPLAIN` returns the TopK query produced by the SQL parser, so you can see what will run before executing it. ### 1.6 Implementation The parser, `topk-sql`, is open source in [github.com/topk-io/topk](https://github.com/topk-io/topk), alongside `topk-py`, `topk-js`, and `topk-rs`. Like the SDKs, it is a thin mapping over the engine: it parses Postgres-flavored SQL into a TopK query rather than implementing a separate query planner, so it provides the same semantics and benefits from all optimizations we make to our planning and execution pipeline. ## Pick your interface SQL is a thin wrapper around TopK, not a second implementation of it. It's one more way in, next to the SDKs, mapping onto the same engine. The query you'd write in Python and the same query in SQL resolve to the same plan. Reach for whichever fits where you're working: a notebook, a service, a dashboard, a psql prompt, or any JDBC-compatible tool. That means TopK can plug into the existing SQL ecosystem — from BI dashboards to federated query engines and warehouses. Start today at [console.topk.io](https://console.topk.io), read the [TopK SQL overview](https://docs.topk.io/sdk/topk-sql/overview), or browse the parser source at [github.com/topk-io/topk](https://github.com/topk-io/topk). ### High-Quality Search, Out of the Box URL: https://www.topk.io/blog/20260611-semantic-index-multi-vector-retrieval Published: 2026-06-11 Summary: TopK's semantic_index annotation brings state-of-the-art multi-vector retrieval to production — no embedding pipeline, no separate vector store, no reranking service. State-of-the-art retrieval quality usually comes with a catch. You start with an embedding pipeline, a vector database, add hybrid search with RRF, and maybe a reranker. But production search systems don't live isolated inside notebooks. They are a distributed system with reliability and latency budgets, constant writes, strict freshness requirements, high-QPS reads, filters, permissions, and much more. That's why we built `semantic_index` — a single schema annotation that abstracts this complexity and enables state-of-the-art retrieval ready for production. Batteries included. ```python from topk_sdk.schema import text, semantic_index client.collections().create( "docs", schema={ "text": text().index(semantic_index()) } ) ``` ```typescript await client.collections().create("docs", { text: text().index(semanticIndex()), }); ``` That's the entire setup. No embedding pipeline, no separate vector store, no reranking service. Under the hood, `semantic_index` is powered by **[Iso-ModernColBERT](https://huggingface.co/topk-io/Iso-ModernColBERT)** (our multi-vector embedding model) and **[sparse multi-vector encoding (SMVE)](/blog/20260311-smve-multi-vector-retrieval)**, which makes late interaction scale to billions of documents with filtering and online index updates. ## Why multi-vector? Single-vector (dense) embeddings compress an entire document into one point in high-dimensional space. That works until your queries get specific — for example, a clause in a contract, a row in a financial table, or a step in a procedure. Late interaction models keep one embedding per token and match at token granularity, which is why they consistently outperform dense models on out-of-domain and long context retrieval. The problem was never quality; it was cost and operational complexity. Multi-vector indexes are an order of magnitude larger, and exact MaxSim scoring is computationally expensive. TopK solves this by identifying a small set of candidates using fast sparse approximations and then refining them to the final **top-k** results using quantized MaxSim reranking. ## Performance - **Ingest 1.5B+ tokens/hour** — embedded and indexed, with sub-second index lag. Your documents are searchable as you write them. - **295 QPS across BEIR with ~75ms p99 latency** — high-quality search without performance tradeoffs. - **52.88% nDCG@10 on BEIR** — end-to-end, on the live system, not an offline eval of the model alone. - **~30% higher recall and nDCG@10 on ViDoRe v3** — state-of-the-art performance beating 80x bigger Qwen3-VL-Embedding-8B model. - **80.48% accuracy on BrowseComp-Plus** — top-5 performing research agent without a complicated search pipeline. ## BEIR: Baseline Most retrieval benchmarks measure the model in isolation. We measured the whole system end-to-end — including document ingestion, embedding inference, indexing, and concurrent queries on our production clusters. Across all 15 BEIR datasets, `semantic_index` averages **52.88% nDCG@10**, only ~1% lower than the base model with exact MaxSim. On the performance side, every dataset cleared 175 QPS, smaller corpora pushed past 390 QPS, and p99 latency stayed between 50ms and 125ms with sub-second index lag throughout. Your documents are searchable as you write them, which is becoming increasingly more important for agentic use cases. ## ViDoRe V3: Enterprise Retrieval BEIR is text-only. Enterprise retrieval is complex PDFs, tables, slides, and multilingual documents, which is exactly where token-level matching pulls away from dense embeddings. We compared `semantic_index` against Qwen3-VL-Embedding-8B, a state-of-the-art dense embedding model **80x larger** than ours. **+34% recall, +30% nDCG@10 improvement on average.** On industrial documentation, recall jumps from 42.05% to 75.97% (+81%). Finance (EN) goes from 58.90% to 80.91%. Pharma from 60.16% to 83.52%. There isn't a single domain where the 8B dense model wins. ## BrowseComp-Plus: Agentic Search Retrieval is increasingly consumed by agents, not humans. We plugged `semantic_index` into an agentic research loop as the retriever for BrowseComp-Plus. A gpt-5 agent with the default harness achieved **80.48% accuracy** and **77.82% recall**, placing it top-5 in the overall leaderboard (as of June 11, 2026). The agent averaged ~14 search calls per task with 88.54% citation precision, which indicates that the retriever is surfacing the right documents early enough for the agent to ground its answers without wasting tokens. ## Try it today Everything above — multi-vector embedding inference, retrieval with quantized MaxSim reranking, online index updates, and filtering — is available today behind simple, high-level abstractions. ```python from topk_sdk.schema import text, semantic_index from topk_sdk.query import select, field, fn # Create collection with semantic_index client.collections().create( "docs", schema={ "text": text().index(semantic_index()), }, ) # Insert documents client.collection("docs").upsert([ {"_id": "doc-1", "text": "..."}, {"_id": "doc-2", "text": "..."}, ]) # Query using maxsim scoring with semantic_similarity docs = client.collection("docs").query( select( "text", score=fn.semantic_similarity( "text", "query string" ), ) .top_k(field("score"), 10) ) ``` ```typescript // Create collection with semanticIndex await client.collections().create("docs", { text: text().index(semanticIndex()), }); // Insert documents await client.collection("docs").upsert([ { _id: "doc-1", text: "..." }, { _id: "doc-2", text: "..." }, ]); // Query using maxsim scoring with semanticSimilarity const docs = await client.collection("docs").query( select({ text: field("text"), score: fn.semanticSimilarity("text", "query string"), }) .topk(field("score"), 10) ); ``` - **Get started:** [console.topk.io](https://console.topk.io) - **Docs:** [docs.topk.io/guides/semantic-search](https://docs.topk.io/guides/semantic-search) ### SMVE: Multi-Vector Retrieval That Just Works URL: https://www.topk.io/blog/20260311-smve-multi-vector-retrieval Published: 2026-03-11 Summary: TopK introduces SMVE, a fast and scalable approach to late-interaction retrieval based on sparse random projections. Late-interaction models are having [a bit of a moment among the search community](https://x.com/lateinteraction/status/2027214531429118128) — and for good reason. They have been shown to be more expressive than single-vector retrieval across many benchmarks and modalities, but until recently they have been too expensive and cumbersome to use as a true first-stage retrieval primitive. At TopK, we believe late-interaction retrieval deserves the same place in the search stack as established semantic retrieval primitives like single-vector embeddings and reranker models. Over the past few months, we have worked to make them a first-class feature of our database. Multi-vector retrieval is now natively supported in TopK ([docs](https://docs.topk.io/concepts/multi-vector-search)), and in this post, we dive into how we made it scale to large datasets while preserving the CRUD properties of our database offering. ## No free lunch with multi-vector retrieval Multi-vector retrieval gets its quality from a richer scoring function: instead of compressing a query and document into one vector each and computing their dot product, it compares sets of token embeddings using the [MaxSim](https://docs.topk.io/concepts/multi-vector-search#maxsim-scoring) operator, matching each query token against its most similar document token. That extra expressiveness is powerful, but not free. Compared to single-vector retrieval, you typically need to store far more data per document and do far more work at scoring time, which makes exhaustive multi-vector retrieval impractical at scale. Prior work tackles this tradeoff in a few different ways: compress the token embeddings, [keep fewer of them](https://www.answer.ai/posts/colbert-pooling.html), build multi-stage retrieval pipelines such as [PLAID](https://arxiv.org/abs/2205.09707), or collapse the multi-vector representation into one large descriptor as in [MUVERA](https://arxiv.org/abs/2405.19504). MUVERA is especially interesting because it comes with theoretical guarantees: with sufficiently large descriptors, its scores approximate MaxSim closely enough to recover candidates similar to those from exhaustive late-interaction retrieval. The catch is that these descriptors need to be very large, which makes them expensive to store and compute with. **Our idea is simple: expressive descriptors may need to be large, but they [do not](https://arxiv.org/abs/2503.01776) [need to be dense](https://arxiv.org/abs/2505.11388).** ## SMVE: Sparse Multi-Vector Encoding SMVE is based on exactly that idea: instead of approximating multi-vector representations with dense descriptors, we convert them into **sparse vectors** whose dot product approximates MaxSim similarity. Sparse vectors have a useful property: storage and computation depend only on the non-zero elements and effectively ignore everything else. This means that storage and computational complexity scale with the number of **non-zero elements**, not with the ambient dimensionality of the embedding space. That lets us build very high-dimensional representations that preserve much of the original expressiveness while remaining compact to store and fast to query. The key question, then, is how to find a transformation that produces a sparse vector while still approximating MaxSim well. ### How SMVE works The answer is surprisingly simple - SMVE consists of just three steps: 1. **Random Projection onto Spherical Anchors**: We sample a large set of random unit vectors that act as anchor directions in the embedding space. Each token embedding is then projected onto these anchors, producing a higher-dimensional vector of cosine similarities. Intuitively, this gives us a sketch of where the token sits relative to many reference directions. 2. **Sparsification**: For each projected token vector, we keep only the Top-K largest values and set the rest to zero. This works well because in high-dimensional spaces, a vector tends to have very small inner products with most random directions, so most projection values stay close to zero while only a small number capture strong alignment with the anchor directions. Keeping only those strongest signals gives us a sparse representation of each token embedding. 3. **Pooling**: Finally, we aggregate the token-level sparse representations into a single sparse vector. In particular, for queries we sum the token vectors, while for documents we average the non-zero contributions in each dimension. Lastly, we can repeat the steps multiple times with different random matrices and concatenate the results to reduce variance and improve retrieval quality. The core SMVE transformation is simple enough to express in just a few lines of code. ```python from torch import randn, Tensor, topk, zeros_like embedding_dim = 128 # dimension of the input token embeddings width = 2048 # dimension of the SMVE-transformed embeddings k = 8 # number of non-zero elements to keep for each token # sample a random matrix B with unit-norm columns # -> a large set of anchor directions spread across the unit sphere B = randn(embedding_dim, width) # shape (embedding_dim, width) B /= B.norm(dim=0, keepdim=True) # normalize each column to unit length def smve(token_embeddings: Tensor, B: Tensor, k: int, is_query: bool) -> Tensor: # random projection: num_tokens x embedding_dim -> num_tokens x width projections = token_embeddings @ B # top-k sparsification: keep only the k largest values for each token # shape remains num_tokens x width values, indices = topk(projections, k) sparse = zeros_like(projections) sparse.scatter(dim=-1, index=indices, src=values) # pooling: num_tokens x width -> width # for both queries and documents, we first sum the token vectors pooled = sparse.sum(dim=0) # nothing else to do for queries if is_query: return pooled # for documents, we average the non-zero values in each dimension: # we compute the count of non-zero contributions in each dimension, # clamp to 1 to avoid division by zero, # and divide the sum-pooled vector by the count to get the average non_zero_counts = (sparse != 0).sum(dim=0).clamp(min=1) return pooled / non_zero_counts ``` Apart from the input embedding dimension set by the underlying late-interaction model, SMVE has only two hyperparameters: the number of anchor directions (`width`) and the number of non-zero values kept per token (`k`). Importantly, storage and compute are controlled by `k`, not by `width`. Each token contributes at most `k` non-zero values, so a pooled embedding for an input with `n` tokens has at most `k * n` non-zero elements. In practice, query and document embeddings can be pruned even further. TopK's production implementation uses optimized sparse representations and compute kernels to achieve the best possible performance when applying SMVE in both query and write paths. ### Multi-vector retrieval with SMVE We employ SMVE in the first stage of multi-vector retrieval in TopK. In particular, we store both the SMVE-transformed document representations and the original multi-vector representations. At query time, we first retrieve a candidate set using fast sparse retrieval over the SMVE-transformed representations, possibly overfetching by a small constant factor to compensate for approximation error. This step benefits directly from sparsity: documents that do not overlap with the non-zero structure of the query can be filtered out very efficiently. We then score the candidates using their original multi-vector representations and the full MaxSim operator. *This two-step process allows us to essentially match the quality of exhaustive MaxSim retrieval while speeding up the process significantly.* ## Experimental results We evaluated SMVE on several large-scale datasets from the [BEIR benchmark](https://arxiv.org/abs/2104.08663) using embeddings produced by [ColBERTv2](https://huggingface.co/colbert-ir/colbertv2.0). Experiments were run on **MS MARCO**, **Natural Questions (NQ)**, and **HotpotQA**, using a production implementation inside TopK. Our experiments focused on the trade-off between latency and recall in the end-to-end late-interaction retrieval pipeline. Specifically, we measured **end-to-end query latency**, including query embedding, candidate retrieval using SMVE, and final reranking with MaxSim over quantized multi-vector representations. To verify that these efficiency gains do not come at the cost of retrieval quality, we also measured **recall@k** for the full retrieval pipeline. Across all datasets, SMVE substantially reduces end-to-end latency while maintaining competitive retrieval quality. Compared to prior late-interaction systems such as **PLAID** and **MUVERA**, SMVE achieves roughly **5–8× lower latency** at common retrieval cutoffs. For example, on **MS MARCO**, a dataset with 8.8 million documents, average query latency is **39.9 ms for k=100** and **60.3 ms for k=1000**, compared to **221–318 ms for PLAID** and **310–444 ms for MUVERA**. Similar trends hold on **NQ** and **HotpotQA**, where SMVE consistently delivers the fastest response times. Crucially, these latency improvements do not come at the cost of retrieval quality. **Recall@k** remains competitive with the strongest baselines across datasets. On **MS MARCO** and **NQ**, SMVE closely matches the recall of both PLAID and MUVERA, while on **HotpotQA** it significantly outperforms PLAID and remains close to MUVERA. These results demonstrate that late-interaction retrieval can operate as a practical first-stage retrieval primitive on large datasets, achieving high recall with low end-to-end latency. ## Production-ready setup with TopK Our goal with this work was not just to make late-interaction retrieval faster in theory, but to make it practical to deploy in real systems. SMVE gives us a straightforward path to production: instead of relying on specialized multi-stage pipelines, we can transform multi-vector representations into sparse embeddings at write time and query them efficiently using our scalable sparse retrieval infrastructure. These ideas are now built directly into **TopK**, so you can use late-interaction models on large and dynamic corpora without assembling a complex retrieval stack. Simply ingest your representations and let TopK [index](https://docs.topk.io/sdk/topk-py/schema#multi_vector_index) and [query](https://docs.topk.io/sdk/topk-py/query#multi_vector_distance) them cost-effectively at scale. ### TopK Bench: Benchmarking Real-World Vector Search URL: https://www.topk.io/blog/20251201-topk-bench Published: 2025-12-01 Summary: This post evaluates how managed vector databases handle large-scale, production-like workloads—including millions of vectors, concurrent querying, real-world filtering, and continuous ingestion—using reproducible benchmarks across several leading providers. Vector databases are often evaluated on isolated metrics like query latency or recall, but production workloads depend on more than that. Databases need to be able to ingest data continuously, scale under concurrency, handle filters efficiently, and maintain recall across dataset sizes. In this benchmark, we evaluate how several of the most widely used managed vector databases (both serverless and instance-based) perform under simulated production-like workloads. We run five core benchmarks across multiple dataset sizes (100k, 1M, and 10M vectors): 1. **Ingest**: Measures total ingestion time and throughput of the write path (100k → 10M vectors), along with **freshness**—the delay from write acknowledgement to data being available in query results. 2. **Concurrency**: Assesses how latency and QPS change as the number of concurrent client workers increases (1, 2, 4, 8). 3. **Filtering**: Examines how metadata and keyword filters affect latency and QPS at different selectivity levels (100%, 10%, 1%). 4. **Recall**: Reports how well systems maintain recall as datasets scale and filters are applied. 5. **Read-write**: Evaluates how query performance degrades, if at all, while the database handles concurrent writes. ## Scope This benchmark evaluates **dense retrieval** performance across a set of popular managed vector databases, including both serverless and instance-based providers. All providers are tested with the same datasets, identical request patterns, and the same evaluation logic. Only TopK's absolute numbers are made public; other providers are anonymized to keep the focus on behavior rather than direct comparison while still reflecting how real, widely used services behave. ## 1. Dataset The benchmark is built from MS MARCO passages and queries, using 768-dimensional embeddings generated with [nomic-ai/modernbert-embed-base](https://huggingface.co/nomic-ai/modernbert-embed-base). We provide three corpus sizes—100k, 1M, and 10M vectors—each bundled with 1,000 evaluation queries and precomputed ground-truth nearest neighbors so recall can be measured consistently across systems. All datasets are hosted on S3 under `s3://topk-bench` as `docs-{100k,1m,10m}.parquet` and `queries-{100k,1m,10m}.parquet`. The [topk-io/bench](https://github.com/topk-io/bench) repository contains the benchmarking tool and further details on the dataset format. Each document includes the original passage text, its dense embedding, and two synthetic filter fields designed for controlled selectivity experiments: - `int_filter`: An integer field whose value are sampled from a uniform distribution `[0, 10_000]`. Predicates like `int_filter <= 10_000`, `int_filter <= 1_000`, and `int_filter <= 100` are then constructed to match ~100%, ~10%, and ~1% of documents. - `keyword_filter`: A categorical field populated with tokens chosen so that different keyword predicates produce similar 100% / 10% / 1% selectivity levels. Queries like `text_match(keyword_filter, "10000")`, `text_match(keyword_filter, "01000")`, or `text_match(keyword_filter, "00100")` are then constructed to match ~100%, ~10%, and ~1% of documents. These fields let you precisely control how many documents match a query, making it possible to evaluate how different systems behave as filter selectivity changes. ## 2. Methodology All benchmarks run in AWS containers in the same region as each provider. Most providers are serverless; for non-serverless providers, we use minimal viable configurations based on provider recommendations to ensure fair comparison. To reduce noise from transient issues like network hiccups, tail latencies, and noisy neighbors in serverless environments, each test configuration runs 5 times. We drop the worst run and report the **mean of the remaining 4 runs**, where "worst" is defined as: - **Latency**: highest p99 latency - **QPS**: lowest QPS - **Recall**: lowest recall Before each measurement, we perform a warmup run to ensure systems are in a steady state. Query benchmarks warm up with `concurrency=1` and a 60-second timeout (2x the measurement timeout). Filter benchmarks warm up with both filters enabled to exercise all code paths. For measuring latency & QPS, we send queries in a 30-second window. For recall, we run a set of 1000 queries and measure `recall@10`. ## 3. The `topk_bench` library Designing benchmarks that look like real production workloads is non-trivial, so we encoded these workloads into [`topk_bench`](https://github.com/topk-io/bench), an open-source, reproducible benchmarking tool. The tool includes the datasets, query sets, and evaluation logic used in this benchmark, making it easy to re-run experiments or benchmark your own deployments. You can find the full benchmarking suite at [topk-io/bench](https://github.com/topk-io/bench), which is open source and ready to use. The repository also includes a comprehensive Jupyter notebook for in-depth result analysis and visualization. ```python # Ingest documents tb.ingest( provider=tb.TopKProvider(), config=tb.IngestConfig( input="s3://topk-bench/docs-1m.parquet", # ... ), ) # Run queries tb.query( provider=tb.TopKProvider(), config=tb.QueryConfig( queries="s3://topk-bench/queries-1m.parquet", concurrency=4, # ... ), ) # Write metrics (locally or to s3://) tb.write_metrics("bench-1m.parquet") ``` You can find the full benchmarking suite at [topk-io/bench](https://github.com/topk-io/bench), which is open source and ready to use. The repository also includes a comprehensive Jupyter notebook for in-depth result analysis and visualization. ## 4. Ingest Performance Ingest performance evaluates how efficiently systems accept new data. We report ingestion time for each dataset size, and then examine how throughput behaves during long ingests. **Lower ingestion time is better; higher throughput (MB/s) is better.** For each provider, we performed a grid search to find the optimal combination of batch size and concurrency that yields the highest throughput. ### 4.1 Ingest Time We measure the total time required for systems to accept all data for each dataset size (100k, 1M, and 10M vectors). This is the time from the first write request to the final acknowledgment that all data has been ingested. **Lower ingestion time is better.** ### 4.2 Ingest Throughput Throughput measures the rate at which systems can accept data during ingestion, reported as MB per second. Higher throughput indicates more efficient write paths and better utilization of available resources. **Higher throughput is better.** ### 4.3 Freshness Freshness measures the time between write acknowledgment and the time when the written document becomes visible to queries. We report p50, p90, and p99 percentiles. **Lower freshness (time to visibility) is better.** Some providers offer strong consistency guarantees, which we disable in this benchmark since they would hide freshness characteristics at the cost of higher latency. We test with the default (eventual) consistency setup to measure true freshness behavior. ## 5. Query Throughput & Concurrency Scaling Here we look at how query throughput and latency change as client-side concurrency increases. For each provider and dataset size (100k, 1M, 10M), we run the same fixed query set with **1, 2, 4, and 8 concurrent clients** and observe the overall throughput (QPS) and P99 latency. ### 5.1 Latency We measure p99 latency as client-side concurrency increases. Systems that scale well should maintain stable or only slightly increasing latency, while systems that hit bottlenecks will show significant latency degradation at higher concurrency. **Lower latency is better.** **Note**: Provider C shows extreme latency outliers at 8 concurrency (436ms at 100k, 439ms at 1M), 20–40x worse than other providers. At concurrency 4→8, latency increases dramatically (159ms→436.5ms at 100k, 71.5ms→439ms at 1M) while QPS remains flat. This strongly suggests a query execution strategy that degrades sharply beyond a certain concurrency threshold, which makes it a risky choice for workloads that need predictable tail latencies as they scale out. ### 5.2 QPS We measure query throughput (queries per second) as client-side concurrency increases. Systems that scale well should show increasing QPS with higher concurrency, while systems that hit bottlenecks will plateau or degrade. **Higher QPS is better.** ## 6. Filtering Performance Many practical workloads combine vector search with metadata and keyword predicates, which modify the candidate set and affect both query performance and recall. We evaluate how systems handle filters at different selectivities (100%, 10%, 1%). As selectivity decreases, systems that efficiently prune candidates should show improved performance without degrading in result quality. **Lower latency is better; higher QPS is better.** Each provider runs the same filtered query sets across all dataset sizes using a single concurrent client. ### 6.1 No Filter vs 100% Filter A 100% filter selects the entire dataset but still exercises the filtering path. This isolates the overhead of the filtering mechanism itself. #### 6.1.1 Latency #### 6.1.2 QPS ### 6.2 Metadata Filters (100% / 10% / 1% Selectivity) We exercise metadata filtering using integer predicates at three selectivities. Systems that efficiently filter on metadata should show decreasing latency and increasing QPS as selectivity decreases from 100% to 1%, since fewer documents need to be processed. **Lower latency is better; higher QPS is better.** #### 6.2.1 Latency #### 6.2.2 QPS ### 6.3 Keyword Filters (100% / 10% / 1% Selectivity) We test keyword filtering using text predicates at three selectivities. Keyword filters exercise a different execution path than metadata filters. Systems that efficiently handle keyword predicates should show decreasing latency and increasing QPS as selectivity decreases from 100% to 1%. **Lower latency is better; higher QPS is better.** #### 6.3.1 Latency #### 6.3.2 QPS ## 7. Recall We measure recall at **top_k=10** using pre-computed ground truth for each dataset size and filter type. Ground truth was computed using exact search, ensuring we have the true nearest neighbors for each query. The recall tests use the same query sets as the previous benchmarks. We examine whether systems maintain high recall accuracy across dataset sizes and filter selectivities. **Higher recall is better.** #### 7.1 Metadata Filters Recall **Note**: Provider D shows lower recall (0.872) for int 1% filter at 10M dataset compared to other providers (~0.98-1.0). This suggests it may be using post-filtering, which can hurt recall performance when filters are highly selective. For applications that depend on high recall under narrow metadata filters, this kind of behavior would be a clear red flag. Perfect recalls (1.0) at high-selectivity filters on small datasets (100k) may indicate that implementations switch to exact search algorithms for small filtered candidate sets. #### 7.2 Keyword Filters Recall **Note**: Perfect recalls (1.0) at high-selectivity filters on small datasets (100k) may be due to exact matching behavior. ## 8. Read-Write Performance To see how systems behave under mixed workloads, we run the same query tests while a background writer continuously updates documents from the dataset, then compare read-only against read-write runs. The writer updates unrelated metadata fields (vectors and filter fields remain unchanged), and we query the same documents being updated to exercise this path. **Lower latency degradation is better; higher QPS maintenance is better.** #### 8.1 Latency #### 8.2 QPS ## 9. Cost Analysis (Simulated Workload) To get a sense of operating costs, we approximate a production-like workload and apply each provider’s **public on‑demand pricing** (late 2025). We assume a 10M‑item collection (768‑dimensional vectors with ~1 KB of metadata), **10M writes** and **50M queries** over the course of a month. We then run this workload through each pricing model (compute/scan units, read–write units, and storage) to get an order‑of‑magnitude cost comparison across TopK and other managed services. Under this model, **TopK** costs roughly **$29/month**, while other providers' costs span roughly **$120–$650/month**. The goal is to show how total cost compares on for a fixed, production-like, workload. **Avoid costly surprises when evaluating hosted vector databases.** Benchmarking can quickly get expensive, especially when testing the production details that matter: concurrency, p99 latency, metadata filtering, recall, ingest speed, and long-term cost. TopK is **built for production-ready** AI retrieval from day one — fast ingest, low-latency filtered search, high recall, and predictable cost. See our [Pricing](/pricing). ## 10. Conclusion These benchmarks show how managed vector databases differ on ingestion speed, query concurrency, filtering, recall, mixed read-write workload handling, and opearting cost. They reveal key tradeoffs—write speed vs. freshness, latency vs. concurrency, and search quality vs. filter complexity. **Why TopK behaves this way.** TopK’s results in this benchmark are a consequence of its design from first principles, not any single tuning choice. A few key architectural decisions matter in practice: - **Write path tuned for throughput**: a log‑structured write path with scalable compaction and indexing lets TopK accept writes quickly and make data promptly available for querying. - **Separation of read and write paths**: queries read optimized data files via our vectorized query engine (`reactor`), which gives us predictable tail latencies under load. - **Separation of storage and compute**: TopK uses object storage for durability and elastic compute, making it easy to scale and cost-effective for different workload shapes. You can explore more benchmarks and engineering deep dives in the TopK blog and benchmarks pages, starting from [www.topk.io/benchmarks](https://www.topk.io/benchmarks). Together, these choices are what enable the “just works” behavior these benchmarks surface: systems that remain fast, predictable, and cost‑efficient even as scale, filters, and hybrid scoring are added. For more on how TopK is built, see [Why Vector DBs Are the Wrong Abstraction](https://www.topk.io/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch) and [Billion-Scale Hybrid Search](https://www.topk.io/blog/20250722-billion-scale-hybrid-search). EDIT: TopK results were updated on 2025-12-08 to reflect the performance of the latest version. ### [Podcast] Building a Search Database from First Principles URL: https://www.topk.io/blog/20250907-podcast-building-search-database-first-principles Published: 2025-09-07 Summary: Listen to our founder Marek Galovic discuss the challenges and solutions in building a search database from the ground up, covering everything from vector search to hybrid retrieval and the architectural decisions that shaped TopK. **Apple Podcasts** **Spotify** ## In this episode Modern search is broken. There are too many pieces that are glued together. - Vector databases for semantic search - Text engines for keywords - Rerankers to fix the results - LLMs to understand queries - Metadata filters for precision Each piece works well alone. Together, they often become a mess. When you glue these systems together, you create: - **Data Consistency Gaps** - Your vector store knows about documents your text engine doesn't. Which is right? - **Timing Mismatches** - New content appears in one system before another. Users see different results depending on which path their query takes. - **Complexity Explosion** - Every new component doubles your integration points. Three components means three connections. Five means ten. - **Performance Bottlenecks** - Each hop between systems adds latency. A 200ms search becomes 800ms after passing through four components. - **Brittle Chains** - When one system fails, your entire search breaks. More pieces mean more breaking points. I recently built a system where we had query specific post-filters but the requirement to deliver a fixed number of results to the user. A lot of times, the query had to be run multiple times to achieve the desired amount. So we had an unpredictable latency, a high load on the backend where some queries hammered the database 10+ times, and a relevance cliff where results 1-6 look great, but the later ones were poor matches. Today on How AI Is Built, we are talking to Marek Galovic from TopK about how they built a new search database with modern components. "How would search work if we built it today?" Cloud storage is cheap. Compute is fast. Memory is plentiful. One system that handles vectors, text, and filters together - not three systems duct-taped into one. One pass handles everything: Vector search + Text search + Filters → Single sorted result Built with hand-optimized Rust kernels for both x86 and ARM, the system scales to 100M documents with 200ms P99 latency. The goal is to do search in 5 lines of code. ## Why This Matters The current approach to building search systems is fundamentally flawed. By treating each component as a separate system and gluing them together, we've created a fragile, complex, and slow architecture that fails under real-world conditions. This podcast episode explores a different approach: building search as a unified system from the ground up. Instead of duct-taping together vector databases, text engines, rerankers, and filters, TopK handles everything in a single pass with consistent data, predictable performance, and simple integration. Whether you're dealing with the complexity of multi-system search architectures, struggling with performance bottlenecks, or simply curious about how search could work better, this episode provides valuable insights into rethinking search infrastructure from first principles. ## Learn More - Explore our [documentation](https://docs.topk.io) to see how TopK works - Check out our [benchmarks](/benchmarks) to see performance comparisons - Read our [blog](/blog) for deep technical insights _Interested in building the future of search? We're hiring! Check out our [careers page](/careers) for open positions._ ### Beyond RRF: How TopK Improves Hybrid Search by up to 7.8% URL: https://www.topk.io/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality Published: 2025-07-24 Summary: A case study on how TopK hybrid retrieval outperforms traditional RRF-based methods, improving nDCG@10 by an average of 4.5% (and up to 7.8% on some datasets) by leveraging score-aware ranking, dynamic weighting, and efficient result merging. Modern information retrieval systems increasingly combine sparse and dense representations to balance lexical precision with semantic generalization. Traditionally, hybrid retrieval pipelines fetch partial results from multiple indices (e.g., dense embeddings and keyword-based models) and merge them using rank aggregation methods such as Reciprocal Rank Fusion (RRF). While effective, these pipelines often fail to fully leverage the scoring signals of individual retrievers and introduce ranking inconsistencies. This case study explores using TopK true hybrid retrieval capabilities to improve results quality over rank-fusion approaches. We benchmark four retrieval configurations across several datasets from the BEIR benchmark suite and observe that the TopK hybrid retrieval consistently improves nDCG@10 by up to 7.8% over traditional rank-fusion methods. ## Background Retrieval quality is a critical determinant of downstream application performance in search, recommendation, and question answering systems. Sparse retrieval, powered by models like SPLADE, excels at matching exact terms and handling structured queries, while dense retrieval models capture semantic similarity even when query and document vocabularies diverge. However, neither paradigm is universally dominant: - Sparse retrievers falter on paraphrased or conceptually rich queries. - Dense retrievers often overlook rare terms or domain-specific keywords. To address this, hybrid systems aggregate results from both models. The most common method is Reciprocal Rank Fusion (RRF), which normalizes ranks from each retriever and combines them into a unified ranking. While RRF is simple and effective, it ignores raw score magnitudes, applies uniform fusion weights, and often limits candidate lists to partial top-k results from each retriever. This can suppress relevant documents ranked moderately by both systems but overlooked by either individually. ## TopK Hybrid Search Our approach leverages [TopK's hybrid retrieval](https://docs.topk.io/concepts/true-hybrid-search) capabilities to provide direct, score-aware ranking across multiple retrieval methods. Instead of truncating to partial result sets and applying rank-only fusion, TopK: 1. **Scores and normalizes results directly** from each retriever (dense and sparse), respecting the magnitude of relevance scores rather than solely their ranks. 2. **Applies a tunable custom scoring function** that weights dense vs. sparse contributions dynamically (e.g., emphasizing sparse scores when exact term matches are present, and dense scores otherwise). 3. **Merges candidates globally** rather than pre-truncating, ensuring that documents moderately ranked by both retrievers are surfaced if their combined score is competitive. 4. **Selects the final top-k results** (e.g., top 10 or 100) directly, minimizing recall loss from early-stage truncation. This approach effectively removes a key bottleneck in hybrid search pipelines: the disconnect between partial recall from individual retrievers and the final relevance ordering. Here is how the query looks like in [TopK SDK](https://github.com/topk-io/topk): ```Python from topk_sdk.data import f32_vector, f32_sparse_vector from topk_sdk.query import select, field, fn collection.query( select( # Dense vector score dense_score=fn.vector_distance("dense", f32_vector([...])), # Sparse vector score sparse_score=fn.vector_distance("sparse", f32_sparse_vector({...})) ) .topk( # Merge dense and sparse scores 0.7 * field("dense_score") + 0.3 * (field("sparse_score") / 100.0), # Select top-10 results 10 ) ) ``` ## Experiments We evaluated four configurations across multiple datasets from the BEIR benchmark: 1. Dense-only retrieval using ModernBERT-base. 2. Sparse-only retrieval using SPLADE-v3. 3. Traditional hybrid retrieval using Reciprocal Rank Fusion (RRF). 4. Hybrid retrieval with TopK, employing a custom scoring function: `alpha * dense_score + (1 - alpha) * sparse_score`. We used nDCG@10 as the primary metric, reflecting both relevance and ranking position with `top_k = 10` results per query. By incorporating scores from both dense and sparse models inside a single query, we achieved an average improvement of 4.58% over RRF-based hybrid systems. | Dataset | Dense-only | Sparse-only | RRF | TopK Hybrid | Improvement | | ---------- | ---------- | ----------- | ------- | ------------ | ----------- | | FiQA | 0.40661 | 0.38023 | 0.4123 | **0.42853** | 3.94% | | TREC-COVID | 0.81431 | 0.66741 | 0.76779 | **0.82798** | 7.84% | | NQ | 0.52029 | 0.51405 | 0.53885 | **0.55271** | 2.57% | | NFCorpus | 0.32458 | 0.35837 | 0.34593 | **0.36803** | 6.39% | | FEVER | 0.85213 | 0.79154 | 0.84643 | **0.86464** | 2.15% | | Average | 0.583584 | 0.54232 | 0.58226 | **0.608378** | 4.58% | In practice, people often overfetch _k' > k_ results from individual retrievers and then apply RRF. While this is a valid approach to improve results quality, it often leads to slower queries and higher resource usage. For the sake of completeness, we evaluated RRF with 100 candidates per retriever to get the final top-10 results. | Dataset | RRF (10 candidates) | RRF (100 candidates) | TopK Hybrid | Improvement | | ---------- | ------------------- | -------------------- | ------------ | ----------- | | FiQA | 0.4123 | 0.41458 | **0.42853** | 3.36% | | TREC-COVID | 0.76779 | 0.80907 | **0.82798** | 2.34% | | NQ | 0.53885 | 0.54093 | **0.55271** | 2.18% | | NFCorpus | 0.34593 | 0.35027 | **0.36803** | 5.07% | | FEVER | 0.84643 | 0.84316 | **0.86464** | 2.55% | | Average | 0.58226 | 0.591602 | **0.608378** | 3.10% | As the table above shows, RRF with 100 candidates per retriever improves the overall results quality but TopK's hybrid retrieval still outperforms it by 3.10% on average while being more efficient at the same time. ## Summary Our evaluation demonstrates that TopK hybrid retrieval consistently improves results relevance across multiple datasets, achieving a 4.5% average increase in nDCG@10 over reciprocal rank fusion. By directly integrating normalized scores from dense and sparse retrievers, applying tunable weightings, and selecting the final top-k results without intermediate truncation, TopK mitigates recall loss and ranking inconsistencies inherent to partial list aggregation. These results underscore TopK's value as a more principled and efficient alternative to conventional hybrid search pipelines. If you want to learn more about TopK's hybrid search capabilities, check out our [documentation](https://docs.topk.io/concepts/true-hybrid-search). _If you are interested in building high-quality search infrastructure, shoot me an email at [marek@topk.io](mailto:marek@topk.io). **We’re hiring!**_ ## References - [TopK Hybrid Search](https://docs.topk.io/concepts/true-hybrid-search) - [ModernBERT](https://huggingface.co/blog/modernbert) - [SPLADE-V3](https://huggingface.co/naver/splade-v3) - [BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, Thakur et al., 2021](https://arxiv.org/abs/2104.08663) - [Discounted Cumulative Gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) ### Scaling Without Complexity: Billion-Scale Hybrid Search with TopK URL: https://www.topk.io/blog/20250722-billion-scale-hybrid-search Published: 2025-07-22 Summary: TopK enables billion-scale hybrid search with <100ms latency, fast indexing, and high-quality results. Enterprises are generating unprecedented volumes of data, and AI agents can only deliver value when they can access and reason over that information. But as datasets grow into the billions, most search and vector databases break down, leaving teams to choose between slow results, spiraling costs, or operational overhead. TopK removes those tradeoffs. With our latest release, TopK supports billions of documents inside a single collection while maintaining predictable, low-latency performance — without the need for manual sharding or complicated infrastructure. ## Why This Matters As organizations adopt AI-powered applications and knowledge retrieval pipelines, search infrastructure has become a critical bottleneck. Teams often spend significant engineering resources maintaining shards, tuning indexes, and building layers of caching just to meet latency targets. TopK’s unified architecture eliminates these pain points. You can grow a single collection to billions of records, query with hybrid search (dense and sparse vectors, keywords, and metadata filters), and still achieve sub-100 millisecond latencies without introducing operational overhead. For AI teams, this means their search infrastructure can scale to enterprise knowledge bases without degraded user experience. For product teams, it means search features stay fast and reliable as customer data grows. For technology leaders, it means fewer infrastructure components to manage, reduced engineering burden, and predictable costs as your datasets scale. ## Predictable Performance at Scale TopK was built from the ground up to handle the challenges of scale. Its distributed indexing engine, vector-aware storage, and adaptive caching work together to deliver consistent performance without manual tuning. Whether you are supporting AI agents, enterprise search, or analytics platforms, TopK allows your teams to focus on delivering value—not babysitting infrastructure. In our [benchmarks](/benchmarks), we show that TopK can index billions of documents in hours, and deliver ~50ms query latencies for both dense and sparse vectors. Additionally, TopK's hybrid query engine supports text search, filtering, and flexible scoring at this scale, giving users full flexibility without sacrificing quality of results. For dense vector search, TopK can deliver ~60ms p99 latency which improves to ~30ms when using filters with no degradation in results quality. This is contrary to other solutions which often degrade performance and results quality when using filters. For sparse vector search, TopK can deliver ~50ms p99 latency which improves to ~40ms when using filters. Similar to dense vector search, filtering does not degrade results quality. If you’re building AI agents, powering RAG pipelines, or modernizing your enterprise search stack for the AI era, we’d love to hear about your challenges and goals. Reach out to our team at [TopK](https://www.topk.io) to share your use case and explore how TopK can help you scale without compromise. ### Binary Vector Search at 350GB/s using ARM NEON URL: https://www.topk.io/blog/binary-vector-search-arm-neon Published: 2025-07-17 Summary: Optimizing binary vector search using ARM NEON instructions to achieve 350GB/s throughput. At TopK, performance and efficiency are the core principles that enable our hybrid retrieval engine (see our [benchmarks](/benchmarks), it's fast and it scales). In our journey to optimize dense vector retrieval, we turned our focus to one critical component — the Hamming distance function. This metric, essential in measuring bitwise similarity between binary vectors, plays a foundational role in tasks like approximate nearest neighbor (ANN) search. When you operate at scale, thousands of queries per second over millions of documents, even small improvements in this function can translate to massive gains. This post outlines our exploration and results from leveraging ARM NEON instructions to optimize the Hamming distance kernel. ## Baseline implementation Our starting point was a straightforward Rust implementation. The function iterates through two byte slices, computes the XOR byte-by-byte, and accumulates the number of differing bits using `count_ones()`. ```rust pub fn hamming_distance(x: &[u8], y: &[u8]) -> u32 { assert_eq!(x.len(), y.len()); let mut total = 0; for i in 0..x.len() { total += (x[i] ^ y[i]).count_ones(); } total } ``` While simple and portable, this approach left a lot on the table in terms of vectorization and memory throughput. | Threads | Samples/sec | Throughput (GB/s) | | :-----: | :-----------: | :---------------: | | 1 | 194,791,802 | 23.22 | | 10 | 1,743,320,290 | 207.82 | ## SIMD 101: Vanilla NEON Let’s take a look at our first NEON-based implementation that loads 16 bytes (128 bits) from x and y, computes `POPCOUNT(XOR(x, y))`, sums the register horizontally and adds the partial result to the accumulator. Here are the instructions we’ll need: - `veorq_u8` bitwise XOR of 16 bytes - `vcntq_u8` per-byte popcount across 16 lanes - `vaddvq_u8` horizontal add of all lanes into a single value ```rust fn hamming_distance_neon(x: &[u8], y: &[u8]) -> u32 { assert_eq!(x.len(), y.len()); let x_ptr = x.as_ptr(); let y_ptr = y.as_ptr(); // Accumulators let mut acc = 0_u32; // Operate on 128 bits (16 x u8) at a time let n = x.len() / 16; for i in 0..n { unsafe { let x = vld1q_u8(x_ptr.add(i * 16)); let y = vld1q_u8(y_ptr.add(i * 16)); // popcount(XOR(x, y)) // Each of the 16xu8 lanes has value <= 8 let xor = vcntq_u8(veorq_u8(x, y)); // Accumulate as u32 acc += vaddvq_u8(xor) as u32; } } // Handle the remaining bytes for i in (n * 16)..x.len() { acc += (x[i] ^ y[i]).count_ones(); } acc } ``` | Threads | Samples/sec | Throughput (GB/s) | | :-----: | :-----------: | :---------------: | | 1 | 424,912,063 | 50.65 | | 10 | 2,566,529,278 | 305.95 | Even the most basic NEON-based implementation gave us 2x improvement in single-threaded throughput and ~1.5x improvement in multi-threaded throughput. Noice! ## Exploiting Instruction Level Parallelism (ILP) The above implementation accumulates the result into a single scalar register which creates a dependency between memory loads and stores. To leverage instruction level parallelism available in modern CPUs, we changed the implementation to use two separate accumulators with effectively creates two separate pipelines no with load-store dependencies. ```rust fn hamming_distance_neon_ilp(x: &[u8], y: &[u8]) -> u32 { assert_eq!(x.len(), y.len()); let x_ptr = x.as_ptr(); let y_ptr = y.as_ptr(); // Accumulators let mut acc = unsafe { [vdupq_n_u8(0); 2] }; // Operate on 256 bits (4 x u64, 32 x u8) at a time let n = x.len() / 32; for i in 0..n { let i = i * 32; unroll! { for j in 0..2 { unsafe { let x = vld1q_u8(x_ptr.add(i + j * 16)); let y = vld1q_u8(y_ptr.add(i + j * 16)); // popcount(XOR(x, y)) // Each of the 16xu8 lanes has value <= 8 let xor_popcnt = vcntq_u8(veorq_u8(x, y)); // Accumulate as u32 acc[j] = vaddq_u8(acc[j], xor_popcnt); } } } } let mut res = unsafe { // Horizontal add vaddvq_u16(vpaddlq_u8(vaddq_u8(acc[0], acc[1]))) as u32 }; // Handle the remaining bytes for i in (n * 32)..x.len() { res += (x[i] ^ y[i]).count_ones(); } res } ``` | Threads | Samples/sec | Throughput (GB/s) | | :-----: | :-----------: | :---------------: | | 1 | 431,638,800 | 51.46 | | 10 | 2,697,048,892 | 321.51 | This version marginally outperformed the single-lane NEON implementation, particularly under high concurrency. It benefited from better utilization of NEON execution units, reducing stall cycles and enhancing memory-level parallelism. ## Fewer Instructions, More Speed Dense vector retrieval operates on vectors with fixed dimension - usually between 384 and 1536. This allows us to optimize our compute kernels even more by fully unrolling the loops and minimizing the control instructions overhead. Additionally, fully unrolled loops allow the compiler to better optimize register placement which further improves our effective throughput. ```rust fn hamming_distance_neon_1024b(x: &[u8], y: &[u8]) -> u32 { assert_eq!(x.len(), y.len()); let x_ptr = x.as_ptr(); let y_ptr = y.as_ptr(); unsafe { let mut acc = [vdupq_n_u8(0); 2]; unroll! { for i in 0..4 { let x = vld1q_u8(x_ptr.add(i * 32)); let y = vld1q_u8(y_ptr.add(i * 32)); let xp1 = vcntq_u8(veorq_u8(x, y)); acc[0] = vaddq_u8(acc[0], xp1); let x = vld1q_u8(x_ptr.add(i * 32 + 16)); let y = vld1q_u8(y_ptr.add(i * 32 + 16)); let xp2 = vcntq_u8(veorq_u8(x, y)); acc[1] = vaddq_u8(acc[1], xp2); } } // Horizontal add vaddvq_u16(vpaddlq_u8(vaddq_u8(acc[0], acc[1]))) as u32 } } ``` | Threads | Samples/sec | Throughput (GB/s) | | :-----: | :-----------: | :---------------: | | 1 | 525,551,619 | 62.65 | | 10 | 2,934,356,752 | 349.80 | This final variant delivered the best throughput for both single-threaded and multi-threaded workloads. With minimal branching and perfectly aligned memory access, it approaches the architectural limits of the CPU. ## Conclusion This journey underscores how performance engineering at the instruction level can help us get the most out of the underlying hardware. Leveraging ILP and problem-specific assumptions significantly improved both single-threaded and multi-threaded throughput of our Hamming distance kernel. The 1024-bit optimized NEON kernel now sustains nearly 350 GB/s which makes our billion-scale retrieval lower latency and improves the overall cost efficiency of our offering. Check out our [benchmarks](/benchmarks) to see how optimizations like this allow us to offer highest performance on the market. _If you are passionate about squeezing performance at both low-level and system-level, shoot me an email at [marek@topk.io](mailto:marek@topk.io). **We’re hiring!**_ ## References - [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) - [ARM NEON](https://developer.arm.com/Architectures/Neon) - [RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search, Gao et al., SIGMOD 2024](https://arxiv.org/abs/2405.12497) ### We Raised $5.5 Million to build an AI-Native Search Engine for Enterprises URL: https://www.topk.io/blog/seed-round Published: 2025-07-01 Summary: TopK has raised $5.5 million in seed funding to accelerate the development of its unified, AI-native search platform, enabling organizations to seamlessly combine vector, keyword, and custom ranking in a single system. The round was led by top-tier investors, fueling our mission to redefine search for the AI era. We’re thrilled to announce that TopK has raised $5.5 million in seed funding to build the world’s first AI-native, true hybrid search engine for enterprise. This milestone is backed by [Earlybird](https://www.earlybird.com/), [KAYA](https://www.kaya.vc/), [Irregular Expressions](https://www.irregex.vc/), and a group of exceptional angel investors — who share our conviction that search must evolve to meet the demands of modern AI-driven applications. ## The Search Landscape Is Changing The explosion of large language models like ChatGPT has accelerated enterprise demand for vector databases—and exposed the limits of legacy search systems. Today’s companies face: - **Fragmented infrastructure**: Piecing together multiple databases, APIs, and ranking layers. - **Performance bottlenecks**: Traditional search engines struggle to deliver low-latency hybrid results at scale. - **Rigid ranking**: Little to no flexibility in tailoring results to domain-specific needs. TopK was born to address these gaps, enabling developers and AI agents to search both structured and unstructured data—with lightning speed and deep control over relevance. ## True Hybrid Retrieval, Unified in One Engine At the heart of TopK is our hybrid retrieval approach, which seamlessly blends: - **Vector search** (semantic embeddings) - **Keyword matching** (precise keyword filtering) - **Advanced filtering & ranking** (domain-specific tuners) By unifying these capabilities under one roof, teams can fine-tune their query pipelines, optimize ranking strategies, and serve production workloads at web scale—no stitching together of disparate tools required. ## Meet the Founders - **Marek Galovič**, CEO & Co-Founder “The database market is undergoing a fundamental transformation. Our platform provides the unified query layer that powers the world’s most advanced AI-driven applications.” - **Jerguš Lejko**, CTO & Co-Founder “We’re challenging the status quo with a small but mighty team—design partnerships today will help us tackle tomorrow’s toughest search problems.” With proven track records in scalable architecture and AI tooling, they’re transforming the foundation of search. ## Backed by Top-Tier Investors We’re honored to be joined by: - **Earlybird** (General Partner: Andre Retterath) “TopK brings a fundamental shift in how infrastructure is built… delivering intelligent search for the AI era.” - **KAYA** (Partner: Karel Zheng) “With an exceptionally capable team, TopK’s core engine is the fastest, most cost-effective hybrid solution on the market.” - **Irregular Expressions** and leading angels This support will turbocharge our product development and global expansion, ensuring TopK becomes the go-to platform for enterprises across e-commerce, finance, healthcare, law, and beyond. ## What’s Next? With seed capital secured, we're investing heavily in enterprise-level capabilities that will serve our growing customer base: - Advanced enterprise features: Comprehensive security, compliance, and reliability capabilities - Enhanced query understanding: Sophisticated document transforms and intelligent query processing capabilities - Expanded data support: Beyond traditional text search to handle diverse content types The funding will fuel our team expansion, bringing in top engineering talent to accelerate our platform's evolution. Most importantly, we're actively seeking large-scale partnerships that will challenge us and push the boundaries of what's possible in search technology. We want to work with teams tackling the most complex, high-impact use cases that will help us build the most powerful search platform in the market. ## Ready to revolutionize your search? 👉 Visit our [homepage](/) to learn more or [sign up](https://console.topk.io) to get started. Follow our journey on [Twitter](https://x.com/topk_io) and join the conversation. ### TopK at Data Council AI Launchpad 2025 URL: https://www.topk.io/blog/topk-at-data-council-2025 Published: 2025-05-29 Summary: At Data Council 2025, we introduced TopK, a unified cloud-native query engine that combines vector search, keyword matching, and flexible scoring in a single system. We demonstrated how TopK overcomes the limitations of traditional vector databases by supporting hybrid queries with custom ranking logic while maintaining high performance. At [Data Council 2025 AI Launchpad](https://www.datacouncil.ai/talks25/ai-launchpad-2025-topk?hsLang=en), we introduced TopK, our new retrieval engine built to radically simplify and modernize search infrastructure. We walked through why existing vector databases fall short—especially when it comes to combining vector similarity with filters, keyword relevance, and domain-specific ranking. These systems often hard-code similarity metrics at index time and can't efficiently support flexible scoring or real-world metadata filtering, leading to poor relevance and unnecessary complexity. In response, we built TopK: a unified, cloud-native query engine that supports dense and sparse vector search, BM25 keyword scoring, multi-vector retrieval, and expressive scoring logic—all in a single query on a single dataset. It lets you efficiently blend signals like recency, quality, or geolocation into your ranking without sacrificing performance. We showcased a [live demo](https://youtu.be/viHRWx5n-T0?t=300) using a medical research dataset to illustrate how TopK handles everything from basic vector search to hybrid queries with keyword matching, metadata filters, and custom scoring expressions. We also showed how TopK can take care of embedding and reranking out of the box—making it easy to go from raw data to production-grade search with minimal setup. ### Why Vector DBs Are the Wrong Abstraction – And What We Built Instead URL: https://www.topk.io/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch Published: 2025-03-17 Summary: We’ve spent the last three years building the most popular vector database on the market. In that time we realized that a database built around vectors as a primary key is simply the wrong abstraction, creating an unnecessary obstacle for users in production. We’ve spent the last three years building the most popular vector database on the market. In that time we realized that a database built around vectors as a primary key is simply the wrong abstraction, creating an unnecessary obstacle for users in production. In most real-world applications, you want to combine vector search with traditional text search, metadata filtering, and rank results using custom scoring functions to get the best relevance. Doing all of this is very inefficient if your database is built around a vector index that assumes vector-only queries. The primary reason for this inefficiency is that the distribution of embeddings and the distribution of metadata are not strongly correlated. Another problem is the cost of legacy databases that couple compute and storage. In this architecture, write traffic can negatively affect queries which usually leads to over-provisioning of resources to maintain SLAs. Secondly, the data needs to be replicated to multiple nodes for durability and high availability. This gets expensive at $0.02 per GB. ## Architecture On a high level, we separate writes, indexing/compaction and query execution into multiple independent services. First, user writes are handled by the log writer service that appends them to a durable write ahead log (WAL) backed by object storage. Next, the compactor service consumes WAL entries and produces a read-optimized representation with additional indexes to make search queries fast. Similarly to the log, the indexed files are also durably persisted in object storage. Finally, the most interesting part, queries. The router service receives application queries, validates them and converts to a logical plan representation used by our distributed query engine **reactor**. Reactor then distributes the query over a set of executor nodes that read indexed files from object storage, execute the query plan, and return partial results back to the router which computes the final result and returns it back to the client. Executor nodes cache data locally on NVMe SSDs and in-memory to improve latencies for subsequent queries. In principle, any instance can handle requests for any collection in the cluster which gives us high availability. We don’t want random routing though, since that would result in poor cache utilization, so we consistently assign collections and data files to executors to get high cache hit rates. ## Query Engine Both query routers and query executors use the same query engine, reactor, to run distributed queries. Internally, reactor uses Apache Arrow for in-memory data extended with custom layouts for types like dense/sparse matrix and posting lists. The choice of Arrow allows us to use off-the-shelf compute kernels for undifferentiated ops like filtering and focus our effort on developing high performance kernels for search. We initially looked at using DataFusion, which is the default choice for most new databases these days, but ultimately decided against it since it’s better suited for analytical workloads and doesn’t natively support external indexes. Furthermore, the co-design of storage format and execution engine allows us to write compute kernels that operate directly on compressed data which massively improves performance on the search workloads that we target. ## Storage Format Using object storage as the primary storage medium posed a number of challenges for us when it comes to delivering low query latencies. First, time-to-first-byte (TTFB) request latencies are roughly 190ms p95 which is much higher than locally attached disks. Second, per-request pricing model implies optimal I/O request sizes to minimize cost and maximize throughput. Given these constraints, we found off-the-shelf file formats such as Parquet to perform quite poorly since they have serial dependencies in their I/O and couple statistics/metadata granularity with I/O granularity. The latter is particularly problematic for reading row groups with columns that have drastically different value size because of the imbalance in column chunk sizes. To address these issues we built a columnar file format (.bob) that has very wide I/O trees to maximize concurrent I/O and decouples the logical file structure from the physical data layout. This enabled us to achieve both optimal I/O request size for object storage and granular statistics for effective block pruning at the same time. ## What does this enable? Having a search database that’s 10-100x cheaper is great but not enough. Ultimately, our goal is to enable developers to build production-ready search with highly relevant results in the most intuitive way possible. ### True hybrid retrieval Existing search systems that support hybrid retrieval usually do so using reciprocal rank fusion (RRF) where partial results from multiple indexes are merged and resorted based on their ranks. This requires over-fetching candidates from individual indexes and doesn’t consider relevance scores of candidates which hurts performance. To fix these issues, TopK supports true hybrid retrieval in a single query against the same index which enables users to combine multiple vectors with text filters and metadata filters. ```python client.collection("books").query( select( "title", score=fn.vector_distance("title_embedding", [...]), # vector scoring ) .filter(match("catcher")) # text filtering .filter(field("published_year") > 1920) # metadata filtering .top_k(field("score"), 100), ) ``` ### Flexible scoring Custom scoring rules based on document attributes (e.g. boosting) are instrumental for getting highly relevant results in your application. TopK gives you the ability to combine multiple scoring functions with custom expressions to optimize the ranking of results you show to your users. An example use case for this is a medical search application where we want to combine relevance of abstract and passage embedding with BM25 score and boost documents based on a quality of the journal they were published in. See the code below to understand what I have in mind: ```python client.collection("medical_papers").query( select( "passage", abstract_sim=fn.vector_distance("abstract_embedding", [...]), passage_sim=fn.vector_distance("passage_embedding", [...]), bm25=fn.bm25_score(), ) .filter(match("respiratory disease")) .filter(field("published_year") >= 2019) .top_k( field("journal_quality") * (field("abstract_sim") + field("passage_sim") + field("bm25")), 100 ), ) ``` ## Benchmarks ### Vector search with selective filters Performance of vector databases degrades with highly selective filters even though this is the most common access pattern in production. TopK’s indexing and execution engine which results in highly selective queries being faster, not slower. The benchmarks below show vector-only queries with filters that select 100%, 10% and 1% of the indexed documents. For a collection with 1M documents and 768-dimensional vectors we achieve ~62ms p99 latency, for collection with 10M documents and 768-dimensional vectors, we achieve ~115ms p99 latency. ### Text-only filtering and ranking Not every application needs vector-based retrieval, especially when traditional keyword filtering with BM25 scoring works fine as a first-stage retriever. Our query engine supports text-only queries with or without BM25 and achieves ~32ms p99 latency for conjunction queries (all query terms must match) and ~60ms p99 latency for disjunction queries (any query term must match). If you are building a semantic search, AI-driven application, agents, RAG or similar and would like to try this for yourself, you can now head over to [console.topk.io](https://console.topk.io), generate your API key and start upserting and querying your data. ## References 1. [Exploiting Cloud Object Storage for High-Performance Analytics, Durner et al., VLDB'23](https://www.durner.dev/app/media/papers/anyblob-vldb23.pdf?i=1) 2. [An Empirical Evaluation of Columnar Storage Formats, Zeng et al., VLDB'23](https://arxiv.org/pdf/2304.05028) 3. [Reciprocal Rank Fusion, Cormack et al., SIGIR'09](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) ## Answers ### Can You Convert Vectors Between Models? URL: https://www.topk.io/learn/converting-embeddings Category: Embeddings in Production Summary: No. Different models produce incompatible spaces, and learned mappings lose the distinctions you upgraded for. Plan to re-embed, and make it cheap. **No. Embeddings from different models, including different versions of the same model, live in geometrically incompatible spaces. A query embedded with model B scored against documents embedded with model A returns noise, not degraded-but-usable results.** **Learned translations between spaces exist in research, but they blur exactly the fine distinctions you're upgrading to get, and their failure modes are unpredictable on your data. Treat vectors as derived data: keep the source text, re-embed on every model change, and put your effort into making re-embedding cheap.** ## Why don't spaces transfer? An embedding's coordinates mean nothing outside the model that produced them; dimension 412 encodes whatever that particular training run made it encode. Two models trained on different data with different objectives put "similar" documents in entirely different places. There's no shared frame for a simple transform to recover. ## What about learned mappings? Research shows rough translation between embedding spaces is possible: [vec2vec (Jha et al., 2025)](https://arxiv.org/abs/2505.12540) translates embeddings across models without paired data. But "possible" means enough to be interesting for analysis, not enough to serve retrieval. A mapping trained to minimize average error preserves coarse topology and sacrifices the fine-grained neighborhoods that decide top-10 rankings, and it drifts hardest on exactly the out-of-distribution content where you'd least notice. For a production index, "approximately converted" means silently worse results with no error to alert on. ## So what makes re-embedding cheap? Three things cut the cost, in order of impact: 1. **Embed less:** deduplicate before embedding, since redundant corpora shrink dramatically ([how to chunk redundant documents](/learn/chunking-redundant-documents)). 2. **Batch offline:** backfills are throughput problems, not latency problems; batched inference is a fraction of per-request pricing on any provider. 3. **Let the database do it:** when the engine owns embedding (managed inference), a backfill is a server-side operation instead of an external pipeline you build, monitor, and keep version-consistent. The full migration pattern is in [how do you upgrade embedding models](/learn/upgrading-embedding-models). With TopK's [`semantic_index`](/blog/20260611-semantic-index-multi-vector-retrieval), embedding runs inside the database, so re-embedding is a backfill rather than a data-engineering project, and "we should upgrade the model" turns from a quarter-long migration into a routine operation. ### Can You Do Hybrid Search Through the Elasticsearch API? URL: https://www.topk.io/learn/hybrid-search-elasticsearch-api Category: Hybrid Search & Ranking Summary: Yes. Send query and knn in one request and the scores add with per-clause boosts, or use rank.rrf to fuse by rank. semantic_text covers having no embeddings. **Yes. The search body accepts a `query` and a `knn` clause in one request, returns the union of both result sets, and scores each hit as the keyword score plus the vector score, weighted by the `boost` on each clause. Reciprocal rank fusion replaces the sum when you ask for it.** **If you have no embeddings of your own, map the field as `semantic_text` and use a `semantic` query, and the engine embeds at index and query time. TopK serves all three shapes through its Elasticsearch-compatible API, with managed embedding behind `semantic_text`.** ## How do you send both retrievers in one request? You put the keyword query under `query` and the vector search under `knn`, in the same body. Elasticsearch runs both, combines the matches as a disjunction, and scores each hit as the sum of its keyword score and its vector score, with a `boost` on either clause weighting its share of the sum ([Elastic kNN guide](https://www.elastic.co/docs/solutions/search/vector/knn)). A document that both retrievers return outranks a document that only one returns. ```json POST /books/_search { "query": { "match": { "body": { "query": "dragons", "boost": 0.9 } } }, "knn": { "field": "embedding", "query_vector": [0.12, -0.03, ...], "k": 20, "boost": 0.1 }, "size": 10 } ``` The `k` on the `knn` clause is how many vector matches enter the union, and `size` is how many hits you get back after the scores are combined. TopK's Elasticsearch API keeps the same arithmetic. A document that matches both retrievers scores above one that matches either alone, and raising the `knn` boost raises the vector share of the score. ## Should you add scores or fuse by rank? Adding scores is a weighted fusion, and it works when the two scores live on scales you understand. A BM25 score is unbounded and a normalized cosine sits between 0 and 1, so the `boost` values do the calibration, and you should expect to tune them per corpus. Reciprocal rank fusion ignores the score values and combines positions instead, which needs no calibration and is the reason it is the default choice when you have no way to compare the two scales ([Elastic RRF reference](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion)). Elasticsearch expresses it today as an `rrf` retriever that wraps a `standard` and a `knn` retriever, with `rank_window_size` setting how many hits each contributes and `rank_constant` defaulting to 60. TopK's Elasticsearch API accepts the earlier form, a top-level `rank` block with `rrf` next to `query` and `knn`, which Elasticsearch documented from 8.8 to 8.13, and it does not accept the `retriever` syntax yet. Both methods merge two lists that were ranked separately, so a document that sits just outside both top lists never gets ranked on the combined signal. [RRF versus true hybrid](/learn/rrf-vs-true-hybrid) covers that limit and TopK's measurement of it, where scoring once over the full candidate set [improved nDCG@10 by 4.58% on average over RRF](/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality) across five BEIR datasets (July 2025). Through the Elasticsearch API you get the two fusion methods above, and through TopK's native query you get the single ranking expression. ## What if you have no embeddings of your own? Map the text field as `semantic_text` and query it with a `semantic` clause. In Elasticsearch, a `semantic_text` field is backed by an inference endpoint that embeds the text at index time and the query at search time, so you never handle vectors in your application ([Elastic semantic_text reference](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text)). A `semantic` query can stand alone, or it can share a request with a `knn` clause over a separate `dense_vector` field, and the two combine the same way a keyword query and `knn` do. ```json POST /books/_search { "query": { "semantic": { "field": "content", "query": "a quest to destroy a cursed ring" } }, "size": 10 } ``` On TopK, a `semantic` clause is a scoring clause, so it is valid under `query`, `must`, or `should`, and a `must_not` around it returns a 400. Adding a `sort` on another field returns hits in sort order with a null `_score`, as Elasticsearch does for any scored query. On TopK, a `semantic_text` field is embedded by TopK's managed inference, so the same request works without an inference endpoint of your own. TopK accepts `inference_id`, `search_inference_id`, and `chunking_settings` on the mapping, so an existing mapping loads without edits, and TopK's managed model does the embedding whatever `inference_id` names. ## What does this look like on TopK? The request is one of the bodies above, sent by the Elasticsearch client you already use, with the endpoint and the API key pointed at TopK. ```python from elasticsearch import Elasticsearch es = Elasticsearch("https://", api_key="") res = es.search( index="books", query={"semantic": {"field": "content", "query": "a quest to destroy a cursed ring"}}, knn={"field": "embedding", "query_vector": query_embedding, "k": 20}, size=10, ) ``` The [compatibility overview](/learn/elasticsearch-api-compatibility) covers what TopK accepts and rejects, and the [kNN guide](/learn/knn-search-elasticsearch-api) covers the vector clause on its own. If you want keyword, vector, and [multi-vector](/learn/multi-vector-retrieval) signals in one ranking expression instead of a fusion of two lists, the [native TopK query](/learn/rrf-vs-true-hybrid) does that in one line. ### Can You Do Multi-Vector Search Through the Elasticsearch API? URL: https://www.topk.io/learn/multi-vector-elasticsearch-api Category: Multi-Vector Retrieval Summary: In Elasticsearch, rank_vectors only rescores candidates through a script. TopK indexes the field for MaxSim retrieval, searched by a knn clause with a matrix. **Only through a script. Elasticsearch stores token embeddings in a `rank_vectors` field, and Elastic's reference describes it as a way to rerank documents with MaxSim through a `script_score` query, which scores whatever the inner query returns, every document under `match_all` or the shortlist another retriever picked. The field has no index and no place in the `knn` clause.** **TopK indexes multi-vector fields so MaxSim can retrieve, and rank, on its own. Through its Elasticsearch-compatible API, a `rank_vectors` mapping becomes a MaxSim index, and a `knn` clause whose `query_vector` is a matrix of token embeddings searches it directly.** ## What does Elasticsearch offer for multi-vector? Elasticsearch stores a variable number of vectors per document in a `rank_vectors` field, and it scores them with a `script_score` query that calls `maxSimDotProduct` on the query matrix ([Elastic rank_vectors reference](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/rank-vectors)). The reference describes the field as a way to rerank documents with MaxSim, and the field takes `element_type` and `dims` and nothing else. There is no `index` or `similarity` parameter, so the engine cannot search the field the way it searches a `dense_vector`. The `script_score` query scores whatever its inner query returns. Under `match_all`, which is the example in the reference, the script computes MaxSim for every document in the index, one at a time and without an index to narrow the work. Under a `match` or a single-vector `knn`, the script rescores that shortlist, and the multi-vector signal only reaches documents the first retriever already found. ## Why is rescoring different from retrieval? Rescoring can only reorder the shortlist, so a document that a single vector or BM25 misses never gets a MaxSim score, and the brute-force alternative reaches every document by scanning every document. [Multi-vector retrieval](/learn/multi-vector-retrieval) exists because one pooled vector loses the detail inside a passage that precise queries need, and the documents it loses are the ones that only look relevant token by token. A pipeline that retrieves with the pooled vector and rescores with MaxSim inherits the miss, and the rescoring step cannot recover it. The script also costs the full MaxSim per candidate, which is the query token count times the document token count in dot products, for every document in the shortlist. Widening the shortlist to catch more misses raises that cost in proportion, and it still never reaches the documents outside it. ## What does an indexed multi-vector field change? An index lets MaxSim choose the candidates instead of reordering someone else's, so a document that only looks relevant token by token can still be found. TopK's multi-vector index, [SMVE](/blog/20260311-smve-multi-vector-retrieval), does that selection and then scores the shortlist with exact MaxSim. In TopK's measurement (March 2026, ColBERTv2 on BEIR), that gave [roughly 5 to 8 times lower end-to-end latency than PLAID and MUVERA](/blog/20260311-smve-multi-vector-retrieval) at competitive recall, and on MS MARCO, with 8.8M documents, SMVE averaged 39.9 ms for k=100 against 221 to 318 ms for PLAID and 310 to 444 ms for MUVERA. The production system built on it, [`semantic_index`](/blog/20260611-semantic-index-multi-vector-retrieval), measured 52.88% nDCG@10 across all 15 BEIR datasets, within about 1% of exhaustive MaxSim, at 295 QPS and about 75 ms p99, while ingesting over 1.5B tokens per hour (June 2026). [Is multi-vector retrieval too expensive](/learn/multi-vector-at-scale) covers the cost side of the same measurements. ## What does this look like through the Elasticsearch API? You map the field as `rank_vectors`, index a matrix per document, and search with a `knn` clause whose `query_vector` is a matrix. On TopK, that field is served by a multi-vector index scored by [MaxSim](/learn/maxsim), and the search ranks every document in the index by MaxSim rather than rescoring a shortlist. ```python from elasticsearch import Elasticsearch es = Elasticsearch("https://", api_key="") es.indices.create(index="passages", mappings={"properties": { "token_embeddings": {"type": "rank_vectors", "dims": 128}, }}) es.index(index="passages", id="p1", document={ "token_embeddings": passage_token_vectors, # one vector per token }) res = es.search(index="passages", knn={ "field": "token_embeddings", "query_vector": query_token_vectors, # a matrix, one vector per query token "k": 10, }) ``` TopK checks the shapes both ways. A flat vector against a `rank_vectors` field returns a 400, and a matrix against a `dense_vector` field returns a 400, so a mismatch fails at the request rather than producing a wrong ranking. The [compatibility overview](/learn/elasticsearch-api-compatibility) covers the rest of the API, and [when to use multi-vector embedding models](/learn/when-to-use-multi-vector-embeddings) covers whether your workload needs the field at all. ### Can You Replace Elasticsearch Without Rewriting Queries? URL: https://www.topk.io/learn/elasticsearch-api-compatibility Category: Scale & Architecture Summary: Yes, if the engine speaks the Elasticsearch API well enough for the official clients: the product header, your mappings, bulk and search, and the query DSL. **Yes, if the new engine implements the Elasticsearch REST API closely enough that the official clients accept it and your requests keep working. Compatibility has to cover the product header the clients check for, your mappings accepted as written, the document and bulk endpoints, and the query DSL your application already sends.** **TopK exposes an Elasticsearch-compatible API to the same engine behind its SDKs and SQL, so an existing Elasticsearch client can create indexes, bulk load, and run kNN, keyword, and hybrid queries against TopK by changing the endpoint and the API key.** ## What does compatibility have to cover? The first thing an official client does is refuse to talk to a server it does not recognize. The Elasticsearch clients look for an `X-Elastic-Product: Elasticsearch` header on every successful response, and the Python client [raises `UnsupportedProductError`](https://github.com/elastic/elasticsearch-py/blob/main/elasticsearch/_sync/client/_base.py) when the header is missing. An engine that wants to sit behind those clients has to answer the root endpoint with that header and a version block, and it has to echo request headers such as `X-Opaque-Id` that tracing tools depend on. After the handshake, four groups of endpoints do almost all the work. First, index creation with a `mappings` block, so field types such as `text`, `keyword`, `integer`, `boolean`, `object`, and `dense_vector` are accepted as written. Second, the document endpoints, which include `_doc`, `_bulk`, `_mget`, and `_count`. Third, `_search` and `_msearch` with the query DSL your application uses, which for most applications means `match`, `term`, `terms`, `range`, `bool`, `prefix`, `regexp`, `ids`, and `exists`, plus `sort` and `size`. Fourth, aggregations, at least `terms` on keyword fields. ## What has to match beyond the endpoints? Accepting the request is the easier half, because the client also expects Elasticsearch's answers. Scores are the clearest example. Elasticsearch reports a `dense_vector` cosine hit as `(1 + cosine) / 2`, so identical vectors score 1.0 and orthogonal vectors score 0.5, and it reports `dot_product` the same way ([Elastic field reference](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector)). An application that thresholds on `_score` or reads `max_score` breaks if a replacement returns the raw cosine instead, so the replacement has to return the normalized score, and TopK does. Field semantics have to match too. A `match` on a `keyword` field must compare the whole value, case sensitively, while a `term` on a `text` field must match one indexed token. Sorting or aggregating on an analyzed `text` field must fail with a 400 the way Elasticsearch fails, because an application that relies on the error to pick a `.keyword` subfield would otherwise get a wrong answer instead of an exception. TopK's Elasticsearch API behaves the same way in each of those cases. ## What will not carry over? Any engine that implements the Elasticsearch API without being Elasticsearch has gaps, and the useful gaps are the ones that fail loudly. TopK answers with a 400 and a message for mappings and clauses it does not implement, which includes `date` fields, custom `analyzer` settings, `minimum_should_match`, and scripted updates in `_bulk`. A migration test run finds each of those on the first pass, because nothing is silently reinterpreted. If your application depends on date math, custom analyzers, or Painless scripts, budget for changing those call sites, and the rest of the request path stays as it is. ## When should you keep Elasticsearch? Keep Elasticsearch when the workload is what Elasticsearch was built for. If your indexes are logs and metrics, your queries lean on date histograms and pipeline aggregations, and your team lives in Kibana, a retrieval engine behind the same API does not help you. Move when the workload is retrieval for an application, which means kNN over your own embeddings, keyword and semantic search over documents, filters that have to stay fast, and hybrid ranking, and you want a different engine underneath without a rewrite. ## What does this look like on TopK? You keep the client and change the endpoint and the key. The client library, the mappings, the bulk loader, and the queries are the ones you already have. ```python from elasticsearch import Elasticsearch es = Elasticsearch("https://", api_key="") es.indices.create(index="books", mappings={"properties": { "title": {"type": "text"}, "genre": {"type": "keyword"}, "embedding": {"type": "dense_vector", "dims": 768, "similarity": "cosine"}, }}) es.search(index="books", knn={ "field": "embedding", "query_vector": query_embedding, "k": 10, "filter": {"term": {"genre": "fantasy"}}, }) ``` Your mapping decides what TopK indexes. A `dense_vector` field is served by a vector index with the metric you named, a `semantic_text` field is embedded and served by TopK's managed inference, and Elastic's `rank_vectors` type is served by a multi-vector index scored by MaxSim, which [turns a rescoring field into a retriever](/learn/multi-vector-elasticsearch-api). The implementation is [open source](https://github.com/topk-io/topk/tree/main/topk-es), and the [kNN](/learn/knn-search-elasticsearch-api) and [hybrid](/learn/hybrid-search-elasticsearch-api) guides cover the query side. ### Can You Run Vector Search in SQL? URL: https://www.topk.io/learn/sql-vector-search Category: Hybrid Search & Ranking Summary: Yes, two ways: extensions like pgvector add distance operators to Postgres, and TopK speaks the Postgres protocol so hybrid search is ordinary SQL. **Yes, and there are two different ways to get there. A vector extension such as [pgvector](https://github.com/pgvector/pgvector) adds a vector column type and distance operators inside a relational database. A search engine that speaks the Postgres wire protocol, which is what [TopK SQL](/blog/20260614-topk-sql) is, exposes semantic, keyword, multi-vector, and hybrid scoring as ordinary SQL to any Postgres client.** **The difference is what the SQL can express. An extension gives you `ORDER BY embedding <-> query`, one distance against one column. A search dialect treats scores as values: BM25, semantic similarity, and vector distance can be selected, weighted, and combined in a single `ORDER BY` expression, which is true hybrid ranking in one statement.** ## What does a vector extension give you? [pgvector](https://github.com/pgvector/pgvector) adds a `vector` column type, distance operators, and ANN indexes to Postgres, so similarity search runs next to your relational data. You keep one system, transactional writes, and joins against the tables you already have. If your data already lives in Postgres, the corpus is modest, and a single distance ranking is all the search you need, the extension is the right choice, and you should take it. ## What does search-native SQL add? Retrieval modes become scoring functions, and scores become ordinary values you can alias, filter, and combine. TopK SQL exposes each of its index types through one function: | Function | What it scores | |---|---| | `semantic_similarity(field, query)` | managed semantic search, embedded and reranked by the engine | | `vector_distance(field, vector)` | dense or sparse ANN against your own embeddings | | `multi_vector_distance(field, matrix)` | [late-interaction MaxSim](/learn/multi-vector-retrieval) retrieval | | `bm25_score()` | keyword relevance from `match_any` / `match_all` predicates | Tables are schemaless, so rows can carry undeclared fields that remain filterable, and the type system includes dense, sparse, multi-vector, and binary vector shapes natively. Indexes are declared inline on the column: ```sql CREATE TABLE books ( title TEXT, published_year INTEGER, bio TEXT INDEX semantic_index(), embedding f32_vector(768) INDEX vector_index(metric = 'cosine') ); ``` The row `multi_vector_distance` deserves a second look: that is MaxSim, the scoring operator behind [multi-vector retrieval](/learn/multi-vector-retrieval), available as a SQL function. ## How does hybrid ranking work in one statement? Score each signal, alias it, and write the blend directly in `ORDER BY`: ```sql SELECT _id, title, bm25_score() AS keyword_score, semantic_similarity(bio, 'an epic fantasy quest') AS semantic_score, vector_distance(embedding, '[...]'::f32_vector) AS vector_score FROM books WHERE match_any(bio, 'dragon wizard') AND published_year > 1950 ORDER BY 0.2 * keyword_score + 0.5 * semantic_score + 0.3 * vector_score DESC LIMIT 10; ``` As the [TopK SQL announcement](/blog/20260614-topk-sql) puts it, this is "hybrid search without multiple queries, client-side fusion, or reciprocal-rank fusion." It is the [true-hybrid model](/learn/rrf-vs-true-hybrid) expressed in SQL: every signal is scored inside one query, and the ranking is a single expression instead of a merge of separate result lists. Metadata folds into the same expression, so `boost(semantic_score, published_year > 2010, 1.5)` promotes recent books without a second pass. ## What can connect to it? Anything that talks to Postgres. The SQL layer implements the Postgres wire protocol in both simple and extended query modes, so `psql`, application drivers, ORMs, prepared statements, and dashboard tools connect without adapters. The connection is one line, with an API key as the password: ```bash psql "host=elastica.sql.topk.io password=" ``` Tables are inspectable through `information_schema`, and `EXPLAIN` shows the engine query your SQL was parsed into before it runs. The dialect is a thin mapping, and its parser is [open source](https://github.com/topk-io/topk): the query you write in SQL and the same query in the Python SDK resolve to the same plan, so you choose the interface per surface, a notebook, a service, a BI dashboard, without changing what executes. ## When is the extension the right choice? Stay with pgvector when Postgres is already your source of truth, your corpus and query volume are modest, and search means one similarity ranking joined against relational data. Reach for search-native SQL when ranking is the product: when you need keyword, semantic, and [multi-vector](/learn/multi-vector-retrieval) signals in one expression, engine-side embedding, and [filters that stay fast at high selectivity](/learn/filtered-vector-search), while keeping the Postgres tooling your team already uses. TopK SQL ships today: the [language overview](https://docs.topk.io/sdk/topk-sql/overview) covers the full dialect, and the [announcement post](/blog/20260614-topk-sql) (June 2026) walks through the design. ### Collections or Filters for Multi-Tenancy? URL: https://www.topk.io/learn/multi-tenant-vector-search Category: Filtering & Multi-Tenancy Summary: Default to one shared collection with a mandatory tenant filter; split out only huge or contractually isolated tenants. Idle-collection cost decides. **For thousands of small tenants, default to a shared collection with a mandatory tenant filter, and split out only the tenants that are huge, noisy, or contractually required to be isolated.** **The deciding variable is your engine's cost of an idle collection: in memory-resident engines every collection carries fixed RAM and index overhead, so thousands of collections mean paying for thousands of mostly-idle indexes; in storage-backed engines idle data costs storage prices, which weakens the penalty on either layout.** **What never changes: the tenant filter must be enforced by the engine at query time, and it must stay fast at high selectivity.** ## What does the shared-collection pattern look like? Everything lives in one index: every document carries `tenant_id`, and the application attaches a non-negotiable `tenant_id = X` filter to every query. That is the same mechanism as [RBAC filtering](/learn/rbac-vector-search), since tenancy is just the coarsest ACL. Onboarding a tenant is writing rows, not provisioning infrastructure, which is what makes thousands of customers operable by a small team. In TopK the tenant filter is one predicate on the query: ```python from topk_sdk.query import select, field, fn docs = client.collection("docs").query( select("title", score=fn.semantic_similarity("content", user_query)) .filter(field("tenant_id") == "acme") # mandatory, attached by the app .sort(field("score"), asc=False) .limit(10) ) ``` The [query documentation](https://docs.topk.io/collections/query) covers the predicate syntax. ## When do separate collections win? Separate collections win in three cases. A **scale outlier**, a tenant so large it degrades everyone's latency, deserves its own home. **Compliance isolation** wins when contracts mandate physical separation that a filter can't provide. **Divergent configuration** wins when a tenant needs a different embedding model or schema and can't share an index built on other assumptions. All three describe your largest few customers. Building per-tenant collections for *everyone* designs the whole system around the exceptions. ## What's the failure mode of each layout? The shared layout fails through its filter: a tenant occupying 0.1% of a shared index is a 0.1%-selectivity filter on every query, which is exactly where naive engines [time out or under-fill](/learn/filtered-vector-search). The separate layout fails through fixed-cost multiplication and operational sprawl, and in stateful engines per-tenant growth reintroduces [shard-rebalancing pain](/learn/shard-rebalancing) one collection at a time. TopK's architecture keeps both layouts viable. [Each collection is its own prefix on object storage](https://docs.topk.io/architecture): isolation by construction, with idle data at storage prices rather than pinned to compute. Natively-evaluated filters are [benchmarked across selectivities](https://www.topk.io/benchmarks) for the shared layout, and for tenants who need physical isolation contractually, [dedicated single-tenant regions](https://docs.topk.io/architecture) cover the compliance case. The choice can follow your tenants rather than your database. ### Do You Still Need a Reranker? URL: https://www.topk.io/learn/reranking Category: Hybrid Search & Ranking Summary: Add a reranker when the right documents reach your top 100 but not your top 5. With a multi-vector first stage, measure first: it may be redundant. **Add a reranker when your first-stage retrieval reliably gets the right documents into the top 50–100 but not the top 5, and you can afford a model pass per candidate at query time.** **Skip it, or at least measure first, when your first stage already scores at token level (multi-vector / late interaction): a reranker exploits largely the same signal and may add latency for little gain.** ## What does a reranker actually add? A cross-encoder reads the query and a candidate document *together* and judges relevance jointly ([Nogueira & Cho, 2019](https://arxiv.org/abs/1901.04085)), catching interactions that independently computed embeddings miss. First-stage retrieval optimizes recall: get the relevant documents somewhere into the candidate set. The reranker optimizes precision: get the best ones to the very top. It runs second because a model pass over the whole corpus would be far too slow. ## What does it cost? Reranking costs one model forward pass per candidate, at query time, in series with retrieval. Reranking 200 candidates means 200 model calls of latency budget: acceptable for some search products, disqualifying for agent loops that fire dozens of queries in parallel. ## When does multi-vector make it redundant? The [ColBERT paper](https://arxiv.org/abs/2004.12832) introduced [late interaction](/learn/multi-vector-retrieval) precisely as an efficient alternative to cross-encoder reranking: [MaxSim](/learn/maxsim) does token-level matching in the retrieval stage itself, from precomputed embeddings. If your first stage is multi-vector, much of what a reranker would recover is already in the ranking. The honest caveat: cross-encoders still hold the accuracy ceiling. If top-5 precision is everything and latency is nothing, rerank anyway, but the gap over a multi-vector first stage is far smaller than over a single-vector one, so benchmark before paying for both. ## The decision in one line If you run a single-vector first stage and hear quality complaints, add a reranker. If you run a multi-vector first stage, measure the reranker's lift before you ship it. TopK's multi-vector retrieval and custom ranking run in a single query, which often removes the separate reranking hop entirely. It is worth testing against your current two-stage stack. ### Does Quantization Hurt Accuracy? URL: https://www.topk.io/learn/vector-quantization-accuracy Category: Scale & Architecture Summary: Yes, but recoverably: quantize for candidate selection, then rescore the shortlist at full precision. The loss lands in the tail, not the top results. **Yes, quantization loses information, but the loss is recoverable if you spend it where rankings aren't decided. The production pattern is two-stage: use aggressively quantized vectors (int8, [product quantization](https://inria.hal.science/inria-00514462v2/document), even 1-bit binary) to select candidates fast and cheap, then rescore that shortlist with full-precision vectors.** **Quantization's ranking mistakes concentrate in the long tail of near-ties, not in the clear top results, so rescoring recovers nearly all the quality while the memory and speed savings stay.** ## Where does the accuracy actually go? Quantization compresses the space between vectors, so documents with genuinely different similarities start to tie. For the clear winners, the passages obviously close to the query, the ordering survives even harsh compression. For the marginal candidates ranked 50–200, ties break randomly. Naively returning quantized scores as final rankings surfaces that tail noise; using them only to build a candidate set hides it. ## Why does rescoring fix it? Rescoring works because the candidate set only needs to *contain* the right documents, not order them. If the true top-10 reliably lands somewhere in the quantized top-100, exact rescoring of 100 vectors (trivial compute) reconstructs the true order. You keep quantized speed on the expensive full-corpus scan and pay full precision only on a shortlist. ## How far can you push it? You can push surprisingly far: binary quantization stores one bit per dimension (a 32× reduction over float32), and Hamming distance over binary codes is fast enough to scan huge candidate sets, with rescoring cleaning up the ranking. TopK's production kernel makes the point concrete: hand-optimized ARM NEON code sustains [nearly 350 GB/s of Hamming-distance throughput](/blog/binary-vector-search-arm-neon) (measured July 2025, ~15× the scalar baseline on a single thread), which is what makes brute-force scanning of binary candidates viable at billion scale. ## When should you be careful? Measure recall on *your* data before trusting any compression level: out-of-distribution embeddings and very anisotropic spaces quantize worse than benchmark datasets suggest, and the fix (a larger candidate set) costs some of the speed back. The knob is candidate-set size; tune it against a fixed [recall target](/learn/benchmarking-vector-search), not by feel. ### How Do You Audit an AI Answer? URL: https://www.topk.io/learn/auditable-ai-answers Category: Agents & RAG Summary: An answer is auditable when every claim cites a document and page a human can open. That requires citations from retrieval, not citations written by the model. **An AI answer is auditable when every claim carries a reference to a specific document and page that a human can open and check. That requires citations produced by the retrieval layer, which records exactly which passages the answer was built from. Citations written by the model are text like any other text: fluent, plausible, and easy to fabricate.** **The measurable form of this property is citation precision: of the references an answer gives, how many actually support the claim they are attached to.** ## Why can't you trust citations the model writes? A language model generates plausible text, and a reference is just more text. Asked to cite, a model produces something citation-shaped whether or not a source exists, and the fabrications look identical to the real thing. The failure is documented at scale: [Dahl et al. (2024)](https://arxiv.org/abs/2401.01301) document pervasive fabricated case citations in LLM legal outputs, and a fabricated reference in a filing or a client memo is a compliance incident, not a formatting bug. Your compliance officer asks where a number in the answer came from. "The model said so" is not something you can file. ## What does a retrieval-produced citation look like? The retrieval layer knows something the model never can: which passages were actually fetched and placed in context. An auditable pipeline keeps that record and attaches it to the output, so each fact in the answer points at a retrieved passage with its document ID, file name, and page numbers. The citation is a pointer into evidence the system verifiably used, not a claim the model composed. In TopK's [ask API](https://docs.topk.io/datasets/ask), that structure is the response format. Answers come back as facts, each carrying reference IDs that map to the matched passages with document name and page-level locations, plus a 0–100 confidence score: ```python for message in client.ask("What was the total net income of Bank of America in 2024?", ["my-docs"]): print(message) ``` A reviewer takes the reference, opens the named file at the named page, and checks the claim. That round trip is what "auditable" means in practice. ## How do you measure auditability? Citation precision: the fraction of an answer's references that genuinely support the claims they back. On [BrowseComp-Plus](/blog/20260611-semantic-index-multi-vector-retrieval) (June 2026), an agent running on TopK's retrieval stack reached 88.54% citation precision alongside 80.48% task accuracy, averaging about 14 search calls per task. High citation precision indicates the retriever surfaces the right documents early enough for the model to ground every claim instead of improvising. ## What should a finance team demand? Ask any vendor three questions. Does every claim carry a reference a human can open, at page granularity? Are the references produced by retrieval rather than generated by the model? Is citation precision measured and published, with a date? A vendor that cannot answer all three is asking to be trusted, and audits do not run on trust. Apply the same scrutiny to [the recall numbers on the box](/learn/recall-vs-retrieval-quality). Retrieval quality is what makes the citations good in the first place: an answer grounded in the [wrong table row](/learn/rag-financial-tables) cites its wrong source faithfully. Auditability and retrieval accuracy are the same investment, which is why TopK publishes both numbers together. ### How Do You Benchmark Vector Search? URL: https://www.topk.io/learn/benchmarking-vector-search Category: Scale & Architecture Summary: Benchmark your own workload: mixed read/write load, your filters, and latency percentiles at a fixed recall target. Read-only QPS flatters every engine. **Benchmark the workload you'll actually serve, not the one that flatters the engine: concurrent reads and writes in your real ratio (80/20 is a common spike shape), your embedding dimensionality, your metadata filters, reported as p50/p95/p99 latency at a stated recall target.** **Read-only QPS on a freshly built index is the number everyone quotes and nobody serves in production.** ## Why do read-only benchmarks mislead? Writes degrade reads. Index mutation contends with traversal, tombstones accumulate, background compaction steals CPU at the worst moment ([why upserts slow HNSW reads](/learn/hnsw-upsert-performance)). An engine can post excellent read-only numbers and lose a large fraction of that throughput under a modest write load. You only find out by measuring the mix. ## What must be held fixed? **Recall.** Approximate search lets any engine trade accuracy for speed silently, so QPS without a recall target is meaningless. That is why [ANN-Benchmarks](https://arxiv.org/abs/1807.05614), the standard methodology in the field, plots speed *against* recall rather than reporting either alone. Tune everything to the same recall (e.g. 95% against exact search on your data), then compare latency. Report percentiles, not averages: retrieval sits in a request path, and p99 is what your users feel. ## How should the run be structured? Load real data at real dimensionality, include your filters at realistic selectivity, warm the system, then measure three phases: steady state, the spike (reads and writes ramped together), and the recovery window after. The last phase catches deferred costs like compaction. Long enough runs matter; mutation damage compounds over minutes, not seconds. TopK's published methodology ([topk-bench](/blog/20251201-topk-bench), December 2025) is a concrete template: five benchmark dimensions (ingest, concurrency scaling, filtering, recall, and read-write interference), each configuration run five times with the worst run dropped and the mean of the remaining four reported, warmup at twice the measurement timeout, 30-second query windows, and recall@10 over 1,000 queries against precomputed ground truth. Its datasets carry synthetic filter fields tuned to match exactly 100%, 10%, and 1% of documents, so selectivity is a controlled variable instead of an accident of the data. One dimension almost every benchmark skips is **freshness**, the delay between a write being acknowledged and the document appearing in query results. If your application writes continuously, an engine that acknowledges fast but surfaces slowly is lying to you about its write path; measure the gap. ## What makes results trustworthy? Trust requires reproducibility: pinned datasets, a published harness, versioned configs, and a date. A vendor number you can't rerun is marketing. TopK's results are published this way: the [benchmarks page](https://www.topk.io/benchmarks) covers 1M–1B vectors, filter selectivities, and concurrent load, reproducible via the open-source [topk-bench harness](https://github.com/topk-io/bench) with its datasets hosted publicly on S3. Rerunning it against your own deployment takes a few lines: ```python tb.ingest( provider=tb.TopKProvider(), config=tb.IngestConfig(input="s3://topk-bench/docs-1m.parquet"), ) tb.query( provider=tb.TopKProvider(), config=tb.QueryConfig(queries="s3://topk-bench/queries-1m.parquet", concurrency=4), ) tb.write_metrics("bench-1m.parquet") ``` The [methodology post](/blog/20251201-topk-bench) also makes an honesty choice worth copying: only TopK's absolute numbers are public, and other providers stay anonymized, keeping the focus on behavior rather than scoreboard marketing. ### How Do You Chunk Redundant Documents? URL: https://www.topk.io/learn/chunking-redundant-documents Category: Scale & Architecture Summary: Deduplicate before you embed: canonicalize versions, hash-dedupe chunks, and reach history through filters. Indexing every version multiplies cost. **Collapse the redundancy before it reaches the embedding model. Version-controlled corpora are mostly near-duplicates of themselves; chunking and embedding every version multiplies storage and compute for zero recall gain, and pollutes results with five variants of the same passage.** **The pattern: canonicalize to the current version of each document, deduplicate chunks by content hash before embedding, attach version and date as metadata, and serve history through filters only when explicitly requested.** ## Why does redundancy hurt more than it costs? The bill is the visible half. The quality damage is worse: your top-*k* has finite slots, and if a passage exists in ten near-identical revisions, retrieval happily fills the result list with ten copies of one idea, crowding out the second and third relevant passages a RAG prompt or agent actually needed. Deduplication is a relevance fix that happens to also be a cost fix. ## What does a working pipeline look like? 1. **Canonicalize:** resolve each document to its current version; strip boilerplate (headers, footers, legal blocks) that repeats across the corpus. 2. **Chunk, then hash:** normalize whitespace and hash each chunk's content. A chunk whose hash you've already indexed gets a reference, not a new embedding. 3. **Embed once per unique chunk,** and let metadata carry which documents and versions contain it. 4. **Default queries to `latest`,** exposing older versions through an explicit version filter. ## What if you must keep every version? Compliance corpora often require it. Keep them, but keep them as metadata-filtered history rather than as default-searchable content. The index answers day-to-day queries from the deduplicated current set; an audit query opts into the archive with a filter. Storage-backed engines make the archived portion cheap to retain. TopK's metadata filtering makes the "latest by default, history on demand" split a query-time concern, and usage-based pricing means every chunk you dedupe away is ingest and storage you visibly don't pay for. ### How Do You Enforce RBAC in Vector Search? URL: https://www.topk.io/learn/rbac-vector-search Category: Filtering & Multi-Tenancy Summary: Store access metadata on every document and enforce entitlements as a query-time filter in the engine. Post-filtering leaks; per-role indexes explode. **The pattern is filter-at-query: store access metadata on every document (tenant, groups, explicit ACL entries) and have the application attach the user's resolved entitlements to every query as a mandatory filter that the engine evaluates *during* retrieval.** **The two alternatives both fail. Post-filtering is a security and recall bug: restricted documents participate in scoring before being dropped, and selective permissions empty the result set. Per-user or per-role indexes explode combinatorially and go stale the moment permissions change.** **Query-time filtering keeps one index, honors permission changes instantly, and puts enforcement in one place.** ## Why is post-filtering a security problem, not just a recall problem? "Remove it from the results afterward" still lets restricted content shape the query: it occupies candidate slots, and its removal is observable. A user who can see 2% of the corpus gets near-empty answers ([why filtering breaks graph indexes](/learn/filtered-vector-search)) while the system did scoring work over documents they were never entitled to touch. Access control belongs in retrieval, not in cleanup. ## Why not an index per role? Real RBAC means groups, nested groups, per-document shares, and revocations, far beyond a handful of static roles. Materializing indexes per access pattern means combinatorial storage, rebuild-on-permission-change, and a window where revoked access still serves. It only looks viable at whiteboard scale. ## What does the working pattern look like? 1. **On ingest:** write ACL metadata on each document: owner, allowed groups/roles, tenant. 2. **On query:** the application resolves the user's entitlements (expand groups from your identity system) and attaches them as a filter: `tenant = X AND groups ∩ user_groups ≠ ∅`. 3. **In the engine:** the filter executes as part of retrieval, so scoring only ever sees authorized documents. Permission changes are metadata updates, not index rebuilds. Deny-rules and audit stay in the application layer, where identity lives. ## What does this demand from the engine? Everything rests on filtered search staying fast and correct at high selectivity. A user who can access 1% of the corpus is a 1%-selectivity filter on every single query, which is [exactly where graph-index engines fall over](/learn/filtered-vector-search). TopK's query engine treats filter predicates as first-class query operations (vectorized filtering inside its [reactor engine](https://docs.topk.io/architecture), designed so [selective queries get faster, not slower](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch)) and publishes [filtered-search performance across selectivities](https://www.topk.io/benchmarks), which is precisely the capability RBAC-at-query-time depends on. ### How Do You Index Text for Regex Search? URL: https://www.topk.io/learn/regex-index-ngrams Category: Scale & Architecture Summary: Split text into n-grams, keep a posting list per gram, and compile the regex into a boolean expression over the grams it needs. Sparse grams keep it selective. **You index text for regex search by splitting every document into n-grams, storing a posting list for each gram, and compiling each regex into a boolean expression over the grams that any match must contain. Intersecting the posting lists gives a candidate set, and the exact regex verifies only the candidates.** **Plain trigrams work but filter weakly, because short grams are common. TopK's sparse grams keep every trigram for correctness and add longer, rarer grams selectively, so the index stays small while it gains the selectivity of long grams.** ## How does a trigram index work? Split each document into every overlapping run of three characters and store a posting list for each trigram. The string `alan turing` produces nine grams. ```text ["ala", "lan", "an ", "n t", " tu", "tur", "uri", "rin", "ing"] ``` Any document that contains `alan turing` must contain all nine grams, so intersecting their posting lists gives a candidate set. Russ Cox described the same approach for [Google Code Search](https://swtch.com/~rsc/regexp/regexp4.html), and it has the property that keeps the whole scheme safe, which is that it cannot produce false negatives. ## Why are trigrams a compromise? Short grams are common. A trigram such as `the` or `ing` occurs in a large fraction of any corpus, so its posting list is expensive to read and removes almost nothing. Trigrams also lose adjacency past three characters, so a document can contain every trigram from a query in unrelated places and still become a candidate. Longer fixed grams are more selective, but they create a different problem. An index over one width cannot help with required fragments shorter than that width, and indexing every width repeats the same text at each width, so the index grows with the maximum width. Sparse grams resolve the tension between the two. ## What do sparse grams change? Sparse-gram tokenization assigns a deterministic weight to every bigram, and TopK uses CRC32 for the weight. It then selects an n-gram only when its two boundary bigrams outweigh every bigram inside it. The rule depends only on the contents of the span, so if a query string occurs inside a document, every gram selected from the query is also selected from the same interval in the document. Text before or after the match cannot change the decision. The selection probability for a span of n characters is `2 / ((n-1)(n-2))`, so every trigram is selected and longer grams appear less and less often. The expected gram count telescopes to about `2L` for a document of length `L`, compared with order `LN` for a dense index over every width up to `N`. The difference is what lets the index admit long, selective terms without paying for every long interval. The derivation is in [the fast regex search post](/blog/20260814-fast-regex-search-sparse-grams). ## How does a regex become a boolean expression? The planner parses the regex into the same representation the exact matcher uses. It then tracks what it can prove about every possible match, which includes required literal runs, required prefixes and suffixes, and whether a subexpression can match the empty string. Concatenation combines constraints with `AND`, alternation combines them with `OR`, and a wildcard contributes nothing. For `colou?r of the (sky|sea)`, a conservative plan looks like the following. ```text " the" AND "r of t" AND ("e se" OR "e sk") ``` The plan states only facts that every match must satisfy. A document that fails the plan cannot match and gets skipped, and a document that passes is a candidate. Because the guarantee only goes one way, the planner can drop a weak gram or cap a large alternation whenever that is cheaper. Each of those choices admits more candidates and never loses a true match. Patterns made mostly of wildcards fall back to a scan, and the fallback is safe for the same reason. ## What decides which grams the planner uses? Information decides it, not length. The usefulness of a gram is `I(g) = -log2 P(g)`, so rare grams filter well and common grams do not, and two grams of the same length can have very different posting lists. A familiar phrase stays common even when it is long, and an unusual name becomes selective after a few characters. The planner therefore prefers the terms with the lowest measured document frequency, whatever their length. Raising the maximum gram length adds only about `2L / (N(N-1))` terms per step under the sparse model, so it stays cheap and still helps. TopK ships the index behind its ordinary regex filter, so `field("message").regexp_match(pattern)` plans through sparse grams automatically. The design and its measurements are in the [fast regex search post](/blog/20260814-fast-regex-search-sparse-grams) (August 2026), and the [interactive gram explorer](/blog/20260814-fast-regex-search-sparse-grams#content-defined-n-grams) in the post shows which grams a given string produces. ### How Do You Run kNN Search Through the Elasticsearch API? URL: https://www.topk.io/learn/knn-search-elasticsearch-api Category: Scale & Architecture Summary: Map a dense_vector field with dims and similarity, send a knn clause with field, query_vector, and k, and put the filter inside knn so it runs before ranking. **You map a field as `dense_vector` with `dims` and a `similarity`, index documents with a vector in that field, and send a `knn` clause with `field`, `query_vector`, and `k`. The response is an ordinary hits list, and the score is Elasticsearch's normalized similarity, so a cosine hit comes back as `(1 + cosine) / 2`.** **Put any filter inside the `knn` clause rather than in `post_filter`, so it narrows the candidates before ranking and still returns `k` matches. TopK serves the same request shape through its Elasticsearch-compatible API, and its engine gets faster as the filter gets tighter.** ## How do you map a vector field? A vector field is declared in the index mapping with a type of `dense_vector`, the number of dimensions, and the similarity the engine should rank by. Elasticsearch offers `cosine`, `dot_product`, `l2_norm`, and `max_inner_product`, and it defaults to `cosine` for float vectors ([Elastic field reference](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector)). TopK's Elasticsearch API accepts the first three. The `dot_product` option requires every vector, including the query vector, to be unit length, so use `cosine` unless you normalize your embeddings yourself. ```json PUT /books { "mappings": { "properties": { "genre": { "type": "keyword" }, "embedding": { "type": "dense_vector", "dims": 768, "similarity": "cosine" } } } } ``` Elasticsearch lets you omit `dims` and infers it from the first vector you index. TopK requires `dims` in the mapping and returns a 400 at index creation without it, as it does for a `bit` element type whose `dims` is not a multiple of 8, so a mistake in the mapping never reaches the data. ## What does the knn clause look like? The `knn` clause names the field, carries the query vector, and asks for `k` nearest neighbors. It sits at the top level of the search body, next to `size` and `sort`, and the response is the same hits structure every other query returns. ```json POST /books/_search { "knn": { "field": "embedding", "query_vector": [0.12, -0.03, ...], "k": 10 }, "size": 10 } ``` The vector has to match the mapped dimensions, and it has to contain finite numbers. A `k` of zero, a `num_candidates` smaller than `k`, and a `knn` against a field that is not a vector all return a 400 rather than an empty result. ## What do the scores mean? The `_score` on a kNN hit is a normalized similarity, and the normalization depends on the metric. For `cosine` and for float `dot_product`, Elasticsearch reports `(1 + similarity) / 2`, so an identical vector scores 1.0 and an orthogonal vector scores 0.5 ([Elastic field reference](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector)). For `l2_norm`, the score falls from 1.0 as the distance grows and stays positive. Read the score as a ranking value, and if you threshold on it, threshold on the normalized number rather than on a raw cosine. TopK's Elasticsearch API returns the same numbers. An identical cosine vector scores 1.0, an orthogonal one scores 0.5, and `max_score` equals the top hit, so an application that reads `_score` gets what it read before. ## Does the filter run before or after ranking? A filter inside the `knn` clause runs before ranking. Elasticsearch applies it during the approximate search so that `k` matching documents come back, while a `post_filter` applies after the kNN step and can return fewer than `k` ([Elastic kNN guide](https://www.elastic.co/docs/solutions/search/vector/knn)). If you filter by tenant, category, or permission, the filter belongs inside `knn`, or a small tenant sees a short or empty result list even when it has plenty of matching documents. ```json { "knn": { "field": "embedding", "query_vector": [0.12, -0.03, ...], "k": 10, "filter": { "term": { "genre": "fantasy" } } } } ``` The inner filter is necessary for the same reason that [filtering breaks HNSW](/learn/filtered-vector-search). A graph index organizes vectors by neighborhood, and the documents that satisfy a selective filter are scattered across it, so an engine that filters after traversal runs out of candidates. TopK's engine treats the filter as part of the query plan rather than as a mask over the index, and its stated design goal is that [highly selective queries get faster](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch), because the filter shrinks the set the engine scores. Through the Elasticsearch-compatible API, the `knn.filter` restricts candidates before ranking and `hits.total` reports the number that matched. ## What does this look like on TopK? The request is the one above, sent by the Elasticsearch client you already use, with the endpoint and the API key pointed at TopK. ```python from elasticsearch import Elasticsearch es = Elasticsearch("https://", api_key="") res = es.search(index="books", knn={ "field": "embedding", "query_vector": query_embedding, "k": 10, "filter": {"term": {"genre": "fantasy"}}, }, size=10) for hit in res["hits"]["hits"]: print(hit["_id"], hit["_score"]) ``` On TopK, a `dense_vector` field is served by a vector index with the metric you named, and Elastic's `rank_vectors` type is served by a multi-vector index that a matrix `query_vector` [searches by MaxSim](/learn/multi-vector-elasticsearch-api). The [compatibility overview](/learn/elasticsearch-api-compatibility) covers what TopK accepts and rejects, and the [hybrid guide](/learn/hybrid-search-elasticsearch-api) covers combining `knn` with a keyword or semantic query. ### How Do You Upgrade Embedding Models? URL: https://www.topk.io/learn/upgrading-embedding-models Category: Embeddings in Production Summary: Blue-green the index: backfill with the new model, dual-write, shadow-test, flip reads, keep the old index for rollback. Model vectors never mix. **The safe pattern is blue-green at the index level: build a second index with the new model, backfill it from source text, dual-write both indexes during the transition, compare quality in shadow, flip reads to the new index, and keep the old one warm until you'd no longer roll back.** **There is no in-place upgrade, because vectors from different models occupy [incompatible spaces](/learn/converting-embeddings), so a half-migrated index is a broken index. The cutover is the easy part. The real work is the re-embedding pipeline: you own it end to end, and query-time and ingest-time embeddings must stay in lockstep.** ## Why can't you upgrade in place? A query embedded with the new model scores garbage against documents embedded with the old one. Any state where one index holds both is serving wrong results. That single fact forces the parallel-index shape of every safe migration. ## What's the sequence? 1. **Backfill:** re-embed the corpus with the new model into a new index (dedupe first: [don't pay twice for redundancy](/learn/chunking-redundant-documents)). 2. **Dual-write:** new and updated documents flow to both indexes so neither goes stale. 3. **Shadow-test:** run real traffic against both; compare recall and answer quality offline before any user sees a change. 4. **Flip reads, then watch:** cut over behind a flag; the old index is your instant rollback. ## Where does it actually go wrong? Embedder skew causes the classic production incident: one version of the model (or one preprocessing path) embeds the queries while another embedded the documents. Nothing errors; results are just quietly worse. Any migration plan is mostly a plan for keeping the ingest-side and query-side embedder identical at every moment. ## What changes when the database owns embedding? Most of the pipeline disappears. When the engine embeds at both ingest and query time (managed inference), skew becomes structurally impossible (one embedder, one place), backfill becomes a database-side operation instead of an external batch job you build, and the migration collapses to: create index with model B, backfill, flip. TopK's [`semantic_index`](/blog/20260611-semantic-index-multi-vector-retrieval) is the concrete version: one schema annotation, no external embedding pipeline, with embedding inference running inside the system at [over 1.5B tokens per hour](/blog/20260611-semantic-index-multi-vector-retrieval), which makes a model upgrade a schema change rather than an infrastructure project. ### How Should You Weight Hybrid Scores? URL: https://www.topk.io/learn/hybrid-score-weighting Category: Hybrid Search & Ranking Summary: There is no single right weight: short queries lean lexical, verbose ones lean dense. Condition weights on the query, or use RRF and skip them. **There is no single right weight, because the right balance is a property of the query, not the corpus. A two-word, entity-heavy query ("A17-B manual") is a lexical query, where exact-token evidence should dominate. A twenty-word natural-language question is a semantic query, where dense similarity should dominate.** **You have three options, in order of effort: use rank-based fusion (RRF) and skip weights entirely; set weights per query class from cheap features like token count and term rarity; or learn the weighting function from labeled data.** **If you can't measure quality yet, use RRF. If you can, condition the weights on the query.** ## Why does one static weight fail? A static weight is one answer to two different questions. Tune α on your verbose queries and short identifier lookups drown in semantic noise; tune it on the lookups and paraphrased questions go blind. When query shape varies wildly, any fixed blend is wrong for a large fraction of traffic. No single value can fix that, because a constant is the wrong type for a query-dependent quantity. ## What per-query signals actually work? Cheap, computable-at-query-time features carry most of the signal: **length** (short queries lean lexical, long ones lean dense), **rare-token presence** (IDs, codes, camelCase, and digits push lexical, since rare tokens are what embeddings [blur away](/learn/hybrid-search)), and **quote or operator syntax** (explicit exact-match intent). A handful of if-then rules over these features (three query classes, three weight profiles) captures most of the win before any learning is involved. ## When should you learn it instead? When you have labeled relevance data and enough traffic diversity that hand rules visibly leave quality behind. [Bruch et al. (2022)](https://arxiv.org/abs/2210.11934) show that tuned convex score combinations generally beat untuned rank fusion, which is the case for learning weights once you can measure. Then per-class constants become a small model predicting weights from query features. Don't start here: learned fusion adds training, serving, and drift-monitoring costs that only pay off after the simple version is measurably insufficient. ## Where does the engine matter? Per-query weighting is only cheap if the scores meet in one place. In a two-engine setup, changing the blend means re-normalizing two incompatible score distributions in application code ([RRF vs score fusion vs true hybrid](/learn/rrf-vs-true-hybrid)). In a true-hybrid engine the blend is a ranking expression evaluated inside the query. In TopK it's literally `0.7 * dense_score + 0.3 * sparse_score` [in the query itself](https://docs.topk.io/guides/true-hybrid-search), so query-conditional weighting is a per-request parameter change, not a pipeline rebuild. TopK's [BEIR case study](/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality) (July 2025) measured this score-aware, tunable approach at 4.58% higher nDCG@10 on average than RRF's fixed rank fusion. ### Is Multi-Vector Retrieval Too Expensive? URL: https://www.topk.io/learn/multi-vector-at-scale Category: Multi-Vector Retrieval Summary: It was: token-level embeddings cost 10–100× more storage. Two-stage designs and MaxSim approximations like SMVE now make it a viable first stage. **It was. For years the cost was disqualifying, which is why late interaction stayed a reranking trick. One embedding per token means roughly 10–100× more storage than one embedding per document, and MaxSim scoring does orders of magnitude more work than a single dot product.** **That has changed: modern engines approximate MaxSim cheaply to select candidates, then score only the shortlist exactly. If you ruled multi-vector out on cost a year ago, the math has moved.** ## Where does the cost come from? **Storage.** A 500-token document stores 500 embeddings instead of one. Quantization and token pooling shrink this substantially, but token-level indexes remain larger than single-vector ones. **Compute.** [MaxSim](/learn/maxsim) evaluates a query-token × document-token similarity matrix per candidate. Run naively across a large corpus, that is thousands of times the scoring work of single-vector search. ## How do engines make it affordable? The answer is two stages: a cheap approximation narrows millions of documents to a small candidate set, then exact MaxSim ranks only the survivors. [PLAID](https://arxiv.org/abs/2205.09707) prunes candidates using quantized centroids. [MUVERA](https://arxiv.org/abs/2405.19504) builds fixed-dimensional encodings whose inner product approximates MaxSim, and it comes with theoretical guarantees, but the descriptors must be very large and dense to earn them. TopK's [SMVE](/blog/20260311-smve-multi-vector-retrieval) (Sparse Multi-Vector Encoding) starts from the observation that expressive descriptors may need to be large, but they don't need to be *dense*. It projects each token embedding onto a large set of random spherical anchors, keeps only the top-k strongest projections per token, and pools the result into one sparse vector whose dot product approximates MaxSim. Cost therefore scales with the non-zero count rather than the dimensionality, and candidate selection runs on ordinary sparse-retrieval infrastructure. Exact MaxSim then reranks only the shortlist. The measured result (March 2026, ColBERTv2 on BEIR): [roughly 5–8× lower end-to-end latency than PLAID and MUVERA](/blog/20260311-smve-multi-vector-retrieval) at competitive recall. On MS MARCO's 8.8M documents, SMVE averages 39.9ms for k=100, versus 221–318ms for PLAID and 310–444ms for MUVERA. ## What does this change in practice? Multi-vector stops being a reranker bolted onto some other retriever and becomes the first stage itself: token-level precision over the whole corpus at a cost curve close to conventional search. Affordability is settled. The remaining question is [whether your queries need it](/learn/when-to-use-multi-vector-embeddings). TopK runs this stack in production as [`semantic_index`](/blog/20260611-semantic-index-multi-vector-retrieval), one schema annotation that embeds, indexes, and serves late interaction end to end. Its measured system numbers (June 2026): 52.88% nDCG@10 across all 15 BEIR datasets (within about 1% of exact exhaustive MaxSim) at 295 QPS with ~75ms p99 latency, while ingesting over 1.5B tokens per hour with sub-second index lag. ### RRF vs Score Fusion vs True Hybrid: Which Should You Use? URL: https://www.topk.io/learn/rrf-vs-true-hybrid Category: Hybrid Search & Ranking Summary: RRF merges lists by position, score fusion tunes weighted scores, and true hybrid scores every signal in one query. Start with RRF; move up as you measure. **Start with RRF: it needs no tuning and is hard to beat out of the box. Move to score fusion once you have evaluation data to tune weights against. Reach for true hybrid when post-hoc merging is itself the limitation.** **The three differ in *when* the signals meet: Reciprocal Rank Fusion (RRF) merges two finished ranked lists using only positions; score fusion normalizes each retriever's raw scores and combines them with weights; true hybrid never produces two separate lists, because it scores every signal inside a single query and ranks once.** ## The quick comparison | | How signals combine | Tuning | Best when | |---|---|---|---| | **RRF** | Sum of `1/(k+rank)` across lists (positions only) | None | You have no eval data; you want a robust default | | **Score fusion** | Normalize scores, then weighted sum | Weights + normalization per workload | You can measure quality and tune weights | | **True hybrid** | One query scores all signals, ranks once | Ranking expression | You want full control and no two-stage merge | ## How does RRF work? RRF combines lists using each document's rank, ignoring raw scores. Every document scores `Σ 1/(k + rankᵢ)` summed over the lists it appears in, with `k` conventionally 60 (from [Cormack, Clarke & Büttcher, 2009](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf)). Because it only reads positions, it fuses lists whose scores live on incompatible scales (BM25's unbounded scores and cosine similarity's bounded ones) without any normalization. Its strength is robustness with zero tuning, which makes it the standard first choice. Its weakness is that discarding scores discards information: RRF can't tell a document that barely made a list from one that dominated it, since both are judged only by rank. ## How does score fusion work? Score fusion keeps the raw scores, normalizes them onto a common scale, and combines them with weights, typically `α · vector + (1−α) · lexical`. Because it preserves each retriever's confidence, a well-tuned weight can beat RRF. [Bruch et al. (2022)](https://arxiv.org/abs/2210.11934) analyze fusion functions and find tuned convex combinations generally outperform rank-based fusion. You can lean on vector similarity for paraphrase-heavy queries and on lexical scores for exact-term queries. The cost is tuning and fragility. Different score distributions need normalization (min-max, z-score, or similar), the weight `α` must be tuned per workload against evaluation data, and both can drift as your data changes. Score fusion rewards teams that can measure quality; it punishes teams that can't. ## What is "true hybrid"? True hybrid computes a single combined score in one retrieval pass instead of merging two independently-retrieved lists. Rather than running lexical and vector search separately and stitching the results, the engine evaluates all signals (keyword, dense, sparse, [multi-vector](/learn/multi-vector-retrieval), metadata) inside one query and produces one ranking. This removes a structural limitation of both fusion methods: post-hoc merges can only reorder documents that each retriever already returned in its top-*k*. A document strong on the combined signal but outside both individual top-*k* lists never gets the chance to rank. As TopK's case study [puts it](/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality), rank fusion "can suppress relevant documents ranked moderately by both systems but overlooked by either individually." Scoring once, over the full candidate set, avoids that blind spot and gives you direct control over the ranking expression. The gap is measurable. TopK benchmarked the three approaches on five BEIR datasets (July 2025, ModernBERT dense + SPLADE-v3 sparse): [true hybrid improved nDCG@10 by 4.58% on average over RRF](/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality), up to 7.84% on TREC-COVID. The standard mitigation, overfetching 100 candidates per retriever before fusing, closed some of the gap but still lost by 3.10% on average, while doing more work per query. ## Which should you use? Start with **RRF**: it needs no tuning and is hard to beat out of the box, especially before you have labeled data. Move to **score fusion** once you can measure quality and want to exploit the confidence signal in raw scores. Reach for **true hybrid** when post-hoc merging is the limitation itself: when you need signals beyond two lists, want a document's combined score to decide its rank directly, or want to express custom ranking logic rather than tune a fusion weight. TopK is built around the true-hybrid model: keyword, vector, and multi-vector retrieval with a custom ranking expression evaluated in a single query, so you don't assemble two retrievers and a fusion layer yourself. The fusion weights are literally one line: ```python from topk_sdk.query import select, field, fn from topk_sdk.data import f32_vector, f32_sparse_vector docs = client.collection("docs").query( select( dense_score=fn.vector_distance("dense", f32_vector([...])), sparse_score=fn.vector_distance("sparse", f32_sparse_vector({...})), ) .sort(0.7 * field("dense_score") + 0.3 * field("sparse_score"), asc=False) .limit(10) ) ``` The [true hybrid search guide](https://docs.topk.io/guides/true-hybrid-search) covers score normalization and the full expression syntax. ### Top-k (Retrieval) URL: https://www.topk.io/learn/top-k Category: Concepts Summary: Top-k means returning the k highest-scoring results from a ranked search or model. It is a ranking cutoff, distinct from TopK, the search engine. **Top-k (lowercase, hyphenated) is the operation of keeping the *k* highest-scoring items from a ranked list. In retrieval it means "return the k most relevant documents"; in language models it means "consider only the k most probable next tokens." The *k* is simply the cutoff: top-10 retrieval returns ten results, top-5 returns five.** > **Not to be confused with [TopK](https://www.topk.io):** TopK (capital T, capital K) is a search engine for accuracy-critical AI applications: hybrid search, multi-vector retrieval, custom ranking, and managed inference in one API. This page is about *top-k*, the general ranking concept. ## What does top-k mean in search? In search and retrieval, top-k is the number of results you ask for. A vector or [hybrid](/learn/hybrid-search) query scores every candidate document, sorts by score, and returns the top *k*. Choosing *k* is a recall-versus-noise tradeoff: a larger *k* is more likely to include the right document but feeds more irrelevant material to whatever consumes the results. In TopK's [query API](https://docs.topk.io/collections/query), the top-k cutoff is the sort-and-limit at the end of every query: ```python .sort(field("score"), asc=False).limit(10) # top-10 by score ``` In two-stage pipelines the value of *k* often differs per stage: a first stage might retrieve top-200 candidates, and a [reranker](/learn/reranking) narrows them to top-5 for the final answer. ## What does top-k mean in language models? In text generation, top-k sampling restricts the model to the *k* most probable next tokens, then samples from just those. It caps how adventurous the model can be: top-k of 1 is greedy decoding (always the single most likely token), while a larger *k* allows more diversity. It is a sibling of top-p (nucleus) sampling, which selects the smallest set of tokens whose probabilities sum to *p* instead of a fixed count. ## Why does the top-k / TopK distinction matter? Search terminology and a product name collide here. "Top-k retrieval" is a decades-old concept in information retrieval, so context and capitalization are the only way to tell whether a page means the ranking cutoff or the search engine. When precision matters, write "top-k retrieval" for the concept and "TopK" for the product, and don't rely on capitalization alone. ### What Is MaxSim? URL: https://www.topk.io/learn/maxsim Category: Multi-Vector Retrieval Summary: MaxSim sums each query token's best match against a document's tokens. It is the scoring operator behind multi-vector (late-interaction) retrieval. **MaxSim (maximum similarity) is the operator that scores a document in late-interaction retrieval. For each query token embedding, it finds that token's highest similarity against every document token, then sums those per-token maxima into a single score.** **It is the mechanism that lets [multi-vector retrieval](/learn/multi-vector-retrieval) match parts of a query to parts of a document instead of comparing two whole-document summaries.** ## How is MaxSim computed? For a query with token embeddings `q₁…qₙ` and a document with token embeddings `d₁…dₘ`: ``` MaxSim(Q, D) = Σᵢ maxⱼ sim(qᵢ, dⱼ) ``` Read it inside-out: for each query token `qᵢ`, compute its similarity (usually cosine or dot product) against **all** document tokens and keep only the maximum, which is that token's best match in the document. Then **sum** those maxima across every query token. Each query term contributes the evidence of its single strongest match, so a document is rewarded when it contains a good match for *each* part of the query. Khattab & Zaharia introduced MaxSim in [ColBERT](https://arxiv.org/abs/2004.12832) (2020) as the core of "late interaction." ## Why "late" interaction? The name marks the timing: query and document interact at scoring time, not encoding time. A standard single-vector model interacts early. It collapses each side into one vector before comparison, so the two are only ever compared as whole summaries. MaxSim defers the comparison until both are represented as sets of token embeddings, keeping fine-grained detail available. That is what recovers precise, long-tail matches a summary would average away. ## Why does the "max" matter? The maximum keeps each query token's score independent and noise-resistant. A query token only needs *one* strong match somewhere in the document, so a relevant term buried in a long document still contributes fully; it isn't diluted by all the unrelated tokens around it. Summing across query tokens then rewards documents that satisfy the whole query, one term at a time. ## What does MaxSim cost? MaxSim costs a similarity matrix per candidate document. Instead of one dot product, it computes an `n × m` grid of query-token × document-token similarities, takes a row-wise max, and sums. That is on the order of thousands of times more scoring work than a single-vector comparison, plus the storage for one embedding per token. Those costs are why late interaction ran second, as a reranker, for years. If you have wondered why your engine only offers late interaction as a reranking step, this is the reason. ## How is MaxSim made affordable at scale? Engines approximate MaxSim in a cheap first stage and score only the survivors exactly. Techniques like PLAID and MUVERA, and TopK's [SMVE](/blog/20260311-smve-multi-vector-retrieval), which turns multi-vector representations into sparse vectors whose dot product approximates MaxSim, let the first stage run at sparse-retrieval speed while final ordering comes from exact MaxSim on a small candidate set. TopK measured this design at [roughly 5–8× lower end-to-end latency than PLAID and MUVERA](/blog/20260311-smve-multi-vector-retrieval) at competitive recall (March 2026, ColBERTv2 on BEIR). That is what lets multi-vector work as a first-stage retrieval primitive rather than just a reranker. The full decision framework is in [when to use multi-vector embeddings](/learn/when-to-use-multi-vector-embeddings). ### What Is Multi-Vector Retrieval? URL: https://www.topk.io/learn/multi-vector-retrieval Category: Multi-Vector Retrieval Summary: Multi-vector retrieval stores one embedding per token and scores documents with MaxSim, so precise queries match exact passages instead of a lossy document summary. **Multi-vector retrieval (also called late interaction) represents a document as one embedding per token instead of one embedding per document, and scores it with [MaxSim](/learn/maxsim): each query token finds its best match among the document's tokens, and those matches sum into the score.** **Use it when queries target specific details that a single summary vector loses; use single-vector search for broad topical matching.** ## Why isn't one embedding per document enough? A single embedding is a lossy summary. Compressing a whole document into one ~1,000-dimensional vector keeps the broad topic and discards specifics: exact figures, rare terms, a definition buried in one clause. Queries that depend on those specifics land far from the document's summary vector, and the right passage never surfaces. This is a capacity limit, not a training gap: a [2025 result](https://arxiv.org/abs/2508.21038) proves that fixed-size single vectors cannot represent all the combinations of documents that may be jointly relevant to a query. ## How does it work? 1. Encode each document into one embedding **per token** and store all of them. 2. Encode the query the same way. 3. Score with MaxSim: each query token takes its maximum similarity over the document's tokens; the maxima sum into the document's score. Because parts match parts, a query term can align with the exact passage that answers it instead of competing against an averaged summary. The canonical model is [ColBERT](https://arxiv.org/abs/2004.12832); ColPali and ColQwen extend the idea to document images, tables, and scans. ## What does it cost? It costs storage and compute: one embedding per token is roughly 10–100× more data than one per document, and MaxSim does far more scoring work than a single dot product. That cost kept late interaction in the reranker seat for years. See [is multi-vector retrieval too expensive?](/learn/multi-vector-at-scale) for how modern engines changed the math. ## When should you use it? Use it when precision on specific passages decides the outcome: agent retrieval, technical docs, legal and financial search, complex PDFs. For broad topical queries over short texts, single-vector search is cheaper and usually sufficient. The full decision framework is in [when to use multi-vector embedding models](/learn/when-to-use-multi-vector-embeddings). TopK supports multi-vector retrieval natively as a first-stage query option, not a separate system to operate. Its SMVE encoding runs late interaction at [roughly 5–8× lower latency than PLAID and MUVERA](/blog/20260311-smve-multi-vector-retrieval) at competitive recall (March 2026). Enabling it is a single schema annotation: ```python from topk_sdk.schema import text, semantic_index client.collections().create( "docs", schema={"text": text().index(semantic_index())}, ) ``` TopK embeds the text with its late-interaction model and serves MaxSim-scored retrieval from that one line. The [multi-vector search guide](https://docs.topk.io/guides/multi-vector-search) covers querying and bring-your-own-embeddings setups. ### When Do You Need Hybrid Search? URL: https://www.topk.io/learn/hybrid-search Category: Hybrid Search & Ranking Summary: Use hybrid search when queries mix natural language with exact tokens like names and codes. Vectors blur exact terms; keywords miss paraphrases. **You need hybrid search when your queries mix natural language with exact tokens: product codes, names, error strings, domain jargon. Vector search alone blurs exact tokens into their semantic neighborhood, while keyword search alone misses synonyms and paraphrases.** **If your failed queries show both patterns, run both retrievals and combine them. Pure prose queries can live on semantic search alone, and pure identifier lookups can live on keyword search alone, but most real query streams are neither.** ## Why does vector search miss exact matches? An embedding is a summary of meaning, and rare tokens carry almost none of it. Search for `SKU A17-B` and the embedding dissolves it into a rough neighborhood of product-code-ish text, so the exact match is gone before scoring even starts. A keyword index does the opposite: BM25 scoring on term frequency, term rarity, and document length ([Robertson & Zaragoza, 2009](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf)) matches those tokens exactly, which is why keyword search still anchors most production systems. ## Why does keyword search miss paraphrases? To a keyword index, "car" and "automobile" are unrelated strings. Any query phrased differently from the document scores zero, no matter how close the meaning. Your user asked a perfectly good question, phrased it differently from the document, and got nothing. Embeddings exist precisely to close that gap. ## How do the two get combined? This is the real design decision, and you have three options: merge two ranked lists by position (RRF), normalize and weight the raw scores (score fusion), or score every signal inside one query (true hybrid). Each one trades tuning effort for control over the final ranking. We compare all three in [RRF vs score fusion vs true hybrid](/learn/rrf-vs-true-hybrid). ## How do you know you actually need it? Look at your failed queries. If they cluster around exact identifiers, you're missing keyword search; if they cluster around rephrasings, you're missing vectors. When you see both patterns, you need hybrid, and most real query streams get there fast. TopK runs keyword, vector, and [multi-vector](/learn/multi-vector-retrieval) scoring in a single query with one ranking expression, so you never stitch together a second engine or a fusion layer. In TopK's [BEIR case study](/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality) (July 2025), that single-query approach scored 4.58% higher nDCG@10 on average than fusing two result lists with RRF. The whole thing is one query: ```python from topk_sdk.query import select, field, fn, match docs = client.collection("articles").query( select( "title", semantic=fn.semantic_similarity("content", "climate change policies"), keyword=fn.bm25_score(), ) .filter(match("carbon") | match("renewable energy")) .sort(field("semantic") * 0.6 + field("keyword") * 0.4, asc=False) .limit(10) ) ``` The [true hybrid search guide](https://docs.topk.io/guides/true-hybrid-search) covers the full API. ### When to Use Multi-Vector Embedding Models URL: https://www.topk.io/learn/when-to-use-multi-vector-embeddings Category: Multi-Vector Retrieval Summary: Use multi-vector models when queries target specific details in long or complex documents; use single-vector when queries are broad or cost dominates. **Use a multi-vector (late-interaction) embedding model when your queries target specific details (a clause, a figure, a rare term) inside long or complex documents. Use a single-vector model when queries are broad and topical, texts are short, or storage and compute cost is the binding constraint.** **The two are not competitors so much as tools for different query shapes: multi-vector matches parts of a query to parts of a document; single-vector matches a whole query to a whole document.** ## The short version If you only read the table, read this one: | Your situation | Better default | |---|---| | Queries name specific details (IDs, clauses, numbers, function names) | **Multi-vector** | | Long documents where the answer is one passage among many | **Multi-vector** | | Multi-part queries ("X with Y and Z") | **Multi-vector** | | Documents with tables, forms, or scanned pages | **Multi-vector** | | Out-of-domain vocabulary the model wasn't trained on | **Multi-vector** | | Agents issuing many precise queries in parallel | **Multi-vector** | | Broad, topical queries ("articles about coffee") | **Single-vector** | | Short texts: titles, tweets, FAQ questions | **Single-vector** | | Clustering, deduplication, recommendation by similarity | **Single-vector** | | Cost or storage is the hard constraint and quality is "good enough" | **Single-vector** | Most production systems end up using both: a cheap single-vector or sparse stage to narrow the field, then multi-vector scoring where precision pays off. ## What is the actual difference? A single-vector model compresses an entire document into one embedding, a fixed-length summary. Search compares one query vector to one document vector. This is fast and cheap, but the summary is lossy: whatever detail didn't survive the compression can't be matched later. A multi-vector (late-interaction) model keeps one embedding **per token**. At query time it uses the [MaxSim](/learn/maxsim) operator: each query token independently finds its most similar document token, and those best matches sum into the score. Because parts match parts, low-level detail survives. The canonical late-interaction model is ColBERT; newer ones (ColPali, ColQwen) extend the idea to images and documents. This is not only an empirical difference. A [2025 result](https://arxiv.org/abs/2508.21038) shows that fixed-dimension single vectors have a provable ceiling: past a certain point, one vector of a given size cannot represent all the combinations of documents that could be jointly relevant to a query. Adding detail is a capacity problem, not a training problem. ## When do multi-vector models win? Multi-vector wins wherever the answer lives in a *part* of the document that a summary would average away. **Specific, long-tail queries.** Product codes, legal clauses, function names, rare terminology: exactly the tokens a document-level summary sacrifices first. Each query token gets to hunt for its own match. **Long documents.** In a single vector, a long document drifts toward its dominant topic and everything else becomes invisible. Token-level representations keep the minor passages findable. **Complex and visual documents.** On [ViDoRe v3](https://www.topk.io/benchmarks), a benchmark of real enterprise documents including PDFs, tables, and slides across multiple languages, a compact multi-vector retriever outperformed a single-vector model **80× its size** by an average of +34% recall and +30% nDCG@10, and won in every domain tested. Industrial-document recall roughly doubled (from ~42% to ~76%). The large single-vector model won zero domains. **Agentic retrieval.** Humans ask broad questions; agents fire many precise ones in parallel, often probing the same topic from different angles. That query distribution is the worst case for single vectors. On [BrowseComp-Plus](https://www.topk.io/benchmarks), an agent paired with late-interaction retrieval reached ~80% task accuracy with ~89% citation precision (top-5 on the leaderboard as of mid-2026). ## When should you stick with single-vector? Multi-vector is not a universal upgrade, and treating it as one wastes money. Single-vector models are the right default for **broad semantic matching**, where topical similarity is the whole job and there's no fine detail to preserve. They're better for **short texts**, since a tweet or a title has little internal structure to lose in compression. They're the standard tool for **clustering, deduplication, and recommendation**, which operate on whole-item similarity, not passage-level matching. And they win outright when **cost or latency is the binding constraint** and single-vector quality already clears your bar, where the correct engineering answer is often "the cheaper method is good enough here." Honest reporting bears this out: independent evaluations find late interaction sometimes *underperforms* single-vector on datasets where queries are already broad, so the gain is real but not guaranteed. Measure on your own data before committing. ## What is the catch with multi-vector? The catch is cost, and historically it was disqualifying. **Storage.** One embedding per token instead of one per document means roughly 10–100× more data to store, depending on document length, dimension, and precision. Even with aggressive quantization, token-level indexes are larger than a single high-dimensional vector per chunk. **Compute.** MaxSim scores a query-token × document-token similarity matrix per candidate, not a single dot product, which is on the order of thousands of times more scoring work than single-vector comparison. These two costs are the entire reason late interaction stayed a research technique for years despite its quality advantage. The quality was never in doubt; the bill was. ## How do production systems make multi-vector affordable? The modern answer is a two-stage pipeline: a cheap first stage narrows millions of documents to a small candidate set, then exact multi-vector scoring reranks only the survivors. The candidate stage can be single-vector, sparse, or a specialized approximation; several approaches exist (PLAID, MUVERA, and others), each trading complexity or descriptor size for speed. TopK's approach, [SMVE](https://www.topk.io/blog/20260311-smve-multi-vector-retrieval), converts multi-vector representations into **sparse vectors** whose dot product approximates MaxSim, so the first stage runs at sparse-retrieval speed and cost while final ordering comes from exact MaxSim on the candidates. In TopK's published evaluation this stays within about 1% of exhaustive MaxSim quality while running several times faster than prior approaches. The practical effect: multi-vector becomes usable as a first-stage retrieval primitive at scale, not just a reranker on a handful of results. The takeaway for a build decision: the old rule "multi-vector is too expensive for first-stage retrieval" is increasingly out of date. If you ruled it out on cost a year ago, the tradeoff has moved. ## FAQ **Is multi-vector always more accurate than single-vector?** No. It's more accurate on queries that depend on specific detail, long documents, or complex layouts. On broad topical queries and short texts the gap narrows or disappears, and single-vector is cheaper. Test on your workload. **Can I use both?** Yes, and most production systems do. A common pattern is a cheap first stage (single-vector or sparse) followed by multi-vector reranking, so you pay for precision only where it changes the answer. **Which databases support multi-vector retrieval?** Several now do, with different cost and scale characteristics: TopK, Qdrant, Vespa, Weaviate, and Elasticsearch among them. Native support is now common; the differences are in how efficiently each runs it at scale. **Do I need to retrain or replace my embedding model?** Not necessarily. Late-interaction models are a distinct model class, but you can adopt one for the retrieval path while keeping single-vector embeddings elsewhere. Some models can also expose token-level output embeddings for late interaction. **Does multi-vector help with images and non-text documents?** Yes, and this is one of its clearest advantages. Late-interaction variants like ColPali and ColQwen apply the same part-matches-part idea to document images, tables, and scanned pages, where a single summary vector loses the most. --- _Related: [What is MaxSim?](/learn/maxsim) · [RRF vs Score Fusion vs True Hybrid](/learn/rrf-vs-true-hybrid)_ ### Which Recall Number Actually Matters? URL: https://www.topk.io/learn/recall-vs-retrieval-quality Category: Scale & Architecture Summary: Index recall measures whether ANN approximates exhaustive search. Retrieval quality measures whether the right document surfaces. Evaluate quality first. **Two different numbers share the name "recall," and they answer different questions. Index recall measures whether an approximate index reproduces exhaustive search over the same vectors, which is a fidelity claim about infrastructure. Retrieval quality measures whether the correct document surfaces at all, judged against human relevance labels, which is a claim about your answers.** **A vector database can score 99% on the first while your product fails the second, because the embedding sets the quality ceiling and the index only preserves it. Evaluate retrieval quality first; it decides whether the answer is right. Check index recall second; it decides whether the infrastructure wastes any of the quality you have.** ## What does index recall measure? Index recall compares an approximate nearest-neighbor index against brute-force search over the same embeddings: of the true top-10 nearest vectors, how many did the index return? It is the standard axis of [ANN-Benchmarks](https://arxiv.org/abs/1807.05614), and it is the number most vector databases advertise, because it measures the thing an index actually controls. A database advertising 99% recall@10 is making exactly this claim: our approximation is faithful to exhaustive search. That claim is honest and useful. It says nothing about your answers. ## What does retrieval quality measure? Retrieval quality compares search results against human relevance judgments: of the documents a person would call correct for this query, how many surfaced? Benchmarks like BEIR and ViDoRe measure this form, reported as recall@k or nDCG@k against labeled ground truth. The denominator is what is *true*, not what exhaustive search over your embeddings would have returned. ## Why can both numbers be true while the answer is wrong? The index can only find what the embedding preserved. If a single-vector embedding [averaged the table row away](/learn/rag-financial-tables), exhaustive search misses it too, and a 100%-recall index faithfully retrieves the wrong pages. This is the confusing moment in a vendor bake-off: two systems both report 99% recall, and their answers differ wildly, because the number never measured the answers. The scale of the ceiling effect is measurable. On [ViDoRe v3](/blog/20260611-semantic-index-multi-vector-retrieval) (June 2026), TopK's compact late-interaction model beat a dense embedding model 80× its size by +34% recall and +30% nDCG@10 on average. No index setting closes a gap like that, because the gap lives in the representation, not the approximation. ## Which should you evaluate? Evaluate both, in this order, and insist they are labeled. Measure retrieval quality first, on your own documents and queries, because it decides whether the product answers correctly. Check index recall second, at a fixed quality level, because it tells you what the approximation costs ([how to benchmark](/learn/benchmarking-vector-search)). TopK publishes both forms and labels them. Its [topk-bench harness](/blog/20251201-topk-bench) measures recall@10 against precomputed exact nearest neighbors, which is index recall. Its [semantic_index evaluation](/blog/20260611-semantic-index-multi-vector-retrieval) reports 52.88% nDCG@10 end to end across BEIR against relevance labels, within about 1% of exhaustive MaxSim, which is retrieval quality plus the fidelity cost, stated separately. That is the reporting pattern to demand from any vendor. ### Why Do AI Agents Search With Regex? URL: https://www.topk.io/learn/agents-use-regex Category: Agents & RAG Summary: A phrase is already a valid regex; case-folding and alternatives are small edits to it. One tool covers every query, so agents use it. The engine must be cheap. **Agents search with regex because it is the one tool that covers every query they produce. A literal phrase is already a valid regular expression, and case insensitivity, optional spelling, and alternatives are small edits to the same string, so the agent uses regex for everything.** **That makes a good interface and a costly operator. Regex is fast per string and slow per corpus, and an agent issues many such queries at once. The engine underneath has to make regex cheap at scale, or the agent's own search behavior becomes the bottleneck.** ## What do agent regex queries look like? Watch an agent work and the pattern shows quickly. It sends `invoice` when it means a keyword. It sends `(?i)nobel prize` when it wants a phrase search that ignores case. It sends `colou?r` to cover a spelling variant, and it sends `error (reading|writing) file` to test two hypotheses in one call. None of those queries need the power of regex, but regex expresses all of them, so the agent never has to choose a tool. TopK observed the same behavior in production, and it is what motivated the [sparse-gram regex index](/blog/20260814-fast-regex-search-sparse-grams). ## Why does this break a naive engine? It breaks because the cost is per document. A regex filter over a collection reads and tests every row, and a person's occasional query can absorb that. An agent's burst of concurrent regex filters cannot absorb it, because the scans compete to read the same body of text and throughput stops growing almost as soon as concurrency does. The same thing happens in [why RAG fails for agents](/learn/rag-for-agents), where agent workloads take a mechanism built for one human query at a time and run it many times in parallel, which exposes costs a person never notices. Why regex in particular scales this way is covered in [why is regex search slow at scale](/learn/regex-search-slow). ## What does the engine have to do about it? It has to use the literals. The useful observation in TopK's design is that agent regexes contain many literals, and even a pattern with optional sections or alternatives says that certain characters must occur. An index over those characters turns each regex into a boolean expression over required n-grams, generates a small candidate set, and runs the exact regex only on the candidates. The agent keeps the interface it prefers, and the engine keeps the work small. How the index is built is in [how do you index text for regex search](/learn/regex-index-ngrams). ## How much does it matter? It decides whether agent search is viable at all. On TopK's cluster measurement at concurrency sixteen, the sparse-gram path delivered [55.3 times the throughput and 61 times lower average latency](/blog/20260814-fast-regex-search-sparse-grams) of a full scan. Indexed throughput also grew almost thirteen times from concurrency one to sixteen, and throughput on the scan path barely moved over the same range (August 2026). Agents run at those concurrency levels. TopK's regex filter plans through the index automatically. ```python from topk_sdk.query import select, field # Whatever literals the agent's pattern contains become the candidate plan. docs = client.collection("docs").query( select("text").filter( field("text").regexp_match("(?i)nobel prize") ).limit(50) ) ``` The [fast regex search post](/blog/20260814-fast-regex-search-sparse-grams) has the full design, and the [query documentation](https://docs.topk.io/collections/query) covers the filter syntax. ### Why Do Upserts Slow HNSW Reads? URL: https://www.topk.io/learn/hnsw-upsert-performance Category: Scale & Architecture Summary: Reads and writes contend on one graph: inserts rewire lists mid-traversal and deletes leave tombstones. Sustained upserts tax every read. **Upserts slow reads because both operations fight over one data structure. Inserting into an HNSW graph rewires neighbor lists across multiple layers (the same lists queries are traversing at that moment), so writes take locks that reads feel.** **Deletes are worse: the graph can't cheaply heal around a removed node, so engines tombstone it, and recall and latency decay until a rebuild or compaction that itself competes for the same CPU and memory. Under sustained upserts, p99 latency rises and recall drifts down at the same time.** ## What happens on an insert? The new vector must find its neighbors and be linked into every layer it appears in. [HNSW's insertion algorithm](https://arxiv.org/abs/1603.09320) runs greedy searches through the graph plus mutations to existing nodes' neighbor lists. Each mutation is a synchronization point with concurrent queries. One insert is cheap; thousands per second are a standing tax on every traversal. ## Why are deletes worse than inserts? HNSW has no cheap delete. Removing a node properly means repairing every neighbor list that pointed at it, so engines mark it dead instead and filter tombstones at query time. Traversals now visit nodes that yield nothing, recall degrades as live paths thin out, and the eventual cleanup (rebuild or compaction) is a heavyweight job running alongside your traffic. The recall-degradation-under-updates problem for graph indexes is documented in [FreshDiskANN (Singh et al., 2021)](https://arxiv.org/abs/2105.09613), which exists precisely because naive streaming updates break graph ANN quality. ## How do systems cope? The common patterns all reduce contention on the serving structure: segment the index and merge in the background (LSM-style), route writes to a separate ingest path that builds fresh segments off the serving path, or decouple indexing from serving entirely so queries never share locks with ingestion. Look for one property: reads served from immutable structures, writes absorbed elsewhere, and merges running asynchronously. This is how TopK's write path actually works: a log-writer service appends writes to a [WAL backed by object storage](https://docs.topk.io/architecture), a separate compactor service consumes the log and builds read-optimized indexed files (LSM-style, with compaction planned and executed in parallel), and query executors only ever read immutable segments. [Read and write paths run on separate node pools](https://docs.topk.io/architecture), so ingestion never takes a lock a query can feel. TopK's stated motivation is the failure mode this page describes: in coupled architectures, ["write traffic can negatively affect queries, which usually leads to over-provisioning of resources to maintain SLAs"](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch). ## What should you do? If your workload has real write volume, benchmark under mixed read/write load, never read-only ([how to benchmark](/learn/benchmarking-vector-search)), and prefer engines whose ingest path is decoupled from the query path. The measured effect of this design in TopK: dynamic write batching sustains [about 70MB/s of ingest, over 30,000 vectors per second](https://docs.topk.io/architecture), without sharing resources with the read path; concurrent-load measurements are on the [benchmarks page](https://www.topk.io/benchmarks). ### Why Do Vector Databases Re-Shard? URL: https://www.topk.io/learn/shard-rebalancing Category: Scale & Architecture Summary: Stateful engines pin data to nodes, so growth forces risky migrations. Storage-compute-separated engines pin nothing; there is nothing to rebalance. **Databases re-shard because, in a stateful architecture, data lives on specific machines. When a tenant grows past its shard's capacity, or shards drift out of balance, the database must physically move vectors between nodes while serving traffic: double-writing during migration, rebuilding index structures on the destination, and eating I/O and CPU that queries needed.** **Architectures that separate storage from compute don't have this problem: the corpus lives on object storage, query nodes are stateless caches over it, and a capacity change is a cache-warming event, not a data migration.** ## What triggers rebalancing? Uneven growth triggers it. Hash-based placement balances document *counts*, not tenant *sizes*, so one customer 100× larger than the median concentrates load on whichever shards hold it. Add hardware and the new nodes are empty until data moves to them. Either way, the fix is migration, and migration is the risky operation. ## Why is it so painful for vector indexes specifically? It hurts because graph indexes don't move cheaply. An [HNSW](https://arxiv.org/abs/1603.09320) shard isn't a flat file you can stream to another node. Either you ship the graph and its memory footprint wholesale, or you re-insert vectors on the destination and pay the graph's insert-time construction cost for every vector at migration time. Both happen while the shard keeps serving queries, which is why rebalances show up as latency incidents and are scheduled like surgeries. ## Which architectures avoid it entirely? Storage-compute separation avoids it entirely. When the source of truth is object storage and query nodes hold only caches, "where data lives" stops being a per-node fact: any node can serve any partition by warming its cache. Scaling out is adding stateless readers; tenant growth is just more objects in storage ([why object storage for vector search](/learn/object-storage-vector-search)). TopK's architecture is a concrete example of the pattern: executor nodes are ["semi-ephemeral and fungible"](https://docs.topk.io/architecture), and in principle [any instance can handle requests for any collection](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch). Collections are *consistently assigned* to executors purely for cache hit rates, not data ownership. Losing an executor loses neither data nor availability; the router just redistributes requests. There is no rebalancing operation because there is nothing to rebalance. ## What should you ask a vendor? One question exposes the architecture: *"What happens operationally when one tenant grows 100×?"* If the answer involves a migration plan, data lives on nodes. If it involves nothing, storage and compute are separated. TopK is built storage-compute separated on object storage. The design rationale is in [how we built a new search database from scratch](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch). ### Why Does Filtering Break HNSW? URL: https://www.topk.io/learn/filtered-vector-search Category: Filtering & Multi-Tenancy Summary: The graph was built without knowing your filter. Post-filtering empties results; in-traversal filtering breaks connectivity. Native filtering fixes both. **HNSW breaks under filters because the graph only knows the whole corpus; your filter was never part of its construction. A graph index has exactly two bad options when a filter arrives.** **Post-filtering searches normally and discards non-matching results afterward: under a selective filter, all *k* results can be discarded and the query returns nothing. In-traversal filtering skips non-matching nodes during search: under a selective filter, so few nodes qualify that the graph loses connectivity, and the search either wanders (latency spikes, timeouts) or terminates early with poor recall.** **The fix is an engine that evaluates filters natively as part of retrieval, adapting its strategy to how selective the filter actually is.** ## What goes wrong with post-filtering? The math is unforgiving: fetch top-100 and apply a filter that matches 1% of the corpus, and the expected number of surviving results is about one. Fetch-more-and-hope (top-1,000, top-10,000) burns latency and still offers no guarantee. That is the "zero results" failure mode, and it appears exactly on the queries where the filter mattered most. ## What goes wrong with filtering during traversal? [HNSW](https://arxiv.org/abs/1603.09320)'s speed comes from greedy hops through a well-connected neighborhood graph. Mask off 99% of nodes and the connected structure the search depends on effectively disappears: paths to matching regions run through non-matching nodes. Engines compensate by visiting more of the graph, which is the timeout failure mode: latency grows as selectivity tightens. Both failure modes, and why they worsen with predicate selectivity, are analyzed in [ACORN (Patel et al., SIGMOD 2024)](https://arxiv.org/abs/2403.04871). ## What does a real fix look like? The fix is selectivity-adaptive execution inside the engine. Broad filters can stay close to normal graph search; highly selective filters should flip to searching the matching subset directly (at the extreme, brute-forcing a few thousand matching vectors beats wandering a masked graph). Demand one thing from any vendor: published filtered-search latency *across selectivities*, not just unfiltered numbers. There's a deeper reason bolted-on filtering can't be fixed: as TopK's engineering team [puts it](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch), "the distribution of embeddings and the distribution of metadata are not strongly correlated". A vector index organizes data by semantic neighborhood, so the documents matching your filter are scattered uniformly across it, and no traversal order can find them efficiently. Filtering has to be a first-class operation of the query engine, not a mask over a vector index. TopK's engine was built to invert the usual curve: its stated design goal is that [highly selective queries get *faster*, not slower](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch), since a tight filter shrinks the candidate set the engine actually scores. The measurements bear it out at both ends of the scale: ~62ms p99 on 1M documents and ~115ms on 10M across filters selecting 100%, 10%, and 1% of the corpus (March 2025), and at [one billion documents](/blog/20250722-billion-scale-hybrid-search) (July 2025), dense search improves from ~60ms p99 unfiltered to ~30ms with filters, with no degradation in result quality. Fuller selectivity sweeps are on the [benchmarks page](https://www.topk.io/benchmarks). In practice, filters compose directly with the vector query: ```python from topk_sdk.query import select, field, fn, match docs = client.collection("books").query( select( "title", similarity=fn.vector_distance("title_embedding", [0.1, 0.2, ...]), ) .filter(match("catcher")) # keyword predicate .filter(field("published_year") > 1980) # metadata predicate .sort(field("similarity"), asc=False) .limit(10) ) ``` The [query documentation](https://docs.topk.io/collections/query) covers the full predicate syntax. ### Why Does RAG Fail for Agents? URL: https://www.topk.io/learn/rag-for-agents Category: Agents & RAG Summary: Agents fire many precise, long-tail queries: the worst case for single-vector RAG, which summarizes away the details. Fix retrieval, not the model. **Classic RAG fails for agents because it targets human questions: broad, topical, one at a time. Agents do the opposite. They fire many precise, long-tail queries in parallel, probing for specific facts.** **Single-vector retrieval compresses each document into a lossy summary, so precisely the details agents hunt for are what the index has already thrown away. The model then reasons over noisy context and produces confident wrong answers.** **The fix is detail-preserving retrieval, not a bigger model.** ## What do agent queries look like? Watch an agent work through a task and you will see it decompose the job into narrow probes: an error string, a function name, a clause number, the same topic asked from three angles at once. Each probe has one specific right answer living in one passage. That query distribution is the worst case for a retriever that only knows each document's overall gist. ## Why doesn't "just read the files" work? Grep-style harnesses (giving the agent `ls`, `grep`, and file-reading tools) produce better answers than naive RAG, which is why coding agents use them. But the cost scales linearly with corpus size, and your token spend on serial tool calls balloons. Past a few thousand documents it becomes a latency and cost dead end. The tradeoff is worked through in detail in [RAG Is Broken for Agents](/blog/20260628-rag-is-broken-for-agents). ## What actually fixes agent retrieval? Preserve the details in the index and let the agent query them directly: - **[Multi-vector retrieval](/learn/multi-vector-retrieval)** keeps one embedding per token, so a precise query matches the exact passage that answers it ([when to use it](/learn/when-to-use-multi-vector-embeddings)). - **[Hybrid search](/learn/hybrid-search)** covers the exact tokens (IDs, error codes, names) that embeddings blur away. - **Metadata filters** scope each probe to the slice of the corpus the agent actually means. TopK combines all three in one query. Its `semantic_index` stack (multi-vector retrieval via SMVE, hybrid scoring, filters) reached [80.48% accuracy on BrowseComp-Plus](/blog/20260611-semantic-index-multi-vector-retrieval), a top-5 research agent as of June 2026, with sub-second index freshness so agents search documents as they're written. ### Why Does RAG Fail on Financial Tables? URL: https://www.topk.io/learn/rag-financial-tables Category: Agents & RAG Summary: Single-vector embeddings compress a whole page into one summary, so the row you need averages away into noise. Token-level retrieval matches at the cell. **RAG fails on financial tables because single-vector retrieval compresses a whole page into one summary embedding. A table is hundreds of independent facts, and a summary keeps the topic ("a quarterly income statement") while averaging away which row said what. The query that needed one cell lands on the right document and the wrong number.** **Token-level retrieval fixes the failure at its source: with one embedding per token, the query's "net interest margin" and "Q3 2024" match the exact row and column that answer it, instead of dissolving into an average of every row on the page.** ## Why are tables the worst case for embeddings? A financial table is not one fact; it is hundreds of independent facts packed into rows that look almost identical. The meaning of any cell lives at the intersection of its row label, its column header, and its value, and a single ~1,000-dimensional summary vector has nowhere to keep those intersections. This is a capacity limit, not a model-quality problem: a [2025 result](https://arxiv.org/abs/2508.21038) proves that fixed-size single vectors cannot represent all the combinations of content that may be jointly relevant to a query. Prose degrades gracefully under this compression, because a paragraph usually makes one point and the summary keeps it. A table degrades catastrophically, because the summary keeps "this page is financial results" and drops the 400 numbers that made it useful. ## Why does finance feel this hardest? Financial documents are table-dense, and the answers people need are numbers. Your analyst asks for Q3 2024 net interest margin, the pipeline retrieves the right 10-Q, and the model gets a page whose summary vector matched the topic. It answers fluently with a figure from the wrong column, and a plausible wrong number in finance is worse than no answer at all. Agent workloads compound the problem, because [agents fire many precise probes](/learn/rag-for-agents) like "FY2024 provision for credit losses" where each probe has exactly one right cell. ## What does the evidence show? TopK evaluated its compact late-interaction model against Qwen3-VL-Embedding-8B, a dense embedding model 80× larger, on [ViDoRe v3](/blog/20260611-semantic-index-multi-vector-retrieval) enterprise documents (June 2026). On the finance set, recall rose from 58.90% to 80.91%, part of a +34% recall and +30% nDCG@10 average improvement, and the dense model won no domain at all. ## What actually fixes it? Index at token granularity. [Multi-vector retrieval](/learn/multi-vector-retrieval) keeps one embedding per token and scores with MaxSim, so query terms match cells instead of page summaries. Visual late-interaction models (ColPali, ColQwen) extend the same part-matches-part idea to scanned pages and rendered tables, where cell structure never survives text extraction in the first place. The full decision framework is in [when to use multi-vector embeddings](/learn/when-to-use-multi-vector-embeddings). TopK runs this stack in production: `semantic_index` embeds and serves token-level retrieval from one schema annotation, so a query for a specific line item matches the cell that holds it rather than the page that mentions it. Retrieval that records which passage an answer came from is also what makes the answer [auditable](/learn/auditable-ai-answers). The finance numbers above are its published, dated results. ### Why Does RAG Retrieve the Wrong Chunks? URL: https://www.topk.io/learn/rag-wrong-chunks Category: Agents & RAG Summary: Wrong chunks come from four places: chunk boundaries, lossy embeddings, missed exact tokens, and filters that empty results. Your failed queries tell you which. **Wrong chunks come from four places, and your failed queries tell you which one you have. The chunking cut the answer apart. The embedding summarized the detail away. The query needed exact tokens that vectors blur. Or a filter emptied the candidate set after retrieval already ran.** **Work through the four in that order, because they fail at different pipeline stages and each has a specific fix. The model is almost never the problem: by the time context reaches it, the wrong chunks were already chosen.** ## What does each failure look like? You asked for the termination clause and got the definitions page. Before changing anything, collect twenty failed queries and match them against this table, because each failure class leaves a different fingerprint. | What your failed queries look like | Likely cause | The fix | |---|---|---| | The retrieved chunk starts or ends mid-thought; the answer sits just outside it | Chunk boundaries cut the answer apart | [Chunking strategy](/learn/chunking-redundant-documents) | | Broad questions work; precise ones (a clause, a figure, a row) miss | Single-vector embeddings average details away | [Multi-vector retrieval](/learn/multi-vector-retrieval) | | Queries with IDs, codes, or names return topically similar but wrong text | Exact tokens blurred into a semantic neighborhood | [Hybrid search](/learn/hybrid-search) | | Results go sparse or empty the moment you add a filter | Post-filtering discards the candidates after search | [Native filtering](/learn/filtered-vector-search) | ## Is the chunking destroying the answer? Fixed-size splitting cuts documents at arbitrary character counts, so answers straddle boundaries and each half looks irrelevant on its own. Redundant corpora make it worse: ten near-identical revisions of one passage can fill the entire top-k with copies of a single idea, crowding out the second passage you needed. Deduplicate and chunk on structure before touching anything else, because every downstream stage inherits these chunks ([how to chunk redundant documents](/learn/chunking-redundant-documents)). ## Did the embedding average the detail away? A single embedding per chunk is a lossy summary that keeps the topic and drops the specifics. Precise queries fail against it structurally, and [tables fail hardest](/learn/rag-financial-tables). If your misses cluster on detail-seeking queries, the fix is token-level retrieval: on ViDoRe v3 enterprise documents (June 2026), TopK's compact late-interaction model beat a dense model 80× its size by [+34% recall on average](/blog/20260611-semantic-index-multi-vector-retrieval). [Agents make this failure mode the default](/learn/rag-for-agents), because agent probes are precise by nature. ## Did the query need exact tokens? To an embedding, `SKU A17-B` is a rough neighborhood, not a string. If your misses cluster on identifiers, part numbers, error codes, and names, you are missing keyword scoring next to the vectors. Running both in one ranking measurably beats fusing separate result lists: [4.58% higher nDCG@10 on average in TopK's BEIR case study](/blog/20250724-beyond-rff-how-topk-improves-hybrid-search-quality) (July 2025), covered in [when do you need hybrid search](/learn/hybrid-search). ## Are filters emptying the candidate set? Fetch top-100, apply a filter that matches 1% of the corpus, and the expected number of survivors is about one. If results collapse whenever you add a tenant, date, or permission filter, your engine filters after search instead of during it. That failure and its fix are worked through in [why does filtering break HNSW](/learn/filtered-vector-search). ## What if it is more than one? It usually is, and that is the operational trap: four failure classes tend to mean four systems (a chunker, an embedding pipeline, a keyword engine, a filter layer) patched together in application code. TopK ships the retrieval side as one engine: token-level [multi-vector](/learn/multi-vector-retrieval), keyword, and vector scoring with [natively evaluated filters](/learn/filtered-vector-search) in a single query, with the quality measured and published on its [benchmarks page](https://www.topk.io/benchmarks). Fix the chunking upstream, and let one query handle the rest. ### Why Is Regex Search Slow at Scale? URL: https://www.topk.io/learn/regex-search-slow Category: Scale & Architecture Summary: Regex is fast per string and slow per corpus, because every document gets read and tested. An n-gram index that finds candidates first removes most of the work. **Regex is slow at scale because the cost is per document, not per query. A regular expression evaluates quickly against one string, but a regex filter over a collection has to read every document and run the pattern on each one. A faster regex engine only improves that by a constant factor.** **The fix is to ask how many documents need the full regex at all. A sparse n-gram index finds a small candidate set from the literals the pattern requires, and the exact regex runs only on those candidates. On TopK's measurement, that gave 55.3 times the throughput of a full scan.** ## Why does one fast operation become slow? A regex engine is designed to match one string. It compiles the pattern once and then reads the input one character at a time, so a single log line or document field takes microseconds. Against a collection, the engine repeats that work for every document, and the total cost grows with the number of documents times the average field length. Even an engine built on automata, with no exponential backtracking and no [ReDoS](https://en.wikipedia.org/wiki/ReDoS) risk, has the same cost curve with a smaller constant. Regex filters become the most expensive operator in a plan for that reason, and nobody has to do anything wrong for it to happen. The regex is fast. The scan over every document is slow. ## Why do agents make it worse? A person types an occasional regex and can wait for a scan. An agent issues many regex filters at once, because [it uses regex for ordinary keyword search too](/learn/agents-use-regex), and it issues them while it explores, verifies, and refines an answer. Concurrent full scans compete to read and test the same body of text, so throughput stops growing almost as soon as concurrency starts. In [TopK's measurement](/blog/20260814-fast-regex-search-sparse-grams), throughput on the unindexed scan path grew almost nowhere from concurrency one to sixteen, and throughput on the indexed path grew almost thirteen times over the same range. ## What makes it fast again? The engine has to stop testing every document. Regexes written by agents usually contain literals, and even a pattern with optional sections or alternatives says that some characters must occur. An index over those characters can name a small candidate set, and the exact regex only has to check the candidates. The rule that makes the two stages safe only goes one way. The index may return false positives, and the exact regex removes them. The index must never lose a real match, so the engine can skip any document that fails the index expression without checking it. How the index is built so that the rule holds is covered in [how do you index text for regex search](/learn/regex-index-ngrams). ## How much faster is it? TopK measured the query path end to end on one of its clusters. The measurement compared sparse-gram candidate generation plus exact verification against an unindexed full scan. At concurrency sixteen, the [indexed path delivered 55.3 times the throughput, 61 times lower average latency, and 51.2 times lower p99 latency](/blog/20260814-fast-regex-search-sparse-grams) (August 2026). The gap grows as concurrency grows, and high concurrency is how agent workloads run. ## What does this look like in TopK? The interface does not change. You write a regular expression as a filter, and the planner turns whatever literals it contains into a candidate plan before the exact regex runs. ```python from topk_sdk.query import select, field docs = client.collection("logs").query( select("message").filter( field("message").regexp_match("error (reading|writing) file") ).limit(100) ) ``` Pass `"i"` as a second argument for a case-insensitive match. The [query documentation](https://docs.topk.io/collections/query) covers the full filter syntax, and the [fast regex search post](/blog/20260814-fast-regex-search-sparse-grams) walks through the design. ### Why Object Storage for Vector Search? URL: https://www.topk.io/learn/object-storage-vector-search Category: Scale & Architecture Summary: At billions of vectors the dominant cost is keeping data hot. Object storage decouples storage from compute, so cost tracks query load, not data size. **At billions of vectors, the dominant cost is keeping data hot. A memory-resident HNSW index couples storage to compute: growing the corpus means growing the cluster, whether or not query volume grew.** **An object-storage-backed engine keeps the corpus at commodity storage prices, caches the hot working set, and scales compute with query load instead. That cost gap widens with every added vector, traded against higher latency on cold, rarely-touched data.** ## What does memory-resident HNSW cost at scale? An [HNSW](https://arxiv.org/abs/1603.09320) graph must be resident to be fast: vectors plus neighbor lists, in RAM or pinned to fast SSD, multiplied by replicas. Every new vector permanently occupies the most expensive tier of the hierarchy, so the bill grows with data size even if nobody queries the new data. At millions of vectors this is fine; at billions it dominates everything else you spend. ## What changes with object storage? Storage and compute stop being the same bill. The corpus lives on object storage at cents per GB-month ([current S3 pricing](https://aws.amazon.com/s3/pricing/)); query nodes are caches over it, sized for the traffic you actually serve. Data that's rarely queried costs storage prices, not RAM prices. Capacity changes become cache-warming events rather than data migrations, which is also why this architecture sidesteps [shard rebalancing](/learn/shard-rebalancing). Durability flips too. A coupled architecture pays for durability by replicating data across nodes (TopK [measured this at $0.02 per GB](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch) when analyzing legacy designs), while object storage provides durability as a property of the storage tier itself: in TopK's design, [a write acknowledged into the object-storage-backed WAL is already durable](https://docs.topk.io/architecture), with no replica fleet to fund. ## What's the honest tradeoff? The tradeoff is cold reads. A query touching data outside the cache pays an object-storage round trip that a fully-resident index never pays. TopK measured raw object-storage time-to-first-byte at [roughly 190ms p95](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch) (March 2025), which is the honest size of the penalty an engine must engineer around. Engineering around it is where object-storage engines differ. TopK's approach: a multi-tier read-through cache (memory over NVMe over object storage), and a purpose-built columnar format, [.bob](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch), designed with wide I/O trees for maximally concurrent reads, after finding that off-the-shelf formats like Parquet "have serial dependencies in their I/O" that keep object-storage latency in the request path. A workload that uniformly hammers the entire corpus at ultra-low latency is still the memory-resident architecture's home turf. ## When is memory-resident still right? Memory-resident wins for small-to-mid corpora that fit in RAM affordably, for hard single-digit-millisecond latency floors on every query, and for workloads whose hot set *is* the whole dataset. If your corpus fits in memory and the bill doesn't hurt, the simpler architecture wins. TopK is built object-storage-first: [all data is persisted on object storage as the only durable store](https://docs.topk.io/architecture). The measured results: ~62ms p99 on 1M documents and ~115ms p99 on 10M (768-dim, March 2025), and [a single collection scaling to one billion documents](/blog/20250722-billion-scale-hybrid-search), indexed in hours and queried at sub-100ms latency with no manual sharding (July 2025). Fuller numbers live on the [benchmarks page](https://www.topk.io/benchmarks), and the design rationale is in [how we built a new search database from scratch](/blog/vector-dbs-are-the-wrong-abstraction-how-we-built-a-new-search-database-from-scratch). ## Optional - Security: https://www.topk.io/security - Responsible disclosure and bug bounty policy for TopK. - Trust Center: https://trust.topk.io - Security compliance and trust information for TopK. - Privacy Policy: https://www.topk.io/privacy-policy - Privacy policy - Terms of Service: https://www.topk.io/terms-of-service - Terms of service