How Do You Benchmark Vector Search?

Why do read-only benchmarks mislead?

Writes degrade reads. Index mutation contends with traversal, tombstones accumulate, background compaction steals CPU at the worst moment (why upserts slow HNSW reads). An engine can post excellent read-only numbers and lose a large fraction of that throughput under a modest write load. You only find out by measuring the mix.

What must be held fixed?

Recall. Approximate search lets any engine trade accuracy for speed silently, so QPS without a recall target is meaningless. That is why ANN-Benchmarks, the standard methodology in the field, plots speed against recall rather than reporting either alone. Tune everything to the same recall (e.g. 95% against exact search on your data), then compare latency. Report percentiles, not averages: retrieval sits in a request path, and p99 is what your users feel.

How should the run be structured?

Load real data at real dimensionality, include your filters at realistic selectivity, warm the system, then measure three phases: steady state, the spike (reads and writes ramped together), and the recovery window after. The last phase catches deferred costs like compaction. Long enough runs matter; mutation damage compounds over minutes, not seconds.

TopK's published methodology (topk-bench, December 2025) is a concrete template: five benchmark dimensions (ingest, concurrency scaling, filtering, recall, and read-write interference), each configuration run five times with the worst run dropped and the mean of the remaining four reported, warmup at twice the measurement timeout, 30-second query windows, and recall@10 over 1,000 queries against precomputed ground truth.

Its datasets carry synthetic filter fields tuned to match exactly 100%, 10%, and 1% of documents, so selectivity is a controlled variable instead of an accident of the data.

One dimension almost every benchmark skips is freshness, the delay between a write being acknowledged and the document appearing in query results. If your application writes continuously, an engine that acknowledges fast but surfaces slowly is lying to you about its write path; measure the gap.

What makes results trustworthy?

Trust requires reproducibility: pinned datasets, a published harness, versioned configs, and a date. A vendor number you can't rerun is marketing.

TopK's results are published this way: the benchmarks page covers 1M–1B vectors, filter selectivities, and concurrent load, reproducible via the open-source topk-bench harness with its datasets hosted publicly on S3. Rerunning it against your own deployment takes a few lines:

import topk_bench as tb

tb.ingest(
    provider=tb.TopKProvider(),
    config=tb.IngestConfig(input="s3://topk-bench/docs-1m.parquet"),
)
tb.query(
    provider=tb.TopKProvider(),
    config=tb.QueryConfig(queries="s3://topk-bench/queries-1m.parquet", concurrency=4),
)
tb.write_metrics("bench-1m.parquet")

The methodology post also makes an honesty choice worth copying: only TopK's absolute numbers are public, and other providers stay anonymized, keeping the focus on behavior rather than scoreboard marketing.