What does memory-resident HNSW cost at scale?
An HNSW graph must be resident to be fast: vectors plus neighbor lists, in RAM or pinned to fast SSD, multiplied by replicas. Every new vector permanently occupies the most expensive tier of the hierarchy, so the bill grows with data size even if nobody queries the new data. At millions of vectors this is fine; at billions it dominates everything else you spend.
What changes with object storage?
Storage and compute stop being the same bill. The corpus lives on object storage at cents per GB-month (current S3 pricing); query nodes are caches over it, sized for the traffic you actually serve. Data that's rarely queried costs storage prices, not RAM prices. Capacity changes become cache-warming events rather than data migrations, which is also why this architecture sidesteps shard rebalancing.
Durability flips too. A coupled architecture pays for durability by replicating data across nodes (TopK measured this at $0.02 per GB when analyzing legacy designs), while object storage provides durability as a property of the storage tier itself: in TopK's design, a write acknowledged into the object-storage-backed WAL is already durable, with no replica fleet to fund.
Memory-resident
Corpus pinned in RAM
times replicas
Data growth forces cluster growth
Bill grows with data, not usage
Object-storage-backed
Corpus on object storage
cents per GB-month
Query nodes cache the hot set
Compute scales with query load
What's the honest tradeoff?
The tradeoff is cold reads. A query touching data outside the cache pays an object-storage round trip that a fully-resident index never pays. TopK measured raw object-storage time-to-first-byte at roughly 190ms p95 (March 2025), which is the honest size of the penalty an engine must engineer around.
Engineering around it is where object-storage engines differ. TopK's approach: a multi-tier read-through cache (memory over NVMe over object storage), and a purpose-built columnar format, .bob, designed with wide I/O trees for maximally concurrent reads, after finding that off-the-shelf formats like Parquet "have serial dependencies in their I/O" that keep object-storage latency in the request path.
A workload that uniformly hammers the entire corpus at ultra-low latency is still the memory-resident architecture's home turf.
When is memory-resident still right?
Memory-resident wins for small-to-mid corpora that fit in RAM affordably, for hard single-digit-millisecond latency floors on every query, and for workloads whose hot set is the whole dataset. If your corpus fits in memory and the bill doesn't hurt, the simpler architecture wins.
TopK is built object-storage-first: all data is persisted on object storage as the only durable store. The measured results: ~62ms p99 on 1M documents and ~115ms p99 on 10M (768-dim, March 2025), and a single collection scaling to one billion documents, indexed in hours and queried at sub-100ms latency with no manual sharding (July 2025). Fuller numbers live on the benchmarks page, and the design rationale is in how we built a new search database from scratch.