AnswersScale & Architecture

How Do You Chunk Redundant Documents?

How do you keep index size and cost under control when chunking highly redundant, version-controlled enterprise documents?

2 min readJuly 2026

The short answer

Collapse the redundancy before it reaches the embedding model. Version-controlled corpora are mostly near-duplicates of themselves; chunking and embedding every version multiplies storage and compute for zero recall gain, and pollutes results with five variants of the same passage.

The pattern: canonicalize to the current version of each document, deduplicate chunks by content hash before embedding, attach version and date as metadata, and serve history through filters only when explicitly requested.

Why does redundancy hurt more than it costs?

The bill is the visible half. The quality damage is worse: your top-k has finite slots, and if a passage exists in ten near-identical revisions, retrieval happily fills the result list with ten copies of one idea, crowding out the second and third relevant passages a RAG prompt or agent actually needed. Deduplication is a relevance fix that happens to also be a cost fix.

What does a working pipeline look like?

  1. Canonicalize: resolve each document to its current version; strip boilerplate (headers, footers, legal blocks) that repeats across the corpus.
  2. Chunk, then hash: normalize whitespace and hash each chunk's content. A chunk whose hash you've already indexed gets a reference, not a new embedding.
  3. Embed once per unique chunk, and let metadata carry which documents and versions contain it.
  4. Default queries to latest, exposing older versions through an explicit version filter.
Collapse redundancy before the embedding bill, not after.

Versioned corpus

mostly near-duplicates

Canonicalize to latest

Hash-dedupe chunks

Embed unique chunks once

History via metadata filters

What if you must keep every version?

Compliance corpora often require it. Keep them, but keep them as metadata-filtered history rather than as default-searchable content. The index answers day-to-day queries from the deduplicated current set; an audit query opts into the archive with a filter. Storage-backed engines make the archived portion cheap to retain.

TopK's metadata filtering makes the "latest by default, history on demand" split a query-time concern, and usage-based pricing means every chunk you dedupe away is ingest and storage you visibly don't pay for.

Keep reading

Collections or Filters for Multi-Tenancy?Default to one shared collection with a mandatory tenant filter; split out only huge or contractually isolated tenants. Idle-collection cost decides.Why Object Storage for Vector Search?At billions of vectors the dominant cost is keeping data hot. Object storage decouples storage from compute, so cost tracks query load, not data size.Why Does RAG Fail for Agents?Agents fire many precise, long-tail queries: the worst case for single-vector RAG, which summarizes away the details. Fix retrieval, not the model.