AnswersAgents & RAG

Why Does RAG Retrieve the Wrong Chunks?

Q: Your RAG pipeline worked in the demo and misses in production, so where do wrong chunks actually come from, and how do you tell which failure you have?

Wrong chunks come from four places, and your failed queries tell you which one you have. The chunking cut the answer apart. The embedding summarized the detail away. The query needed exact tokens that vectors blur. Or a filter emptied the candidate set after retrieval already ran. Work through the four in that order, because they fail at different pipeline stages and each has a specific fix. The model is almost never the problem: by the time context reaches it, the wrong chunks were already chosen.

Your RAG pipeline worked in the demo and misses in production, so where do wrong chunks actually come from, and how do you tell which failure you have?

3 min readJuly 2026

What does each failure look like?

You asked for the termination clause and got the definitions page. Before changing anything, collect twenty failed queries and match them against this table, because each failure class leaves a different fingerprint.

What your failed queries look like	Likely cause	The fix
The retrieved chunk starts or ends mid-thought; the answer sits just outside it	Chunk boundaries cut the answer apart	Chunking strategy
Broad questions work; precise ones (a clause, a figure, a row) miss	Single-vector embeddings average details away	Multi-vector retrieval
Queries with IDs, codes, or names return topically similar but wrong text	Exact tokens blurred into a semantic neighborhood	Hybrid search
Results go sparse or empty the moment you add a filter	Post-filtering discards the candidates after search	Native filtering

Four stages, four independent ways the wrong chunks get chosen.

Chunk

boundaries can cut answers

Embed

summaries can drop details

Query

exact tokens can blur

Filter

candidates can empty

Context

Is the chunking destroying the answer?

Fixed-size splitting cuts documents at arbitrary character counts, so answers straddle boundaries and each half looks irrelevant on its own. Redundant corpora make it worse: ten near-identical revisions of one passage can fill the entire top-k with copies of a single idea, crowding out the second passage you needed. Deduplicate and chunk on structure before touching anything else, because every downstream stage inherits these chunks (how to chunk redundant documents).

Did the embedding average the detail away?

A single embedding per chunk is a lossy summary that keeps the topic and drops the specifics. Precise queries fail against it structurally, and tables fail hardest. If your misses cluster on detail-seeking queries, the fix is token-level retrieval: on ViDoRe v3 enterprise documents (June 2026), TopK's compact late-interaction model beat a dense model 80× its size by +34% recall on average. Agents make this failure mode the default, because agent probes are precise by nature.

Did the query need exact tokens?

To an embedding, SKU A17-B is a rough neighborhood, not a string. If your misses cluster on identifiers, part numbers, error codes, and names, you are missing keyword scoring next to the vectors. Running both in one ranking measurably beats fusing separate result lists: 4.58% higher nDCG@10 on average in TopK's BEIR case study (July 2025), covered in when do you need hybrid search.

Are filters emptying the candidate set?

Fetch top-100, apply a filter that matches 1% of the corpus, and the expected number of survivors is about one. If results collapse whenever you add a tenant, date, or permission filter, your engine filters after search instead of during it. That failure and its fix are worked through in why does filtering break HNSW.

What if it is more than one?

It usually is, and that is the operational trap: four failure classes tend to mean four systems (a chunker, an embedding pipeline, a keyword engine, a filter layer) patched together in application code. TopK ships the retrieval side as one engine: token-level multi-vector, keyword, and vector scoring with natively evaluated filters in a single query, with the quality measured and published on its benchmarks page. Fix the chunking upstream, and let one query handle the rest.