Every team I work with arrives at the same conclusion independently: "our chunking must be wrong." It almost never is. Chunking is the most visible knob, so it gets turned first, but the accuracy usually leaks somewhere less obvious.
Before you touch a single parameter, split the pipeline into stages you can measure separately.
Measure retrieval and generation apart
A RAG answer fails for one of two reasons: the right context never made it into the prompt, or it did and the model still answered badly. These are completely different bugs with completely different fixes, and a single end-to-end score hides which one you have.
Build a small set of questions where you know which document holds the answer. Then measure two numbers:
- Recall@k — was the correct document anywhere in the top k results?
- Answer accuracy given perfect context — feed the correct document by hand and grade the answer.
If recall@k is low, generation tuning is wasted effort. If recall is high but answers are still wrong, your retriever is fine and the problem is in the prompt or the model.
The stage everyone skips
In most pipelines I see, recall@20 is above 0.9 and recall@3 is around 0.5. The right document is being found — it just is not near the top. The context window then gets filled with twenty documents, nineteen of which are noise, and the model dutifully picks the wrong one.
That gap between recall@20 and recall@3 is what reranking closes:
# Retrieve wide, then rerank down to what actually goes in the prompt.
candidates = vector_store.search(query, k=50)
ranked = reranker.rank(query, candidates)
context = ranked[:4]
A cross-encoder reranker is slower per document than a vector lookup, but you are running it over 50 candidates, not the whole corpus. In practice it adds 80–200 ms and moves recall@3 far closer to recall@20.
Then, and only then, tune retrieval
Once you know reranking is in place, the remaining recall@50 misses are genuine retrieval failures. Those are worth attacking, and the usual culprits are:
- Pure vector search on keyword-shaped queries. Product codes, error strings, and names are exactly what embeddings are bad at. Hybrid search with BM25 recovers them.
- Chunks that split the answer in half. If a table header lands in one chunk and the row in another, no ranking will save you.
- Metadata that was never indexed. "Last quarter" is a filter, not a semantic query.
Chunking matters. It is just the third thing to fix, not the first.
What to do on Monday
Write 40 questions with known answers. Measure recall@3 and recall@20. If the gap is large, add a reranker before you change anything else. It is the single highest-leverage change in most RAG systems, and it takes an afternoon.