Embedder choice is the only retrieval knob that moves the needle
Going from lexical to semantic retrieval adds +0.26 mean recall. Chunking and the bge query instruction do essentially nothing on these short-passage benchmarks. If you tune one thing, tune the embedder.
Finding 2
Better retrieval produces better answers, on both benchmarks
The retrieval gap carries through to answer quality: lexical retrieval costs about 0.24 token-F1 end to end, the same direction on both datasets. The embedder result is not just a retrieval-metric artifact.
Finding 3
More context keeps helping the answer, past where recall maxes out
Exact match climbs from 52% to 70% as k goes 3 to 10, even though recall@10 is a saturated retrieval metric. The generator tolerates distractors and gains from the higher chance the gold passage is present. Retrieve generously.
Finding 4
Judges agree on faithfulness, argue about relevance
Over the answers the judges actually scored (abstentions carry no judge vote and are excluded), two independent judges agree about 0.94 on faithfulness but only 0.78 on relevance, on both benchmarks. Relevance is the soft, noisy axis: panel it and report the agreement number. A single faithfulness judge can stand alone.
The full matrix
Every configuration at a glance
HotpotQA
2Wiki
The whole retrieval sweep as two grids of nDCG@k (darker is better). Two things jump out: every local row is far darker than every hashing row, so semantic retrieval wins at every k and chunk setting; and the whole and chunked rows are nearly identical, so chunking does nothing. Columns brighten left to right as k rises.
How this was measured. The retrieval matrix is free (local embeddings,
deterministic): 36 configurations at 300 questions each. Answer quality and judge
agreement use the Anthropic API, 50 questions per cell, with the generator (Opus) and
the judges (Sonnet + Haiku) kept distinct so no model grades its own answer. Effects
are differences in mean recall; answer metrics are single stochastic runs and carry a
few points of noise. Every number here is regenerated from the saved run files.
Reproduce: scripts/run_matrix.sh, scripts/run_rag_paid.sh,
then python3 scripts/build_findings_page.py.