Nine experiments ran against a frozen expert-labeled golden set under one protocol: tune on dev, declare one candidate, confirm once on holdout, record the result either way. One change survived: swapping the embedding model. Four dev wins evaporated on holdout, which is the eval doing exactly what it was built to do.
300 queries split 200 dev / 100 holdout (every 3rd by sorted query id, byte-reproducible). All tuning reads dev; holdout is touched once per declared candidate, never iterated. Wins are judged by per-query win/loss counts and an exact sign test, not aggregate deltas alone. Five retrievers re-baselined on dev: BM25 0.641, dense@256 0.686, dense@768 0.715, dense@1536 0.722, hybrid 0.730.
Low b was significantly worse everywhere; every b ≥ 0.8 beat the baseline aggregate, a clean length-normalization pattern. The dev winner looked ironclad and still failed.
| candidate k1=1.5, b=0.9 | Δ nDCG@10 | W/L | sign test |
|---|---|---|---|
| dev (200 q) | +0.0124 | 14/1 | p=0.001 |
| holdout (100 q) | +0.0017 | 6/5 | p=1.0 |
A p=0.001 dev win, selected from a 34-cell grid, can still be selection luck. Defaults stayed.
Split verdict. The candidate w=0.6, rrf-k=10 was confirmed on holdout as a real improvement over default hybrid fusion (+0.0258, 14/4, p=0.031): rrf-k=60 discounts ranks too flatly for this corpus. But even tuned, hybrid trails plain dense@1536 on holdout (0.732 vs 0.744, p=0.86). Fusion fixes hybrid, not retrieval. Code defaults unchanged; the recommendation lives in the findings doc.
The unused BEIR SciFact train split carries 809 more expert-labeled queries. Merged, re-split 740 dev / 369 holdout. The embed step gained incremental cache extension, with old vectors verified byte-identical, because provider embeddings are not bit-stable and a rewrite would have silently broken every committed baseline. Statistical power was the binding constraint all day: effects invisible at 200 dev queries became measurable at 740.
The one change that survived holdout, and the largest single gain of the project: +0.0501 nDCG@10 on the 369-query holdout, 84/38, p < 0.0001. The ladder also answers a practical sizing question: large@256 ties small@1536 exactly, the same quality at one sixth the vector width.
Each model scored the same 55,250 (query, doc) pairs, dense-3large's top-50 per query, into a committed content-addressed cache: one-time local GPU work, $0 in API, rebuildable forever from pinned open weights. The result is a clean model-generation gradient.
A 2021 reranker actively destroys a strong 2025 bi-encoder's ordering, monotonically worse with depth. A 2024 reranker matches it. A 2025 reranker edges it on dev and gives it all back on holdout. Cross-encoder architecture buys nothing by itself; on chunk-sized abstracts, dense-3large's top-10 ordering is already at parity with sub-1B cross-encoder judgment.
Everything above was greedy forward selection, so this round went back for the interactions. Three measurements, all free from committed caches, one declared candidate at the end:
| question | best dev result | answer |
|---|---|---|
| Blend reranker with dense instead of replacing? (--rerank-alpha) | +0.0233, 137/88, p=0.0013 | all 9 configs significant on dev, the only clean sweep of the campaign |
| Does a 12x-smaller index + rerank match full width? | +0.0087 vs full width, p=0.16 | yes: @256 + blend top-20 is indistinguishable from @3072 (dev level) |
| Does fusion help the new champion? | +0.0049, p=0.073 | no: best corner is a rounding error, the rest flat to negative |
The declared candidate (blend α=0.7, top-20) went to holdout and returned +0.0086, 54/48, p=0.62: not promoted. The first rejection where the direction survived, which is what a small real effect looks like below the power floor. Resolution paths: more holdout queries, or a larger reranker (Qwen3-4B/8B) that widens the effect.
A second, user-directed holdout shot then confirmed the cheap-index stack in its own role: @256 + blend vs full width -0.0048, 48/55, p=0.55 (parity holds; dev and holdout gaps are tiny and point in opposite directions), and vs bare @256 +0.0392, 77/35, p=0.0001 — the first reranking effect of the campaign to survive holdout. The reranker cannot improve a strong base, but it fully repairs a cheap one.
The sweep-inflation problem is not theoretical. Best-of-N selection on a finite query set inflates the winner's score; the holdout exists to take that inflation back out. It did, four times in two days:
| candidate | selected from | dev | holdout | verdict |
|---|---|---|---|---|
| BM25 k1=1.5, b=0.9 | 34 configs | +0.0124, p=0.001 | +0.0017, p=1.0 | rejected |
| hybrid w=0.6, rrf-k=10 (vs dense) | ~17 configs | +0.0289, p=0.19 | -0.0120, p=0.86 | rejected |
| rerank Qwen3-0.6B top-10 | 9 configs | +0.0170, p=0.072 | -0.0037, p=0.17 | rejected |
| blend Qwen3 α=0.7, top-20 | 9 configs | +0.0233, p=0.0013 | +0.0086, p=0.62 | rejected |
| dense 3-large@3072 | 4 dims | +0.0394, p<0.0001 | +0.0501, p<0.0001 | promoted |
The one survivor got stronger on holdout. Real effects replicate; selection artifacts regress.
Champion: dense, text-embedding-3-large at 3072 dims (0.777 test-300, 0.783 full-holdout). Storage-constrained deployments now have a holdout-confirmed alternative: @256 plus the local Qwen blend (top-20, α=0.7) runs a 12x-smaller index with no detectable quality loss on either split, and its lift over bare @256 (+0.039, p=0.0001) is the campaign's only confirmed reranking effect. Fusion is a measured dead end; hard reranking below 1B params is too; blend-over-strong-base is the one live thread, directionally positive but below the power floor. Remaining open items: larger rerankers (Qwen3-4B/8B on the 3090), typed query growth via genq to raise that floor, and an optional stemming branch.
Reproduce: every run reads committed caches (two embedding, three rerank) with no keys and no network; scripts/signtest.py recomputes any comparison; full methodology and per-experiment numbers in docs/campaign/tuning-findings.md, docs/campaign/comparison-findings.md, and docs/reranker-design.md.