This is the story of a two-day tuning campaign run with evalharness, this repoâs deterministic retrieval evaluation harness, written for a reader who is technically curious but has never worked in information retrieval; every piece of jargon gets explained on first use. The short version: over two days we ran roughly a dozen experiments, promoted exactly one change, and the tool caught our own enthusiasm three separate times. The promoted change and every rejected one are equally reproducible from this repo, offline, for a total API spend of about twenty cents.
First, the problem being measured. A retrieval system takes a question and returns the documents most likely to answer it. It is the âRâ in RAG (retrieval-augmented generation): before a language model can answer from your documents, something has to find the right documents. If retrieval fails, everything downstream fails, so this project measures retrieval in isolation.
Measuring it takes three things. A corpus (here, about five thousand scientific paper abstracts from a benchmark called SciFact). A set of queries with known correct answers, which this project calls the golden set: each entry is a query plus the document IDs an expert marked as correct. Those expert relevance labels are called qrels in the retrieval literature, short for âquery relevance judgments,â and ours reproduce the upstream SciFact expert qrels exactly. And finally a score. Ours is nDCG@10, ânormalized discounted cumulative gain at rank 10.â Strip the name away and the idea is simple: look at the top ten results a system returns, give credit for each correct document found, give more credit when it appears higher in the list, and scale the total so that a perfect ordering scores 1.0. Average that over every query and you have one number that rewards both finding the right documents and ranking them well.
The starting point: a five-way comparison with a surprise in it
The campaign opened from a baseline established the day before: a five-way comparison of retrievers on the 300-query SciFact golden set.
The contestants need introductions. BM25 is the classic lexical approach, a refinement of the word-matching search that has powered search engines for decades: it scores a document by how often the queryâs words appear in it, weighted so that rare words count more and long documents do not win just by containing more words. It knows nothing about meaning; âheart attackâ and âmyocardial infarctionâ are unrelated strings to it. Dense retrieval is the modern alternative: a neural network called an embedding model (also called a bi-encoder, because it encodes queries and documents separately) converts each text into a long list of numbers, a vector, positioned so that texts with similar meaning end up near each other. To search, you embed the query and return the documents whose vectors point in the closest direction; synonyms and paraphrases come along for free because it matches meaning rather than spelling. Hybrid retrieval runs both and merges their ranked lists, here with a recipe called reciprocal rank fusion (RRF): each document earns points from each list based purely on its rank position, higher ranks earn more, and the fused ranking sorts by total points. It lets two systems vote without needing their raw scores to be comparable.
The numbers: BM25 0.652 nDCG@10, dense (OpenAIâs text-embedding-3-small at full vector width) 0.730, hybrid 0.722.
Two of those results contradicted expectations. The influential 2021 BEIR benchmark paper found dense models losing to BM25 on scientific text, and that finding hardened into folklore: âdense underperforms on specialist domains.â Here dense beat BM25 by eight points. The folklore was not wrong in 2021; it had a hidden timestamp. That eraâs dense models transferred poorly to scientific claims, and 2026-era encoders are simply much better. The second surprise was that hybrid, the standard âbest of both worldsâ pitch, lost to plain dense. The per-query data showed why: RRF averages the two lists with equal weight, and when one arm is eight points stronger, averaging in the weaker arm costs more on queries the strong arm already got right than it recovers on rescues. Hybrid fixed 10 of denseâs misses and broke 14 of its wins. Fusion needs a reason, not a reflex.
So the campaign began with a clear champion (dense at 1536 dimensions, 0.730) and a clear question: how much better can we make this, and how do we know a âbetterâ number is real?
Day one, step zero: split the data before touching any knob
The first action of the campaign was not an experiment. It was building a guardrail: splitting the golden set into a dev set (200 queries) and a holdout set (100 queries), deterministically, before turning a single dial.
Here is why that matters, because it is the hinge of the entire story. When you tune a system against a fixed set of test queries, every adjustment that raises the score is doing some mixture of two things: genuinely improving retrieval, and accidentally fitting the quirks of those particular queries. The second kind of âimprovementâ is worthless; it evaporates on any new query. With only 300 queries and most experiments differing on just a handful of them, luck can easily masquerade as progress. The defense is old and simple: tune only on the dev set, and keep the holdout locked away, touched exactly once per candidate, at the moment of deciding whether to promote it. Because the holdout queries never influenced any tuning decision, a win that replicates there is probably real. A team that tunes against its whole eval set produces numbers that go up while the system does not get better. The numbers lie, politely and confidently.
One more piece of statistical machinery appears throughout this story: the sign test. Instead of just comparing average scores, we count queries. For every query where two systems disagree, which one won? If system A beats B on 92 queries and loses 47, we ask: how likely would a fair coin be to come up heads 92 times out of 139? The sign test answers with a p-value, the probability of seeing a split that lopsided by pure chance. It deliberately ignores how big each win was, which throws away information but makes it conservative, and conservative is the right bias for promotion decisions. A half-point average gain with a 55-to-52 win/loss record is indistinguishable from noise, and the findings docs say so when it happens.
Experiment one: the BM25 sweep, and the first catch
BM25 has two tuning parameters, k1 and b, controlling how quickly repeated words saturate and how strongly long documents are penalized. We swept a grid of 34 combinations on the dev set. The winner, k1=1.5 with b=0.9, looked excellent: +0.0124 nDCG@10, fourteen wins against one loss, p=0.001, nominally âone in a thousand this is chance,â and the whole b=0.9 column was positive, so it was not even a lone lucky cell.
It failed the holdout completely: +0.0017, six wins to five, p=1.0. Chance.
This is worth sitting with, because it is the least intuitive lesson of the two days. How does a one-in-a-thousand result vanish? The answer is called sweep inflation, or in statistics, the multiple comparisons problem. We did not run one experiment; we ran 34 and reported the best. When you draw 34 tickets from a noisy lottery, the best ticket is biased upward by construction: partly quality, partly the luck of the draw on those specific dev queries. The more configurations tried, the bigger the inflation, and the p-value inherits it because it was computed as if this were the only experiment. The holdout exists for exactly this failure mode, and it worked on its first day. BM25 stayed at its defaults, and the roadmapâs warning that dev winners shrink on holdout went from theory to lived experience within hours.
Experiment two: fixing fusion is not the same as improving the system
The hybrid retriever had two knobs of its own: a dense-weight flag (how much more the dense armâs votes count than BM25âs) and the RRF constant controlling how steeply rank positions are discounted. The dev sweep found that favoring the dense arm (weight 0.6) and discounting ranks much more sharply (constant 10 instead of the default 60) was the best fusion recipe.
The holdout shot asked two separate questions and got different answers. Versus default hybrid: +0.0258, fourteen wins to four, p=0.031. Confirmed, a genuine improvement to the fusion component; the default constant discounts ranks too flatly for this corpus. Versus plain dense: -0.0120, fifteen wins to seventeen. Not confirmed. The tuned hybrid, now demonstrably the best hybrid we could build, still lost to its own stronger ingredient used alone.
The lesson generalizes well beyond retrieval: fixing a component is not the same as improving the system. We had made fusion better at its job, and its job was still not worth doing here. The tuned operating point was recorded as a recommendation rather than changed in code, and dense remained champion.
The pivot: the eval itself was the bottleneck
At this point a pattern was visible: experiments kept landing at âpositive direction, not significant.â With 200 dev queries and most configurations tying on the majority of them, effects of a point or two simply could not be resolved. The most valuable next move was not another retriever idea; it was more statistical power, meaning more labeled queries.
The fix was sitting in the benchmark all along. The golden set had been built from SciFactâs 300-query test split, but the datasetâs train split carries 809 more queries with the same expert qrels, unused. Merging them grew the eval from 300 to 1,109 expert-labeled queries, re-split into 740 dev and 369 holdout. No labels were invented or repaired: a general-purpose model should never overrule domain experts who had context it lacks, so the audit tooling flags suspicious labels for human review but never edits them.
Growing the query set required embedding the new queries, which surfaced a subtle engineering requirement: the embedding cache had to become incrementally extensible, with old vectors verified byte-identical after extension. That sounds like pedantry until you know that provider embedding APIs are not bit-stable: embed the same text twice and you can get microscopically different numbers back. If growing the query set meant re-embedding everything, every committed baseline would silently shift. Freezing old vectors and only appending new ones keeps every historical number exactly reproducible; in this project the committed caches, not the API, are the source of truth.
The payoff was immediate: an effect that was unresolvable at 200 dev queries (the tuned hybridâs edge) became a measurable trend at 740. The instrument had been sharpened.
The one promoted change: a better encoder
Phase 3 was the least mechanically interesting experiment and the biggest win: swap the embedding model from text-embedding-3-small to text-embedding-3-large. Same API, same everything, just a stronger encoder, at roughly 6.5 times the embedding cost, which for this corpus meant about twenty cents.
Both models are Matryoshka-trained, named after the nesting dolls: the training objective forces the earliest positions in each vector to carry the most information, so you can truncate a 3072-number vector to its first 256 numbers and still have a usable embedding rather than an arbitrary slice. One committed full-width cache therefore serves every smaller dimension for free, turning âwhat vector width do I actually need?â from an assertion into a measurement.
The dimension ladder on the large model read 0.7295 at width 256, 0.7663 at 768, 0.7701 at 1536, and 0.7772 at full 3072 width. The holdout confirmation against the old champion was unambiguous: +0.0501, 84 wins to 38 losses, p below 0.0001. Promoted, the only change of the campaign to survive the holdout.
The ladder also produced the most practically quotable result of the project: the large model truncated to 256 dimensions ties the small model at 1536 dimensions exactly, 0.7295 versus 0.7296. For anyone sizing a vector store, that is a sixth of the storage and compute for the same quality. A better encoder squeezed into fewer numbers beats a weaker encoder at full width.
One side effect mattered for what came next: with the stronger dense arm, fusion lost even more of its remaining value, because the small pocket of queries BM25 uniquely served had largely been absorbed by the better encoder. If more quality was available anywhere, it would be in reordering the shortlist, not in mixing in a lexical arm.
Day two: the reranker bakeoff
The motivation for reranking came straight from the metrics: across every retriever, the right document was in the top five far more often than it was in first place. The remaining headroom was ordering, not finding.
A cross-encoder reranker attacks exactly that. Where a bi-encoder embeds query and document separately and compares vectors, a cross-encoder reads the query and one candidate document together, as a single input, and outputs a relevance score. Reading them jointly lets it notice interactions a bi-encoder structurally cannot, which makes it more accurate and far more expensive per pair. So the standard pattern is a pipeline: cheap retrieval narrows five thousand documents to fifty candidates, and the cross-encoder re-scores only those fifty.
The design decision was where those scores come from. A rerank API vendor would cost a few dollars and could deprecate the model out from under the results; instead the scorer is a local cross-encoder on our own GPU, with pinned open weights, writing scores into a committed cache keyed by hashes of the query and document text. One offline scoring step per model, and every subsequent eval run reads frozen scores with no key and no network. The whole bakeoff cost zero dollars in API spend.
Three models were bracketed deliberately across generations: MiniLM (2021, 22 million parameters, the classic tiny reranker), bge-reranker-v2-m3 (2024, 568 million, the expected default), and Qwen3-Reranker-0.6B (2025, the newest generation at a size that fits an 8 GB laptop GPU). Each scored the same 55,250 query-document pairs.
The results formed a clean gradient by model generation. MiniLM was actively harmful at every depth, up to five and a half points worse than not reranking at all, and monotonically worse the deeper it reached: a dated 2021 model confidently reshuffling a strong 2025 bi-encoderâs ordering makes it worse. bge-v2-m3 was a statistical wash at every depth. Qwen3 was positive at every depth, best at +0.0170 on dev with p=0.072. The architecture lesson writes itself: cross-encoders are not magic. The joint reading buys nothing by itself; model quality and recency are everything.
Then the now-familiar ending. Qwen3âs best configuration was the winner of nine (model, depth) cells, maximal exposure to sweep inflation, and the holdout returned -0.0037, p=0.17. The +1.7-point dev margin evaporated entirely. Third catch in two days, and the largest. Nothing was promoted, the dense model alone remained champion, and the negative result went into the findings doc in full: âwe tried three rerankers and here is exactly how each lostâ is precisely the institutional memory this tool exists to keep.
The revisit round
Every experiment so far followed greedy forward selection: take the current champion, try one addition, keep it only if it wins outright. Greedy search is cheap but blind to interactions, and the bakeoff left one clue worth following: Qwen3 was consistently positive on dev at every depth, not useless, just not better than dense as a replacement. So the current round asks a different question: what if the rerankerâs opinion is blended with the dense ranking instead of replacing it?
The mechanism is rank-level interpolation with a single knob, alpha: each documentâs final position is a weighted combination of its rank in the dense list and its rank in the reranked list. Alpha=1 is pure reranker, alpha=0 is pure dense, and the interesting territory is between.
The first result is the best dev number of the entire project: alpha=0.7 over the top 20 candidates scores +0.0233 over the champion, 137 wins against 88 losses across the 740-query dev set, p=0.0013. Two mediocre soloists appear to make a good duet: the reranker is right often enough to help, and anchoring 70 percent of the weight to the dense ranking contains the damage when it is wrong.
If this story has taught you anything, though, you already know the correct posture: that was a dev number, selected from a sweep of alphas and depths, and three dev darlings had already died on the holdout in two days.
The round finished with two more measurements and then the single holdout shot. The fusion recheck came back empty: against the new encoder, even the best fusion recipe managed +0.005 with p=0.07, and most of the grid was flat or negative, so fusion is now a measured dead end rather than an assumed one. The cheap-index experiment came back genuinely useful: 256-wide vectors alone lose badly to full width (-0.044), but 256-wide vectors plus the blended reranker over the top 20 are statistically indistinguishable from full width. A vector store one twelfth the size, plus a few milliseconds of local reranking, costs nothing in quality.
And the declared candidate? The blend over the full-width champion went to holdout and returned +0.0086, 54 wins against 48 losses, p=0.62. Not promoted. The fourth catch is the subtlest: unlike every earlier rejection, the direction survived (the other three flipped to zero or negative), which is what a small real effect looks like when the confirmation set is too small to prove it. The discipline does not award partial credit, so the champion stands unchanged: dense retrieval, text-embedding-3-large, 3072 dimensions.
The cheap-index result then earned its own confirmation. It had initially been left at dev level, because each round rations itself to one holdout consultation, but it was testing a different claim (parity on a small index, not victory over the champion) and a deliberate second shot was spent on it. It passed, both ways. Against the full-width champion the stack measured -0.0048 with p=0.55: the dev gap was +0.0087, the holdout gap -0.0048, two tiny differences pointing in opposite directions, which is exactly what genuine parity looks like. And against bare 256-wide retrieval the stackâs lift measured +0.0392 with 77 wins against 35 losses, p=0.0001, making it the only reranking effect in the entire campaign to survive a holdout test.
That gives the campaign a second confirmed configuration and arguably its most practical result. For anyone paying for a vector database, the choice is now measured on both splits: run full-width vectors for the confirmed maximum, or store one twelfth the data, spend a few milliseconds of local CPU-or-GPU reranking per query, and get quality the eval cannot tell apart from the champion. It also completes the rerankerâs characterization in one sentence: a sub-1B local reranker cannot improve a strong retrieverâs ordering, but it fully repairs a cheap one.
Two honest paths could settle the blend question. More holdout queries, via the genq tooling that generates typed candidate queries (factoid, synthesis, ambiguous) admitted only after human review, would raise the power floor. Or a stronger reranker could widen the effect past it: the Qwen familyâs 4B and 8B models do not fit the laptopâs 8 GB GPU but run on the desktopâs 24 GB RTX 3090, and the 0.6B modelâs consistent positive trend is exactly the evidence the roadmap said would justify that trip.
The 3090 round: the big rerankers
Two weeks later the desktop with the 24 GB card was back in reach, and the round the roadmap had promised got its turn. It opened with a quiet victory for the whole design: the desktop had none of the projectâs generated data, so everything was rebuilt from the repo alone: fresh virtual environment, re-downloaded corpus, regenerated golden set and split. The first act of the new machine was to run BM25 against a baseline the laptop had committed, and the gate passed with identical numbers, byte for byte. Two machines, one truth; the determinism thesis held in the field on its first real test.
The 4B model scored its 55,250 pairs in about two and a half GPU-hours and produced the campaignâs first fully significant sweep: all twelve depth-and-alpha configurations beat the champion on dev, every one with a solid p-value, the best at +0.0368. Two patterns shifted from the 0.6B story. Pure replacement, which had failed at 0.6B, was now clearly positive: a better judge needs less of the baseâs veto. And the gains grew with depth rather than flattening, meaning the 4B was finding real promotions deep in the candidate list where the small model could not.
The 8B leg turned into an unplanned systems lesson before it produced a number. The modelâs 16 GB of weights plus batch activations quietly overflowed the card, and the driver, rather than erroring, silently spilled the overflow to system RAM at a thirtieth of the speed: the GPU read â100% busyâ while throughput halved and kept degrading. The tell, in hindsight, was power draw: 190 watts at full utilization on a 350-watt card means starved cores, not a working one. The night produced, in order, a diagnosis, a driver setting that turns silent spill into a loud out-of-memory error, a batch-size arithmetic, and a fifteen-line automatic backoff that halves the batch and retries when memory runs out; the backoff fired in production twenty minutes after it was written and the run finished the night at triple its spilled pace. The whole incident, with numbers, became its own explainer (docs/explainers/gpu-memory-explained.md).
The 8B results were the strongest of the campaign and continued the scale gradient without a wrinkle: every cell significant, the best at +0.0460, with 8B beating 4B beating 0.6B at every single configuration. At 8B the blend became fully obsolete: pure replacement was the best cell outright. Whatever a bigger judge buys, the campaign now had three cleanly spaced points saying it buys steadily more.
Then the roundâs single declared candidate, the 8B pure rerank over the top 50, went to the holdout. It returned +0.0278, with 68 wins against 49 losses, p=0.096. Not promoted. Fifth catch, and the most informative rejection of the campaign: the direction held for the third round in a row, the shrinkage from dev was the ordinary 40 percent, and the aggregate gap was nearly three points, but 252 of the 369 holdout queries tied, and 68 against 49 among the 117 that moved is just short of what the bar demands. The arithmetic in the findings doc puts it plainly: at that tie rate, this holdout can only certify an effect roughly the size dev claimed, arriving unshrunk, which is not how dev margins behave.
So the champion stands, and the round closes on the campaignâs most useful kind of conclusion: not a promotion but a diagnosis. A real effect near three points very likely exists; the instrument can no longer see it. The bottleneck moved from the retriever to the eval itself, and the next investment is not a bigger model but a bigger holdout, through the genq pipeline where a model drafts typed queries and a human admits them one by one. The 3090 round set out to buy a better reranker and instead measured its way back to the projectâs founding claim: the eval instrument is the product.
Lessons learned
The eval instrument is the product. The most consequential change of the campaign was not a retriever improvement; it was growing the query set from 300 to 1,109. Statistical power gates everything: an underpowered eval cannot distinguish a real two-point gain from noise, so every tuning effort upstream of it is guesswork with extra steps. Sharpen the instrument before turning knobs.
The holdout catches what dev enthusiasm cannot. Five times across the campaign a dev result with a respectable story failed its holdout confirmation: a p=0.001 BM25 sweep winner, a +2.9-point fusion margin, a nine-config reranker champion, a p=0.0013 blend candidate, and finally an 8B rerank candidate whose +4.6-point dev margin shrank to +2.8 and landed at p=0.096. None was a bug or a bad idea; each was the ordinary arithmetic of picking the best of many noisy tries, and the last one is probably a real effect the holdout is simply too small to certify. Any tuning loop without a locked holdout set is generating exactly these numbers and believing them. The flip side got demonstrated too: the claims that were real (the encoder swap, the cheap-index stack) passed their confirmations without drama.
Negative results recorded are institutional memory. The findings docs give losing experiments the same treatment as winners: exact deltas, win/loss counts, p-values, and reproduction commands. Six months from now, âshould we add a reranker?â has a documented answer with numbers attached instead of being re-run, and dev-overfitted, all over again.
Determinism plus committed caches makes every claim checkable. Every number in this story reproduces from the repo with no API key: embedding and rerank scores are frozen in committed caches, eval-time scoring is pure arithmetic, scores are rounded before ranking so float noise cannot flip near-ties across machines, and every run file carries a fingerprint of its configuration and a hash of its dataset. Total API spend for the campaign, including the promoted encoder swap, was about twenty cents. Reproducibility was not expensive; it was a design habit.
Hardware has its own lessons. Local GPU scoring earned some operational scar tissue worth writing down. Probe with a small batch before committing to a long GPU run, because a config error discovered forty minutes in costs forty minutes. Checkpoint incrementally, so an interrupted scoring run resumes instead of restarting; the extendable cache design gave this for free. Emit visible progress lines with a rate and an ETA, because a silent multi-minute process is indistinguishable from a hung one and a decaying rate is the fingerprint of memory pressure. Watch shared GPU memory and power draw, not utilization: a card oversubscribed into silent system-RAM spill reports 100% busy while starved cores idle at half wattage. And make failure loud, cheap, and self-correcting, in that order: an error you can see, a checkpoint that caps the loss, and a backoff that halves the batch and retries. The full case study is docs/explainers/gpu-memory-explained.md.
Glossary
- nDCG@10: a 0-to-1 score for a ranked result list that gives credit for each relevant document in the top ten, more credit the higher it ranks, normalized so a perfect ordering scores 1.0.
- BM25: the classic lexical (word-matching) retrieval algorithm, scoring documents by query-word frequency with adjustments for rare words and document length.
- Dense retrieval / bi-encoder: search using a neural embedding model that converts queries and documents separately into vectors, then ranks documents by vector similarity, matching meaning rather than exact words.
- Hybrid retrieval / reciprocal rank fusion (RRF): running lexical and dense retrieval together and merging their ranked lists by awarding each document points based on its rank position in each list.
- Cross-encoder reranker: a model that reads a query and one candidate document jointly and scores their relevance; too slow to search a whole corpus, so it re-orders a shortlist produced by a cheaper retriever.
- Golden set: the evaluation dataset of queries paired with their known correct documents, against which every retriever is scored.
- Qrels: query relevance judgments, the expert-made labels saying which documents are correct answers for which queries.
- Dev/holdout split: dividing the golden set so tuning uses only the dev portion, while the holdout portion is scored once per candidate to check whether a dev win is real.
- Sign test: a conservative statistical test that counts per-query wins and losses between two systems and asks how likely that split would be if the systems were actually equal.
- Sweep inflation / multiple comparisons: the upward bias in the best result of many tried configurations, because the winner is partly quality and partly luck; the reason dev winners routinely shrink on holdout.
- Matryoshka embeddings: vectors trained so their earliest positions carry the most information, letting one full-width vector be truncated to any smaller dimension and still work.
- Statistical power: an evalâs ability to detect a real difference of a given size; more labeled queries means smaller effects become measurable.