evalharness (Deterministic Retrieval Evaluation)

Role: Solo, end-to-end. Design, metrics engine, CLI, tuning campaign, docs
Tech Stack: Python 3.11+, numpy, rank-bm25, sentence-transformers, pytest, GitHub Actions
Repository: Private. Demo available on request

Overview

evalharness is a retrieval evaluation harness built the way a test-framework engineer would build one: deterministic, reproducible, CI-gated, with the LLM strictly as an auditor. The one-line thesis is “evals as regression tests, not vibes.”

An eval harness is a test suite for a nondeterministic system, and most eval tooling handles that by leaning on LLM-as-judge, which is expensive, non-reproducible, and unauditable. I spend my working life building test infrastructure, and that pattern reads to me like a test suite whose assertions change every run. So this project inverts it. Every score the tool produces (hit@k, MRR, nDCG, gold rank, confuser rank) is a pure function: same corpus, same queries, same retriever, same numbers, on any machine. The optional LLM audit can flag suspect labels for human review, but it writes to a separate file and cannot touch a score. Pull the LLM out entirely and every number is unchanged.

It measures retrieval only, on purpose. Conflating retrieval quality with generation quality is the most common mistake in RAG evaluation, because when the answer is bad you can’t tell which half failed. This tool isolates the “R.”

The codebase is about 1,550 lines of core Python with nearly the same volume of tests (121 of them), which is roughly the test-to-code ratio you’d expect from someone who builds test frameworks for a living.

Determinism as an Engineering Requirement

“Deterministic” is easy to claim and hard to keep, so the harness defends it mechanically rather than by policy:

• Runs are timestamp-free JSON. A run file contains nothing that varies between identical executions, so two runs of the same configuration are byte-comparable and diff cleanly in git.
• Every run fingerprints its configuration. Retriever type, embedding model, vector width, reranker model and revision, cache content, depth, and blend all hash into a self-describing label like rerank@bge-reranker-v2-m3(dense@3072,top20,a0.7), so no two configurations are ever silently conflated.
• Float noise is clamped before it can matter. Dense scoring is numpy over committed vectors, with cosine scores rounded to 6 decimals before ranking, so platform-level float differences can never flip a near-tie between machines.
• run --verify proves the retriever’s end. It re-derives a sample and fails loudly if the retriever isn’t holding up its half of the reproducibility contract.

Baselines are committed to the repo and gate CI: the workflow regenerates a run from scratch and fails the build if hit@3 regresses past threshold. The committed baseline was produced on Windows and CI runs on Linux, so the gate doubles as a cross-OS regression check on the metrics themselves. That’s not an accident, that’s the point.

The Golden Set: Labels with Provenance

Scores are only as trustworthy as the labels behind them, so the golden set is managed like evidence:

• Each entry is a query, its expert-judged relevant documents (qrels), and confusers: hard negatives that look right but aren’t, which the retriever must rank below gold. A dedicated metric, confusers_above_gold, measures how often it fails to.
• Every label carries provenance (source, author, reviewer, review date). Nothing enters the set without a source.
• The tooling can propose candidates two ways: mine-confusers derives hard negatives deterministically with no LLM, and genq bootstraps candidate queries with one. Either way, candidates are just candidates. The review command is the only door into the golden set, and a human holds it.
• After a run, audit lets an LLM flag suspect labels and defensible misses for re-review. It writes a separate audit.json and has no code path to a score.

On BEIR SciFact, the bundled BM25 retriever scores nDCG@10 = 0.652 against the published Elasticsearch BM25 baseline of 0.665; the gap comes from the BM25 variant and tokenizer, not the harness, and the metric implementations follow trec_eval conventions. I checked because a harness that can’t reproduce known numbers shouldn’t be trusted to produce new ones.

Four Retrievers, One Protocol

The comparison surface is deliberately wide and deliberately cheap to run. bm25 (lexical), dense (embedding cosine at a selectable width), and hybrid (reciprocal rank fusion of both) share one three-method protocol, and --rerank wraps any of them in a cross-encoder second stage. Evaluating your own retriever means implementing three members:

class MyRetriever:
    name = "my-retriever"
    def retrieve(self, query: str, k: int) -> list[str]: ...  # doc_ids, best first
    def fingerprint(self) -> str: ...  # stable hash of your config

The expensive, nondeterministic work is pushed to one-time cache builds that get committed:

• Embeddings: the repo commits two embedding caches for SciFact (text-embedding-3-small at 1536 and text-embedding-3-large at 3072), and --dim derives any smaller width at load time via Matryoshka truncate-and-renormalize. The quality-versus-size tradeoff gets measured, not asserted, and the full comparison runs with no OpenAI key.
• Reranker scores: a cross-encoder’s score depends only on (model, query text, doc text), so rerank-score runs the model once locally and writes a content-addressed cache of pair scores. Eval time reads the cache with no model, no key, no network. The scorers are pinned open-weight models, so unlike an API-based judge, anyone can rebuild the cache, forever.
• Blending: --rerank-alpha mixes base and reranker orderings rank-wise, because a mediocre reranker’s failure mode is confidently demoting documents the base had ranked correctly. The base retains a veto.

The Tuning Campaign: One Promotion, Five Catches

The harness exists to be used, so I ran a multi-day tuning campaign on SciFact (about five thousand scientific abstracts) and wrote the whole thing down. Every experiment tuned on a dev split, and only declared candidates got one shot at a held-out split, with sign tests deciding significance. Roughly a dozen experiments produced exactly one promoted change, and the harness caught my own enthusiasm five separate times. Dev-split wins that looked obviously real went to holdout and died: a BM25 parameter sweep that came back +0.0017 at p=1.0, literal chance, was the first catch.

The holdout-confirmed results (nDCG@10, SciFact test split):

The other confirmed result is a deployment story rather than a table row: dense 3-large @ 256 plus a Qwen3-0.6B rerank blend recovers full-width quality on a 12x smaller vector store (+0.039 over bare @256, p=0.0001, statistically indistinguishable from the champion on both splits). On this corpus a reranker can’t improve a strong base’s ordering, but it fully repairs a cheap one, and a 12x smaller index is real money at scale.

Findings worth stealing: dense beats BM25 on scientific text with modern encoders, which 2021-era BEIR folklore says shouldn’t happen; RRF fusion helps a weak dense arm but subtracts from a strong one; and reranker model generation is everything while architecture is nothing (a 2021 MiniLM cross-encoder actively degrades a strong bi-encoder at every depth, a 2024 bge-v2-m3 is a wash, and 2025 Qwen3 models help).

The last round is my favorite failure. I scored Qwen3-4B and 8B reranker caches locally on my RTX 3090, and both dev sweeps beat the champion in all 12 top_n x alpha configurations, the first all-significant sweeps of the campaign, with a clean scale gradient (8B > 4B > 0.6B). The declared 8B candidate then went +0.0278 on holdout, 68 wins to 49 losses, p=0.096: not promoted. The direction held and the power fell short, and the honest conclusion is that the eval’s own statistical power, not reranker quality, is now the binding constraint. That diagnosis drove the query set from 300 to 1,109 expert-labeled queries, because past a certain point, improving the system means improving the instrument.

Total API spend for the original two-day campaign: about twenty cents. The caches did the rest.

Running Local Models Without Babysitting Them

The reranker caches for the larger models are built by hours-long GPU jobs (the 4B scoring pass alone took about 2.5 GPU-hours on the 3090), and long unattended jobs need to fail loudly and resume cheaply:

• OOM backoff: an out-of-memory error during scoring halves the batch size and retries, and the reduction sticks for the rest of the run instead of thrashing.
• Resume-aware progress: scoring checkpoints into the content-addressed cache, so a killed job resumes where it stopped, and the progress heartbeat (timestamps, pairs/s, ETA) excludes already-cached pairs so the rate estimate is honest rather than flattering.
• The docs include a GPU memory explainer written off a real incident, where CUDA’s silent system-memory fallback turned a fast run into a slow one and the power-draw reading was the tell.

CLI Interface

# get a benchmark and score a retriever
evalharness fetch-scifact --out data/scifact
evalharness run --retriever dense --dim 256 --embeddings data/scifact/embeddings \
  --corpus data/scifact/corpus.jsonl --dataset data/scifact/golden.jsonl --out runs/dense256.json

# compare configurations, gate against a committed baseline
evalharness compare runs/*.json --out compare.html
evalharness gate baselines/scifact_bm25.json runs/scifact_bm25.json
evalharness report runs/dense256.json --out report.html

# build a golden set for your own corpus, human review required
evalharness mine-confusers --corpus corpus.jsonl --dataset golden.jsonl --out candidates.json
evalharness genq --corpus corpus.jsonl --n 20 --seed 7 --out qcandidates.json
evalharness review --candidates candidates.json --dataset golden.jsonl --reviewer you

# one-time cache builds (the only steps that touch a model)
evalharness embed --corpus corpus.jsonl --queries golden.jsonl --out data/mycorpus/embeddings
evalharness rerank-score --model BAAI/bge-reranker-v2-m3 --top-n 50 --out data/scifact/rerank-bge ...

# let the llm flag suspect labels for re-review (never scores)
evalharness audit runs/run.json --corpus corpus.jsonl --dataset golden.jsonl

Features

• Deterministic scoring - hit@k, MRR, nDCG, gold rank, and confuser rank as pure functions; timestamp-free JSON runs that diff cleanly in git
• CI gating - committed baselines fail the build on metric regression; the Windows-baseline-versus-Linux-CI pairing doubles as a cross-OS reproducibility check
• Configuration fingerprinting - every run self-labels with retriever, model, width, rerank depth, and blend so results are never silently conflated
• Three retriever families plus reranking - BM25, dense (with Matryoshka width selection), hybrid RRF, and cross-encoder reranking with rank-wise blending
• Committed caches - embeddings and reranker pair scores are one-time builds; eval time needs no API key, no network, and no GPU
• Provenance-gated golden set - every label carries source and reviewer; deterministic confuser mining and LLM query bootstrapping both feed a mandatory human review step
• LLM as auditor, never scorer - the audit flags suspect labels to a separate file with no code path into scores
• Tuning discipline built in - dev/holdout splits and sign tests; the campaign promoted one change in a dozen experiments and rejected five results that looked good on dev
• Unattended GPU jobs - OOM backoff with sticky batch reduction, checkpointed resume, honest ETAs that exclude cached work
• Reader-facing docs - seven single-concept explainers plus full campaign records, so every number in the findings is traceable to a run

Tech Stack

• Language: Python 3.11+, stdlib-first; core dependencies are just numpy and rank-bm25
• Optional extras: openai (one-time embedding builds), sentence-transformers (local cross-encoder scoring), anthropic (label audit)
• Testing: pytest, 121 tests, test volume roughly equal to core code
• CI: GitHub Actions, test job plus a regenerate-and-gate job with the HTML report as a build artifact
• Reporting: self-contained HTML reports and multi-run comparison tables