The Two Axes Nobody Guards
input and consequence, the bookends
previously → who makes the decision (part one), and who wrote the frame it happens inside (part two). both of those live in the middle of the pipeline, which is exactly where everyone already looks.
here’s the trap in getting the middle right. you make the decision deterministic, you keep the frame yours, you feel safe, and a decision is still only ever as good as the evidence handed to it and only as safe as whatever the system does with its answer. those are the two ends, and they turned out to be axes three and four. what took me longest to see is what they have in common → they bracket the decision, and neither of them looks like a decision, which is exactly why nobody guards them. the model choosing what to look at doesn’t feel like a verdict. the system acting on what the model said doesn’t feel like one either.
start with the downstream end, because it’s the more visible of the two.
the third axis: what the output becomes
i ran the instrument over another tool expecting a higher-stakes rerun of what i already had. it was, right up until it exposed a third axis i’d been blind to for the same reason i was blind to the second → every tool i’d audited happened to sit on the safe side of it.
look at what the model’s output became in each earlier repo. a score. a summary. a category. a newsletter. every one of them is an annotation, something a human reads before doing anything about it. and that human is doing unpaid audit work i never credited. when a summary is wrong, i notice, because i read it. when a newsletter picks a weak story, a person sees the weak story and moves on. displaying a model’s output silently outsources the final audit to the reader, so “naked judge” was never quite as naked as i made it sound. there was always someone standing at the exit, catching the worst of it on the way out.
then i ran it against a tool i’m still building, a trading engine, and the model’s output stopped being something a human reads and became something the system does. the model decides trade-or-no-trade and authors every parameter, direction, instrument, strike, contract count, entry premium, and those values are copied into a trade record and booked, with downstream P&L computed off them. nobody reads it first. the human who was quietly serving as my last auditor in every other tool is simply gone, because the output executes instead of displays. and this is exactly the kind of thing you want the audit to catch while the code is still open in your editor rather than after it’s live.
that’s a third axis, orthogonal to the other two. not how checked is the decision and not who wrote the frame, but what happens to the output → does it pause for a human, or flow straight into action and computation. and it’s the amplifier, the one that turns the earlier failures from expensive into unrecoverable, because it removes the backstop that had been silently covering for them. a naked judge you read is recoverable. a naked judge you execute is not. same rung, different universe. that’s a stage-independent truth, it holds whether the tool ships tomorrow or never.
this unfinished tool also handed me the sharpest reducible-but-unwired illustration i’ve got, and because it’s unfinished it handed it to me before it could ever hurt anything. as it currently stands, the model invents the entry premium, the cost basis of the trade, which means total cost, max risk, position size, and every P&L number downstream would compute off an invented figure. here’s what the audit flagged: a deterministic option pricer already exists in the codebase, imported into the same file, already in active use everywhere except entry, for marking open positions to market. so the correct answer is sitting right there, imported, one function over, and the trade path is still routed through the model because that’s the wire i hadn’t run yet. on a shipped tool that’s a confession. on a workbench it’s the entire point of auditing early → it found the number i still need to take back from the model, before a single real dollar of P&L ever gets computed off a hallucination. a wrong annotation is a wrong sentence. a wrong number that gets computed upon is a wrong world, and the cheapest time to notice that is while the wiring is still exposed.
the same audit surfaced an inversion i’ll want to resolve before this thing goes anywhere near live. the detection half is deterministic through and through → every alert, every escalation score, every state transition a pure function of config thresholds, no model in the loop. and as it stands, the model can veto a signal that passed every one of those gates, on nothing but a free-text rationale. code proposes, model disposes → the exact backwards of the first thing auditing myself taught me. the deterministic layer is supposed to hold the veto over the model, and right now, in this half-built tool, it’s wired the other way around. catching that on a finished system would sting. catching it on a workbench is just design.
the fourth axis, the one i made earn its seat
now the upstream end, and the sneakier of the two. i could feel this one coming early, and it felt elegant, which is precisely the sensation the last part is going to tell you not to trust. so instead of seating it on a hunch i wrote down the test that would earn it → point the instrument at something retrieval-heavy and check whether the real failure is something the first three axes genuinely can’t describe. then i ran that test against a hybrid-search knowledge tool i’d built, and it came back positive cleanly enough that refusing the axis would have been the dishonest move. so here it is, earned rather than assumed.
input-authority → who chose the evidence the decision runs on.
the tool works like this: a model classifies and rewrites the query, deterministic code retrieves and ranks candidate articles, and a model summarizes the top few. the ranking is the cleanest auditor in the whole tour → reciprocal-rank fusion over stored vectors, every position traceable to a rule, and even a garbage topic self-corrects, because too few matches trips a deterministic fallback to the full corpus. on the decision axis this thing scores beautifully. the decision is genuinely audited.
and it can still be silently, invisibly wrong, for a reason none of the first three axes can see, because the model chose the evidence that flawless ranking runs on. two places do it. first, the classifier’s topic narrows the candidate set before ranking, and the fallback only guards against too few matches → a confident-but-wrong topic that happens to be populous excludes the correct article from the pool before any ranking runs. the auditor then flawlessly, reproducibly ranks a candidate set the right answer was never in. second, the model rewrites the user’s query, and that rewrite replaces the original for both the vector search and the keyword search, with no fallback and nothing comparing rewrite-recall against original-recall. a bad rewrite quietly degrades every deterministic step downstream, and each one faithfully executes the degradation.
that is a failure the other three axes score as clean. the frame is fixed and mine. the ranking decision is a textbook auditor. nothing here is about display-versus-execute. and yet the outcome is compromised from upstream, because a perfectly-audited decision is only ever as good as the inputs it was handed, and the model handed them to itself. that’s the whole reason input-authority can’t fold into decision-authority: decision-authority measures how checked the verdict is, and here the verdict is fully checked and still wrong, because the checking ran over evidence the model curated. a clean auditor over model-selected inputs is a model deciding the outcome through the back door, in the one spot every downstream guard is structurally blind to.
top-k retrieval is an input-authority call wearing infrastructure’s costume. so is query rewriting. so is which weak labels reach a training set. and the fix is the same shape as every other fix in this argument → guard the selection, not just the decision. compare rewrite-recall against original-recall and fall back when the rewrite loses ground. make a confident topic a ranking boost rather than a hard filter, or always union in a slice of the full corpus so the right answer can’t be excluded before ranking even starts. the move is always to take the pen back from the model at the exact point it was quietly holding it, and here that point is what gets retrieved, one step upstream of everything you thought you were auditing.
put the two ends together and the trap is obvious in hindsight. the decision is the part everyone instruments, and it sits between two axes almost nobody does → what the model was allowed to look at, and what gets done with what it said. a spotless decision in the middle buys you nothing if the model picked its own evidence going in, or if its answer executes before a human ever reads it. guard the middle all you like. the ends are where it walks out the back.
which leaves me holding four axes. four is a suspiciously round, suspiciously tidy number, and this entire argument is a case against trusting tidy things that feel true just because they’re well-organized. so the last part does the only honest thing left → it turns the instrument on the taxonomy itself.