The Model Doesn't Get to Decide

Part 1 · Jul 5, 2026 · 15 min read

the decision axis

i build a lot of unrelated tools. a personal deal-finder that scores classified listings against a rulebook. a tool that reads an entire codebase and grades it. a parser that turns unstructured books into structured data. a land-intelligence platform that reads LiDAR and tells me things about dirt. on paper they share nothing, and they don’t even lean on the model to the same degree, some are almost pure rules, some are doing real llm work on every item. but every time i finish one and step back, it’s the same shape staring at me. i’ve stopped calling that a coincidence and started calling it a position.

the shape is three layers, in this order:

opinionated taxonomy → deterministic scoring → llm as auditor.

and the order is the whole argument, because it runs almost exactly backwards from where the industry is sprinting right now.

the default everyone is building

the 2026 reflex is: point the model at the problem and let it decide. score the leads with an llm. rank the deals with an llm. read the document and just ask the model what matters. the model sits in the decision seat, end to end, and the pitch is that this is the sophisticated, modern, agentic move.

it’s seductive because it stands up in an afternoon and it demos beautifully. it’s also wrong for anything you actually need to trust, repeat, or debug, for one deeply boring reason: you can’t get the same answer twice, and you can’t explain the answer you got. the instant someone asks “why did this one rank above that one,” an end-to-end llm system has no reply except a shrug wearing a confidence score.

so i invert it. the model gets the last word, and it doesn’t get the deciding word at all.

layer one: opinionated taxonomy

the first thing i build is never the scraper or the model. it’s a point of view, written down as structure. what makes a truck a good deal. what makes a parcel worth a second look. the categories, the thresholds, the disqualifiers, the things i care about and the things i deliberately refuse to.

the key word is opinionated. most systems pretend to be neutral, “data-driven,” objective, as if a ranking falls out of the numbers on its own. it never does. somebody’s judgment is always encoded in there. the only real question is whether it’s encoded legibly and on purpose or smuggled in and then denied. i’d rather put my opinion at the top, in the open, where i can argue with it and tune it next week. the opinion isn’t a flaw in the system. the opinion is the system. a deal-scorer that doesn’t encode what i think a good deal is would be worthless no matter how clever the model bolted on top of it.

layer two: deterministic scoring

then the opinion becomes math. explicit, boring, reproducible math. an item comes in, gets scored against the taxonomy, and the same input yields the same output every single time, with every point of that score traceable back to a rule i can physically point at.

this is the layer the industry is quietly embarrassed by, and it’s the one i’d surrender last. determinism buys me the three things the end-to-end model structurally cannot:

• reproducibility → same input, same score, forever. today’s run is comparable to last month’s run.
• debuggability → a score looks wrong, i walk the rules and find exactly where it went wrong, in minutes, with certainty.
• legibility → i can tell you precisely why this ranked above that, in one sentence, with numbers.

it’s also nearly free. no tokens, no latency, no drift, no model-version roulette. the deterministic core is where the actual decision lives, and i keep the decision somewhere i can see it, reproduce it, and defend it.

layer three: llm as auditor

then, and only then, the model shows up. and its job is not to decide. its job is to audit.

the deterministic scorer is rigid by design, which means it’s blind by design. it does exactly what i told it and not one thing more, so it misses whatever i didn’t think to encode, the weird listing, the context that won’t fit in a column, the thing that’s obviously off to a human and invisible to a rule. that gap is precisely where an llm is genuinely good: fuzzy judgment on the edges, catching what the rules couldn’t reach. so i point it at the already-scored output and give it a narrow brief. does anything here look mis-scored. is there a signal the rules missed. explain this ranking in plain language.

the property that makes this safe is that the llm’s output is checkable against a score i already trust. it isn’t the source of truth, it’s a challenger to a source of truth that already exists. when it disagrees, the disagreement is a flag for me to investigate, not a verdict i have to accept. it augments judgment exactly where judgment is fuzzy, and it stays out of the seat where consistency is the whole point. auditor, never judge.

the real test: i audited my own tools

i didn’t want to take my own word for whether i actually follow this pattern, so i built a prompt that audits a repo against the three layers and, more importantly, tries to disprove the thesis, to hunt down every place the model quietly decides something. then i validated it the only way that means anything → i pointed it at my own portfolio, project after project, at every stage of development, and told it not to flatter me. things that have been running in production for months, things built but not shipped, things still half-assembled on the workbench. the results were eye-opening, and far more useful than a clean pass would have been for two reasons. the first is that they turned a slogan (“the model doesn’t get to decide”) into a gradient with four rungs, and the distance between those rungs is the entire craft. the second is that the stage a project got caught at changed what its failures even meant. a gap in a tool that shipped six months ago is a lapse. the identical gap in a tool i’m still wiring together is a TODO the audit found before i did. both are worth knowing. they are emphatically not the same fact, and i’ll flag which is which as we go.

rung one, bounded → the model can only move the answer in the safe direction. the cleanest example i have is a codebase maturity grader i wrote. it scores a repo across three axes (cognitive, operational, agentic), and the model is unavoidably involved, you can’t grade a codebase’s sophistication with a regex. but its only power is to reject signals. it validates the pattern-detected signals, hands back a set of rejections, code filters those out, and then the levels are computed by deterministic decision trees over whatever survived, min-capped in code. the model literally cannot push a level higher than a deterministic pattern already found. it can only subtract. i don’t even have to trust it here, because its worst-case move is making the score more conservative. that’s the strongest form of the whole pattern: not “the model is checked” but “the model can’t move the number the dangerous way in the first place.”

rung two, checked → the model proposes freely, and code holds a veto with teeth. a book-parser i built does this for character identity. when the model reconciles two references to what might be the same character, its answer is accepted only if it names a member of the existing candidate set, and event types get coerced to a fixed enum or dropped. the model gets to talk. code gets to say no. the veto is real and it fires against a fixed universe the model doesn’t control.

rung three, laundered → a deterministic function fed unchecked model output. this is a judge wearing an auditor’s robe, and it fooled me. the maturity grader also produces a “fit” verdict, is this repo under-built, appropriately built, or over-built for its intent. that verdict looks airtight: it’s pure code comparing two numbers. one of those numbers comes from the trustworthy bounded vector above. but the other number derives from a handful of “intent” values that are free model output, checked only for shape, never for correctness. there’s a deterministic prior meant to seed them, and the model overrides it at will. so the verdict is a deterministic function of one input i can defend and one unchecked model judgment. the determinism is real and cosmetic, because the actual decision quietly migrated one step upstream to a spot where nothing guards it. and here’s the part worth the whole series: this is more dangerous than an honest judge. when the book-parser ships a model-written scene summary, it announces itself as model judgment and i scrutinize it. the fit verdict launders a model judgment through a deterministic-looking function and earns trust it never paid for. i believed that verdict more than i’d earned the right to.

rung four, naked → the model’s output is the artifact, no deterministic step between it and me. and this rung has two tenants i’d lazily filed together until the audit split them:

irreducible, which is legitimate. a scene’s summary, its mood, its time-of-day. you cannot write a deterministic gate for “is this the right read of the scene’s mood” without just reimplementing the model. when the judgment is genuinely subjective, an honest naked judge is the correct design, and bolting on a fake check to rubber-stamp it would be worse than nothing.

reducible-but-unwired, which is a bug i’d been calling a design. the book-parser’s duplicate-merge verdict is the model’s yes-or-no, final and permanent. and the maddening part is that i had already written the deterministic rule that “Carl” and “Carl’s father” are not the same entity. it just sits upstream, filtering which pairs get sent to the model, instead of sitting at the decision with the power to veto the model’s answer. the guard exists. it’s in the wrong place.

and here’s the move that shrinks “irreducible” down to its true size, because it’s usually smaller than it looks. even when the judgment genuinely can’t be gated, the facts the judgment rests on almost always can. a model that emits a recommendation is making one call you can’t check (is this the right pick) wrapped around several you can (do the things it cites actually exist, are the numbers real, are the links real). the discipline is to decompose the naked output and gate the factual substrate while letting the opinion run free. the failure i keep finding is shipping the whole thing naked because part of it was irreducible, and letting a checkable, hallucinated fact ride out of the building on the back of an uncheckable, perfectly legitimate opinion.

the things auditing myself actually taught me

the same instrument over a handful of unrelated codebases surfaced the same failures in each one, which is how i know they belong to me and not to the domain. the cleanest proof of that came from a tool where two model outputs came back from the same call and i’d gated one and shipped the other raw. same function, same model, same domain, opposite discipline. it isn’t the problem that decides whether the model gets to judge. it’s me, output by output.

one: my determinism kept showing up as a prior when it needed to be a veto. the book-parser knows family members aren’t the same person, and that knowledge filters candidates instead of vetoing the merge. the maturity grader computes an intent prior, and the model overrides it freely. in both, the deterministic opinion is positioned upstream of the decision as a suggestion, so the model still gets the last word, and a model with the last word is a judge no matter how much code is standing around it. the fix is identical in both places, and it’s the whole argument compressed to one line → put the deterministic opinion at the decision point with override power, not upstream of it as a seed. prior-versus-veto is the entire difference between an auditor and a judge in a costume.

two: i wired my guards onto the paths that least needed them. the tools gate the structured path heavily and leave the free path open. one grades a repo and runs pattern-derived findings through a confirm-or-dismiss gate, then lets the model’s net-new “semantic” findings through ungated. the book-parser validates IDs and positions rigorously and ships model prose untouched. it’s backwards. the path where the model has the most freedom is the path that most needs a gate, and it’s exactly the one where writing the gate is hardest, so it’s the one i skipped. the discipline decayed precisely where it mattered most.

three: when i couldn’t write a rule, i wrote nothing, where a bound would have done. the places the model runs free are the places where i couldn’t state the correct answer in code, what’s the right score adjustment, what’s the right rating out of ten. but “i can’t compute the right number” is not the same as “i can’t constrain the wrong ones,” and i kept treating them as if they were. i don’t need a rubric that reproduces the model’s judgment. i need a bound that caps how far its judgment can move the result, a ceiling on the adjustment, a requirement that it cite a reason from a fixed list, a check that the size of the change is proportional to the evidence for it. a bound is drastically cheaper than a rule and usually enough, and it’s the exact thing i skip, because the moment i decide i can’t write the perfect check i quietly write no check. rung one, the bounded model that can only move the number the safe way, was never reserved for the easy cases. it’s available almost everywhere i currently have a naked judge, the second i’m willing to settle for constraining the model instead of replacing it.

four: i built the check, wired it to the right place, and then turned it off with a comment. this is the one that stings. one of my generators runs a deterministic validator that catches fabricated citations, usernames and permalinks that don’t exist in the source data. it runs on every document. it correctly returns invalid when the model invents an attribution. and then the very next step ships the document anyway, with a line sitting in version control that amounts to continue anyway, it’s still usable. the guard is present, correctly placed, and firing, and i overruled it in advance. that is worse than the misplaced guard and worse than the missing one, because a missing guard is ignorance and a misplaced guard is a mistake, but a disarmed guard is a decision, and i left mine in a comment where it could indict me later. note the exact shape of the gap too: i enforce citation presence with a retry and disarm citation correctness. so i verify that the model cited something and decline to verify that the something is real. a fabricated source passes the presence gate, gets caught by the correctness gate, and the correctness gate’s verdict is thrown on the floor. across my repos the checkable layer fails in three escalating ways → absent from the path that actually runs, present but upstream of the decision, and present-firing-and-ignored. only the third one isn’t an accident.

five: my error handling laundered failure into success. in that same generator, when the model failed to assign an item to a category, the fallback quietly dropped it into the first subcategory of the first bucket and moved on. so a failure to classify renders in the output as a confidently-placed item, indistinguishable from a real assignment. this is the sneakiest bug in any of the repos, because it never surfaces as an error, it surfaces as a plausible result. every other failure mode at least leaves a mark → a bad score, a wrong merge, a dead link. this one erases its own evidence. the error handler is supposed to be the place where you catch the model failing; i’d built one that disguises it. a fallback that turns “i don’t know” into a specific, confident, wrong answer is not a safety net. it’s a laundromat.

none of those five is a reason to distrust the pattern. they’re the pattern telling me, in my own code, exactly where i stopped applying it. that’s what a good auditor is supposed to do, and it’s a little funny that i had to build one to point it at myself.

and when the problem genuinely needs the model, when it’s too fuzzy to rules-engine your way around and you can’t just write the taxonomy as a lookup, there’s a move stronger than any runtime guard: capture the judgment offline and compile it into something deterministic, so the model isn’t in the room when the decision happens at all. one of my cleanest tools does exactly this. it has to classify messy owner-name strings into a fixed set of types, is “Erma Banks” a person or a bank, which is real fuzzy judgment no regex settles cleanly. so the model does that judgment once, offline, generating training labels. a deterministic map normalizes those labels to the fixed categories. a plain classifier trains on them. and at runtime the model never runs → the prediction is frozen sklearn, fully reproducible, and the model’s contribution survives only as weights it can no longer move. that’s the auditor pattern taken to its limit: the model’s judgment is present in the artifact and absent from the decision. it’s also the hardest kind of clean to earn, because unlike a threshold engine the problem actually required the model, and you got it out of the loop anyway by paying the cost up front. distillation is what “the model doesn’t get to decide” looks like when the model was genuinely necessary.

all of that assumed one thing i never said out loud → that the frame the decision happens inside was mine to begin with. the next part is about the tool that didn’t get the memo.