I Asked Three Rival Models to Kill My Spec

Aug 1, 2026 · 8 min read

a sequel of sorts to the model doesn’t get to decide: same posture, applied one layer earlier, before the system exists at all.

i’m building a CRM. i know, i know. but there’s a straight line to it from the rest of my work: i test enterprise CRM systems for a living, and SF-Assistant exists because i wanted to pull the business rules buried in an org’s metadata out into plain language. once you can do that, the question kind of asks itself: if the spec can be extracted, what would you rebuild it on? aicrm is my attempt at an answer, and the long game, somewhere down the road, is exactly that pipeline: point SF-Assistant at an org, get a full plain-language breakdown of what it actually does, and rebuild it on something like this.

the thing itself is basically my whole blog thesis turned into a product: a business system that lives in one sqlite file, where the configuration is plain text instead of forty setup screens, and where nobody, human or AI, ever just changes the system. an AI drafts your change, a deterministic rulebook validates it, a second model plays devil’s advocate, and a human approves it with the access implications spelled out in writing. receipts for everything.

but this post isn’t about the CRM. it’s about what happened before i wrote any code for it, because i did something i’ve wanted to do properly for a while: i treated the spec itself as the first artifact under test.

specs are where bugs are cheapest

i’ve spent my career in QA, and the thing QA teaches you that nobody wants to hear is that the expensive bugs were all cheap once. they were cheap when they were sentences. a contradiction between section 6 and section 9 costs nothing to fix in a document and costs a rewrite once both sections have shipped as code. everybody nods at this and then nobody actually reviews specs with anything like the rigor they apply to a pull request, i think because review is boring and specs are long and the author is usually the only person who cares.

so the question i wanted to answer: can you get pull-request-grade adversarial review on a design document, before the design has any code to defend it, using models as the reviewers? not models as oracles, models as hostile readers. the distinction matters and the whole experiment hangs on it.

the setup was a fixed pipeline, and i want to lay it out plainly because i’d reuse it for anything spec-shaped:

1. draft. every decision made explicitly, marked decided or open, rationale inline. no “we’ll figure that out later” without a name on the gate where later happens.
2. in-context adversarial pass. the drafting party attacks its own text before anyone else sees it. this found the internal contradictions and the philosophy-level holes, the stuff you can see from inside.
3. revision, with a punch list mapping every finding to its fix.
4. cold-context reviews, three of them, sequentially. the spec goes to a reviewer with no conversation history and nothing invested in it. i used three frontier models from three different vendors. cold context is the entire point: a reviewer who shared the drafting context would share the drafting assumptions, and shared assumptions are exactly where spec bugs live.
5. disposition, per pass. every finding gets explicitly accepted, adapted with the divergence argued, or deferred to a named gate. nothing is dropped silently. even the misfires get recorded with their correction, and i’ll come back to why, because that rule paid for itself in a way i didn’t expect.
6. fix replay. every fix gets walked through every section that references the behavior it changed, before the revision ships. i did not design this rule. i learned it, the hard way, on pass three.
7. a freeze criterion. stop when a pass’s most severe findings are about the fix process itself rather than the architecture. at that point review costs more than it catches, and the next reviewer should be executable code.

four passes. twenty-five recorded fixes. the spec froze at v1.0 with zero open decisions, and its closing line is the honest summary: the verbs are in writing, and the writing survived four assassins.

what each assassin was good at

the three cold reviewers were not interchangeable, and the differences turned out to be the most interesting data in the whole exercise.

the first cold review was the heavyweight one on paper, and it delivered: seven blocking findings, including one that quietly invalidated a piece of the architecture before it existed. part of the design leaned on parameterized sqlite views, and the reviewer pointed out that sqlite does not allow parameters in view definitions. that is not an opinion or a design taste, that is a fact about the ground, checkable in thirty seconds, and it was sitting under a frozen-looking decision. the same pass forced me to pin null-handling semantics for every context an expression can run in, which later became an entire compiler phase with its own truth-table tests. i’ve started calling this executable-truth reviewing: findings that verify claims against reality rather than against vibes. highest value per finding of anything in the process.

the second reviewer was the confident one. five findings, and two of them were misfires, fast persuasive exploit narratives that fell apart when i actually traced them through the spec. here’s the part i want to be honest about, because it’s the part that makes people give up on model review: a plausibility-style reviewer will absolutely hand you a wrong exploit sketch with total confidence. if my process had been “believe the reviewer,” i’d have patched phantom holes and made the document worse. but the disposition rule, nothing dropped silently, misfires recorded with their correction, turned even the wrong findings into value: every single one of the five, including the two misfires, marked a place where the spec had no written policy. the exploit was wrong; the unwritten policy it tripped over was real. two paragraphs of correction bought five paragraphs of policy that didn’t exist before.

the third reviewer got the document last, after two rounds of fixes, and found twenty things, of which the four most severe were all in the fixes. one fix had relaxed a concurrency rule in a way that contradicted the approval-gate binding two sections away. another had shipped an error-semantics table that, read carefully, pinned approval gates to fail open. fail open. in the system whose one-sentence pitch is “when unsure, get more careful, not less.” those bugs were not in the architecture, they were in the patches, which is how i learned the replay rule: fixes are code too. a fix that is locally sound and globally unchecked is exactly a hotfix merged without running the suite, and it fails the same way.

and one more thing about that third pass, because it’s the detail i keep chewing on: the drafting party and the adjudicator across all these rounds was claude, working with me in a long-running session. the third cold reviewer was also claude, same family, cold context. it caught the in-context adjudicator’s own shipped errors, the ones neither of us could see from inside. same model, different context, different findings. whatever is doing the work here, it is not the vendor. it’s the epistemics: cold context sees what shared context can’t, the way a developer can’t proofread their own code five minutes after writing it.

the convergence curve

the finding classes descended in an orderly way that i did not plan and was a little spooked to watch: pass one found architecture holes. pass two found authorization semantics. pass three found operational edge policy. pass four found problems the earlier fixes had introduced. when the review process starts finding bugs mostly in its own output, you’re done, that’s the freeze criterion firing, and pushing past it would just be paying reviewers to disagree about wording.

i want to resist the triumphant version of this story, because the process has real weaknesses and i watched them. the panel is only as good as the disposition discipline; a lazy adjudicator with the same three reviewers would have shipped phantom fixes for confident misfires and skipped the replay that catches patch-on-patch contradictions. and the adjudicator is fallible in a way the process has to assume rather than hope against, which is the whole reason the pipeline ends with a criterion that fires it and hands the job to something executable. the safety isn’t in any participant, model or human. it’s in the system: panel plus disposition plus, eventually, fixtures.

which is where it stands now. the spec’s last line before the appendix says the next reviewer is code, and that turned out to be true: the config format parser and the expression compiler exist, built test-first against the frozen document, and the fixture corpus is slowly becoming what the review panel was, a hostile reader that never gets tired. the highest-yield lesson of the whole exercise came from exactly that direction, too: in every single pass, the most reliable finding class was the spec’s own examples violating the spec’s own rules. an example automation wrote to a field no object declared. the flagship example object omitted a marking its own permission model required. examples are executable claims. treat them as the first fixture and validate them like one, because your reviewers will, and they’ll be right.

the model still doesn’t get to decide. but it turns out four of them, kept cold, kept sequential, and kept on a disposition leash, make a hell of a review panel.