The Audit Prompt
the instrument
this is the thing i keep referring to → the prompt i pointed at my own repos to hunt down every place the model was quietly deciding something. one honest warning before you run it: it is not a neutral instrument, and it doesn’t pretend to be. it’s an opinionated taxonomy with a cover letter. the four definitions up top, auditor and judge and the two layers between them, are a point of view i authored, and everything the prompt does happens inside that frame. which is the whole series in miniature → the first thing i build is never the tool, it’s the spec, the opinion written down first. this prompt is that spec. it shows the rough edges because it came out of brainstorming and drafting rather than a polished release, and i’m handing you the draft on purpose. it works better as a thing you argue with than a thing you revere.
the reason its output is worth anything, given that a model generates it, is the single rule running through the whole thing → cite file:line for every claim, no citation, no claim. that is what keeps the audit an auditor and not a judge. the model reports where it thinks the cracks are, the citations let you walk to the exact line and see for yourself, and you dispose. so treat the verdict as a challenger, never a source of truth, and take that literally, because the instrument is not infallible. pointed at a half-built trading engine of mine, it read a wire i hadn’t run yet as a shipped confession and indicted me for a decision i hadn’t made. the file:line citations, plus a ten-second look at the lines it points at, are the entire reason to trust it. the audit doesn’t get to decide either. run it on your own repos, and disagree with it out loud.
## objective
you are auditing THIS repository against a single architectural thesis:
**opinionated taxonomy → deterministic scoring/decision → llm as auditor (never judge).**
locate each layer in the real code, and above all determine *honestly* whether the LLM is an AUDITOR (its output feeds a checkable, deterministic step) or a JUDGE (its output is itself the final, unchecked result). do NOT flatter the thesis. a clean "judge" finding is more useful than a flattering "auditor" fiction. if the model decides, say so and show exactly where.
## definitions (use these exactly)
- **opinionated taxonomy**: an author-defined point of view encoded as structure, categories, thresholds, level definitions, disqualifiers, weights. defined up front and version-controlled, NOT generated by the model at runtime.
- **deterministic decision**: the step that produces the final result from its inputs via explicit code, such that identical inputs always yield an identical result and every part of the result traces to a rule you can point at.
- **llm-as-auditor**: the model's output is an INPUT to a downstream checkable/deterministic step (a rubric mapping, a schema validation, a scored comparison). the model reports; code decides.
- **llm-as-judge**: the model's output IS the final result, with no deterministic step between the model and the thing the user acts on.
## method (do this first, before answering)
1. map the data flow from raw input to final output. list the pipeline stages in order, each tagged `[code]` or `[llm]`, with the file/function for each.
2. identify the single point where the FINAL result is determined. name file, function, line.
3. state plainly what sets that result: deterministic code, or model output.
## questions (cite `file:line` for every factual claim, or mark "not present")
**Q1 — taxonomy.** where is the opinionated taxonomy defined? is it fixed/version-controlled or model-generated per run? quote 3-4 concrete rules, thresholds, or level definitions verbatim.
**Q2 — decision locus.** trace input to output. at the point the final result is set, is it set by deterministic code or by model output? is there a deterministic mapping step *between* the model's output and the final result? show it, or show its absence.
**Q3 — VERDICT (the important one).** classify this repo as AUDITOR, JUDGE, or HYBRID, and justify with the exact call site.
- auditor = model output feeds a checkable step before it becomes the result.
- judge = model output is the final word with no deterministic check between it and the user.
- hybrid = some decisions checked, some not. list precisely which are which.
**Q4 — honesty layer.** when the LLM does heavy lifting, what constrains or validates its output? check for each and report presence/absence: enforced output schema, deterministic validation/rejection pass, retry-on-invalid, a rubric mapping model evidence to a result, a separate second-pass audit call. cite whichever exist. if NONE exist, state that explicitly.
**Q5 — reproducibility.** same input to same output? note temperature, model pinning, seeds, caching. is output persisted/snapshotted so runs are comparable over time?
**Q6 — receipt.** find ONE concrete artifact where the checkable layer caught (or would catch) a model error, OR where the taxonomy forced a non-obvious call: a test, a log line, a commit message, a code comment, an example output, a fixture. quote it and cite it. if none exists, say so and name the smallest change that would create one.
**Q7 — overreach (be adversarial).** where does the LLM currently hold MORE authority than the thesis wants? name any place it silently decides something the user acts on with no deterministic check. if the answer is genuinely "nowhere," prove it by pointing at the check that guards each model output.
## output format (return exactly this, nothing else)
```
REPO: <name>
ONE-LINE VERDICT: <auditor | judge | hybrid> — <one clause on where it bends>
PIPELINE:
<stage> [code|llm] (<file>)
<stage> [code|llm] (<file>)
...
Q1 TAXONOMY: <where, file:line> | <fixed | per-run>
- <rule/threshold/level quoted verbatim>
- <...>
- <...>
Q2 DECISION LOCUS: <file:func:line> — set by <code | model>
deterministic mapping between model output and final result? <yes/no> — <describe>
Q3 VERDICT: <auditor | judge | hybrid>
justification + call site: <...>
Q4 HONESTY LAYER: <schema | validation | retry | rubric | second-pass | NONE>
cite: <file:line>
Q5 REPRODUCIBILITY: <deterministic? temp / model pin / cache> | snapshotted: <yes/no>
Q6 RECEIPT: "<quote>" (<file:line | commit>)
Q7 OVERREACH: <the honest answer, or "nowhere: each model output guarded by <check>">
```
## rules
- cite `file:line` for every claim. no citation, no claim.
- do not invent. anything absent from the repo is reported as "not present."
- do not soften Q3 or Q7 to fit the thesis. the point of this audit is to find the cracks, not to confirm a story.