RepoAudit (Multi-Dimensional Code Intelligence)

Role: Solo, end-to-end. Architecture, analysis engine, CLI, reporting
Tech Stack: Python 3.10+, Click, Tree-sitter, Anthropic/OpenAI/Gemini, Pydantic, Rich
Repository: Private. Demo available on request

Overview

RepoAudit is a multi-profile code analysis tool that scans repositories across six dimensions (AI maturity, security posture, architecture quality, documentation coverage, portfolio readiness, and auto-generated diagrams) and produces structured reports with optional LLM validation.

Most static analysis tools answer one question. RepoAudit answers six at once, and it does it deterministic-first: every analysis works without an API key. LLM validation is progressive enhancement, not a requirement. That means the tool runs just as reliably in CI/CD pipelines, air-gapped environments, and on local dev machines.

It operates in two modes: developer local (scan any directory) and CI/CD central (analyze multiple GitHub repositories on a schedule via GitHub Actions, up to 10 concurrent workers). Reports come out as Markdown, JSON, or interactive tabbed HTML dashboards.

I run it against its own codebase regularly. The most recent self-scan rated it S4/D4 (Well-Structured / Polished) on architecture, S3 (Solid) on security, and C4/Q4 (Comprehensive / Excellent) on documentation. Full 6-profile scan with LLM validation costs about $3.79 total.

System Architecture

RepoAudit follows a pipeline architecture with three main layers: Interfaces (CLI and local scanner for local analysis, Runner and GitHub client for remote repos), Core (scanner extracts signals using tree-sitter, classifier determines maturity levels, reporter generates outputs), and LLM (router selects models, validator confirms signals, analyzer provides reasoning). The scanner uses tree-sitter for syntax-aware pattern detection, and the LLM layer is entirely optional: deterministic classification works without it, but validation and deep analysis benefit from it.

graph TD
    subgraph "Extraction Layer"
    A[Target Repository] --> B{Tree-sitter Parser}
    A --> C{Regex Fallback Scanner}
    A --> S[Skeleton + Core Files]
    B -->|AST-aware| D[Cognitive / Operational / Agentic Signals]
    C --> D
    end

    subgraph "Validation Layer"
    D --> F{Profile Filter}
    F -->|Deterministic| G[Classification Engine]
    F -->|LLM-Enabled| H[Batch Validator]
    S --> P[Layered Prompt Cache: skeleton, core, rubric]
    P -->|Anthropic/OpenAI/Gemini| H
    H -->|Two-turn context requests| J[Validated Signals]
    J --> G
    end

    subgraph "Fit Layer"
    G --> K[Tri-Axis Levels C0-C5 / O0-O5 / A0-A5]
    D --> I[Intent Drivers]
    I --> Q[Intent Audit + Target Bands]
    K --> V{Per-Axis Fit Verdict}
    Q --> V
    end

    subgraph "Output Layer"
    V -->|under-built / appropriate / over-built| L[Markdown Reports]
    K --> L
    K --> M[JSON Scores]
    K --> N[HTML Dashboards]
    V --> N
    end

Components

Engineering Details

1. Tri-Axis AI Maturity Classification

Existing tools classify AI integration as binary: uses AI or doesn’t. Real projects exist on a spectrum though, and a single-shot API call is fundamentally different from a self-healing agentic loop. Treating them as the same category isn’t useful to anyone. So I designed a tri-axis classification model that evaluates AI sophistication across three independent dimensions.

Cognitive Axis (C0-C5): How the system thinks.

LevelNameExample
C0DeterministicPure logic, no AI
C1ReactiveSingle-shot LLM call
C2InstrumentalTool/function calling chains
C3Adaptive PlannerFeedback loops; output affects next input
C4Reflective SynthesizerSelf-critique and refinement cycles
C5Self-EvolvingGenerates its own tools at runtime

Operational Axis (O0-O5): How the system survives.

LevelNameExample
O0EphemeralLocal/manual only
O1ScriptedPortable (Docker, requirements.txt)
O2Event-DrivenCI/CD integrated
O3Bounded LoopsFault tolerance, retries, circuit breakers
O4PersistentDurable state across runs
O5GovernedBudget enforcement, audit trails, permissions

Agentic Axis (A0-A5): How the system acts.

LevelNameExample
A0StatelessSingle-shot call, result used or discarded
A1ResponsiveGenerates persistent artifacts for later use
A2ConversationalMulti-turn interactions with maintained context
A3Task-OrientedSelects its own tools and strategies to hit a goal
A4Autonomous AgentSelf-directed loops, cross-session memory, planning
A5Self-GoverningSpawns sub-agents, manages its own budget

High-level signals (C3+/O3+/A3+) require LLM validation because distinguishing real implementation from test fixtures, comments, or superficial patterns takes semantic understanding that pattern matching can’t do. Deterministic analysis caps at C2/O2/A2, honest about what it can and can’t prove.

The levels then get graded for fit, not just altitude. Deterministic intent drivers plus an LLM intent audit derive a target band per axis, and each axis gets a three-outcome verdict: under-built, appropriate, or over-built. A weekend script scoring C1/O1/A0 can be a better piece of engineering than a C4 system that never needed to be one, and the report says so.

2. Hybrid Signal Detection with Tree-sitter

Regex-based pattern matching for AI usage signals produced way too many false positives from code examples in comments, string literals, and test fixtures, making deterministic classification unreliable. So I implemented syntax-aware extraction using tree-sitter parsers with 100+ language-specific queries (Python, JavaScript/TypeScript, Java).

Each language has a dedicated query definition file with queries organized by signal type and confidence level. Each query includes refinement callbacks that check AST context, so the system can distinguish a real feedback loop from one mentioned in a docstring. The system walks parent nodes to exclude matches inside string or comment AST nodes and extracts full function bodies for proper context.

3. Iterative LLM Validation with Context Requests

Single-pass LLM validation often lacks sufficient context to confirm high-level signals. A C3+ feedback loop detection might match a code snippet that doesn’t show the full pattern. So I built a two-turn validation system where the LLM can request additional context after initial analysis.

Increased C3+ validation accuracy from 72% to 91% on reference projects. Reduced per-repo validation cost by 40% through batching and context reuse.

4. Multi-Phase Async Security Analysis

Security scanning spans multiple independent dimensions: code patterns, dependency vulnerabilities, architecture exposure, and endpoint reachability all matter, and none of them feed into each other. So I built a four-phase parallel pipeline with semaphore-based throttling.

The distinctive piece here is exposure classification. The scanner traces call graphs to determine whether a vulnerability is directly_exposed (reachable from an HTTP endpoint), indirectly_exposed, or internal. A SQL injection in a utility function called from nowhere is very different from one sitting behind a public API route, and I didn’t want both showing up with the same severity.

Above the four phases sits a deep-verify jury that gates high-severity findings before they reach the report. Each candidate is re-examined with refute-first prompting: the model is instructed to argue the finding isn’t real and to keep it only if it can’t. For High and Critical severity I run several independent jurors and take consensus, enriching each candidate with on-demand data flow (resolving the enclosing function, class, and callers only for findings that reach this stage, not for every hit). This is what clears the false positives that survive pattern matching, the documentation examples, test fixtures, and already-mitigated patterns, without paying for deep analysis on low-severity noise. On the last self-scan it recalibrated or dismissed 7 findings that the earlier phases had flagged, including a Critical that was really a deliberate design choice.

5. Profile-Driven Analysis

Different stakeholders care about different things. A security team wants vulnerability findings. A hiring manager wants architecture quality. A tech lead wants documentation gaps. Rather than try to serve all of them at once, I built a YAML-based profile system where each profile controls the entire analysis pipeline.

name: security
display_name: Security Audit
analysis_mode: security
cognitive_filter:
  min_confidence: 0.7
classification:
  require_validation_for_c3_plus: true

6. Central Model Registry and Cost Control

Supporting multiple LLM providers with different pricing, context limits, and capabilities is a maintenance burden, and cost tracking gets messy fast without a central source of truth. So I built a single-source-of-truth model registry with integrated budget enforcement.

The tool never surprises you with a bill.

7. Layered Prompt Caching

The LLM-first profiles (portfolio, architecture, docs, diagrams, security) all re-send the same repository skeleton and core files on every call. That is one large prefix paid for six times over. My first fix was to warm the cache with one profile, then fire the rest in parallel off that warm prefix. It fell apart on large repos, where the warm profile alone took longer than Anthropic’s 5-minute cache TTL, so the cache had already expired before the parallel calls started.

So I redesigned it as a three-layer cached prefix, skeleton then core then rubric, with byte-stable rendering so the prefix hashes identically on every call. The skeleton (file tree plus signatures) and the core (a set of high-centrality files) are built once, primed with cheap max_tokens=1 calls per model, then reused across every profile. Prime the shared layers once and burst, instead of warming through a whole profile first.

Code Snippet: Security Exposure Classification

This is how the security scanner determines whether a detected vulnerability is actually reachable from an attack surface.

class ExposureClassifier:
    """Traces call graphs to classify vulnerability exposure."""

    def classify(self, signal: SecuritySignal,
                 endpoints: list[Endpoint],
                 call_graph: CallGraph) -> ExposureLevel:
        # Check if the signal's location is directly in an endpoint handler
        if self._is_endpoint_handler(signal.file_path, signal.line, endpoints):
            return ExposureLevel.DIRECTLY_EXPOSED

        # Trace backwards through call graph from known endpoints
        for endpoint in endpoints:
            if call_graph.is_reachable(
                source=endpoint.handler,
                target=signal.function_name,
                max_depth=5
            ):
                return ExposureLevel.INDIRECTLY_EXPOSED

        return ExposureLevel.INTERNAL

CLI Interface

RepoAudit provides both local and CI/CD modes through a Click-based CLI.

# Local developer mode - scan any directory
repoaudit scan ./my-repo                                    # Default: AI maturity
repoaudit scan ./my-repo --profile security --validate      # Security audit with LLM
repoaudit scan ./my-repo --profile architecture --validate  # Architecture analysis
repoaudit scan ./my-repo --profile docs-audit --validate    # Documentation gaps
repoaudit scan ./my-repo --profile diagrams --validate      # Auto-generate Mermaid diagrams
repoaudit full-scan ./my-repo                               # All profiles in parallel

# CI/CD mode - analyze multiple repos via GitHub API
repoaudit central-run --config runner/config.yml
repoaudit status                                            # Dashboard of all tracked repos

# Options
--model anthropic|openai|gemini     # Choose LLM provider
--method treesitter|regex|both     # Extraction method
--format json|markdown|both        # Output format
--render-mode svg|js|both          # Diagram rendering

Features

Tech Stack