RepoAudit (Multi-Dimensional Code Intelligence)

Role: Solo, end-to-end. Architecture, analysis engine, CLI, reporting
Tech Stack: Python 3.10+, Click, Tree-sitter, Anthropic/OpenAI, Pydantic, Rich
Repository: Private. Demo available on request

Overview

RepoAudit is a multi-profile code analysis tool that scans repositories across six dimensions (AI maturity, security posture, architecture quality, documentation coverage, portfolio readiness, and auto-generated diagrams) and produces structured reports with optional LLM validation.

Most static analysis tools answer one question. RepoAudit answers six at once, and it’s deterministic-first where the analysis allows it: AI-maturity classification is fully deterministic and the security scan has a free deterministic tier, both usable with no API key at all. The LLM-first profiles (architecture, docs, portfolio, diagrams) need a key. The deterministic pieces run just as reliably in CI/CD pipelines, air-gapped environments, and on local dev machines.

It operates in two modes: developer local (scan any directory) and CI/CD central (analyze multiple GitHub repositories on a schedule via GitHub Actions, up to 10 concurrent workers). Reports come out as Markdown, JSON, or interactive tabbed HTML dashboards.

I run it against its own codebase regularly. The most recent self-scan rated it S4/D4 (Well-Structured / Polished) on architecture, C4/Q4 (Comprehensive / Excellent) on documentation, and S2 (Basic) on security, which handed me a prioritized fix list. That last grade stings, but a tool that flatters its own author isn’t much of an auditor. The full 6-profile scan with LLM validation cost about $4.74.

System Architecture

RepoAudit follows a pipeline architecture with three main layers: Interfaces (CLI and local scanner for local analysis, Runner and GitHub client for remote repos), Core (scanner extracts signals using tree-sitter, classifier determines maturity levels, reporter generates outputs), and LLM (router selects models, validator confirms signals, analyzer provides reasoning). The scanner uses tree-sitter for syntax-aware pattern detection, and the LLM layer is entirely optional: deterministic classification works without it, but validation and deep analysis benefit from it.

graph TD
    subgraph "Extraction Layer"
    A[Target Repository] --> B{Tree-sitter Parser}
    A --> C{Regex Fallback Scanner}
    A --> S[Skeleton + Core Files]
    B -->|AST-aware| D[Cognitive / Operational / Agentic Signals]
    C --> D
    end

    subgraph "Validation Layer"
    D --> F{Profile Filter}
    F -->|Deterministic| G[Classification Engine]
    F -->|LLM-Enabled| H[Batch Validator]
    S --> P[Layered Prompt Cache: skeleton, core]
    P -->|Anthropic/OpenAI| H
    H -->|Two-turn context requests| J[Validated Signals]
    J --> G
    end

    subgraph "Fit Layer (in development)"
    G --> K[Tri-Axis Levels C0-C5 / O0-O5 / A0-A5]
    D --> I[Intent Drivers]
    I --> Q[Intent Audit + Target Bands]
    K --> V{Per-Axis Fit Verdict}
    Q --> V
    end

    subgraph "Output Layer"
    V -->|under-built / appropriate / over-built| L[Markdown Reports]
    K --> L
    K --> M[JSON Scores]
    K --> N[HTML Dashboards]
    V --> N
    end

Components

• Profile System: six built-in analysis profiles (AI maturity, security, architecture, docs audit, portfolio, diagrams), each defined in YAML with signal filtering, classification overrides, and output customization. Custom profiles supported. Filtering happens at query selection time rather than post-processing, which means the scanner skips unnecessary AST traversal entirely for profiles that don’t need it.
• Tree-sitter Extraction: 100+ language-specific queries across Python, JavaScript/TypeScript, and Java. Each query includes refinement callbacks that check AST context (e.g., is_llm_feedback_loop verifies while loops contain LLM calls with result-dependent exits). The system walks parent nodes to exclude matches inside string or comment AST nodes, and extracts full function bodies instead of just line snippets for proper context. Cut the false positive rate from roughly 40% (regex only) to under 10%, my own estimate against reference projects, not a formal benchmark.
• LLM Provider Abstraction: unified interface across Anthropic and OpenAI with a central model registry tracking pricing, context limits, and capabilities (Gemini is registered and has a provider stub, but isn’t routable from the CLI yet). Deterministic routing with configurable strategies (balanced_fallback, prefer_cheap, fixed) and fallback chains.
• Security Scanner: multi-phase async analysis with pattern detection, dependency CVE scanning (OSV API, free, no auth), endpoint/attack surface classification, and call-graph tracing to determine whether vulnerabilities are actually reachable from HTTP endpoints.
• Token Metering: budget enforcement at four scopes: per-call (prevent runaway single requests), per-repo (cap total spend on one analysis), per-run (cap a CI/CD batch job), and monthly (prevent surprise bills). Conservative pre-call estimation (char_count/4 * 1.15 safety factor) and authoritative post-call metering from provider responses. Every decision logged to JSON ledgers with reason codes.

Engineering Details

1. Tri-Axis AI Maturity Classification

Existing tools classify AI integration as binary: uses AI or doesn’t. Real projects exist on a spectrum though, and a single-shot API call is fundamentally different from a self-healing agentic loop. Treating them as the same category isn’t useful to anyone. So I designed a tri-axis classification model that evaluates AI sophistication across three independent dimensions.

Cognitive Axis (C0-C5): How the system thinks.

Operational Axis (O0-O5): How the system survives.

Agentic Axis (A0-A5): How the system acts.

High-level signals (C3+/O3+/A3+) require LLM validation because distinguishing real implementation from test fixtures, comments, or superficial patterns takes semantic understanding that pattern matching can’t do. Deterministic analysis caps at C2/O2/A2, honest about what it can and can’t prove.

The levels then get graded for fit, not just altitude. Deterministic intent drivers plus an LLM intent audit derive a target band per axis, and each axis gets a three-outcome verdict: under-built, appropriate, or over-built. A weekend script scoring C1/O1/A0 can be a better piece of engineering than a C4 system that never needed to be one, and the report says so. (This fit-grading layer is in development on a feature branch; current builds report the tri-axis levels, and the target-band verdicts land when it merges.)

2. Hybrid Signal Detection with Tree-sitter

Regex-based pattern matching for AI usage signals produced way too many false positives from code examples in comments, string literals, and test fixtures, making deterministic classification unreliable. So I implemented syntax-aware extraction using tree-sitter parsers with 100+ language-specific queries (Python, JavaScript/TypeScript, Java).

Each language has a dedicated query definition file with queries organized by signal type and confidence level. Each query includes refinement callbacks that check AST context, so the system can distinguish a real feedback loop from one mentioned in a docstring. The system walks parent nodes to exclude matches inside string or comment AST nodes and extracts full function bodies for proper context.

• Cut the false positive rate from roughly 40% to under 10% against reference projects (my estimate from spot-checking, not a formal benchmark).
• Regex remains as fallback for unsupported languages, but for the three supported ones, tree-sitter is the primary extractor.

3. Iterative LLM Validation with Context Requests

Single-pass LLM validation often lacks sufficient context to confirm high-level signals. A C3+ feedback loop detection might match a code snippet that doesn’t show the full pattern. So I built a two-turn validation system where the LLM can request additional context after initial analysis.

• Context extractors use language-specific AST traversal (Python’s stdlib ast, JavaScript regex-based) to extract full function bodies, imports, and enclosing class context.
• Signals are batched by file to minimize redundant file fetches, with parallel batch processing (max 3 concurrent batches).
• The LLM sees the initial signal, and if it needs more information, it can request specific functions, classes, or caller locations before making a determination.

The numbers I can actually stand behind: in a validated pilot I wrote down predictions for five reference repos before running the validator, every prediction was confirmed, and the whole pilot cost $0.27 in API spend. Batching and context reuse are what keep the per-repo cost that low.

4. Multi-Phase Async Security Analysis

Security scanning spans multiple independent dimensions: code patterns, dependency vulnerabilities, architecture exposure, and endpoint reachability all matter, and none of them feed into each other. So I built a four-phase parallel pipeline with semaphore-based throttling.

• Phase 1 (Signal Validation): batched LLM validation of detected security patterns (SQL injection, XSS, path traversal, hardcoded secrets, insecure crypto). 50+ deterministic vulnerability patterns with OWASP/CWE mapping.
• Phase 2 (Dependency Scanning): parses manifests (requirements.txt, package.json, Cargo.toml, go.mod) and queries the OSV API for known CVEs. Free, no auth required.
• Phase 3 (Architectural Analysis): classifies code structure (MVC, event-driven, microservices) and analyzes security posture based on architecture patterns.
• Phase 4 (Rating Synthesis): aggregates findings with OWASP Top 10 and CWE mappings, severity ratings, and prioritized remediation recommendations.

The distinctive piece here is exposure classification. The scanner traces call graphs to determine whether a vulnerability is directly_exposed (reachable from an HTTP endpoint), indirectly_exposed, or internal. A SQL injection in a utility function called from nowhere is very different from one sitting behind a public API route, and I didn’t want both showing up with the same severity.

Above the four phases sits a deep-verify jury that gates high-severity findings before they reach the report. Each candidate is re-examined with refute-first prompting: the model is instructed to argue the finding isn’t real and to keep it only if it can’t. For High and Critical severity I run several independent jurors and take consensus, enriching each candidate with on-demand data flow (resolving the enclosing function, class, and callers only for findings that reach this stage, not for every hit). This is what clears the false positives that survive pattern matching, the documentation examples, test fixtures, and already-mitigated patterns, without paying for deep analysis on low-severity noise. On the last self-scan it recalibrated or dismissed 7 findings that the earlier phases had flagged, including a Critical that was really a deliberate design choice.

5. Profile-Driven Analysis

Different stakeholders care about different things. A security team wants vulnerability findings. A hiring manager wants architecture quality. A tech lead wants documentation gaps. Rather than try to serve all of them at once, I built a YAML-based profile system where each profile controls the entire analysis pipeline.

name: security
display_name: Security Audit
analysis_mode: security
cognitive_filter:
  min_confidence: 0.7
classification:
  require_validation_for_c3_plus: true

• Profiles control which signals are extracted, how they’re filtered, whether LLM validation is required, and what output format gets generated. Filtering at query selection time means portfolio scans finish in a fraction of the time of a full scan by skipping C3+ queries entirely.
• The full-scan command runs all profiles in parallel via ThreadPoolExecutor, producing a tabbed HTML dashboard with each dimension as a separate view. Parallel repo analysis supports up to 10 concurrent workers.
• Custom profiles can override any built-in behavior: signal thresholds, classification caps, output templates.

6. Central Model Registry and Cost Control

Supporting multiple LLM providers with different pricing, context limits, and capabilities is a maintenance burden, and cost tracking gets messy fast without a central source of truth. So I built a single-source-of-truth model registry with integrated budget enforcement.

• Every supported model (Claude, GPT, Gemini variants) is registered with pricing per million tokens, context window size, and capability flags.
• Token estimation uses a character-based heuristic with a 1.15x safety factor, fast enough for real-time budget checks without requiring a tokenizer. Authoritative post-call metering from provider responses for accurate cost tracking.
• Budgets enforce at four scopes: per-call, per-repo, per-run, and monthly. Monthly enforcement prevents surprise bills across multiple runs.
• Graceful degradation strategies are configurable: skip_component, choose_cheaper, or fail_run.
• All LLM interactions logged to JSON ledgers with reason codes for auditability.

The tool never surprises you with a bill.

7. Layered Prompt Caching

The LLM-first profiles (portfolio, architecture, docs, diagrams, security) all re-send the same repository skeleton and core files on every call. That is one large prefix paid for six times over. My first fix was to warm the cache with one profile, then fire the rest in parallel off that warm prefix. It fell apart on large repos, where the warm profile alone took longer than Anthropic’s 5-minute cache TTL, so the cache had already expired before the parallel calls started.

So I redesigned it as a layered cached prefix, skeleton then core, with byte-stable rendering so the prefix hashes identically on every call. The skeleton (file tree plus signatures) and the core (a set of high-centrality files) are built once, primed with cheap max_tokens=1 calls per model, then reused across every profile; each profile’s rubric rides on top, uncached, since it differs per profile. Prime the shared layers once and burst, instead of warming through a whole profile first.

• Cut full-scan input cost by 22% on the self-scan, with 238K cache reads against 44K writes.
• Anthropic-specific, since the cache_control mechanism is provider-specific. OpenAI and Gemini get functional parity but pay full input price on repeated context.

Code Snippet: Security Exposure Classification

Simplified from the real classifier for readability; this is the logic for determining whether a detected vulnerability is actually reachable from an attack surface.

class ExposureClassifier:
    """Traces call graphs to classify vulnerability exposure."""

    def classify(self, signal: SecuritySignal,
                 endpoints: list[Endpoint],
                 call_graph: CallGraph) -> ExposureLevel:
        # Check if the signal's location is directly in an endpoint handler
        if self._is_endpoint_handler(signal.file_path, signal.line, endpoints):
            return ExposureLevel.DIRECTLY_EXPOSED

        # Trace backwards through call graph from known endpoints
        for endpoint in endpoints:
            if call_graph.is_reachable(
                source=endpoint.handler,
                target=signal.function_name,
                max_depth=5
            ):
                return ExposureLevel.INDIRECTLY_EXPOSED

        return ExposureLevel.INTERNAL

CLI Interface

RepoAudit provides both local and CI/CD modes through a Click-based CLI.

# Local developer mode - scan any directory
repoaudit scan ./my-repo                                    # Default: AI maturity
repoaudit scan ./my-repo --profile security --validate      # Security audit with LLM
repoaudit scan ./my-repo --profile architecture --validate  # Architecture analysis
repoaudit scan ./my-repo --profile docs-audit --validate    # Documentation gaps
repoaudit scan ./my-repo --profile diagrams --validate      # Auto-generate Mermaid diagrams
repoaudit full-scan ./my-repo                               # All profiles in parallel

# CI/CD mode - analyze multiple repos via GitHub API
repoaudit central-run --config runner/config.yml
repoaudit status                                            # Dashboard of all tracked repos

# Options
--model sonnet|opus|haiku|gpt4o     # Choose LLM model
--method treesitter|regex|both     # Extraction method
--format json|markdown|both        # Output format
--render-mode svg|js|both          # Diagram rendering

Features

• Six Analysis Profiles - AI maturity, security, architecture, documentation, portfolio readiness, auto-generated diagrams
• Deterministic-First - AI maturity fully deterministic and security’s base tier free of API keys; the LLM-first profiles need a key, and validation is always opt-in
• Tri-Axis Classification - novel cognitive (C0-C5), operational (O0-O5), and agentic (A0-A5) maturity model for AI integration, graded for fit against intent-derived target bands (under-built / appropriate / over-built)
• Security Scanning - 50+ vulnerability patterns with OWASP/CWE mapping, exposure classification, dependency CVE scanning via OSV, and a deep-verify jury (refute-first prompting, multi-juror consensus) for high-severity findings
• Tree-sitter Extraction - 100+ AST-aware queries across Python, JavaScript, and Java with semantic refinement callbacks that sharply cut false positives vs regex-only extraction
• Iterative LLM Validation - two-turn system where the LLM can request additional context before confirming signals; validated pilot ran five repos of pre-registered predictions for $0.27
• Multi-Provider LLM - unified interface across Anthropic and OpenAI with deterministic routing and fallback chains (Gemini registered, not yet CLI-routable)
• Cost Control - budget enforcement at four scopes (per-call, per-repo, per-run, monthly) with configurable degradation strategies
• Prompt Caching - layered cached prefixes (skeleton, core, rubric) with prime-then-burst scheduling; ~22% input-cost reduction on LLM-heavy scans
• Interactive Reports - tabbed HTML dashboards, Markdown narratives, structured JSON output
• CI/CD Integration - GitHub Actions workflow for scheduled multi-repo analysis with state tracking and change detection
• Parallel Analysis - up to 10 concurrent workers for batch repo scanning, with diff-based gating to minimize API calls
• Monorepo Support - automatic subdirectory discovery with parallel analysis
• Privacy - read-only, secret redaction, never modifies target repositories

Tech Stack

• Language: Python 3.10+
• CLI: Click 8.0+, Rich (terminal UI)
• Parsing: Tree-sitter (Python, JavaScript, Java), regex fallback
• AI/LLM: Anthropic Claude, OpenAI GPT (provider-agnostic with central model registry; Gemini registered, not yet CLI-routable)
• Validation: Pydantic (config schema), PyYAML (profiles)
• Security: OSV API (dependency CVEs), custom pattern detection, exposure classification
• GitHub: PyGithub (API integration), diff-based change gating
• Reporting: Markdown, JSON, HTML dashboards with Mermaid diagram generation
• Build: Hatchling (PEP 517), pip-installable