Active Sep 2026

Docket Lens (FCC Public Comment Analysis Pipeline)

Role: Solo, end-to-end: ECFS download, extraction schema, report pipeline, quote verification, bias measurement, docs. Also one of the commenters.
Tech Stack: Python, requests, pypdf / pdftotext / docx2txt, Anthropic API (Opus 5 for the report, Sonnet 5 for triage and the brief), any OpenAI-compatible endpoint for triage
Repo: GitHub
Evidence: The final report (2026-09-05, 661 filings, mirrored unedited from the repo)

Overview

In July 2026 the FCC put out a Public Notice in PS Docket 26-189, a proposal to stop the import and marketing of previously authorized foreign-made drones it calls “military-grade.” The proposal defines that term with seven capability categories: takeoff weight of 55 lb or more, agricultural spraying, thermal imaging, lidar, docking stations, defense-article integration, and swarming. Anyone could file a comment before the September 2 deadline, and hundreds of people did.

I filed a comment myself, arguing about category 4, lidar. Before and after that I wanted to know what everyone else was saying, and at several hundred PDFs that isn’t a weekend of reading. So I built a pipeline to do the first pass: pull every filing with an attachment from ECFS, classify each one against a fixed schema, and write a report on the whole record where every quote is checked against the document it came from.

The part I’m proudest of is the part that went wrong first. I built the tool while advocating on one category, and the first version of the schema showed it. The report over-covered lidar about 2x relative to the record. The rest of this page is mostly about how I found that, measured it, and changed the design so the tool asks every category the same question.

The pipeline covers 680 filings in the published repo, and the tracked outputs (a per-filing CSV, the JSONL records, and the dated reports) are all public. The final report is mirrored on this site exactly as the pipeline wrote it. Everything underneath is public record on ECFS.

Engineering Details

1. Four Stages, Each One Re-runnable

The pipeline is four scripts, and each one can be re-run without redoing the work before it.

  • Download (fcc_download.py). Pulls every filing with an attachment from ECFS, throttled so it stays polite. Completed filings are appended to a checkpoint file, so a re-run skips everything already on disk. It takes every attachment, not just the first, because exhibits often carry the actual argument.
  • Triage (docket_triage.py). Extracts text (pdftotext when poppler is installed, pypdf otherwise, docx2txt for Word files), collapses form letters, and sends each unique document to an LLM with a strict JSON schema. Records are keyed on a SHA-1 of the content, so a re-run only pays for new or changed files. It works with the Anthropic API or any OpenAI-compatible endpoint, including a local model.
  • Report (docket_report.py). One agentic session with Opus 5 that reads a compact index of every record, a fixed set of full filings, and a small budget of extra reads it has to justify. It then checks its own quotes and its own coverage before writing reports/report_YYYY.MM.DD.md.
  • Brief (docket_brief.py). One cheap call that condenses the newest report into about 700 words. It doesn’t re-read any filings, and any quote it keeps has to travel with its citation and pass the same check again.

2. The Quote That Wasn’t Theirs

On 2026-08-23 I hand-checked the first full report, built from 259 filings. Of 21 substantive quotes, 18 were exact. Two had been silently reworded. The last one was my own sentence, from my own filing, about proximity sensors, credited to a different commenter.

That is exactly the kind of mistake a reader can’t catch and a commenter would rightly be upset about. So quotes stopped being trusted output. Every direct quote of 25 characters or more must now carry an HTML comment naming its source file, and the report script checks that the quote appears in that file after normalizing curly quotes, dashes, soft hyphens and whitespace. A failed check is quoted back to the model for a targeted rewrite, up to two rounds, and after that it ships with a hard warning instead of a silent pass. That misattribution is now a regression test.

I still hand-checked every claim I published or filed. The check makes the report safe to read, not safe to cite without looking.

3. Measuring My Own Bias

The first schema had a field called attacks_category_4 and no matching field for any other category, because category 4 was the one I cared about. The report written from it spent about twice as much of its text on lidar as the record did.

Fixing it meant separating two questions: did the bias get into the extracted data, or only into the writing?

  • Extraction (v3 schema). Every filing now gets a mandatory {stance, arguments, note} object for all seven categories, where “silent” is an explicit answer. I re-ran triage and diffed old against new with a no-API script, finalized on 2026-09-05 at 659 joined records. The number of filings addressing lidar didn’t move (432 to 432). The old per-category bias had not inflated lidar in extraction. What did move was elsewhere: 104 of 659 overall positions changed, 77 of them from “oppose all” to “oppose partial,” so the old schema had been overcounting blanket opposition. Agricultural spraying picked up 36 and swarming 18, while heavy-lift and thermal lost 24 and 16.
  • Writing. With extraction ruled out, the tilt had to be in what the report writer chose to read and emphasize. In the old design it picked its own reads. The new one starts from a deterministic seed list: every supporter, every trade association, and the top two data-citing filings per category with filer-type diversity. Its own extra reads only top that up, each with a printed reason.
  • Salience lint. After drafting, the report counts each category’s share of mentions in the article and compares it to that category’s share in the record. Past 1.5x either way, the numbers go back to the model for a rewrite. For lidar the ratio went 1.48 (first report, 08-21) to 1.30 (09-04) to 0.63 (first v3 report, 09-05).

That last number swung past parity, and I don’t read it as precise. The v3 report was linted by the buggy term list in the next section, so I treat its ratios as directional, and ratios near the threshold as noise. What it does show is that the lidar tilt is gone. The README says it the way I’d say it now: residual coverage deviations track newsworthiness, not my stake.

4. A Lint That Counted “Docket”

The salience lint matches category mentions with term lists. The first version counted docking-station mentions with dock(?:ing)?, which also matches “docket.” In the first v3 report that was 17 of 21 docking hits, 12 of them inside citation comments. Fixed on 2026-09-05 with a negative lookahead and a unit test, and noted in the README as a warning to anyone adapting it: term lists are fragile, so eyeball the lint before trusting it.

5. Making It Fit and Making It Finish

  • Context budget. On 2026-08-23 the original design handed the report writer full compact records for every filing: 1,436 characters per record, about 93K tokens for 259 filings. A slim index cut that to 159 characters per record, about 10.3K tokens, with full records fetched on demand.
  • Caching. Each tool turn resends the whole growing conversation, so the report session caches its prompt prefix with a one-hour TTL. Opus turns on a large context can outlast the default five minutes, and my runbook estimates an expired cache at about $1.20 per turn versus $0.12 cached at 240K tokens.
  • Form letters. Near-duplicates are collapsed with 8-word shingles at Jaccard similarity above 0.85, which catches form letters but keeps lightly edited variants. One agricultural campaign had 12 copies whose personal text lived in three fillable PDF form fields, so plain text extraction saw 12 identical letters. I found that on 2026-09-04 and backfilled the 11 affected records.
  • Refusals and truncation. Three filings failed with an empty refusal for four runs straight. Each had a different cause: prompt wording, a glyph-decoding bug, and one refusal specific to the model, now retried on Haiku. One filing of 659 uses that fallback. Separately, a 6,000-token output cap truncated 4 of 684 filings, which showed up as “Unterminated string” JSON errors. The cap is now 12,000, and a truncation raises a readable error.
  • Scanned PDFs. 27 image-only filings are flagged rather than silently skipped. With the Anthropic backend they can be transcribed (30 to 60 seconds each) into sidecar text that every later stage reads.

6. Saying Which Record

The downloader only takes filings with attachments: 690 as of 2026-09-07. Pilot Institute counted 3,824 “filings with readable text” around the same time (reported by DroneLife on 09-04), a count that includes express comments whose text lives only in the ECFS API body, which this pipeline doesn’t capture. Both numbers are right about different populations, so the runbook says any claim about “the record” has to name which one.

Build Log

The GitHub history is short because the work was done before the repo existed: all seven commits landed on 2026-09-07, when I published it as docket-lens. The runbook’s dated entries are the real build log.

  • 2026-08-21: First full report, 259 filings, old design. Roughly $5 to $10 and 20 to 40 minutes.
  • 2026-08-23: Quote audit (18 of 21 exact, one misattributed to the wrong filer), which led to citation comments and verification. Slim index cuts the report context from about 93K tokens to 10.3K.
  • 2026-09-04: Refusals root-caused, form-field backfill, truncation cap raised, 27 scanned PDFs flagged. v3 symmetric schema re-run.
  • 2026-09-05: v2 vs v3 comparison finalized at 659 records. Salience ratios measured. “Docket” lint bug found and fixed.
  • 2026-09-07: Pipeline, tests, runbook, outputs and reports published; README rewritten for the docket-lens name.

Code Snippet: Every Quote Has to Be Findable

def verify_quotes(article: str, root: Path) -> list:
    """check every direct quote >= MIN_QUOTE chars: it must carry a
    <!-- src: filename --> citation and appear verbatim in that file.
    returns failure descriptions; empty list = verified."""
    art = article.replace("“", '"').replace("”", '"')
    failures = []
    for quote, src in _QUOTE_RE.findall(art):
        if len(quote) < MIN_QUOTE:
            continue
        if not src:
            failures.append(f'missing <!-- src --> citation: "{quote[:70]}"')
            continue
        text = _filing_norm_text(root, src.strip())
        if text is None:
            failures.append(f'cannot read cited file {src.strip()}: "{quote[:70]}"')
        elif _norm(quote) not in text:
            failures.append(f'quote not found in {src.strip()} (misattributed '
                            f'or altered): "{quote[:70]}"')
    return failures

The failures list is what goes back to the model. The file lookup resolves the cited path and refuses anything outside the downloads folder, so a citation can’t point the check at an arbitrary file.

CLI Interface

python fcc_download.py [docket] [N]     # pull filings from ECFS into ./downloads (default docket 26-189)
python docket_triage.py ./downloads     # extract, dedup, classify -> results_v3.jsonl, filings.csv
python docket_report.py                 # narrative report -> reports/report_YYYY.MM.DD.md
python docket_brief.py                  # ~700-word brief from the newest report
python compare_v2_v3.py                 # no-API diff of two extraction runs

Triage takes ANTHROPIC_API_KEY, or LLM_BASE_URL and LLM_MODEL for a local OpenAI-compatible model. The report and brief need the Anthropic key.

Features

  • ECFS download that takes every attachment and resumes from a checkpoint
  • Text extraction from PDF and Word files, with scanned PDFs flagged or transcribed
  • Form-letter collapse with shingle similarity, tunable per docket
  • Symmetric seven-category schema where silence is an explicit stance
  • Idempotent triage keyed on content hash, so re-runs only pay for new files
  • Deterministic seed list for the report’s reading, with justified extra reads
  • Quote verification against source files, with a repair loop and hard warnings
  • Salience lint comparing each category’s share of the report to its share of the record
  • A 700-word brief built from the finished report, with quotes re-verified
  • A runbook with every tuned threshold, the evidence behind it, and dated measured results

Testing

There are 25 test functions across the triage, report and brief stages. They’re plain assert-based scripts, not a pytest suite, and there’s no CI. They cover the pieces that were wrong at least once: the misattributed quote, the “docket” lint match, the seed list, and the schema. The bigger check was manual: I read the source filings behind every claim I published.

Tech Stack

  • Language: Python 3, standard library plus five packages
  • Download: ECFS retrieval with a resumable checkpoint
  • Extraction: pdftotext (poppler) with a pypdf fallback, docx2txt
  • Models: Anthropic API: Opus 5 for the report session, Sonnet 5 for triage and the brief, Haiku 4.5 as a refusal fallback. Triage also runs against any OpenAI-compatible endpoint.
  • Outputs: JSONL records, a flat CSV (one column per category stance), Markdown reports
  • Docs: README with the bias history, RUNBOOK with thresholds and measured results