Classifieds Briefing (An AI-Curated Buying Agent for a Mountain Relocation)

Role: Solo, end-to-end. Architecture, the scraping and anti-fingerprint layer, the deterministic scoring engine, the two Claude passes, the dashboard renderer, and the CI automation.
Tech Stack: Python 3.10+, requests + BeautifulSoup, Jinja2, the Anthropic Claude API, and GitHub Actions for daily automation.
Repository: Private. The product is a self-contained HTML dashboard generated on each run (no hosted service).

Overview

I was planning a move to the western North Carolina mountains and needed to buy a whole rig at once: a 3/4-ton diesel truck, a 4-season camper, a flatbed trailer, a utility ATV, some compact equipment, and a piece of raw land. All of it lives on classifieds sites, spread across six regional markets, and the good listings move fast.

The manual version of this is refreshing six searches across six cities every morning, re-reading listings you already evaluated yesterday, and squinting at whether a truck advertised as a ā€œ7.3 dieselā€ is actually the desirable pre-emissions engine or a troubled later one. This tool does that pass for me: it scrapes all six categories incrementally, scores every candidate 0 to 100 against a tuned buy box, has Claude audit the listings that matter, and hands me a briefing plus a filterable grid.

A search page shows you what is listed. This tells you what is worth driving to see, and what the seller is not telling you.

System Architecture

A CLI-orchestrated batch pipeline with pluggable sources. run.py drives each stage in sequence; listings flow through as Listing dataclasses. A daily GitHub Action runs the scrape stage and commits the refreshed cache and history snapshot back to the repo, so state stays current with no server to maintain.

graph TD
    A[GitHub Action<br/>daily cron] --> B[run.py orchestrator]
    B --> C[Classifieds source<br/>scrape + detail fetch]
    C --> D[Incremental merge<br/>reuse cache, drop delisted]
    D --> E[scoring.py<br/>deterministic 0-100 + profile]
    E --> F[llm.py<br/>Claude rate + audit pass]
    F --> G[briefing.py<br/>editorial narrative + diff math]
    G --> H[dashboard.py<br/>self-contained HTML]
    D --> I[(output/raw.json<br/>committed cache)]
    D --> J[(history/snapshot-*.json<br/>daily snapshots)]
    I --> D
    J --> F

Components

Engineering Details

1. Prompt-cache warming before the parallel rating pass

The per-listing rating pass sends dozens of listings to Claude, each with a large shared system prompt (the buy box, the profile catalogs, the scoring rubric). Firing them all in parallel would have every worker racing to write the same prompt cache, so most calls pay full input price.

The fix is to run the first call sequentially to warm the cache_control: ephemeral system prompt, then fire the rest across a thread pool of 10 workers so they all hit the warm cache. Cached input tokens bill at roughly 10% of the uncached rate, and the shared prompt is the bulk of each request, so this cuts the rating passā€™s input cost substantially while keeping it fast. Failed JSON parses retry once.

2. Incremental merge to stay under the marketplaceā€™s rate limiter

A cold scrape makes about 24 search-result requests plus up to ~500 detail-page fetches. Doing that daily is both slow and a fast way to earn a 30 to 60 minute IP block.

The incremental merge diffs a fresh scrape against the committed output/raw.json cache: matched listings reuse their cached description and AI fields, delisted ones are dropped, and only genuinely new or price-changed listings get a detail fetch and a fresh rating. That takes daily detail fetches from ~500 down to ~30 to 50 (visible in the committed output/run-stats.json, where daily detail_fetches sits around 40). It also drives the day-over-day ā€œWhat changedā€ diff and per-listing price history.

3. Anti-fingerprint scraping

The marketplace throttles uniform-interval scrapers, and blocks persist per IP. Every sleep in the scraper is a random sample from a range rather than a fixed delay (2 to 5s between regions, 0.8 to 2.5s between pages, 2 to 4s between detail fetches). Category and region order shuffle each run so the request sequence is not a fixed pattern, the User-Agent is picked once per process from a pool of plausible browsers and held constant, and the GitHub Action start time jitters 0 to 30 minutes off the cron so it is not ā€œevery day at noon UTC sharp.ā€ A pre-flight check confirms the IP is not already blocked, and a post-scrape watchdog exits non-zero if regions went silent or volume cratered, so a soft block shows up red instead of quietly rotting the cache.

4. The LLM audits the scorer instead of replacing it

The deterministic scorer is fast, free, and transparent, but regex-based classification gets fooled. ā€œTransmission rebuiltā€ can trip a salvage-title flag; a title reading ā€œ7.3 PSDā€ says nothing about the engine having been swapped for a 6.0.

So Claude runs as an auditor, not an oracle. It returns a fit rating, but also score_corrections (deltas that undo false positives or add missed concerns, with llm_corrected_score = clamp(score + sum(deltas))) and an optional profile_correction validated against the typeā€™s catalog. A real example from the cache: a 2003 Chevrolet 3500HD Duramax that the scorer awarded a perfect 100 (mistaking the LB7 for a pre-emissions gold diesel) gets caught by Claude, re-classified to the troubled-diesel profile, and corrected down to 60.

Code Snippet: warming the prompt cache before fanning out

The first call runs alone to write the cache; the remaining calls hit it in parallel at a fraction of the input cost.

# Warm the prompt cache with one sequential call so the (n-1) parallel calls
# that follow all hit the cache instead of racing to write it.
rated = 0
li0, err0 = _rate_one(
    client, targets[0], model, json_instructions,
    timeline=timelines.get(targets[0].post_id or targets[0].url), today_iso=today_iso,
)
if not err0:
    rated += 1
    _log_rated(1, len(targets), li0)

# Now fire the rest in parallel.
remaining = targets[1:]
with ThreadPoolExecutor(max_workers=max_workers) as ex:
    futures = {ex.submit(_rate_one, client, li, model, json_instructions,
                         timeline=timelines.get(li.post_id or li.url), today_iso=today_iso): (idx + 2, li)
               for idx, li in enumerate(remaining)}
    for fut in as_completed(futures):
        i, li_meta = futures[fut]
        li_result, err = fut.result()
        if err:
            continue
        rated += 1
        _log_rated(i, len(targets), li_result)

Sample Output

A rated listing after both the deterministic scorer and the Claude audit have run. Note the corrected score and the specific red flags pulled from the post body.

{
  "vehicle_type": "truck",
  "region": "tricities",
  "title": "2003 Chevrolet 3500hd 4x4 duramax",
  "price": 9000,
  "year": 2003,
  "score": 100,
  "profile": "O",
  "profile_label": "One-ton: F-350, 3500, dually - overkill for TT, ideal for 5th wheel",
  "llm_rating": 6,
  "llm_callout": "This is an LB7 Duramax, the earliest and most problematic emissions-era Duramax, notorious for injector failures; the deterministic scorer wrongly awarded the 'pre-emissions diesel' Gold bonus, and this truck belongs in profile C.",
  "llm_red_flags": [
    "LB7 Duramax - injector failure history, likely due again near 300k",
    "288k miles - high for a troubled-diesel platform",
    "No mention of head gasket or coolant system work"
  ],
  "llm_highlights": [
    "Owned since 2008 - 16 years of known history",
    "S&S Torquemaster injectors replaced at 230k",
    "Clean title, ice cold AC, EFI Live tow tune loaded"
  ],
  "llm_corrected_score": 60,
  "first_seen": "2026-06-24",
  "last_seen": "2026-07-04",
  "mileage": 288000
}

Tech Stack