Reddit Research

Role: Solo, end-to-end
Tech Stack: Python, FastAPI, React/TypeScript, WebSockets, Sentence Transformers, Pydantic
Repository: Private. Demo available on request

Overview

Reddit Research takes a list of subreddits and a time period and hands back a newsletter-style deep dive where every claim links back to the original post or comment it came from.

The workflow is simple from the user’s side: point it at a few subreddits (say MachineLearning, Python, datascience) and a time range (week, month, year). The system scrapes them in parallel, clusters semantically similar comments so you don’t pay to process the same idea 50 times, extracts structured knowledge items with source attribution, and synthesizes everything into a readable guide.

The reason I built it: I wanted to actually learn things from Reddit at scale without spending hours scrolling, and I wanted the output to be trustworthy, meaning I could click any claim and verify it came from a real person who actually said that.

How It Works

1. Resilient Data Collection

Long-running scrape jobs across multiple subreddits and time periods tend to fail partway through. Rate limits, timeouts, the occasional 503, any of it can kill a job that’s been running for an hour. So before I built anything else I hardened the collection layer.

2. Semantic Deduplication

Across 500 comments from three subreddits, 80 people are saying the same thing in slightly different words. Passing all of them through an LLM is a waste of tokens and produces repetitive output. So sentence transformers cluster semantically similar comments before anything hits the LLM.

If 50 people say “Python is great for data science” in 50 different ways, the system treats it as one cluster. Each cluster keeps all the original source metadata, so attribution survives deduplication, I don’t want to lose the authors when I collapse their comments.

3. Citation-Tracked Knowledge Extraction

LLMs hallucinate. If the output says “u/researcher42 recommended PyTorch,” I need to be able to verify that, otherwise the whole guide is untrustworthy.

Every extracted knowledge item carries required author, source.permalink, and source.subreddit fields, enforced at the Pydantic schema level. Extraction prompts explicitly tell the LLM to copy this metadata verbatim from the input. And then a post-generation validator checks that authors and permalinks in the output actually exist in the input data, flagging anything fabricated.

End result: every claim in the final guide links back to the specific Reddit comment it came from. No made-up citations.

4. Dynamic Topic Organization

Pre-set category schemas don’t survive contact with different domains. A scrape of r/MachineLearning needs very different categories than a scrape of r/PipeTobacco. So a second LLM pass organizes the knowledge items into a dynamic hierarchy built for this specific corpus.

Two-stage assignment (main category first, then subcategory) gets better accuracy than one shot. Category count scales with input size with floor and ceiling bounds, so small runs don’t get over-split and large runs don’t get under-organized. The output reads like a curated table of contents instead of a dump of extracted facts.

5. Newsletter Curation

A structured knowledge guide is dense. Useful, but not something you’d sit down and read. So there’s a dedicated curation step that transforms the structured output into a newsletter-style guide, synthesizing insights across subreddits into coherent narratives instead of per-subreddit silos, with inline citations in ([u/author](permalink)) format so you can click through to the original thread. Configurable between OpenAI and Anthropic for that synthesis step.

6. The Web Interface

A React/TypeScript frontend with a FastAPI backend gives real-time visibility into the pipeline. You watch progress from “Scraping” to “Clustering” to “Drafting Guide” live over WebSockets, and everything persists to disk so a crash doesn’t lose the run.

Testing

Tech Stack