Chronicle (Personal Data Warehouse & Digital Twin Engine)

Role: Solo, end-to-end
Tech Stack: Python 3.11, SQLModel (SQLAlchemy), Typer/Click (CLI), Rich (TUI), OpenAI/Anthropic APIs
Repository: Private. Demo available on request

Overview

Chronicle pulls in your chat history, GitHub commits, and Reddit activity, normalizes everything into a single local database, and exports it as fine-tuning datasets.

The idea is simple: your writing style, your reasoning patterns, your decision-making, all of it is scattered across dozens of platforms and completely untapped. Chronicle aggregates that data into a training corpus you can actually use to fine-tune a model on how you think and write.

System Architecture

The system is a modular ETL pipeline controlled via a CLI. Everything stays on your machine, and all data gets normalized into a common schema.

graph TD
    subgraph "Ingestion Layer (Extract)"
    A[ChatGPT Export] --> D
    B[GitHub API] --> D
    C[Claude/Gemini Logs] --> D
    end

    D{Ingestor Strategy} -->|Normalize| E[Standardized Event Schema]

    subgraph "Core Database (Transform/Load)"
    E --> F[(SQLModel / SQLite)]
    end

    subgraph "Application Layer"
    F --> G[CLI Controller]
    G --> H[Corpus Generator]
    G --> I[Analytics Engine]
    end

    H -->|Export| J[Fine-Tuning JSONL]
    I -->|Visualize| K[Rich TUI Dashboard]

Components

How It Works

1. The Polymorphic Ingestion Pattern

Every data source has a different shape. ChatGPT exports are nested JSON trees. GitHub commits are flat API lists. Reddit comments are threaded. No single parser handles all of them.

So I implemented a strict Strategy Pattern for ingestion.

2. Local-First Data Sovereignty

Users are (rightfully) hesitant to connect personal archives to cloud tools. So the entire architecture is built on SQLite and local file systems.

3. Contextual Search & Retrieval

Searching through 100,000+ lines of chat history and code is slow and imprecise with standard regex. SQLModel with full-text search capabilities handles the scale.

Code Snippet: The Ingestion Strategy

How the parsing logic is decoupled from database storage.

class BaseIngestor(ABC):
    """
    Abstract interface for all data sources.
    Enforces a consistent return type (List[Entry]) regardless of input format.
    """

    @abstractmethod
    def parse(self, source_path: Path) -> Generator[Entry, None, None]:
        """Yields normalized Entry objects from the raw source file."""
        pass

    def validate_schema(self, data: Dict) -> bool:
        """Ensures the source file matches expected version/format."""
        try:
            # Pydantic validation logic
            return SchemaModel(**data)
        except ValidationError:
            return False

# Concrete Implementation
class ChatGPTIngestor(BaseIngestor):
    def parse(self, source_path):
        with open(source_path) as f:
            raw_json = json.load(f)
            for conversation in raw_json:
                yield self._normalize_conversation(conversation)

Use Cases

Tech Stack