Chronicle (Personal Data Warehouse & Digital Twin Engine)

Role: Solo, end-to-end
Tech Stack: Python 3.10+, stdlib sqlite3 + dataclasses, argparse CLI, ijson streaming parser, Anthropic SDK (OpenAI/Ollama via lazy provider imports)
Repository: Private. Demo available on request

Overview

Chronicle pulls in your chat history, GitHub commits, and Reddit activity, normalizes everything into a single local database, and exports it as fine-tuning datasets.

The idea is simple: your writing style, your reasoning patterns, your decision-making, all of it is scattered across dozens of platforms and completely untapped. Chronicle aggregates that data into a training corpus you can actually use to fine-tune a model on how you think and write. It also turns that same data into narrative devlogs: chronicle generate hands a day’s (or week’s) events to an LLM and gets back a written account of what you actually did.

System Architecture

The system is a modular ETL pipeline controlled via a CLI. Everything stays on your machine, and all data gets normalized into a common schema.

graph TD
    subgraph "Ingestion Layer (Extract)"
    A[ChatGPT Export] --> D
    B[GitHub API] --> D
    C[Claude/Gemini Logs] --> D
    end

    D{Ingestor Strategy} -->|Normalize| E[Standardized Event Schema]

    subgraph "Core Database (Transform/Load)"
    E --> F[(SQLite via stdlib sqlite3)]
    end

    subgraph "Application Layer"
    F --> G[CLI Controller]
    G --> H[Corpus Generator]
    G --> I[Stats Engine]
    end

    H -->|Export| J[Fine-Tuning JSONL]
    I -->|Report| K[Plain Text / JSON Stats]

Components

Universal Ingestors: a plugin-based system that parses disparate data formats (JSON exports, API responses, CSVs) and maps them to a single Conversation or Event schema.
The Corpus Engine: filters and cleans historical data to create high-quality datasets for LLM fine-tuning.
Stats Reporting: chronicle stats prints storage counts and sync status as plain text or JSON. No TUI framework, on purpose; the output pipes cleanly into other tools.

How It Works

1. The Polymorphic Ingestion Pattern

Every data source has a different shape. ChatGPT exports are nested JSON trees. GitHub commits are flat API lists. Reddit comments are threaded. No single parser handles all of them.

So I implemented a strict Strategy Pattern for ingestion.

Abstract Ingestor class defines the contract: ingest() returns an IngestorResult with counts, errors, and a sync cursor for the next incremental run.
Concrete implementations (ChatGPTIngestor, GitHubIngestor) handle the source-specific parsing.
The core database doesn’t care where data came from. A commit message and a chat prompt are both just Event objects once they’re normalized.

2. Local-First Data Sovereignty

Users are (rightfully) hesitant to connect personal archives to cloud tools. So the entire architecture is built on SQLite and local file systems.

Raw archives stay local. Content only goes to an LLM when you explicitly run narrative generation or LLM-based classification.
PII detection blocks flagged messages from ever entering the training corpus.

3. Structured Filtering & Retrieval

Running regex over years of raw exports is slow and imprecise. Chronicle indexes everything into SQLite instead, so slicing the archive is a query, not a scan.

get_events filters on indexed columns: source, event type, project, and date range.
Session tagging groups related events, so you can pull everything from one work session without touching the rest of the archive.

Code Snippet: The Ingestion Strategy

How the parsing logic is decoupled from database storage.

# condensed from chronicle/ingestors/base.py

@dataclass
class IngestorResult:
    """Result of an ingestion operation."""

    events_found: int = 0
    events_inserted: int = 0
    events_skipped: int = 0
    errors: list[str] = field(default_factory=list)
    # Sync state for next run
    new_cursor: dict[str, Any] = field(default_factory=dict)

    @property
    def success(self) -> bool:
        return len(self.errors) == 0


class Ingestor(ABC):
    """
    Abstract base class for data ingestors.

    Ingestors are responsible for:
    1. Connecting to a data source (git repo, API, file, etc.)
    2. Extracting events from the source
    3. Converting them to Chronicle Event format
    4. Storing them in the database
    5. Tracking sync state for incremental updates
    """

    source_type: str = "unknown"

    @abstractmethod
    def ingest(
        self,
        identifier: str,
        *,
        full_sync: bool = False,
        dry_run: bool = False,
        **kwargs: Any,
    ) -> IngestorResult:
        """Ingest data from the source."""
        ...

    @abstractmethod
    def validate_source(self, identifier: str) -> tuple[bool, str]:
        """Validate that the source is accessible and valid."""
        ...

    def get_sync_state(self, identifier: str) -> Optional[dict[str, Any]]:
        """Get the sync cursor from the last run, for incremental ingestion."""
        return self.db.get_sync_state(self.source_type, identifier)

Use Cases

Personal AI Assistant: “talk to yourself.” Train an LLM on this data and the model can predict how I’d respond to an email or write a function.
Narrative Devlogs: chronicle generate turns a stretch of events into a written devlog, so a week of commits and chats becomes something readable.
Portfolio Generation (planned): a showcase module on the v0.8.0 roadmap would generate documentation by pulling relevant project history and code snippets.
Memory Extension: a searchable “second brain” for everything I’ve ever written or coded.

Tech Stack

Language: Python 3.10+
Storage: stdlib sqlite3 with plain dataclasses. No ORM, deliberately: the schema is small enough that raw SQL stays readable, and a personal-data tool shouldn’t drag in a dependency tree.
CLI: stdlib argparse
Parsing: ijson for streaming large chat exports without loading them into memory
LLM: anthropic SDK; OpenAI and Ollama supported via lazy provider imports, so you only need the package for the provider you use
Testing: Pytest