BookParser (Multi-Agent Narrative Intelligence)

Role: Solo, end-to-end
Tech Stack: Python 3.12, Anthropic + OpenAI APIs (provider-pluggable client), ThreadPoolExecutor
Repository: Private

Overview

BookParser is an ETL pipeline that turns novels into structured knowledge graphs.

Unlike standard “Chat with PDF” tools that rely on transient context, BookParser builds a persistent “World Bible” tracking character states, inventory changes, and locations across hundreds of pages. It uses a multi-agent architecture to maintain narrative continuity and catch hallucinated data before it lands in the final dataset.

System Architecture

The system follows a Hierarchical Agent Pattern where a central Coordinator manages state and delegates tasks to specialized sub-agents.

graph TD
    A[Input: Raw Text/EPUB] --> B(Scene Splitter Agent)
    B --> C{Coordinator Service}

    subgraph "Parallel Extraction Layer"
    C --> D[Character Agent]
    C --> E[Setting Agent]
    C --> F[Item/Inventory Agent]
    end

    D & E & F --> G[Entity Resolution Service]
    G --> H[Global State Tracker]
    H --> C

    C --> I[Validation Gates]
    I --> J[Final JSON Output]

Components

  • The Coordinator: dispatches scenes to a bounded pool of worker agents and aggregates their results.
  • The Registry (Entity Resolution): a deduplication engine that resolves “The Princess” and “Donut” to the same entity ID (princess_donut_the_queen_anne_chonk) using fuzzy matching and LLM reasoning.
  • Validation Gates: a strict logic layer that enforces schema integrity so no hallucinated IDs make it into the final dataset.

Engineering Details

1. Cost Optimization via Selective Model Use

The naive version of this pipeline makes an LLM call for every entity comparison, which gets expensive fast. So I split the work by what actually needs a model.

  • The frontier model (Claude Sonnet) handles the high-volume work: scene-by-scene extraction, where accuracy matters most.
  • A lightweight model (gpt-5-nano) handles one narrow task it’s good enough for: consolidating duplicate character entries.
  • The real cost saver is skipping the LLM entirely. A difflib fuzzy-match gatekeeper in the entity resolver settles the obvious matches deterministically, so only genuinely ambiguous cases ever reach a model.

2. Context Window Management (“RAG-Lite”)

Passing the entire history of the book to the model for every scene saturates the context and confuses the model. So the WorldStateManager builds a chapter-scoped rolling context instead.

  • get_context_injection() produces a running summary of the accumulated world state, trimmed to a configurable max_context_tokens cap.
  • get_entity_context() adds targeted state for just the entities that appear in the current chunk.
  • The result is a rolling window of relevance that maintains continuity without blowing token limits.

3. Data Integrity & Self-Healing

LLMs are probabilistic, they occasionally return malformed JSON or invent IDs (the_huntingrounds vs hunting_grounds). That kind of hallucination is exactly what my QA background trained me to paranoid-proof against, so I built a strict Validator class.

  • Acts as a gatekeeper between LLM generation and the final output.
  • Detects schema violations and ID mismatches.
  • Malformed JSON triggers a bounded retry loop that re-prompts the model to return valid JSON.
  • Dangling references that survive extraction get cleaned up afterward: an LLM-based reconciliation pass resolves them where reasoning helps, and a deterministic IntegrityFixer handles the mechanical repairs.

Code Snippet: The Validation Logic

From src/utils/validator.py (condensed): every entity’s object ID has to match its registry key.

def validate_entity_ids(self, book: Book) -> Dict[str, Any]:
    """Ensure all entities have proper IDs matching their dictionary keys."""
    errors = []
    warnings = []

    for char_id, character in book.characters.items():
        if character.id is None:
            errors.append(f"Character '{char_id}': ID is None")
        elif character.id != char_id:
            errors.append(
                f"Character '{char_id}': ID mismatch "
                f"(dict key: {char_id}, object ID: {character.id})"
            )

    # ... same checks for items; settings key off their name instead

    return {
        "valid": len(errors) == 0,
        "error_count": len(errors),
        "warning_count": len(warnings),
        "errors": errors,
        "warnings": warnings,
    }

Sample Output

A real scene record from the output (condensed): flat JSON ready for frontend visualization.

{
  "scene_id": "01-01",
  "scene_number": 1,
  "summary": "Carl narrates the beginning of an apocalyptic event that kills everyone indoors at 2:23 AM. He survives only because he's outside in below-freezing weather, holding his ex-girlfriend's expensive Persian cat.",
  "character_ids": ["carl", "princess_donut_the_queen_anne_chonk", "beatrice"],
  "setting_id": "seattle",
  "pov_character_id": "carl",
  "mood": "apocalyptic, desperate, darkly humorous",
  "key_events": [
    "A catastrophic transformation event occurs at 2:23 AM",
    "Everyone indoors dies instantly",
    "Carl survives because he is outside"
  ]
}

Tech Stack

  • Language: Python 3.12 (ThreadPoolExecutor-based parallel agent execution, bounded worker pool)
  • AI Integration: Anthropic + OpenAI APIs (provider-pluggable client)
  • Data Validation: Python dataclasses with a custom validation layer
  • Resilience: bounded JSON-repair retry loop (re-prompts the model when a response fails to parse, capped attempts)
  • Utilities: Regex, stdlib logging