BookParser (Multi-Agent Narrative Intelligence)

Role: Solo, end-to-end
Tech Stack: Python 3.12, Anthropic API, Pydantic, AsyncIO
Repository: Private

Overview

BookParser is an ETL pipeline that turns novels into structured knowledge graphs.

Unlike standard “Chat with PDF” tools that rely on transient context, BookParser builds a persistent “World Bible” tracking character states, inventory changes, and location graphs across hundreds of pages. It uses a multi-agent architecture to maintain narrative continuity and catch hallucinated data before it hits the database.

System Architecture

The system follows a Hierarchical Agent Pattern where a central Coordinator manages state and delegates tasks to specialized sub-agents.

graph TD
    A[Input: Raw Text/EPUB] --> B(Scene Splitter Agent)
    B --> C{Coordinator Service}

    subgraph "Parallel Extraction Layer"
    C --> D[Character Agent]
    C --> E[Setting Agent]
    C --> F[Item/Inventory Agent]
    end

    D & E & F --> G[Entity Resolution Service]
    G --> H[Global State Tracker]
    H --> C

    C --> I[Validation Gates]
    I --> J[Final JSON Output]

Components

• The Coordinator: manages the event loop, dispatching scenes to agents and aggregating results.
• The Registry (Entity Resolution): a deduplication engine that resolves “The Princess” and “Donut” to the same entity ID (donut_the_cat) using fuzzy matching and LLM reasoning.
• Validation Gates: a strict logic layer that enforces schema integrity so no hallucinated IDs make it into the final dataset.

Engineering Details

1. Cost Optimization via Model Cascading

Running frontier models on every paragraph was prohibitively expensive and slow. So I implemented a consolidated extraction strategy.

• Tier 1 (Fast): a lightweight model handles high-volume text scanning and raw entity extraction.
• Tier 2 (Smart): a frontier model is reserved for complex reasoning tasks, resolving ambiguous dialogue, summarizing scene sentiment.
• Reduced API costs by ~40% and processing latency by ~60%.

2. Context Window Management (“RAG-Lite”)

Passing the entire history of the book to the model for every scene saturates the context and confuses the model. So I developed a build_filtered_context() method.

• Before processing a scene, the system queries existing state for only the entities relevant to the current text chunk (keyword-mapped).
• Creates a sliding window of relevance that maintains continuity without blowing token limits.

3. Data Integrity & Self-Healing

LLMs are probabilistic, they occasionally return malformed JSON or invent IDs (the_huntingrounds vs hunting_grounds). That kind of hallucination is exactly what my QA background trained me to paranoid-proof against, so I built a strict Validator class.

• Acts as a gatekeeper between LLM generation and the database.
• Detects schema violations and ID mismatches.
• Triggers retry logic with specific error feedback, forcing the model to self-correct its output.

Code Snippet: The Validation Logic

Ensuring agent-generated IDs strictly match the canonical registry.

def validate_entity_ids(self, entities: List[Dict], registry: Dict) -> bool:
    """
    Acts as a circuit breaker. If an Agent invents a new ID
    that hasn't been registered, the batch is flagged for review.
    """
    validation_passed = True
    for entity in entities:
        canonical_id = self._generate_id(entity['name'])
        if canonical_id not in registry:
            logger.warning(f"Hallucination detected: {canonical_id}")
            validation_passed = False
    return validation_passed

Sample Output

The system produces rich, flat JSON ready for frontend visualization.

{
  "scene_id": 3,
  "location": "third_floor_hallway",
  "characters_present": ["carl", "donut", "mordecai"],
  "inventory_changes": [
    {
      "item_id": "holographic_sign",
      "action": "USED",
      "description": "A magical sign showing directional arrows."
    }
  ]
}

Tech Stack

• Language: Python 3.12 (heavy use of asyncio for parallel agent execution)
• AI Integration: Anthropic API
• Data Validation: Pydantic
• Resilience: Tenacity (exponential backoff for API retries)
• Utilities: Regex, Loguru