Skip to content
UnstableLlamaPublic

About

Agentic RP database & dataset generation from source material

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

booktalk

Turn books into trainable roleplay datasets.

booktalk is a pipeline for converting prose fiction (a novel, a series, anything you have the rights to use) into structured, character-grounded roleplay (RP) training data. The name plays on BookTok — booktalk is about making books talk: animating their characters and worlds into dialogue you can train on.

It produces two outputs in lockstep:

  1. A JSONL RP dataset — messages/metadata training examples with original, canon-grounded dialogue scenes (never copied source prose).
  2. A SQLite "world database" — a persistent, queryable canon of characters, their temporal arcs, relationships, factions, locations, items, and events, plus full pipeline bookkeeping (which chunk produced what, raw model extractions, and per-chunk status).

The world DB is what makes the dataset coherent across a long book: each new chunk is generated with knowledge of the reviewed entities discovered so far, so characters stay in voice and canon stays consistent.

Credits

The pipeline design and methodology are by Darkhn, originally developed for his Animus roleplay models. booktalk is a generalized, open-source implementation of that approach, built and maintained by UnstableLlama. See Acknowledgments.


How it works

booktalk is model-agnostic and bring-your-own-model. It never calls a hosted API for you. Instead it:

  1. Chunks your source book at natural boundaries (chapters, scene breaks).
  2. For each chunk, assembles an exact prompt pack — one prompt per pipeline step, pre-filled with the chunk text and the current reviewed world entities.
  3. You run those prompts through any LLM you like and save the outputs as text files.
  4. booktalk validates and finalizes those outputs into the JSONL dataset and the world DB.

This keeps you in full control of the generation model and cost, and means the same workspace works whether your "model" is a local Llama, a hosted API, or an agent running the prompts interactively.

The pipeline steps

Each chunk passes through eight ordered steps (prompts live in prompts/):

# Step Purpose
1 emotion Primary emotional tone of the passage, as felt in-world.
2 features Reusable RP tags (setting, props, traits, genre tags).
2b world_extraction NEW/UPDATED canon entities as structured JSON for the world DB.
3 archetype The narrative archetype and its dynamic.
4 casting Up to four characters/locations for the scene, preferring known canon.
5 scene_brief ~200 words of director's notes grounding an original scene.
6 dialogue The roleplay scene, as conventional prose with integrated dialogue.
7 formatting The final JSONL training object (messages + metadata).

(Step 8, world DB update, is performed by code at finalization — see prompts/world_db_update.md.)


Requirements

  • Python 3.9+ (standard library only — no third-party dependencies).
  • An LLM of your choosing to generate the per-step outputs.

Repository layout

booktalk/
├── scripts/        operational scripts (init, chunk, seed, process, audit, sync)
├── prompts/        editable prompt templates, one per pipeline step
├── schema/         SQLite schema (world DB + pipeline tracking)
├── seeds/          reviewed-entity seed loader input (+ seed.example.json)
├── tests/          unit tests
├── sources/        YOUR source book(s) go here (gitignored)
├── chunks/         emitted chunk text files (gitignored)
├── intermediates/  per-chunk prompt packs + step outputs (gitignored)
├── output/         final JSONL dataset(s) (gitignored)
└── data/           SQLite database (gitignored)

sources/, chunks/, intermediates/, output/, and data/ are intentionally gitignored — they hold copyrighted source text, large derived data, and the database. Only the code, prompts, schema, and the example seed are committed.


Quickstart

# 1. Initialize the SQLite database (world DB + tracking tables)
python3 scripts/init_db.py

# 2. Put your source book in sources/ (plain text, UTF-8), e.g.
#    sources/mybook.txt

# 3. (Optional but recommended) seed a few hand-reviewed canon entities
cp seeds/seed.example.json seeds/mybook_seed.json   # then edit it
python3 scripts/load_seed.py seeds/mybook_seed.json --dry-run
python3 scripts/load_seed.py seeds/mybook_seed.json

# 4. Chunk the book at natural boundaries
python3 scripts/chunk_source.py \
    --source sources/mybook.txt \
    --book "My Book" --book-order 1 \
    --prefix mybook \
    --min-tokens 2000 --max-tokens 8000

# 5. Prepare the prompt pack for the next pending chunk
python3 scripts/process_chunks.py --prefix mybook --next --prepare-only --limit 1

--prefix controls the filenames for emitted chunks and intermediates (mybook_000001.txt, etc.). Use the same --prefix for chunk_source.py and process_chunks.py. It defaults to chunk.

Generate and finalize a chunk

After step 5, a prompt pack exists at intermediates/<prefix>_000001/prompts/. Run each prompt through your LLM and save the raw outputs as text files next to it:

intermediates/<prefix>_000001/emotion.txt
intermediates/<prefix>_000001/features.txt
intermediates/<prefix>_000001/world_extraction.txt
intermediates/<prefix>_000001/archetype.txt
intermediates/<prefix>_000001/casting.txt
intermediates/<prefix>_000001/scene_brief.txt
intermediates/<prefix>_000001/dialogue.txt
intermediates/<prefix>_000001/formatting.txt

Then finalize:

python3 scripts/process_chunks.py --prefix mybook --chunk-id 1 --finalize-artifacts

Finalization:

  • validates the required artifacts exist and that JSON steps parse,
  • validates the formatted example: rejects assistant turns with unbalanced plain double quotes (a clipped closing quote is easy to miss and bad training signal), and trims exact duplicate copies of the final assistant paragraph (a common padding failure of length-extension passes),
  • enforces optional length targets (see below),
  • applies world_extraction rows into the world DB (marked reviewed=0, for later review),
  • appends the formatted example to output/training.jsonl, keeping intermediates/<prefix>_XXXXXX/formatting.txt and the raw_extractions DB copy byte-identical to what was appended,
  • records all eight raw step outputs in raw_extractions,
  • writes a generation_log row and marks the chunk completed.

If anything fails, the chunk is marked failed and an error.txt is written in its intermediates directory; nothing is partially committed to the dataset.

Identity resolution

Before finalizing a chunk, resolve unnamed / epithet-only characters. If a later chunk (or your own correction) identifies one, promote the world DB id to the canonical character and keep the original descriptor as an alias. Seeding canon up front (step 3) greatly reduces this.

Length targets and batch allocation (optional)

Long generation runs drift: assistant turns shrink toward a uniform length unless targets are enforced at finalization. booktalk supports (but does not require) a ranked-allocation workflow — plan a batch of chunks, rank the scenes, and give each row a word-count target — via optional metadata keys on the formatted example:

  • target_word_count + assistant_word_count — the actual assistant word count must match assistant_word_count and meet target_word_count. A justified near miss (>=90% of target) is allowed only with allow_under_target: true and a length_exception_reason.
  • batch_global_range (e.g. "31-35") — once every row of the batch is present in the output JSONL, the batch's summed actuals must be at least 85% of the summed targets, so a healthy-looking average can't hide a uniformly short batch.

Rows without these keys skip the checks entirely.

Two companion scripts help audit and maintain a long run:

  • scripts/check_length_profile.py — reports per-batch length spread (flat vs varied), batch totals against a planning center, and a drift EMA; --style heuristically flags rows whose final paragraph reads like a stated thesis instead of a concrete image.
  • scripts/sync_durable_copies.py — after an in-place row edit in the JSONL (review rewrite, length repair), re-syncs the per-chunk formatting.txt artifact and the raw_extractions DB copy to match.

Output format

Each line of output/training.jsonl is one training example:

{
  "messages": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "metadata": {
    "source_chunk_index": 1,
    "emotion": "...",
    "archetype": "...",
    "characters": ["..."],
    "genre_tags": ["..."],
    "generation_model": "...",
    "pipeline_version": "0.1"
  }
}

The exact shape is set by prompts/formatting.md — edit it to match your trainer's expected schema.


Customizing

  • Prompts — every step is an editable file in prompts/. The script uses these if present and falls back to built-in defaults otherwise. You can also write the defaults out with python3 scripts/process_chunks.py --write-default-prompts.
  • Chunk sizing — tune --min-tokens / --max-tokens on chunk_source.py.
  • World schema — schema/schema.sql is idempotent; extend it to fit your world.

Tests

python3 -m unittest discover -s tests -v

A note on copyright

booktalk ships with no book text and must never be distributed with copyrighted source material. You supply your own sources and are responsible for ensuring you have the right to use them. The sources/ directory is gitignored for this reason.


License

Licensed under the Apache License 2.0 — see LICENSE and NOTICE.


Acknowledgments

  • Darkhn — original pipeline design and methodology, from his Animus RP models. Hugging Face · GitHub
  • UnstableLlama — generalization, open-sourcing, and maintenance of booktalk. Hugging Face

About

Agentic RP database & dataset generation from source material

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages