Turn books into trainable roleplay datasets.
booktalk is a pipeline for converting prose fiction (a novel, a series, anything you have the rights to use) into structured, character-grounded roleplay (RP) training data. The name plays on BookTok — booktalk is about making books talk: animating their characters and worlds into dialogue you can train on.
It produces two outputs in lockstep:
- A JSONL RP dataset —
messages/metadatatraining examples with original, canon-grounded dialogue scenes (never copied source prose). - A SQLite "world database" — a persistent, queryable canon of characters, their temporal arcs, relationships, factions, locations, items, and events, plus full pipeline bookkeeping (which chunk produced what, raw model extractions, and per-chunk status).
The world DB is what makes the dataset coherent across a long book: each new chunk is generated with knowledge of the reviewed entities discovered so far, so characters stay in voice and canon stays consistent.
The pipeline design and methodology are by Darkhn, originally developed for his Animus roleplay models. booktalk is a generalized, open-source implementation of that approach, built and maintained by UnstableLlama. See Acknowledgments.
booktalk is model-agnostic and bring-your-own-model. It never calls a hosted API for you. Instead it:
- Chunks your source book at natural boundaries (chapters, scene breaks).
- For each chunk, assembles an exact prompt pack — one prompt per pipeline step, pre-filled with the chunk text and the current reviewed world entities.
- You run those prompts through any LLM you like and save the outputs as text files.
- booktalk validates and finalizes those outputs into the JSONL dataset and the world DB.
This keeps you in full control of the generation model and cost, and means the same workspace works whether your "model" is a local Llama, a hosted API, or an agent running the prompts interactively.
Each chunk passes through eight ordered steps (prompts live in prompts/):
| # | Step | Purpose |
|---|---|---|
| 1 | emotion |
Primary emotional tone of the passage, as felt in-world. |
| 2 | features |
Reusable RP tags (setting, props, traits, genre tags). |
| 2b | world_extraction |
NEW/UPDATED canon entities as structured JSON for the world DB. |
| 3 | archetype |
The narrative archetype and its dynamic. |
| 4 | casting |
Up to four characters/locations for the scene, preferring known canon. |
| 5 | scene_brief |
~200 words of director's notes grounding an original scene. |
| 6 | dialogue |
The roleplay scene, as conventional prose with integrated dialogue. |
| 7 | formatting |
The final JSONL training object (messages + metadata). |
(Step 8, world DB update, is performed by code at finalization — see
prompts/world_db_update.md.)
- Python 3.9+ (standard library only — no third-party dependencies).
- An LLM of your choosing to generate the per-step outputs.
booktalk/
├── scripts/ operational scripts (init, chunk, seed, process, audit, sync)
├── prompts/ editable prompt templates, one per pipeline step
├── schema/ SQLite schema (world DB + pipeline tracking)
├── seeds/ reviewed-entity seed loader input (+ seed.example.json)
├── tests/ unit tests
├── sources/ YOUR source book(s) go here (gitignored)
├── chunks/ emitted chunk text files (gitignored)
├── intermediates/ per-chunk prompt packs + step outputs (gitignored)
├── output/ final JSONL dataset(s) (gitignored)
└── data/ SQLite database (gitignored)
sources/, chunks/, intermediates/, output/, and data/ are intentionally
gitignored — they hold copyrighted source text, large derived data, and the
database. Only the code, prompts, schema, and the example seed are committed.
# 1. Initialize the SQLite database (world DB + tracking tables)
python3 scripts/init_db.py
# 2. Put your source book in sources/ (plain text, UTF-8), e.g.
# sources/mybook.txt
# 3. (Optional but recommended) seed a few hand-reviewed canon entities
cp seeds/seed.example.json seeds/mybook_seed.json # then edit it
python3 scripts/load_seed.py seeds/mybook_seed.json --dry-run
python3 scripts/load_seed.py seeds/mybook_seed.json
# 4. Chunk the book at natural boundaries
python3 scripts/chunk_source.py \
--source sources/mybook.txt \
--book "My Book" --book-order 1 \
--prefix mybook \
--min-tokens 2000 --max-tokens 8000
# 5. Prepare the prompt pack for the next pending chunk
python3 scripts/process_chunks.py --prefix mybook --next --prepare-only --limit 1--prefix controls the filenames for emitted chunks and intermediates
(mybook_000001.txt, etc.). Use the same --prefix for chunk_source.py and
process_chunks.py. It defaults to chunk.
After step 5, a prompt pack exists at
intermediates/<prefix>_000001/prompts/. Run each prompt through your LLM and save
the raw outputs as text files next to it:
intermediates/<prefix>_000001/emotion.txt
intermediates/<prefix>_000001/features.txt
intermediates/<prefix>_000001/world_extraction.txt
intermediates/<prefix>_000001/archetype.txt
intermediates/<prefix>_000001/casting.txt
intermediates/<prefix>_000001/scene_brief.txt
intermediates/<prefix>_000001/dialogue.txt
intermediates/<prefix>_000001/formatting.txt
Then finalize:
python3 scripts/process_chunks.py --prefix mybook --chunk-id 1 --finalize-artifactsFinalization:
- validates the required artifacts exist and that JSON steps parse,
- validates the formatted example: rejects assistant turns with unbalanced plain double quotes (a clipped closing quote is easy to miss and bad training signal), and trims exact duplicate copies of the final assistant paragraph (a common padding failure of length-extension passes),
- enforces optional length targets (see below),
- applies
world_extractionrows into the world DB (markedreviewed=0, for later review), - appends the formatted example to
output/training.jsonl, keepingintermediates/<prefix>_XXXXXX/formatting.txtand theraw_extractionsDB copy byte-identical to what was appended, - records all eight raw step outputs in
raw_extractions, - writes a
generation_logrow and marks the chunkcompleted.
If anything fails, the chunk is marked failed and an error.txt is written in its
intermediates directory; nothing is partially committed to the dataset.
Before finalizing a chunk, resolve unnamed / epithet-only characters. If a later chunk (or your own correction) identifies one, promote the world DB id to the canonical character and keep the original descriptor as an alias. Seeding canon up front (step 3) greatly reduces this.
Long generation runs drift: assistant turns shrink toward a uniform length unless
targets are enforced at finalization. booktalk supports (but does not require) a
ranked-allocation workflow — plan a batch of chunks, rank the scenes, and give
each row a word-count target — via optional metadata keys on the formatted
example:
target_word_count+assistant_word_count— the actual assistant word count must matchassistant_word_countand meettarget_word_count. A justified near miss (>=90% of target) is allowed only withallow_under_target: trueand alength_exception_reason.batch_global_range(e.g."31-35") — once every row of the batch is present in the output JSONL, the batch's summed actuals must be at least 85% of the summed targets, so a healthy-looking average can't hide a uniformly short batch.
Rows without these keys skip the checks entirely.
Two companion scripts help audit and maintain a long run:
scripts/check_length_profile.py— reports per-batch length spread (flat vs varied), batch totals against a planning center, and a drift EMA;--styleheuristically flags rows whose final paragraph reads like a stated thesis instead of a concrete image.scripts/sync_durable_copies.py— after an in-place row edit in the JSONL (review rewrite, length repair), re-syncs the per-chunkformatting.txtartifact and theraw_extractionsDB copy to match.
Each line of output/training.jsonl is one training example:
{
"messages": [
{"role": "system", "content": "..."},
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."}
],
"metadata": {
"source_chunk_index": 1,
"emotion": "...",
"archetype": "...",
"characters": ["..."],
"genre_tags": ["..."],
"generation_model": "...",
"pipeline_version": "0.1"
}
}The exact shape is set by prompts/formatting.md — edit
it to match your trainer's expected schema.
- Prompts — every step is an editable file in
prompts/. The script uses these if present and falls back to built-in defaults otherwise. You can also write the defaults out withpython3 scripts/process_chunks.py --write-default-prompts. - Chunk sizing — tune
--min-tokens/--max-tokensonchunk_source.py. - World schema —
schema/schema.sqlis idempotent; extend it to fit your world.
python3 -m unittest discover -s tests -vbooktalk ships with no book text and must never be distributed with copyrighted
source material. You supply your own sources and are responsible for ensuring you
have the right to use them. The sources/ directory is gitignored for this reason.
Licensed under the Apache License 2.0 — see LICENSE and NOTICE.
- Darkhn — original pipeline design and methodology, from his Animus RP models. Hugging Face · GitHub
- UnstableLlama — generalization, open-sourcing, and maintenance of booktalk. Hugging Face