The Garden of Forking Prompts: How Users Explore Narrative Space in Story Generation

Advait Deshmukh; Nora Benedict; Melanie Walsh; Maria Antoniak

cs.CLSep 13, 2026

The Garden of Forking Prompts: How Users Explore Narrative Space in Story Generation

Authors: Advait Deshmukh, Nora Benedict, Melanie Walsh, Maria Antoniak

Organizations: University of Colorado Boulder · University of Georgia · University of Washington

Abstract

Large language models (LLMs) have changed the way people engage with stories. Drawing on public chatbot logs, we can see that when users generate stories, they iteratively edit their prompts to explore narrative possibilities, adjusting characters, redirecting plots, and swapping fictional universes. As aggregated data, these prompts represent rich traces of creative preference at scale. Yet story generation evaluation benchmarks rely on static, one-shot prompts that cannot capture this exploratory behavior. In this work, we study how users revise consecutive story prompts in the wild. Using a dataset of naturally occurring user-chatbot conversations, we construct WildStories, a sample of 275,635 story generation prompts (labeled with story format, prompt components, and explicitness), and WildEdits, a collection of 24,291 edit trees that model how users iteratively edit base story prompts and explore branching story possibilities. From these trees we develop a framework of edit types crossing four directions (adding, removing, changing, and extending) with fourteen targets (e.g., plot, character, genre). We then use our datasets and this framework to analyze user behavior in navigating narrative space via LLMs. Finally, we show how automated permutations based on the framework can be used for story generation benchmarking. Content Warning: This paper works with "wild" chatbot logs, which often include toxic and sexually explicit themes.

Figures & tables

Label	Description	Example Edit (Prompt A $\rightarrow$ Prompt B)	add	remove	change	extend
Targets: Narrative Content
plot	Story events, actions, and developments	…have him transform into a poster of the studio logo napkin with the word “nope” scrawled on it …	$18{,}446$	$5{,}368$	$24{,}489$	$18{,}695$
dialogue	Specific words characters say (rewriting existing lines)	…Honestly, I feel like this whole trip has made me feel …	$1{,}876$	$613$	$3{,}084$	$109$
setting	Time, place, or physical environment	…an extended comedic scene in a Canadian Sitcom Korean Family Drama …	$2{,}743$	$615$	$2{,}892$	$78$
Targets: Characters
char. description	Physical appearance, personality, or abilities	…Diane (late forties , tall, red hair ) has been tutoring Marcus…	$11{,}673$	$2{,}891$	$8{,}899$	$188$

Table 1: Our framework of prompt edit types. Each edit is labeled with a (Direction, Target) pair. Underlined text indicates additions; struck-through text indicates removals. The four rightmost columns show action-level predicted label counts across all in-scope retained WildEdits prompt pairs. All examples are paraphrased to protect user privacy.

setting			model instructions			character description			dialogue			wording
word	PMI	SD	word	PMI	SD	word	PMI	SD	word	PMI	SD	word	PMI	SD
apartment	3.05	1.00	reply	1.90	0.06	innocent	3.19	0.19	huh	3.73	0.76	series	0.76	0.81
city	2.94	0.56	responses	1.85	0.09	tail	2.67	0.46	maybe	3.47	0.77	despite	0.49	1.18
town	2.84	0.48	suggestions	1.81	0.11	pink	2.59	0.25	contractions	3.33	0.53	extremely	0.32	0.87
set	2.82	0.57	descriptions	1.78	0.09	fat	2.56	0.74	we	3.14	0.35	together	0.27	0.68
forest	2.78	1.14	sentences	1.78	0.13	confident	2.50	0.35	you’re	3.07	0.64	dragon	0.18	0.76

Table 2: Top words selected by user- and cluster-weighted pointwise mutual information for five edit targets. We report mean and one standard deviation across 300 bootstrap samples of the users, with rows ordered by bootstrap mean within each target. PMI shading is normalized within each target column; SD shading is on a shared scale across the table. Words are diff features in annotated edit spans, not full prompt texts.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Examples of edit trees (paraphrased prompts for privacy).

Figure 2: The pipeline describing the sampling and processing of data we do to obtain WildStories and WildEdits , starting from WildChat.

	All prompts			Non-toxic prompts
Cohort	$N$	Toxic	Explicit	$N$	Explicit
WildChat-4.8M	$2{,}560{,}447$	$33.7$ %	$8.5$ %	$1{,}697{,}403$	$3.3$ %
WildStories	$275{,}635$	$67.9$ %	$43.5$ %	$88{,}451$	$28.5$ %

Appendix

Table 3: Prevalence of toxic (as determined by WildChat) and explicit (as determined by our classifier) content among filtered (English,

\geq 20

words) WildChat-4.8M prompts and WildStories .

Story mode
Label	add	remove	change	extend
Prose	$38{,}678$	$12{,}192$	$48{,}278$	$8{,}000$
Roleplay	$3{,}866$	$1{,}685$	$6{,}462$	$2{,}530$
Script	$13{,}081$	$3{,}763$	$18{,}253$	$8{,}627$
Narration	$186$	$48$	$183$	$14$
Prompt components

Appendix

Table 4: Action-level edit-direction counts by the story structure of the parent prompt across all

100{,}200

in-scope retained WildEdits prompt pairs (

165{,}846

classified edit actions). Story modes are mutually exclusive. Prompt components are binary and may co-occur, so component rows overlap.

Figure 3: Distribution of edit-tree sizes. Histogram of the number of prompts per non-singleton edit tree (

N=24{,}291

). Tree sizes are strongly right-skewed: the median tree contains three prompts, while a small number contain hundreds or thousands.

Figure 4: Distribution of edit-tree activity across inferred users. Histogram of the number of non-singleton edit trees attributed to each inferred user (

N=2{,}583

). User activity is strongly right-skewed: the median user contributes one edit tree, while a small number contribute hundreds or thousands.

Figure 5: Edit-type correlates of narrative arc change. Bars show the change in primary story-arc transition rate, relative to the overall baseline, for chains containing each edit type. Positive values indicate stronger co-occurrence with arc changes.

Figure 6: Bars show the proportion of chains whose primary story arc differs between the first and last response, grouped by chain length in prompts.

Figure 7: Arc transitions showing flows from the initial prompt to the final one in an edit-chain.

Figure 8: Edit transitions showing flows from one edit type to the next, making the most common revision trajectories (to and from plot) visually obvious.

Figure 9: Distribution of edit tree sizes, quantified by the number of nodes per tree, across the requested story formats of the root prompt. We uniformly sample one eligible tree per user, restricting the analysis to trees with at least five nodes.

		L2 distance			Comp.
Method	Matched	Mean	Med.	P90	MAE
Our pipeline	495	0.15	0.12	0.29	0.05
Naive	498	0.28	0.16	0.91	0.10
Luminate	499	0.74	0.75	0.94	0.24

Appendix

Table 5: Distance between synthetic and real edits, where edits are represented by their drift vector

\mathbf{d}

. L2 is the Euclidean distance between synthetic and real drift vectors matched by source prompt; Comp. MAE is the mean absolute difference per drift component, averaged over the four components and prompts. Across all reported distance metrics, our pipeline is significantly closer to real edits than both baselines under paired source-level permutation tests (

p<0.05

Figure 10: Directed transition probabilities between edit targets in consecutive edits, calculated over all eligible adjacent edit pairs in WildEdits .

Figure 11: Pointwise mutual information (PMI) between edit targets co-occurring within the same edit, calculated over all edits in WildEdits .

Figure 12: PMI association of edit targets with prompt components, calculated over all edits in WildEdits .

Figure 13: PMI association of edit targets with story formats, calculated over all edits in WildEdits .

Figure 14: Edit diffs between prompts. Edit volume represents the number of characters added

-

the number of characters removed.

Explore similar work

NARRA-Gym for Evaluating Interactive Narrative Agents
May 8, 2026Yue Huang, Yuchen Ma, Jiayi Ye +14Language Model Generation Evaluation Interactive Storytelling
University of Notre Dame · 2LMU Munich · 11Munich Center for Machine Learning +8
StoryAlign: Evaluating and Training Reward Models for Story Generation
May 6, 2026Haotian Xia, Hao Peng, Yunjia Qi +4Reward Modeling Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University
When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations
Aug 16, 2026Yuqi Chen, Sixuan Li, Yunfeng Cai +3Language Model Generation Evaluation Interactive Storytelling
The University of Hong Kong · Peking University · Beijing Institute of Mathematical Sciences and Applications (BIMSA) +1

May 8, 2026J/K move · Enter open · S save

NARRA-Gym for Evaluating Interactive Narrative Agents

Yue Huang, Yuchen Ma, Jiayi Ye, Wenjie Wang+13

University of Notre Dame · 2LMU Munich · 11Munich Center for Machine Learning+8

Language Model Generation Evaluation Interactive Storytelling LLM Agent Evaluation Long-Horizon LLM Agents

Open paper AskPDF arXiv

Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmarks for this setting are limited: existing evaluations often focus on static prompts, isolated story generations, or post-hoc ratings, and therefore miss whether models can jointly manage story generation, long-context state and pacing, character simulation, empathic personalization, and story-grounded artifacts. We introduce NARRA-Gym, an executable evaluation environment that turns a sparse emotional seed into a complete interactive story episode and logs the full model-in-the-loop trajectory, including story construction, memory updates, planning, pacing interventions, and optional artifact synthesis. We evaluate nine frontier LLMs using a controlled LLM-as-judge sweep over eight benchmark personas and a human evaluation in which participants rate customized model outputs. Our results show substantial variation across models, personas, and evaluation dimensions: models that produce fluent stories can still fail on robustness, user experience, or resistance-sensitive personalization. These findings suggest that interactive narrative offers a useful benchmark for evaluating long-horizon, user-adaptive LLM behavior beyond isolated story quality.

Abstract

Explore similar work

NARRA-Gym for Evaluating Interactive Narrative Agents

StoryAlign: Evaluating and Training Reward Models for Story Generation

When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations