When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations

Yuqi Chen; Sixuan Li; Yunfeng Cai; Xueai Li; Ka Man Yan; Ying Li

cs.CLAug 16, 2026

When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations

Authors: Yuqi Chen, Sixuan Li, Yunfeng Cai, Xueai Li, Ka Man Yan, Ying Li

Organizations: The University of Hong Kong · Peking University · Beijing Institute of Mathematical Sciences and Applications (BIMSA) · Tsinghua University

Abstract

Large language models can write fluent stories, but open-ended storytelling requires more than local fluency. In evolving world simulations and AI-native games, models must preserve facts, relationships, causal dependencies, and character states as the world changes. We introduce WSE-bench, a process benchmark that separately evaluates sustained generation, canonical coherence, and meaningful development in dynamic LLM storytelling. Generation Coverage records the proportion of planned narrative steps produced; Consistency tracks when canon breaks; and Richness measures how meaningfully branching, player-shaped trajectories develop. Across frontier models, Consistency and Richness do not form a smooth trade-off: their empirical Pareto frontier is non-concave, with several non-dominated intermediate configurations that no positive linear weighting can select. Added structure can enrich trajectories, but it does not uniformly improve coherence and may shorten them. Model scale chiefly improves sustained generation, without producing reliable gains in canonical coherence or meaningful development. These results show that sustained generation, canonical coherence, and meaningful development are distinct and sometimes competing capacities. WSE-bench makes those dynamics visible by extending narrative evaluation from finished stories to the processes that create them.

Figures & tables

Figure 2: WSE-bench combines standardized episode construction, recurrent generation loop, and systematic comparison of agent architectures. Each episode contains one player-controlled character and four non-player characters. At each step, $B_{1}$ lets the Story Agent decide consequences directly, $B_{2}$ uses one Character Agent to propose for all non-player characters, and $B_{3}$ uses one isolated Character Agent per non-player character; the Story Agent remains the sole narrator. WMOD optionally organizes proposed actions into a pre-narration PlotPlan of at most 12 beats, whereas WMEM optionally records a structured post-narration world-state update. Crossing the three bases with the two binary modules yields 12 architecture conditions.

Figure 3: Model-by-architecture point estimates for the Qwen and Frontier panels. Columns index models, and rows index the 12 architecture conditions. The three panels show Generation Coverage $G$ , Consistency $C_{\mathrm{obs}}$ , and Richness $R$ ; the dashed rule separates the two analysis groups. Paired architecture contrasts below provide uncertainty intervals.

Figure 4: Consistency–Richness point estimates based on

C_{\mathrm{adj}}

, which sets the survival contribution to zero after a cell loses observational support. Shapes denote

B_{1}

(Story Agent only),

B_{2}

(one centralized Character Agent), and

B_{3}

(distributed Character Agents); white overlays mark WMEM, WMOD, or both. The upper-right boundary identifies non-dominated configurations, and supported configurations lie on its upper concave envelope.

Contrast	$\Delta G$	$\Delta C_{obs}$	$\Delta C_{adj}$	$\Delta R$
Qwen series
B2 – B1	+0.8 [-1.0, +2.5]	-9.2 [-14.5, -3.2]	-3.4 [-8.3, +3.3]	+4.0 [+2.6, +6.0]
B3 – B1	-1.1 [-3.0, +0.9]	-12.0 [-17.7, -5.4]	-8.4 [-10.5, -1.7]	+7.0 [+4.4, +9.0]
B3 – B2	-1.9 [-3.5, -0.3]	-2.8 [-8.5, +3.6]	-5.0 [-7.7, +0.4]	+3.0 [+0.2, +4.8]
WMEM on – off	-18.0 [-19.8, -16.1]	-5.6 [-10.2, -0.8]	-6.0 [-8.8, -0.6]	-1.3 [-2.5, +0.1]
WMOD on – off	-13.7 [-15.6, -11.7]	+4.7 [-1.0, +11.9]	-22.9 [-25.7, -16.0]	-5.9 [-7.2, -4.2]

Table 1: Architecture effects on the four reported outcomes. Each contrast averages over the other architecture factors. Values are mean differences across the evaluated models, weighted equally, with 95% hierarchical cluster-bootstrap intervals.

Figure 5: Per-model Consistency

C_{\mathrm{obs}}

across the three world families, with each model pooled over 12 architectures. Markers and whiskers show model-specific means and 95% hierarchical cluster-bootstrap intervals. The plotted profiles show within-model source-family variation; aggregate source-family contrasts give each of the 12 evaluated models equal weight.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Initial world	Language	Opening snapshot	Seeds
ITP	Bianjing, 1100 CE	Chinese	Bianjing at the opening state in the year 1100 CE.	14
Smallville	25-agent Smallville	English	Base simulation at midnight on February 13, 2023.	14
DraCor	A Doll’s House	Dano–Norwegian	Act I, after the nurse brings the children upstairs.	2
DraCor	Uncle Vanya	Russian	Act I, just after Serebryakov, Yelena, and Sonya return from their walk.	2
DraCor	Three Sisters	Russian	Act I, after Andrei joins the guests at Irina’s name-day gathering.	2
DraCor	The Cherry Orchard	Russian	Act I, immediately after the family’s arrival in the nursery.	2

Appendix

Table 2: Initial worlds used to construct the 42 EpisodeSeed records. The snapshot identifies the shared canonical starting point, not a plot that generated trajectories are expected to follow.

Figure 7: Comparison of the two Consistency summaries across all 144 model–architecture cells. Panel (a) compares $C_{\mathrm{obs}}$ with $C_{\mathrm{adj}}$ . Panel (b) shows where they differ across architectures, and panel (c) relates that difference to Generation Coverage $G$ . The difference is concentrated in lower-coverage Qwen cells and changes no observed judgment or event location.

Figure 8: Human validation of the Judges. (a) Design-weighted Consistency classification on 120 trajectories. (b) One-vs-rest observed agreement across the same sample; the pairwise series average three comparisons. (c–d) Richness agreement and error on a separate 36-trajectory sample.

Figure 9: Horizon sensitivity for the two six-model groups. Panels (a–b) show equal-model mean generation and contradiction-free survival. Panel (c) reports the timing distribution of first contradictions observed by step 20. Panel (d) recomputes the normalized summaries at

h=5,10,15,20

Generation failures
Quantity	Qwen	Frontier
Centralized Character Agent failures	1.1%	5.4%
Distributed Character Agent failures	1.9%	12.8%
World Model failures	33.6%	6.9%
Story Agent failures	24.4%	4.8%
World Memory Agent failures	42.7%	24.8%

Appendix

Table 3: Two complementary error-propagation audits. Generation-failure rates use trajectories in which the component is enabled. Precursor rates use majority-inconsistent trajectories and report the plurality origin of conservative lexical matches to upstream content. These matches identify possible precursors, not causal effects or errors repaired downstream.

Figure 10: Architecture contrasts among paired conditions in which both trajectories reach the full horizon. Bars average the six model-specific estimates in each group. Positive values favor the first-named base or module-on condition. This analysis complements the primary analysis, which includes each trajectory up to its observed length.

Figure 11: Generation cost and coverage under four architecture contrasts. Bars average within-model percentage changes across the six models in each group. Positive values favor the first-named base or module-on condition. B3 invokes four isolated Character Agents per active step, and lower coverage can reduce later calls and tokens.

Figure 12: Generation outcomes by model across all 12 architectures. Each bar represents 504 planned trajectories. Full horizon indicates 20 completed narrative steps. Story ending denotes an accepted ending, such as the player’s death, permanent incapacity, or a genuinely closed story. Only four occur. Output-format and other technical failures reduce Generation Coverage but are not story outcomes.

Figure 13: Finite-horizon survival curves for the Qwen series. Panel (a) shows the probability of generating through step

t

. Panel (b) shows contradiction-free survival, with trajectories that stop for technical reasons right-censored. Dotted tails have fewer than 30 trajectories at risk. Curves end at the experimental horizon

H=20

, and their integrals yield the summaries in Figure 14 . Qwen3.5-397B-A17B is shown separately as an MoE model.

Figure 14: Qwen-series scaling under unequal trajectory coverage. Dense-model fits use log parameter count for Qwen2.5 3B–72B. Qwen3.5-397B-A17B is shown separately as an MoE model. Open markers have fewer than 30 trajectories at risk at step 20.

First-violation reason	Qwen: $n$ (%) / step	Frontier: $n$ (%) / step
Temporal conflict	131 (25.8%) / 8.0	144 (31.0%) / 11.0
World-fact conflict	52 (10.3%) / 10.0	88 (18.9%) / 13.0
Character-state conflict	70 (13.8%) / 7.0	38 (8.2%) / 9.0
Identity conflict	42 (8.3%) / 2.0	23 (4.9%) / 7.0
Relationship conflict	34 (6.7%) / 2.0	6 (1.3%) / 4.0
Knowledge conflict	2 (0.4%) / 10.0	4 (0.9%) / 11.5

Appendix

Table 5: Adjudicated reasons for the first inconsistency, excluding generation failures. A category requires agreement from at least two inconsistent Judges. Percentages use all majority-inconsistent trajectories in each model panel. The value after each slash is the median first contradictory step.

Agreed reason	Initial World	Earlier Narrative	Current Narrative	Unresolved tied	Total
Temporal	21	82	17	155	275
World fact	2	63	2	73	140
Character state	5	54	9	40	108
Identity	48	0	1	16	65
Relationship	39	0	0	1	40
Knowledge	0	3	1	2	6

Appendix

Table 6: Candidate source of the canonical commitment violated at the first inconsistent step, pooled across both model panels. Sources are localized from the Judges’ stated prior commitment. Ambiguous and tied cases are grouped as unresolved. The table identifies where a commitment was recorded, not which agent caused the conflict.

Figure 15: Leave-one-DraCor-play-out sensitivity for the 12-model

C_{\mathrm{obs}}

contrasts. Each point compares DraCor with a simulation-native control after omitting one play. All DraCor–Smallville contrasts remain positive; one DraCor–ITP contrast is slightly below zero (

-0.27

points).

Consistency: estimate [95% bootstrap CI]
Measure	ITP	DraCor	Smallville
$C_{\mathrm{obs}}$	82.1 [78.4, 86.1]	83.5 [79.1, 87.9]	78.1 [75.7, 84.0]
$C_{\mathrm{adj}}$	80.3 [77.3, 82.8]	83.5 [77.1, 87.8]	75.2 [72.4, 77.0]
DraCor contrast: difference [95% bootstrap CI]
$C_{\mathrm{obs}}$	+1.4 [-4.6, +6.9]	—	+5.4 [-2.2, +9.9]
$C_{\mathrm{adj}}$	+3.2 [-3.8, +8.3]	—	+8.3 [+1.8, +13.5]

Appendix

Table 7: Source-family Consistency and trajectory timing. Cells report equal-model means with 95% hierarchical bootstrap intervals. DraCor contrasts subtract ITP or Smallville from DraCor.

Qwen series
Generator	Contrast	$\Delta G$	$\Delta C_{\mathrm{obs}}$	$\Delta C_{\mathrm{adj}}$	$\Delta R$
Qwen2.5-3B	B2 – B1	$+1.8\,[-0.8,+4.4]$	$-37.6\,[-64.2,-8.4]$	$+0.1\,[-24.2,+33.3]$	$+5.8\,[-1.1,+15.0]$
	B3 – B1	$+0.0\,[-1.7,+1.3]$	$-38.9\,[-62.0,-9.8]$	$-20.0\,[-24.3,+6.8]$	$+10.8\,[-2.3,+20.2]$
	B3 – B2	$-1.8\,[-3.7,-0.3]$	$-1.3\,[-29.1,+31.6]$	$-20.0\,[-30.6,+3.2]$	$+5.0\,[-10.0,+13.9]$
	WMEM on – off	$+0.7\,[-1.1,+2.3]$	$-9.3\,[-33.8,+10.6]$	$-25.1\,[-33.5,+5.6]$	$+1.2\,[-5.3,+8.0]$
	WMOD on – off	$-0.8\,[-2.5,+0.7]$	$-1.2\,[-25.0,+27.1]$	$-20.0\,[-29.9,+9.0]$	$-4.9\,[-10.4,+3.5]$

Appendix

Table 8: Per-model counterpart to main-text Table 1 for both six-model groups. Each block reports the same five marginal contrasts as point estimates with 95% hierarchical bootstrap intervals. Models are fixed and are not resampled.

Qwen series
Generator	Conditional contrast	$\Delta G$	$\Delta C_{\mathrm{obs}}$	$\Delta C_{\mathrm{adj}}$	$\Delta R$
Qwen2.5-3B	WMEM at B1	$+0.4\,[-3.2,+2.7]$	$-11.9\,[-47.5,+0.0]$	$-44.4\,[-49.5,+7.5]$	$-1.1\,[-12.1,+10.9]$
	WMEM at B2	$+0.1\,[-4.6,+3.9]$	$-25.1\,[-70.0,+28.2]$	$-30.7\,[-58.6,+13.5]$	$+4.1\,[-11.3,+15.7]$
	WMEM at B3	$+1.5\,[-0.1,+3.0]$	$+9.0\,[-55.1,+64.5]$	$-0.1\,[-11.9,+9.2]$	$+0.5\,[-10.7,+15.8]$
	WMOD at B1	$-0.4\,[-3.9,+1.4]$	$-11.9\,[-47.5,+0.0]$	$-39.4\,[-45.0,+10.0]$	$-2.8\,[-11.0,+11.8]$
	WMOD at B2	$-1.9\,[-6.2,+1.7]$	$-5.4\,[-38.7,+37.9]$	$-18.8\,[-44.4,+22.8]$	$+1.4\,[-9.7,+12.4]$

Appendix

Table 9: Per-model module effects within each base for both six-model groups. Each module is compared on versus off while averaging over the other. Entries are point estimates with 95% hierarchical bootstrap intervals. Models are fixed and are not resampled.

Explore similar work

Sep 26, 2026

World Agent: Can Language Models Keep a World Running?

World models are moving from generating realistic frames to generating playable worlds, yet whether a delivered world can keep running is not tested anywhere. Existing evaluations stop at generation, at delivery, or at single-step transitions, and each stops at a different point along the way. Correct local state transitions or intermediate outcomes do not guarantee a correctly organized causal event flow. We propose the world agent task, which moves the evaluation point of world generation from the moment of delivery to the continued operation that follows. In this task, a model is not asked to generate a world. It is held responsible for keeping the world running, which requires coordinating events and carrying forward their consequences to constrain subsequent evolution. We instantiate the task in WorldAgent-Benchmark with two complementary tracks. In the maintenance track, the model must ground the events of a continuous narrative into correct transitions of the explicit world state while respecting causal, temporal, and concurrency constraints. In the deduction track, the model must predict how the world will evolve under partial observations and act toward a goal. The maintenance track combines LLM-assisted semantic judgments with programmatic validation and scoring, while the deduction track is evaluated entirely programmatically. Individual judgments are auditable against world states and execution logs, and scores can be recomputed from the saved judgments and execution records. Across 8 models, scores decline steadily as pre-built structure is removed from the world, and causal-relation checking is the weakest component for every model. The benchmark makes the continued operation of a world measurable and distinguishes local completion from failures in event organization. Code and dataset will be released on https://github.com/HCPLab-SYSU/WorldAgent-Benchmark.

Weixing Chen, Weipeng Zhang, Nan An +2

Sun Yat-sen University · South China University of Technology · Guangdong Key Laboratory of Big Data Analysis and Processing +2

World Model Evaluation World Models Causal World Models Causal Reasoning in Language Models Temporal Reasoning in Language Models

Ask PDF

Sep 13, 2026

The Garden of Forking Prompts: How Users Explore Narrative Space in Story Generation

Large language models (LLMs) have changed the way people engage with stories. Drawing on public chatbot logs, we can see that when users generate stories, they iteratively edit their prompts to explore narrative possibilities, adjusting characters, redirecting plots, and swapping fictional universes. As aggregated data, these prompts represent rich traces of creative preference at scale. Yet story generation evaluation benchmarks rely on static, one-shot prompts that cannot capture this exploratory behavior. In this work, we study how users revise consecutive story prompts in the wild. Using a dataset of naturally occurring user-chatbot conversations, we construct WildStories, a sample of 275,635 story generation prompts (labeled with story format, prompt components, and explicitness), and WildEdits, a collection of 24,291 edit trees that model how users iteratively edit base story prompts and explore branching story possibilities. From these trees we develop a framework of edit types crossing four directions (adding, removing, changing, and extending) with fourteen targets (e.g., plot, character, genre). We then use our datasets and this framework to analyze user behavior in navigating narrative space via LLMs. Finally, we show how automated permutations based on the framework can be used for story generation benchmarking. Content Warning: This paper works with "wild" chatbot logs, which often include toxic and sexually explicit themes.

Advait Deshmukh, Nora Benedict, Melanie Walsh +1

University of Colorado Boulder · University of Georgia · University of Washington

LLM Prompting Human-AI Co-Creation Text Generation Language Model Generation Evaluation

Ask PDF

Apr 10, 2026

Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling

LLMs have so far failed both to generate consistently compelling stories and to recognize this failure--on the leading creative-writing benchmark (EQ-Bench), LLM judges rank zero-shot AI stories above New Yorker short stories, a gold standard for literary fiction. We argue that existing rubrics overlook a key dimension of compelling human stories: narrative tension. We introduce the 100-Endings metric, which walks through a story sentence by sentence: at each position, a model predicts how the story will end 100 times given only the text so far, and we measure tension as how often predictions fail to match the ground truth. Beyond the mismatch rate, the sentence-level curve also yields complementary statistics that track plot-level twists and revelations, such as the inflection rate, a geometric measure of how frequently the curve reverses direction. Unlike rubric-based judges, 100-Endings correctly ranks New Yorker stories far above LLM outputs. Grounded in narratological principles, we design a story-generation pipeline using structural constraints, including analysis of story templates, idea formulation, and narrative scaffolding. Our pipeline significantly increases narrative tension as measured by the 100-Endings metric, while maintaining performance on the EQ-Bench leaderboard.

Peiqi Sui, Yutong Zhu, Tianyi Cheng +4

McGill University · University of Chicago · University of British Columbia +1

Language Model Generation Evaluation Text Generation

Ask PDF