Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation

Syed Rifat Raiyan; Md. Tariquzzaman; Md Farhan Ishmam; Tenghao Huang

cs.CLOct 15, 2026

Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for Nonlinear Narrative Generation

Authors: Syed Rifat Raiyan, Md. Tariquzzaman, Md Farhan Ishmam, Tenghao Huang

Organizations: Islamic University of Technology, Bangladesh · University of Utah, USA · University of Southern California, USA

Abstract

Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We introduce a benchmark of 83 published works that measures whether language models can tell nonlinear stories. For each work, we annotate one list of events in both orders, together with a story graph of causal dependencies, character knowledge, and the questions the telling raises and resolves. Given events in chronological order, all 19 models tell them, on average, more chronologically than the published works do; asked for a specific structure, they land within the human range about half the time. We then compare a model's stories of the same events under different conditions, using both an automated judge and 71 human readers. Both prefer the chronological story a model writes by default to the nonlinear one it writes on request, which readers find harder to follow in 81% of decisive judgments. Readers also prefer stories written from the story graph to stories constrained to the published telling order (72% of decisive judgments), a difference the judge does not detect. Our results suggest that following an instruction to reorder a story is not yet the same as telling it well out of order.

Figures & tables

Panel of Figure 1 — Figure 1: Without a cue, the model stays near chronology; a backward cue yields exact reversal, while the published telling preserves forward scene runs. Kendall’s $\tau$ measures agreement with chronology: $+1$ denotes chronological order and $-1$ exact reversal.

Figure 2: The same events, two orders: the explanation precedes the consequence in chronology but arrives last in the telling. Opening with Roy’s death and withholding why he became the target across 16 intervening events turns a known outcome into an unresolved question.

Corpus
works, each annotated in both orders	83
gold narrative graphs	83
event nodes	1,433
scenes	1,221
characters	857
mean $\tau$ of the told order	0.456

Table 1: The corpus. Counts are recomputed from the annotation and independently re-derived by a separate check.

Panel of Figure 3 — Figure 3: Story graph of work S60, a musical labeled zigzag , with events in chronological order and told positions beneath. causes and foreshadows arcs run above the line; each reveals arc below points from a later-told explanation to the earlier-told event it explains, answering the reader question $q_{i}$ it carries ( shares edges omitted). Events: 1 a novelist meets an actress; 2 she tells him of the man she nearly settled for; 3 his first book is accepted; 4 a casting panel ignores her; 5 he promises never to leave; 6 they marry; 7 she takes a provincial engagement; 8 his book succeeds; 9 she confronts his absence; 10 a guest propositions him; 11 she resents his success; 12 he is unfaithful; 13 he leaves a letter; 14 she finds the flat empty. Questions: $q_{1}$ what professional hurt lies beneath her accusation; $q_{2}$ how his success became distance; $q_{3}$ why she can no longer be glad for him; $q_{4}$ why the flat is empty.

Panel of Table 2 — Table 2: Benchmark grid, sorted by $\mathrm{T}_{1}$ band hit (left block, then right). $\mathrm{T}_{1}$ : structural compliance. $\mathrm{T}_{2}$ : spontaneous ordering. $\mathrm{T}_{3}$ : reconstruction. $\mathrm{T}_{4}$ : classification. Italic : open-weights. Bold : best per column (in the $\tau$ columns, closest to the human 0.456, descriptive only). Human band hit ( $\dagger$ ) is 1 by definition. Band-hit intervals are about $\pm 0.11$ , so adjacent ranks are not distinguishable (§ I ). Read $\mathrm{T}_{3}$ $\tau_{\mathrm{rec}}$ against the no-story floor (§ 5.2.1 ).

Figure 4: The no-story floor. Top:

\mathrm{T}_{3}

\tau_{\mathrm{rec}}

for each model, against a solver given the events with the telling deleted (dashed). The two come from separate runs, so the panel calibrates the scores and does not rank the models. Bottom: the same solvers with and without the telling, from the de-dating runs.

Condition	$\tau$	band hit	agree	PDS
$\mathrm{G}_{0}$ no graph	0.362	0.411	0.554	3.85
$\mathrm{G}_{1}$ full graph	0.415	0.518	0.554	4.10
$\mathrm{G}_{2}$ told order (oracle)	0.333	0.982	0.998	3.94
$-$ CAUSES / FORESHADOWS	0.417	0.429	0.557	4.18
$-$ pre / post	0.440	0.464	0.557	4.14
$-$ learns	0.399	0.464	0.561	4.05

Table 3: The scaffolding study. Ablations below the rule each remove one graph layer from

\mathrm{G}_{1}

; agreement with the gold telling confirms that

\mathrm{G}_{2}

follows the order it is given. Rows use the 2 models that ran every condition, the only ones that allow a paired comparison; pooling all 4 models over the three conditions they share gives the same ordering at smaller margins.

Comparison	split	rate	$p$	split	rate	$p$	split	rate	$p$	rate
	better story			harder to follow			better ending			adj.
instructed vs. left alone	64–200–30	0.242	$<0.0001$	197–47–50	0.807	$<0.0001$	114–134–46	0.460	$0.23$	0.246
full graph vs. told order	181–70–43	0.721	$<0.0001$	115–108–71	0.516	$0.69$	141–106–47	0.571	$0.0303$	0.707
full graph vs. event list	154–105–35	0.595	$0.0028$	123–102–69	0.547	$0.18$	133–103–58	0.564	$0.0588$	0.518
full graph vs. one layer removed	79–180–35	0.305	$<0.0001$	140–86–68	0.619	$0.0004$	110–119–65	0.480	$0.60$	0.304

Table 4: Reader verdicts. Each pair is one model writing one story twice. split : wins–losses–ties for the first-named condition. rate : share of decisive judgments. adj. : rate net of word count ( Dubois et al., 2024 ) . violet headline, cyan survives every check, magenta against hypothesis, lavender fails length adjustment.

Panel of Figure 5 — Figure 5: Unadjusted preference rates. Readers favor graph-guided stories over those given a telling order or an event list, and the judge’s ties on two comparisons miss clear reader preferences. “Layer removed” pools the five one-layer ablations.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: The seven shapes, each drawn from the corpus work nearest its category’s median in $(\tau,\mathrm{LIS})$ . Told position runs across and chronological position up, so chronological order is the dotted diagonal. Published retrograde descends in steps with short forward runs, not in the straight corner-to-corner line models return (§ 5.1.2 ). Colors match Figure 8 .

	What we found	What to be careful about
RQ1	Every model tells a story more chronologically than its published telling; cued for a shape, band hit is 47.9%, and the two misses are the literal solutions of rules stated on events.	The reference works were chosen for nonlinearity, both free tasks list events chronologically, and no prompt marks scenes. Band hit is negatively associated with reader preference, partly through condition (§ 5.5.4 ).
RQ2	Reconstruction reaches $\tau_{\mathrm{rec}}=0.743$ ; naming the shape reaches 50.2%, with analepsis a sink.	96.1% of that $\tau$ is reachable with the story deleted (by entailment or prior knowledge of the work), and readers reach only 0.537: report the floor, prefer exact match.
RQ3	The graph raises compliance from 41.1% to 51.8%, almost all in one model, and readers prefer it to a supplied told order 72% of the time.	The judge saw no effect, and the two prompts differ in more than the graph. The edge over an event list is a length effect, and readers prefer every one-layer ablation to the full graph.
RQ4	Cued tellings lose to a model’s own free telling under the judge (68–19) and readers (76%), even for chronicle . As scored, the reveal comes after fewer than three events in 21.4% of stories.	Two scoring cases inflate the early rate (8.3% without them). Retrospective coherence gain is capped near 0.12 by the same floor.
RQ5	The judge matches 71 readers on the one large effect, and readers overturn both of its ties ( $\kappa=-0.030$ on layer removal).	Item-level agreement does not track effect size; fourteen stories, two generators in the layer-removal study, no comparison against human-written prose.

Appendix

Table 5: The paper at a glance. The reader study revised the shaded rows.

Figure 7: Story curve of one annotated work. Top: told position across, chronological rank up. Color and slope both encode jump direction, and dashed rules mark where a withheld explanation lands. Bottom: reader questions left open after each told position. This work opens six and closes all of them.

Requested	band	model	human	agree	exact
structure	hit	$\tau$	$\tau$	gold	form.
Zigzag	0.826	0.407	0.423	0.396	—
Prolepsis	0.721	0.671	0.755	0.635	—
Analepsis	0.684	0.637	0.696	0.552	—
Syllepsis	0.507	0.516	0.526	0.355	—
Achrony	0.300	0.171	0.363	0.220	—

Appendix

Table 6:

\mathrm{T}_{1}

by requested shape, pooled over models (§ 5.1.2 ). The last column is the share of exact identities ( chronicle ) or reversals ( retrograde ), which make up the two near-zero band-hit rows.

Figure 8: The structural plane, one panel per requested structure: the

(\tau,\mathrm{LIS})

region the gold annotations occupy (dashed box), those works (large marks), and every valid cued telling (small dots). Outputs that are the exact identity or reversal stack on one coordinate and are drawn once, in violet , with area proportional to their share. The two failing categories are dense stacks at the ends of the scale, not near-misses. Ticks along the two axes give the marginal distribution of the valid tellings, which the overlapping cloud inside the panel obscures.

Figure 9: Judge verdicts for every scaffolding comparison: blind judgment counts with ties centered, and two-sided exact binomial

p

over decisive pairs.

\mathrm{G}_{1}

beats

\mathrm{G}_{0}

and no other difference is detected. Readers revise two of these rows (§ 5.5 ).

Figure 10: Share of decisive verdicts favoring the first-named condition, from 71 readers and from the judge, with 95% intervals (reader-clustered bootstrap for readers, exact binomial for the judge). The right panel repeats the reader rates after length adjustment. The judge’s rate matches the readers’ on instructed versus free and on graph versus event list. On the other two comparisons, graph versus told order and graph versus one layer removed, the judge returns ties, and these are where readers found the clearest differences.

Figure 11: Where a reversal happens, for the 10 retrograde works. Each point is one telling: the share of its within-scene event pairs told in reverse against the share of its between-scene pairs. Published tellings sit on the left wall, reversing scenes and leaving their contents alone. Cued model outputs pile into the top right corner, reversing both. The two populations differ on the horizontal axis only.

Requested structure	rate	$n$	rate	$n$	rate	$n$	rate	$n$
	instructed		graph vs.		graph vs.		graph vs.
	vs. free		told order		event list		layer removed
zigzag	0.12	33	0.96	23	0.65	17	0.43	37
achrony	0.05	40	0.73	37	0.49	39	0.28	36
prolepsis	0.15	41	0.63	35	0.72	29	0.26	39
syllepsis	0.21	33	0.88	34	0.90	40	0.26	38

Appendix

Table 7: Reader preference by requested structure. rate is the share of decisive judgments favoring the first-named condition. The instructed telling loses most on zigzag , achrony and prolepsis , loses on chronicle too, and draws level on retrograde . The graph’s advantage over the told order is largest where the shape is least mechanical, and holds on chronicle .

Event list	Telling	$\tau$	exact
as written	shuffled, none	0.849	0.085
as written	gold told order	0.884	0.202
de-dated	shuffled, none	0.844	0.096
de-dated	gold told order	0.878	0.181

Appendix

Table 8: The de-dating

2\times 2

(§ 5.2.1 ), pooled over three strong models. Removing every absolute time anchor moves the no-story floor by only -0.005

\tau

, so the floor does not come from explicit dates. Prior knowledge of the works is not ruled out.

	full graph
	pref.	of	rate	$p$
by layer removed
$-$ CAUSES / FORESHADOWS	15	54	0.278	$0.0015$
$-$ pre / post	16	49	0.327	$0.0213$
$-$ learns	13	48	0.271	$0.0021$
$-$ questions/ REVEALS	17	51	0.333	$0.0241$

Appendix

Table 9: Reader verdicts on removing one graph layer. pref. counts decisive judgments for the full graph, so a rate below 0.5 favors the reduced graph. Every layer points the same way. The split by generator ( magenta large effect, lavender none) limits the result.

Between readers : Krippendorff’s $\alpha$ , 311 items
	better	harder to	better
	story	follow	ending
all three levels	0.415	0.112	0.073
95% CI	[0.359, 0.471]	[0.065, 0.157]	[0.031, 0.113]
decisive only	0.595	0.213	0.131
Within a reader : hidden repeat, $n=294$ each

Appendix

Table 10: Rater reliability. Between readers: nominal Krippendorff’s

\alpha

with unequal coders per unit ( Krippendorff, 2018 ) , on a {favored side, other side, tie} recoding. Within a reader: agreement on a hidden repeat with sides swapped. Low agreement on the two right-hand questions is why the paper leads with better story .

		cued	full graph against
Reader subset	$n$	vs. free	order	list	ablation
all readers	71	0.242	0.721	0.595	0.305
shape score $\geq$ 60%	68	0.242	0.722	0.602	0.307
shape score $\geq$ 75%	54	0.268	0.725	0.611	0.304
median reading time $\geq$ 120 s	70	0.245	0.722	0.590	0.304
reads $\geq$ 1 book/month	43	0.299	0.731	0.628	0.255

Appendix

Table 11: Headline rates under stricter reader subsets. order , list and ablation compare the full graph against the told order, the event list and the graph with one layer removed. Cells are the share of decisive judgments favoring the first-named condition, and

n

counts readers. Nothing moves by more than about 7 points, including the comparison that length adjustment removes.

Psychological Depth Scale , 1–5
	$\mathrm{G}_{0}$ events	$\mathrm{G}_{1}$ full	instructed
	$+$ shape	graph	prose ( $\mathrm{T}_{5}^{\mathrm{a}}$ )
close readings ( $n$ )	69	77	148
authenticity	3.61	3.99	3.22
empathy	3.33	3.74	3.11
engagement	3.68	3.91	3.42

Appendix

Table 12: The 294 close readings (instrument in § M ).

\mathrm{G}_{0}

and instructed prose share one prompt (

\mathrm{T}_{5}^{\mathrm{a}}

) and differ in generators: the 4 graph-study models against the 8 prose models, with only GLM-5.2 in both. Instructed prose is rated lowest, draws the most consistency complaints, takes longest to read, and is most often guessed machine-written.

Comparison	split	rate	split	rate	$\Delta$	$n$	$\kappa$	95% CI	verdict
	judge		readers			item-level concordance
instructed vs. left alone	19–68	0.218	64–200	0.242	$+0.024$	67	$+0.278$	$[+0.01,+0.54]$	weak
full graph vs. told order	23–22	0.511	181–70	0.721	$+0.210$	35	$+0.242$	$[-0.11,+0.56]$	weak
full graph vs. event list	31–14	0.689	154–105	0.595	$-0.094$	31	$+0.625$	$[+0.28,+0.91]$	agrees
full graph vs. one layer removed	55–50	0.524	79–180	0.305	$-0.219$	77	$-0.030$	$[-0.23,+0.18]$	unrelated

Appendix

Table 13: The judge against 71 readers on the same texts and question. split and rate count decisive verdicts for the first-named condition, and

\Delta

is readers minus judge.

\kappa

compares the judge with the readers’ majority on items where both were decisive, and the last column grades it: cyan for graph against event list (the smallest reader effect), lavender for instructed against free (aggregate rates within 2 points) and for graph against told order, and magenta for one layer removed. Item-level agreement does not follow the size of the reader effect.

Figure 12: Premature disclosure (§ 5.4 ) as the scorer counts it. Left: where each story narrates the reveal, with a tick at the condition mean. Right: share of stories with fewer than three events before the reveal. The bars include a work whose graph designates no reveal, which is scored early in every condition, and a chronicle whose explanation any chronological telling narrates third. § J gives the rates without these two works.

Condition	$n$	$\tau_{\mathrm{pre}}$	$\tau_{\mathrm{post}}$	RCG	guess	early
$\mathrm{G}_{0}$ no graph	44	0.903	0.967	0.064	0.773	0.214
$\mathrm{G}_{1}$ full graph	43	0.884	0.918	0.034	0.674	0.218
$\mathrm{G}_{2}$ told order	52	0.861	0.886	0.025	0.865	0.071
H human telling	13	0.832	0.887	0.054	0.846	0.071

Appendix

Table 14: Retrospective coherence gain, its paired guess task, and premature disclosure as scored ( early : share of stories with fewer than three events before the reveal).

n

counts the stories RCG could be scored on, while early uses every story (56, 55, 56 and 14), since a story that discloses early cannot be scored for RCG. RCG does not separate the conditions, and the text qualifies the early rates.

Figure 13: $\mathrm{T}_{4}$ confusion, row-normalized and pooled over 19 models. 5 of the six other classes send 29–46% of their mass to analepsis , and achrony is recognized 7.1% of the time.

Panel of Table 15 — Table 15: The 19 evaluated models with the exact identifier each was queried under. Access and tier are the roster’s own labels, and the grouping is by lab. All generation ran between 6 and 10 August 2026, and the reader study between 8 and 17 September 2026.

Narrative Generation

See all 32 papers

Apr 10, 2026

Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling

LLMs have so far failed both to generate consistently compelling stories and to recognize this failure--on the leading creative-writing benchmark (EQ-Bench), LLM judges rank zero-shot AI stories above New Yorker short stories, a gold standard for literary fiction. We argue that existing rubrics overlook a key dimension of compelling human stories: narrative tension. We introduce the 100-Endings metric, which walks through a story sentence by sentence: at each position, a model predicts how the story will end 100 times given only the text so far, and we measure tension as how often predictions fail to match the ground truth. Beyond the mismatch rate, the sentence-level curve also yields complementary statistics that track plot-level twists and revelations, such as the inflection rate, a geometric measure of how frequently the curve reverses direction. Unlike rubric-based judges, 100-Endings correctly ranks New Yorker stories far above LLM outputs. Grounded in narratological principles, we design a story-generation pipeline using structural constraints, including analysis of story templates, idea formulation, and narrative scaffolding. Our pipeline significantly increases narrative tension as measured by the 100-Endings metric, while maintaining performance on the EQ-Bench leaderboard.

Peiqi Sui, Yutong Zhu, Tianyi Cheng +4

McGill University · University of Chicago · University of British Columbia +1

Language Model Generation Evaluation Narrative Generation Text Generation

Ask PDF

May 6, 2026

StoryAlign: Evaluating and Training Reward Models for Story Generation

Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly advanced text generation, stories generated by LLMs still diverge from human-authored works regarding complex narrative structure and human-aligned preferences. A key reason is the absence of effective modeling of human story preferences, which are inherently subjective and under-explored. In this work, we systematically evaluate the modeling of human story preferences and introduce StoryRMB, the first benchmark for assessing reward models on story preferences. StoryRMB contains

1,133

high-quality, human-verified instances, each consisting of a prompt, one chosen story, and three rejected stories. We find existing reward models struggle to select human-preferred stories, with the best model achieving only

66.3\%

accuracy. To address this limitation, we construct roughly

100,000

high-quality story preference pairs across diverse domains and develop StoryReward, an advanced reward model for story preference trained on this dataset. StoryReward achieves state-of-the-art (SoTA) performance on StoryRMB, outperforming much larger models. We also adopt StoryReward in downstream test-time scaling applications for best-of-n (BoN) story selection and find that it generally chooses stories better aligned with human preferences. We will release our dataset, model, and code to facilitate future research. Related code and data are available at https://github.com/THU-KEG/StoryReward.

Haotian Xia, Hao Peng, Yunjia Qi +4

Department of Computer Science and Technology, Tsinghua University

Reward Modeling Language Model Generation Evaluation Human Preference Modeling Text Generation Reward Model Evaluation Narrative Generation

Ask PDF

Aug 16, 2026

When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations

Large language models can write fluent stories, but open-ended storytelling requires more than local fluency. In evolving world simulations and AI-native games, models must preserve facts, relationships, causal dependencies, and character states as the world changes. We introduce WSE-bench, a process benchmark that separately evaluates sustained generation, canonical coherence, and meaningful development in dynamic LLM storytelling. Generation Coverage records the proportion of planned narrative steps produced; Consistency tracks when canon breaks; and Richness measures how meaningfully branching, player-shaped trajectories develop. Across frontier models, Consistency and Richness do not form a smooth trade-off: their empirical Pareto frontier is non-concave, with several non-dominated intermediate configurations that no positive linear weighting can select. Added structure can enrich trajectories, but it does not uniformly improve coherence and may shorten them. Model scale chiefly improves sustained generation, without producing reliable gains in canonical coherence or meaningful development. These results show that sustained generation, canonical coherence, and meaningful development are distinct and sometimes competing capacities. WSE-bench makes those dynamics visible by extending narrative evaluation from finished stories to the processes that create them.

Yuqi Chen, Sixuan Li, Yunfeng Cai +3

The University of Hong Kong · Peking University · Beijing Institute of Mathematical Sciences and Applications (BIMSA) +1

Narrative Generation Language Model Generation Evaluation Interactive Storytelling Text Generation AI Agent Benchmarks

Ask PDF

Abstract

Narrative Generation

Explore similar work

Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling

StoryAlign: Evaluating and Training Reward Models for Story Generation

When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations