Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling

Peiqi Sui; Yutong Zhu; Tianyi Cheng; Peter West; Richard Jean So; Hoyt Long; Ari Holtzman

cs.CLApr 10, 2026

Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling

Authors: Peiqi Sui, Yutong Zhu, Tianyi Cheng, Peter West, Richard Jean So, Hoyt Long, Ari Holtzman

Organizations: McGill University · University of Chicago · University of British Columbia · Duke University

Abstract

LLMs have so far failed both to generate consistently compelling stories and to recognize this failure--on the leading creative-writing benchmark (EQ-Bench), LLM judges rank zero-shot AI stories above New Yorker short stories, a gold standard for literary fiction. We argue that existing rubrics overlook a key dimension of compelling human stories: narrative tension. We introduce the 100-Endings metric, which walks through a story sentence by sentence: at each position, a model predicts how the story will end 100 times given only the text so far, and we measure tension as how often predictions fail to match the ground truth. Beyond the mismatch rate, the sentence-level curve also yields complementary statistics that track plot-level twists and revelations, such as the inflection rate, a geometric measure of how frequently the curve reverses direction. Unlike rubric-based judges, 100-Endings correctly ranks New Yorker stories far above LLM outputs. Grounded in narratological principles, we design a story-generation pipeline using structural constraints, including analysis of story templates, idea formulation, and narrative scaffolding. Our pipeline significantly increases narrative tension as measured by the 100-Endings metric, while maintaining performance on the EQ-Bench leaderboard.

Figures & tables

Panel of Figure 1 — Figure 1: Pipeline architecture and effects. Left: the pipeline extracts a beat template from a reference story (Step 0), combines it with a target idea into a thick intermediate plan, i.e. a densely structured beat scaffold (Step 1), and generates the story (Step 2). Right: effect on the shape of 100-Endings tension curves.

Figure 2: Left : Illustration of the 100-Endings metric. At each sentence position, 100 candidate endings are generated and judged against the actual continuation; the fraction that diverges ( no-rate ) tracks narrative tension. Right : Example from Updike’s “Snowing in Greenwich Village” at 91.2% revealed (no-rate = 0.45).

Figure 3: Mean no-rate and inflection rate can diverge. Left: “Poor Girl” (Petrushevskaya), very high mean no-rate (0.925) but a nearly flat curve (IR 60 0.048). Right: “The Fellow” (Joy Williams), low mean no-rate (0.403) but frequent sharp reversals (IR 60 0.339). Triangles mark peaks/valleys with vertex angle $\leq 60°$ .

Source	EQ-Bench
GPT-5.2	83.80
Claude Sonnet 4.6	82.04
Claude Opus 4.6	82.88
New Yorker (professional)	78.61
Tell Me a Story (amateur)	60.89
StoryStar (amateur)	45.81

Table 1: EQ-Bench rubric score (left) vs. 100-Endings (right). Mean : story-level no-rate; Late : no-rate at

\geq

80% revealed; Retain : % of peak no-rate retained after local peaks; Infl 60 : fraction of sharp (

\leq

60°) reversals. New Yorker leads on all four while underperforming top LLMs on the rubric.

Figure 4: Effect of our narrative pipeline on rubric scores and tension metrics (32 EQ-Bench prompts). Left: EQ-Bench rubric leaderboard with our pipeline at #1 (85.03). Right: $100$ -Ending on the same 32 prompts. The pipeline improves tension over vanilla models.

Source	$n$	EQ-Bench	Mean NR	Late NR	Retain	IR 60
Sonnet 4.6	32	82.0 $\pm$ 1.7	0.606 $\pm$ .063	0.137 $\pm$ .042	14.6 $\pm$ 3.9	0.073 $\pm$ .021
+ Pipeline	32	85.0 $\pm$ 2.3	0.749 † $\pm$ .044	0.340 † $\pm$ .068	34.1 † $\pm$ 4.8	0.135 † $\pm$ .024
Opus 4.6	32	82.9 $\pm$ 1.2	0.611 $\pm$ .061	0.185 $\pm$ .047	15.9 $\pm$ 3.1	0.090 $\pm$ .022
+ Pipeline	32	81.8 $\pm$ 2.7	0.697 † $\pm$ .051	0.296 † $\pm$ .064	32.8 † $\pm$ 5.3	0.142 † $\pm$ .031
GPT-5.2	32	83.8 $\pm$ 1.5	0.694 $\pm$ .067	0.329 $\pm$ .072	36.1 $\pm$ 5.2	0.131 $\pm$ .022
+ Pipeline	32	81.2 $\pm$ 2.3	0.740 $\pm$ .059	0.443 † $\pm$ .069	43.6 † $\pm$ 4.9	0.141 $\pm$ .029

Table 2: Pipeline effect on rubric quality and tension metrics (32 EQ-Bench prompts). Values are point estimates

\pm

half-width of a 95% percentile bootstrap interval over stories (exact bounds in Appendix F ), from iteration 1; the 4 Sonnet 4.6 repeats serve only the stability check in Appendix C.7 . † marks a gain significant under a separate paired bootstrap over the 32 shared prompts, which is more powerful than the

\pm

intervals shown.

Figure 5: No-rate curve for Updike’s “Snowing in Greenwich Village.” Tension rises once the husband walks the guest home alone ( ${\sim}60\%$ story revealed) and stays elevated through the ambiguous ending.

Figure 6: No-rate curves for two stories generated by Sonnet 4.6 from the same prompt. Left: Zero-shot. Right: Narrative pipeline (warmup + planning). The zero-shot curve collapses early, while the pipeline sustains high unpredictability. See Appendix K for full story texts.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Quality tier	Total stories	100-Ending $n$	EQ-Bench $n$
New Yorker	Professional	261	261	53
Tell Me a Story	Amateur (curated)	230	100	41
StoryStar	Amateur (scraped)	100	100	31
EQ-Bench LLMs	LLM-generated	32 $\times$ $M$	320	320

Appendix

Table 3: Summary of evaluation corpora. “100-Ending

n

” is the number of stories used for the 100-ending metric (500–5,000 word range); “EQ-Bench

n

” is the number used for EQ-Bench rubric evaluation (2,000–3,000 word range).

Figure 7: Tension-related EQ-Bench sub-criteria across three human-written corpora (higher $=$ better; negative criteria are inverted). The rubric captures some tension-related signal but evaluates it as a holistic judgment —not through the operational lens of how a story manages information and prediction over time.

Source	Overall	Tension 8	$\pm$ 95% CI	$n$
GPT-5.2 (2–3k words)	84.28	83.86	1.81	23
Top-10 LLMs (all lengths)	81.88	80.80	0.55	320
New Yorker (professional)	78.61	78.40	4.33	53
Tell Me a Story (amateur)	60.89	55.85	3.05	41
StoryStar (amateur)	45.81	44.17	4.23	31

Appendix

Table 4: EQ-Bench Creative Writing rubric scores (0–100 scale). Overall : mean across all 22 criteria. Tension 8 : mean across the 8 tension-related criteria only ( Emotionally Engaging , Emotionally Complex , Overall Reader Engagement , Well-earned Lightness/Darkness , and inverted Unsurprising/Uncreative , Meandering , Incongruent Ending Positivity , Unearned Transformations ). Human corpora restricted to 2,000–3,000 words; GPT-5.2 similarly restricted; pooled top-10 includes all lengths. Judge: Claude Sonnet 4.

Panel of Figure 8 — Figure 8: Overall (left) and late-stage ( $\geq$ 80%, right) mean no-rate by corpus. New Yorker fiction is the least predictable on both measures. Top-10 LLMs score lowest, below all human corpora including amateurs, despite outscoring New Yorker on EQ-Bench (Table 4 ).

Parameter	Value	Rationale
Temperature	1.2	Elevated to produce diverse predictions that probe the full space of plausible endings. Lower temperatures collapse the prediction distribution, making it impossible to distinguish “predictable because all readers would agree” from “predictable because the model happens to be right.”
Top- $p$ (nucleus)	0.95	Standard nucleus sampling threshold.
Max tokens	150	Sufficient for 2–3 sentences of plot-level description.
$n$ (samples per prompt)	100	Each position yields exactly 100 independent predictions.
Batch size	64	Positions processed in batches of 64 for GPU utilization.

Appendix

Table 5: Ending generation sampling parameters.

Corpus	$n$	Mean NR	Late NR	Retain	IR 30	IR 60	IR 120
New Yorker (professional)	261	0.765	0.607	51.7%	0.128	0.200	0.271
Tell Me a Story (amateur)	100	0.693	0.311	39.3%	0.058	0.128	0.229
StoryStar (amateur)	100	0.656	0.375	34.7%	0.105	0.166	0.232
Top-10 LLMs (0-shot)	320	0.630	0.215	23.3%	0.049	0.106	0.201

Appendix

Table 6: Full 100-ending metric results across corpora (stories filtered to 500–5,000 words). Mean NR

=

mean no-rate; Late NR

=

mean no-rate at

\geq

80% of story; Retain

=

% of peak no-rate retained after local peaks (higher

=

tension sustained); IR α

=

inflection rate at vertex angle

\leq\alpha

Config	Rep	Mean NR	Late NR	IR 60	Retain
Sonnet 4.6 (vanilla)	Rep 1	0.606	0.137	0.073	14.6%
	Rep 2	0.606	0.139	0.077	16.7%
	Rep 3	0.607	0.137	0.066	13.5%
	Rep 4	0.605	0.137	0.067	12.9%
	Mean $\pm$ Std	0.606 $\pm$ 0.001	0.138 $\pm$ 0.001	0.071 $\pm$ 0.005	14.4 $\pm$ 1.4%
Sonnet 4.6 + Pipeline	Rep 1	0.749	0.340	0.135	34.1%

Appendix

Table 7: Measurement stability across 4 independent evaluation repeats (

T_{\text{gen}}{=}1.2

n{=}32

iter-1 stories). Standard deviations across repeats are

{\leq}0.001

on mean no-rate and

{\leq}0.001

on late-stage no-rate, indicating negligible measurement noise.

Cell	Condition / model	Words	Mean NR	Late NR	Label
short / low	vanilla, Sonnet 4.6	997	0.449	0.009	Story 4
short / med	pipeline, kimi-k2	946	0.685	0.356	Story 5
short / high	vanilla, Opus 4.6	1,043	0.865	0.469	Story 1
long / low	pipeline, Opus 4.6	2,728	0.527	0.336	Story 3
long / med	vanilla, GPT-5.2	2,203	0.694	0.404	Story 6
long / high	pipeline, Sonnet 4.6	2,429	0.858	0.441	Story 2

Appendix

Table 8: The six-story Task 1 sample. Stories were presented under the blinded labels in the last column, in randomized order. Mean NR and Late NR are the per-story values from the main analysis.

Story	Length	No-Rate	Mean NR	Late NR	A	B	C	D	E	Panel Mean
Story 1	Short	High	0.865	0.469	2	2	3	2	4	2.6
Story 2	Long	High	0.858	0.441	4	5	5	5	5	4.8
Story 3	Long	Low	0.527	0.336	1	1	1	3	1	1.4
Story 4	Short	Low	0.449	0.009	3	3	2	3	1	2.4
Story 5	Short	Medium	0.685	0.356	2	2	3	1	3	2.2
Story 6	Long	Medium	0.694	0.404	3	3	3	3	3	3.0

Appendix

Table 9: Per-annotator Task 1 ratings (1–5) and underlying metric values. Mean NR and Late NR are the per-story values from the main analysis; Length and No-Rate are the stratification cells of the

2\times 3

sampling design. Story labels ( Story 1 – Story 6 ) were blinded for annotators.

Annotator	High	Med	Low	Recovers high $>$ med $>$ low?
A	3.00	2.50	2.00	✓
B	3.50	2.50	2.00	✓
C	4.00	3.00	1.50	✓
D	3.50	2.00	3.00	—
E	4.50	3.00	1.00	✓
Panel mean	3.70	2.60	1.90	✓

Appendix

Table 10: Mean Task 1 rating per no-rate band, per annotator. Four of five annotators recover the metric’s high

>

med

>

low ordering.

Pair	$\Delta$ Mean NR	$\Delta$ Panel rating	Sign agrees?
Story 1 vs Story 4	$-0.42$	$-0.20$	✓
Story 2 vs Story 4	$-0.41$	$-2.40$	✓
Story 1 vs Story 3	$-0.34$	$-1.20$	✓
Story 2 vs Story 3	$-0.33$	$-3.40$	✓
Story 4 vs Story 6	$+0.25$	$+0.60$	✓

Appendix

Table 11: The five story pairs with the widest no-rate gap. The panel-mean tension difference agrees with no-rate in sign on all five.

Statistic	Value	Interpretation
Krippendorff’s $\alpha$ (ordinal)	0.926	$\geq 0.80$ : reliable
ICC(2, 5) absolute agreement	0.915	$>0.90$ : excellent
Kendall’s $W$ (tie-corrected)	0.660, $\chi^{2}(5)=16.5$ , $p=0.006$	ranking agreement above chance
ICC(2, 1) single-rater absolute agreement	0.682	moderate (single rater is noisier)
Mean pairwise quadratic-weighted Cohen’s $\kappa$	0.650	substantial
Mean pairwise Spearman $\rho$	0.573	rank-only complement to $W$

Appendix

Table 12: Inter-annotator reliability on Task 1 (5 annotators, 6 stories).

Pair	$n$	% agree	Cohen’s $\kappa$
A vs B	200	69.0	0.381
A vs C	200	71.0	0.378
B vs C	200	71.5	0.410
Human–human pooled	600	70.5	0.395
Judge vs A	400	63.7	0.274
Judge vs B	400	63.7	0.279

Appendix

Table 13: Task 2 pairwise agreement on Y/N verdicts. The judge is included as if it were a fourth rater. Pooled human–human

\kappa

is the ceiling; the judge sits

\Delta\kappa\approx 0.13

below as a peer rater, and

\Delta\kappa\approx 0.03

below when scored against the human-majority gold.

	gold $=$ Y	gold $=$ N
judge $=$ Y	117 (TP)	88 (FP)
judge $=$ N	46 (FN)	172 (TN)
	P(Y) $=$ 57.1%, R(Y) $=$ 71.8%, F1(Y) $=$ 63.6%
	P(N) $=$ 78.9%, R(N) $=$ 66.2%, F1(N) $=$ 72.0%

Appendix

Table 14: Judge vs. human-majority gold (both annotators agree,

n=423

; 177 ties dropped). Accuracy

=68.3\%

, macro-F1

=67.8\%

\kappa=0.362

Stratum	Level	$n$ (gold)	Judge $\kappa$	Human $\kappa$
condition	vanilla	221	0.407	0.441
	pipeline	202	0.302	0.344
no-rate band	low	136	0.238	0.356
	medium	134	0.412	0.342
	high	153	0.330	0.429
length	short	233	0.289	0.457

Appendix

Table 15: Stratified Cohen’s

\kappa

. “Judge

\kappa

” is against the majority-agree gold (ties dropped). “Human

\kappa

” is the pooled

\kappa

across the three annotator pairs within the stratum. The two columns should be read together: a low judge

\kappa

where the human

\kappa

is also low signals task difficulty rather than judge failure; a low judge

\kappa

where the human

\kappa

is high signals a real judge weakness.

Predicted ending	Judge	Humans	Case
Position A: “The Lighthouse Keeper”, 75% revealed (position no-rate 0.92)
“He simply sat in his chair and I sat by the window, the storm howling between us like the silence had always meant to.”	Y	Y, Y	agree (Y)
“Callum found him dead at his desk, the last log unfinished, and the lighthouse dark when it should have burned.”	N	N, N	agree (N)
“I found him in his room, curled in his chair like he’d been there years, the light out and the sea rising like a breath, waiting.”	Y	N, N	judge false positive
Position B: “Court of the Dreaming”, 89.3% revealed (position no-rate 0.36)
“I’ll gnaw on his fears next Yom Kippur,” she whispered, too sweetly. First Love said nothing, but when the boy in Wichita pulled his blankets over his head that night, he dreamed of daisies—and beneath the petals, something hungering, still.	Y	Y, Y	agree (Y)

Appendix

Table 16: Six Task 2 rows at two positions. Position A : “The Lighthouse Keeper” at 75% revealed, where the continuation has Callum deciding to come home in spring and sharing a quiet Christmas with Fergusson, and nobody dies. Position B : “Court of the Dreaming” at 89.3% revealed, where the continuation has Mora and First Love trading parting threats before the court disperses. Verdicts are the judge’s label and the two assigned annotators’ labels.

Parameter	Value	Notes
Temperature	0.7	All pipeline steps
Max tokens	4,000	Per API call
min_p	0.1	Minimum probability sampling
Reasoning	Disabled	max_tokens:0 for reasoning
Parallel threads	4–6	Per configuration run

Appendix

Table 17: Shared generation hyperparameters for story pipeline steps.

Source	$n$	EQ-Bench	Mean NR	Late NR	Retain %	IR 60
Sonnet 4.6 (vanilla)	32	82.0 [80.3, 83.7]	0.606 [0.542, 0.668]	0.137 [0.096, 0.180]	14.6 [10.8, 18.7]	0.073 [0.053, 0.094]
Sonnet 4.6 + Pipeline	32	85.0 [82.5, 87.1]	0.749 [0.704, 0.791]	0.340 [0.273, 0.409]	34.1 [29.2, 38.9]	0.135 [0.112, 0.159]
Opus 4.6 (vanilla)	32	82.9 [81.7, 84.0]	0.611 [0.548, 0.670]	0.185 [0.140, 0.234]	15.9 [12.8, 19.1]	0.090 [0.067, 0.112]
Opus 4.6 + Pipeline	32	81.8 [78.9, 84.3]	0.697 [0.644, 0.747]	0.296 [0.234, 0.363]	32.8 [27.5, 38.1]	0.142 [0.112, 0.175]
GPT-5.2 (vanilla)	32	83.8 [82.3, 85.3]	0.694 [0.623, 0.756]	0.329 [0.260, 0.403]	36.1 [30.7, 41.2]	0.131 [0.108, 0.153]
GPT-5.2 + Pipeline	32	81.2 [78.8, 83.4]	0.740 [0.677, 0.796]	0.443 [0.375, 0.512]	43.6 [38.5, 48.3]	0.141 [0.113, 0.170]

Appendix

Table 18: Main results with 95% bootstrap confidence intervals (10,000 resamples). This is Table 2 with intervals attached. Mean/Late are mean and late-stage no-rate; Retain is post-spike retention; IR 60 is the inflection rate at

60^{\circ}

Model	Mean NR	Late NR	Retain %	EQ-Bench
Sonnet 4.6	$+0.143$ [ $+0.068$ , $+0.221$ ] ✓	$+0.203$ [ $+0.122$ , $+0.285$ ] ✓	$+19.5$ [ $+13.1$ , $+25.6$ ] ✓	$+2.99$ [ $-0.04$ , $+5.68$ ] n.s.
Opus 4.6	$+0.086$ [ $+0.005$ , $+0.166$ ] ✓	$+0.111$ [ $+0.031$ , $+0.191$ ] ✓	$+16.9$ [ $+10.7$ , $+22.9$ ] ✓	$-1.07$ [ $-4.24$ , $+1.73$ ] n.s.
GPT-5.2	$+0.045$ [ $-0.042$ , $+0.135$ ] n.s.	$+0.114$ [ $+0.013$ , $+0.210$ ] ✓	$+7.5$ [ $+0.2$ , $+14.7$ ] ✓	$-2.56$ [ $-5.42$ , $+0.14$ ] n.s.

Appendix

Table 19: Paired pipeline-minus-vanilla differences with 95% bootstrap intervals over the 32 shared prompts. ✓ marks intervals excluding zero.

Corpus	$n$	Mean NR	Sentences	Words	$r$ (NR, sent) / $r$ (NR, words)
New Yorker	261	0.765	$241\pm 114$	$3365\pm 1127$	$+0.290$ ( $p{=}1.9{\times}10^{-6}$ ) / $+0.172$ ( $p{=}0.005$ )
Tell Me a Story	100	0.693	$115\pm 45$	$1411\pm 518$	$+0.343$ ( $p{=}5{\times}10^{-4}$ ) / $+0.283$ ( $p{=}0.004$ )
StoryStar	100	0.656	$217\pm 123$	$2612\pm 1110$	$-0.009$ ( $p{=}0.93$ ) / $+0.115$ ( $p{=}0.25$ )
Top-10 LLMs	320	0.630	$144\pm 67$	$1367\pm 602$	$-0.050$ ( $p{=}0.38$ ) / $-0.065$ ( $p{=}0.25$ )

Appendix

Table 20: Length distributions and within-corpus correlations with mean no-rate.

Term	Coef	Std err	$t$	$p$
Sentences as length
Intercept ( New Yorker )	$+0.753$	0.012	$+65.5$	$<10^{-300}$
Sentences (per $+1$ SD)	$+0.021$	0.007	$+2.99$	$2.9{\times}10^{-3}$
Tell Me a Story	$-0.046$	0.022	$-2.08$	$3.8{\times}10^{-2}$
StoryStar	$-0.104$	0.021	$-5.05$	$5.5{\times}10^{-7}$
Top-10 LLMs	$-0.115$	0.016	$-7.22$	$1.3{\times}10^{-12}$

Appendix

Table 21: OLS regression of mean no-rate on standardized length and corpus (

n=781

). Left : sentences as the length measure (

R^{2}=0.113

; length alone

R^{2}=0.042

;

\Delta R^{2}

from corpus

=0.071

). Right : words as the length measure (

R^{2}=0.109

; length alone

R^{2}=0.068

;

\Delta R^{2}

from corpus

=0.041

Binning	Quartile	New Yorker	Tell Me a Story	StoryStar	Top-10 LLMs
Sentences	Q1 (8–103)	0.664 (33)	0.637 (46)	0.577 (16)	0.610 (102)
	Q2 (103–147)	0.747 (31)	0.734 (35)	0.672 (19)	0.676 (109)
	Q3 (147–240)	0.758 (74)	0.767 (18)	0.709 (28)	0.599 (75)
	Q4 (240–644)	0.800 (123)	0.514 (1)	0.643 (37)	0.609 (34)
Words	Q1 (94–1070)	0.587 (6)	0.640 (33)	0.560 (7)	0.636 (151)
	Q2 (1070–1789)	0.706 (26)	0.704 (40)	0.602 (21)	0.633 (107)

Appendix

Table 22: Mean no-rate within length quartiles (per-corpus mean,

n

in parentheses). Bins are computed over the pooled set, so a corpus contributes unevenly across quartiles.

Binning	Quartile	New Yorker	Tell Me a Story	StoryStar	Top-10 LLMs
Sentences	Q1 (8–103)	0.323 (33)	0.249 (46)	0.272 (16)	0.164 (102)
	Q2 (103–147)	0.545 (31)	0.329 (35)	0.290 (19)	0.229 (109)
	Q3 (147–240)	0.589 (74)	0.437 (18)	0.399 (28)	0.220 (75)
	Q4 (240–644)	0.709 (123)	0.238 (1)	0.446 (37)	0.312 (34)
Words	Q1 (94–1070)	0.328 (6)	0.247 (33)	0.255 (7)	0.177 (151)
	Q2 (1070–1789)	0.325 (26)	0.303 (40)	0.202 (21)	0.222 (107)

Appendix

Table 23: Late-stage no-rate within length quartiles. New Yorker is highest and Top-10 LLMs lowest in all eight bins.

Length bucket	$n$	Mean no-rate	Median no-rate
$<$ 2000w	41	0.696	0.723
2000–3000w	53	0.760	0.794
3000–4000w	72	0.798	0.834
4000–5000w	95	0.772	0.784

Appendix

Table 24: Mean and median no-rate by story length for the full New Yorker corpus (

n=261

Genre	$n$	Mean no-rate	vs. LLM mean
Showbiz/Artistry	7	0.842	$+0.212$
School Stories	5	0.816	$+0.186$
Psychological	17	0.786	$+0.156$
Identity	21	0.770	$+0.140$
Family	55	0.765	$+0.135$
Biography/Character Study	56	0.759	$+0.129$

Appendix

Table 25: Mean no-rate by New Yorker macro-genre (

n\geq 5

), against the Top-10 LLM mean of 0.630. One-way ANOVA across genres:

F=0.61

p=0.79

\eta^{2}=2.8\%

Group	distinct-1	distinct-2	Mean pairwise Jaccard
New Yorker	0.19	0.62	0.143
Tell Me a Story	0.18	0.60	0.150
StoryStar	0.15	0.53	0.190
Model conditions (6)	0.16–0.18	0.59–0.60	0.147–0.155

Appendix

Table 26: Lexical diversity of the 100 generated endings. Lower pairwise Jaccard means more diverse.

Anchoring controls
Condition	$n$	DeepSeek V4 Pro	Gemini 2.5 Flash
New Yorker	200	$3.04\pm 0.81$	$3.92\pm 0.35$
Top-10 LLMs	200	$2.98\pm 0.80$	$3.92\pm 0.28$
100 identical copies	—	$1.00\pm 0.00$	$1.00\pm 0.00$
50 duplicates $+$ 50 real	—	$1.87\pm 0.92$	$1.87\pm 1.19$
Real 100 endings	—	$3.20\pm 0.68$	$3.93\pm 0.46$

Appendix

Table 27: LLM-judged diversity of the 100-ending sets (1–5), and anchoring controls. Controls are run on synthetic ending sets: 100 copies of one ending, a 50/50 mix of duplicates and real endings, the real 100 endings, and 100 endings drawn from different stories.

Stage	GPU-hours	Per story (median)	Per sentence position
Generation (Qwen3-32B)	$\sim$ 61	$\sim$ 1.8 GPU-min	$\sim$ 2.3 GPU-s
Judge (Qwen3-8B)	$\sim$ 37	$\sim$ 1.1 GPU-min	$\sim$ 1.4 GPU-s
Total	$\sim$ 99	$\sim$ 2.9 GPU-min	$\sim$ 3.7 GPU-s

Appendix

Table 28: Measured cost of the 100-Endings metric over

\sim

2,050 stories and

\sim

97k judged positions.

Condition	$n$	Mean NR	$\pm$ SD	Late NR	$\pm$ SD
Sonnet 4.6, vanilla	32	0.606	0.184	0.137	0.125
Sonnet 4.6 $+$ full pipeline	32	0.749	0.126	0.340	0.202
Sonnet 4.6 $+$ pipeline, Step 0 removed	32	0.728	0.137	0.398	0.218

Appendix

Table 29: Removing Step 0 (warmup analysis) from the Sonnet 4.6 pipeline, on the same 32 EQ-Bench prompts.

Contrast	Mean NR $\Delta$	Late NR $\Delta$
Full pipeline $-$ vanilla	$+14.4$ pp ( $t{=}{+}5.04$ , $p{=}4.7{\times}10^{-7}$ ) ✓	$+20.3$ pp ( $t{=}{+}5.94$ , $p{=}2.8{\times}10^{-9}$ ) ✓
No-Step-0 $-$ vanilla	$+12.2$ pp ( $t{=}{+}5.28$ , $p{=}1.3{\times}10^{-7}$ ) ✓	$+26.2$ pp ( $t{=}{+}6.78$ , $p{=}1.2{\times}10^{-11}$ ) ✓
Full pipeline $-$ No-Step-0	$+2.2$ pp ( $t{=}{+}0.79$ , $p{=}0.43$ ) n.s.	$-5.9$ pp ( $t{=}{-}1.45$ , $p{=}0.15$ ) n.s.

Appendix

Table 30: Paired comparisons across the same 32 prompts (paired

t

-test on per-prompt values).

Explore similar work

StoryAlign: Evaluating and Training Reward Models for Story Generation
May 6, 2026Haotian Xia, Hao Peng, Yunjia Qi +4Reward Modeling Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University
When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations
Aug 16, 2026Yuqi Chen, Sixuan Li, Yunfeng Cai +3Language Model Generation Evaluation Interactive Storytelling
The University of Hong Kong · Peking University · Beijing Institute of Mathematical Sciences and Applications (BIMSA) +1
The Garden of Forking Prompts: How Users Explore Narrative Space in Story Generation
Sep 13, 2026Advait Deshmukh, Nora Benedict, Melanie Walsh +1LLM Prompting Human-AI Co-Creation
University of Colorado Boulder · University of Georgia · University of Washington

May 6, 2026J/K move · Enter open · S save

StoryAlign: Evaluating and Training Reward Models for Story Generation

Haotian Xia, Hao Peng, Yunjia Qi, Xiaozhi Wang+3

Department of Computer Science and Technology, Tsinghua University

Reward Modeling Language Model Generation Evaluation Human Preference Modeling Text Generation Reward Model Evaluation

Open paper AskPDF arXiv

Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly advanced text generation, stories generated by LLMs still diverge from human-authored works regarding complex narrative structure and human-aligned preferences. A key reason is the absence of effective modeling of human story preferences, which are inherently subjective and under-explored. In this work, we systematically evaluate the modeling of human story preferences and introduce StoryRMB, the first benchmark for assessing reward models on story preferences. StoryRMB contains

1,133

high-quality, human-verified instances, each consisting of a prompt, one chosen story, and three rejected stories. We find existing reward models struggle to select human-preferred stories, with the best model achieving only

66.3\%

accuracy. To address this limitation, we construct roughly

100,000

high-quality story preference pairs across diverse domains and develop StoryReward, an advanced reward model for story preference trained on this dataset. StoryReward achieves state-of-the-art (SoTA) performance on StoryRMB, outperforming much larger models. We also adopt StoryReward in downstream test-time scaling applications for best-of-n (BoN) story selection and find that it generally chooses stories better aligned with human preferences. We will release our dataset, model, and code to facilitate future research. Related code and data are available at https://github.com/THU-KEG/StoryReward.

Abstract

Explore similar work

StoryAlign: Evaluating and Training Reward Models for Story Generation

When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations

The Garden of Forking Prompts: How Users Explore Narrative Space in Story Generation