What happens when a storyteller forgets its own story? Large Language Models (LLMs) can now generate narratives spanning tens of thousands of words, but they often fail to maintain consistency throughout. When generating long-form narratives, these models can contradict their own established facts, character traits, and world rules. Existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors largely unexplored. To address this gap, we present ConStory-Bench, a benchmark designed to evaluate narrative consistency in long-form story generation. It contains 2,000 prompts across four task scenarios and defines a taxonomy of five error categories with 19 fine-grained subtypes. We also develop ConStory-Checker, an automated pipeline that detects contradictions and grounds each judgment in explicit textual evidence. Evaluating a range of LLMs through five research questions, we find that consistency errors show clear tendencies: they are most common in factual and temporal dimensions, tend to appear around the middle of narratives, occur in text segments with higher token-level entropy, and certain error types tend to co-occur. These findings can inform future efforts to improve consistency in long-form narrative generation. Our project page is available at https://picrew.github.io/constory-bench.github.io/.
Figures & tables
Figure 1: Overview of ConStory-Bench. The framework comprises three components: (a) a 2,000-prompt benchmark for long story generation (Targeting 8,000–10,000 words), (b) ConStory-Checker , a three-stage pipeline that extracts errors across five categories, pairs contradictions, and constructs evidence chains, and (c) standardized scoring via Consistency Error Density (CED) and Group Relative Rank (GRR).
Figure 2: Representative consistency error examples sampled from real LLM-generated stories on ConStory-Bench. Highlighted segments show contradictions in Timeline & Plot Logic , Characterization , World-building & Setting , Factual & Detail Consistency , and Narrative & Style .
Task Type
Count
Percentage
Generation
751
37.5%
Continuation
432
21.6%
Expansion
422
21.1%
Completion
395
19.8%
Total
2,000
100%
Table 1: Statistics of ConStory-Bench across four task types.
Error Type
Sub Error Type
Timeline & Plot
Absolute Time Contradictions
Logic
Duration Contradictions
Simultaneity Contradictions
Causeless Effects
Causal Logic Violations
Abandoned Plot Elements
Table 2: Consistency-error taxonomy used by ConStory-Bench, comprising five categories and 19 subtypes.
Model
CED (errors per 10K words) ↓
GRR ↓
Words
Errors
Total
Overall
Char.
Fact.
Narr.
Time.
World
Proprietary Models
GPT-5-Reasoning
0.113
0.005
0.061
0.003
0.024
0.003
3.05
9050
0.09
1990
Gemini-2.5-Pro
0.305
0.009
0.132
0.015
0.108
0.029
7.79
5584
0.16
1996
Claude-Sonnet-4.5
0.520
0.017
0.224
0.004
0.128
0.043
4.9
8929
0.37
1998
Grok-4
0.670
0.033
0.307
0.065
0.222
0.076
13.38
2765
0.19
2000
Table 3: Comprehensive performance on ConStory-Bench. CED : Consistency Error Density (errors per 10K words; lower is better). Category columns show CED breakdown: Char. (Characterization), Fact. (Factual & Detail Consistency), Narr. (Narrative & Style), Time. (Timeline & Plot Logic), World (World-building & Setting). GRR : Group Relative Rank (lower is better). Words : average output length (words). Errors : average error count per story. Total : number of completed stories. Blue indicates the best model in each column, Green indicates the second best, and Yellow indicates the third best. Models with Total below 2,000 indicate prompts refused due to safety filtering.
Figure 3: Output length distribution across representative models. Stacked bars show the proportion of 0–3K, 3K–6K, and 6K+ word outputs.
Figure 4: Consistency error growth across different story lengths for two models. Lines: Average error count per story at each length bin (cf. “Errors” in Table 3 ); Bars: Number of samples in each bin.
Metric
Qwen3-30B-A3B
Qwen3-4B
Instruct-2507
Instruct-2507
Whole Text Avg Entropy
1.1438
1.0734
Error Content Avg Entropy
1.2814
1.2799
Error vs Whole (% Diff)
+12.03%
+19.24%
Table 4: Entropy comparison between whole text and error-bearing segments. Higher entropy in error content indicates greater unpredictability relative to the whole-text baseline.
Figure 5: Correlation matrix of error categories across all model outputs. Higher values (darker blue) indicate stronger co-occurrence of error types.
Metric
Absolute Time
Core Rules
Quantitative
Geographical
Nomenclature
Memory
Perspective
Contradictions
Violations
Mismatches
Contradictions
Confusions
Contradictions
Confusions
Avg Fact
22.6%
23.7%
23.4%
20.4%
21.6%
21.8%
13.7%
Avg Contradiction
48.9%
39.4%
40.6%
39.2%
34.4%
38.2%
12.2%
Avg Gap
29.7%
23.4%
23.8%
31.0%
23.3%
25.4%
4.7%
Table 5: Positional distribution of seven representative error subtypes. Positions are normalized by story length (0–100%).
Figure 6: Dumbbell plot of error positional distributions. Each row represents an error subtype; blue dots show where facts are first established (fact position), red dots show where contradictions appear, and the connecting line indicates the gap. Columns show four representative models. Values are normalized by story length.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Example prompts for the four narrative generation task types in ConStory-Bench.
Error Category
GT
Direct Detection
Experts Detection (Avg)
Pred
TP
Recall
Prec
F1
Pred
TP
Recall
Prec
F1
Character Consistency
200
126
121
0.605
0.960
0.742
53.5
48.5
0.242
0.891
0.379
Factual Accuracy
200
148
125
0.625
0.845
0.718
16.5
13.5
0.068
0.841
0.124
Narrative Coherence
200
76
70
0.350
0.921
0.507
29.5
18.0
0.090
0.586
0.153
Temporal Logic
200
147
120
0.600
0.816
0.692
84.0
40.5
0.203
0.463
0.281
World Consistency
200
125
114
0.570
0.912
0.702
26.5
18.0
0.090
0.686
0.159
Appendix
Table 6: Performance comparison between ConStory-Checker (Direct Detection) and human expert judgment (Human Detection) on the diagnostic dataset. GT (Ground Truth) indicates the 200 injected errors for each category. The error categories are: Char. (Characterization), Fact. (Factual & Detail Consistency), Narr. (Narrative & Style), Time. (Timeline & Plot Logic), and World (World-building & Setting).
Figure 8: Performance comparison between ConStory-Checker and human expert judgment across five consistency error categories. The evaluation covers recall, precision, F1-score, and comprehensive radar visualization, demonstrating that our automated approach achieves human-competitive performance in detecting narrative inconsistencies.
Figure 9: Complete judge prompt for Timeline & Plot Logic category in ConStory-Checker evaluation protocol.
Figure 10: Complete judge prompt for Characterization category in ConStory-Checker evaluation protocol.
Figure 11: Complete judge prompt for World-building & Setting category in ConStory-Checker evaluation protocol.
Figure 12: Complete judge prompt for Factual & Detail Consistency category in ConStory-Checker evaluation protocol.
Figure 13: Complete judge prompt for Narrative & Style category in ConStory-Checker evaluation protocol.
Figure 14: Representative error detection examples by ConStory-Checker across five consistency dimensions: Timeline & Plot Logic, Characterization, World-building & Setting, Factual & Detail Consistency, and Narrative & Style. Each example demonstrates the system’s ability to identify subtle contradictions through structured extraction of conflicting passages with precise location references.
Figure 15: Performance leaderboard of evaluated models based on GRR scores. Bar length indicates relative performance (longer bars represent better consistency), with color intensity reflecting score magnitude within each model category. Models are grouped by family: proprietary (top), open-source, capability-enhanced LLMs, and agent-enhanced systems (bottom).
Figure 16: The consistency performance (CED) of evaluated models versus average output words.
Model
Overall CED
Generation
Continuation
Expansion
Completion
Avg Words
Total
(748)
(429)
(419)
(394)
Proprietary Models
GPT-5-Reasoning
0.113
0.11
0.093
0.07
0.188
9050
1990
Gemini-2.5-Pro
0.302
0.379
0.233
0.277
0.257
5091
1996
Gemini-2.5-Flash
0.305
0.334
0.243
0.254
0.373
5504
1996
Claude-Sonnet-4.5
0.52
0.67
0.387
0.498
0.402
8929
1998
Appendix
Table 7: Consistency Error Density (CED) disaggregated by prompt task type. CED : errors per 10K words (lower is better). Task type columns show CED for each category: Generation (open-ended story creation), Continuation (extending provided narratives), Expansion (elaborating specific segments), and Completion (filling removed spans). Numbers in parentheses denote the number of prompts per task type. Avg Words : average output length. Total : number of completed stories. Models are grouped by family and sorted by Overall CED in ascending order within each category. Bold values indicate the highest CED among the four task types for each model.
Model
0–1k
1k–3k
3k–5k
5k–8k
8k+
Avg Words
Total
Proprietary Models
GPT-5-Reasoning
24 (1.2%)
48 (2.0%)
58 (2.5%)
554 (27.8%)
1322 (66.4%)
9050
1990
Claude-Sonnet-4.5
54 (2.7%)
40 (2.0%)
39 (2.0%)
531 (26.6%)
1334 (66.8%)
8929
1998
Gemini-2.5-Flash
48 (2.4%)
75 (3.8%)
830 (41.6%)
838 (42.0%)
205 (10.3%)
5504
1996
Gemini-2.5-Pro
43 (2.2%)
43 (2.2%)
845 (42.3%)
1024 (51.3%)
41 (2.1%)
5091
1996
Doubao-1.6-Thinking-2507
40 (2.0%)
548 (27.4%)
1129 (56.5%)
255 (12.8%)
28 (1.4%)
3713
2000
Appendix
Table 8: Detailed output length distribution across evaluated models. Columns show the number and percentage of stories in each word-count bin. Models are sorted by average word count in descending order.
Table 9: Comprehensive token-level uncertainty comparison across three metrics. Each metric compares error-bearing segments against the whole-text baseline. Higher entropy and perplexity, along with lower probability, indicate greater model uncertainty. Relative differences show the percentage change of error content compared to whole text.
Figure 17: Model-specific error correlation matrices across eight representative models. Darker colors indicate stronger positive correlations between error categories.
Model
Metric
Absolute Time
Core Rules
Quantitative
Geographical
Nomenclature
Memory
Perspective
Contradictions
Violations
Mismatches
Contradictions
Confusions
Contradictions
Confusions
Proprietary Models
GPT-5-Reasoning
Fact
48.4%
21.8%
20.0%
7.7%
20.2%
21.4%
13.3%
Contra
90.6%
21.1%
32.2%
30.2%
38.1%
40.5%
4.3%
Gap
47.7%
0.8%
12.7%
22.6%
24.2%
19.1%
15.8%
Claude-Sonnet-4.5
Fact
23.1%
19.1%
25.7%
12.2%
18.6%
16.3%
35.7%
Appendix
Table 10: Per-model positional distribution of seven representative error subtypes across eight models. Positions are normalized by story length (0–100%). Fact : average position where facts are first established; Contra : average position where contradictions appear; Gap : average distance between fact and contradiction positions, computed as Avg Gap=n1∑i=1n∣contrai−facti∣ . Blue = largest Gap per error type.
Recent advances in large language models (LLMs) have enabled the generation of high-quality prose, yet whether these models are capable of generating diverse or creative artifacts remains a contested question. In this work, we investigate the diversity of LLM-generated stories through the framework of narrative similarity. Using a contrastive framework and a dataset of human-written stories and prompts from r/WritingPrompts, we collect narrative similarity judgments across 10 representative LLMs, utilizing both human evaluations and three different automatic annotation methods. Our findings reveal a clear trend: LLM-generated narratives are consistently more similar to each other than human-written stories are. We demonstrate that frontier models in particular converge on a "mean" generic narrative that approximates individual human stories but lacks the collective diversity of human authors. Finally, we show that common mitigation strategies, including negative prompting and temperature scaling, fail to meaningfully address this homogeneity.
Although large language models (LLMs) have demonstrated impressive creative fiction generation, they struggle to maintain narrative consistency and coherent plot lines in long-form stories. In this work, we introduce a unified framework for long-form narrative generation and verification. MAGNET, a multi-agent goal-driven narrative engine for storytelling, generates stories with persona-grounded character agents that propose actions based on a shared world state and evolving story goals, while ATLAS is a graph-based pipeline that compares scene-level world representations across a generated story to detect hallucinations. By evaluating MAGNET using an LLM editor, pairwise rubric scoring, and ATLAS, we show that our framework produces coherent narratives compared to single-model prompting and IBSEN. At 100 pages, MAGNET reduced annotations and hallucinations by 41 and 50%, respectively, compared to the single model baseline and by 34 and 45%, respectively, compared to IBSEN, with pairwise rubric evaluation showing similar results. These results suggest that long-form narratives can emerge from explicit world-state tracking and goal-driven multi-agent generation, providing a foundation for controllable and structurally coherent long-form narrative generation.
Aayush Aluru, Chloe Ho, Muhammad Hammouri +5
Pocket FM · Princeton University · University of Michigan +2
LLMs have demonstrated strong capabilities in creative writing. However, scaling them to full-length novels remains challenging, as maintaining narrative consistency becomes increasingly difficult. Existing story-generation methods typically focus on stories of up to about ten thousand words, leaving their ability to scale to full-length novels underexplored. In this work, we introduce Narrative State Tracking Agent (NstAgent), a training-free agentic framework that allows LLMs to track a structured narrative state including characters, past events and future requirements. We extend an existing benchmark to compare narrative consistency across lengths, and use it together with a writing-quality benchmark to systematically evaluate stories ranging from 10K to 100K words. We show that NstAgent achieves better narrative consistency and writing quality as stories grow longer, and neither of them degrades noticeably as length increases, suggesting that it provides an effective approach to scaling story generation toward full-length novels.
Zhennan Wan, Jianfei Chen
Dept. of Comp. Sci. and Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University