cs.CLMar 6, 2026
SaveLost in Stories: Consistency Bugs in Long Story Generation by LLMs
Organizations: Microsoft, Beijing, China · Singapore University of Technology and Design
Abstract
What happens when a storyteller forgets its own story? Large Language Models (LLMs) can now generate narratives spanning tens of thousands of words, but they often fail to maintain consistency throughout. When generating long-form narratives, these models can contradict their own established facts, character traits, and world rules. Existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors largely unexplored. To address this gap, we present ConStory-Bench, a benchmark designed to evaluate narrative consistency in long-form story generation. It contains 2,000 prompts across four task scenarios and defines a taxonomy of five error categories with 19 fine-grained subtypes. We also develop ConStory-Checker, an automated pipeline that detects contradictions and grounds each judgment in explicit textual evidence. Evaluating a range of LLMs through five research questions, we find that consistency errors show clear tendencies: they are most common in factual and temporal dimensions, tend to appear around the middle of narratives, occur in text segments with higher token-level entropy, and certain error types tend to co-occur. These findings can inform future efforts to improve consistency in long-form narrative generation. Our project page is available at https://picrew.github.io/constory-bench.github.io/.
Figures & tables
Figure 1: Overview of ConStory-Bench. The framework comprises three components: (a) a 2,000-prompt benchmark for long story generation (Targeting 8,000–10,000 words), (b) ConStory-Checker , a three-stage pipeline that extracts errors across five categories, pairs contradictions, and constructs evidence chains, and (c) standardized scoring via Consistency Error Density (CED) and Group Relative Rank (GRR).
Figure 2: Representative consistency error examples sampled from real LLM-generated stories on ConStory-Bench. Highlighted segments show contradictions in Timeline & Plot Logic , Characterization , World-building & Setting , Factual & Detail Consistency , and Narrative & Style .
| Task Type | Count | Percentage |
|---|---|---|
| Generation | 751 | 37.5% |
| Continuation | 432 | 21.6% |
| Expansion | 422 | 21.1% |
| Completion | 395 | 19.8% |
| Total | 2,000 | 100% |
Table 1: Statistics of ConStory-Bench across four task types.
| Error Type | Sub Error Type |
|---|---|
| Timeline & Plot | Absolute Time Contradictions |
| Logic | Duration Contradictions |
| Simultaneity Contradictions | |
| Causeless Effects | |
| Causal Logic Violations | |
| Abandoned Plot Elements |
Table 2: Consistency-error taxonomy used by ConStory-Bench, comprising five categories and 19 subtypes.
| Model | CED (errors per 10K words) | GRR | Words | Errors | Total | |||||
| Overall | Char. | Fact. | Narr. | Time. | World | |||||
| Proprietary Models | ||||||||||
| GPT-5-Reasoning | 0.113 | 0.005 | 0.061 | 0.003 | 0.024 | 0.003 | 3.05 | 9050 | 0.09 | 1990 |
| Gemini-2.5-Pro | 0.305 | 0.009 | 0.132 | 0.015 | 0.108 | 0.029 | 7.79 | 5584 | 0.16 | 1996 |
| Claude-Sonnet-4.5 | 0.520 | 0.017 | 0.224 | 0.004 | 0.128 | 0.043 | 4.9 | 8929 | 0.37 | 1998 |
| Grok-4 | 0.670 | 0.033 | 0.307 | 0.065 | 0.222 | 0.076 | 13.38 | 2765 | 0.19 | 2000 |
Table 3: Comprehensive performance on ConStory-Bench. CED : Consistency Error Density (errors per 10K words; lower is better). Category columns show CED breakdown: Char. (Characterization), Fact. (Factual & Detail Consistency), Narr. (Narrative & Style), Time. (Timeline & Plot Logic), World (World-building & Setting). GRR : Group Relative Rank (lower is better). Words : average output length (words). Errors : average error count per story. Total : number of completed stories. Blue indicates the best model in each column, Green indicates the second best, and Yellow indicates the third best. Models with Total below 2,000 indicate prompts refused due to safety filtering.
Figure 3: Output length distribution across representative models. Stacked bars show the proportion of 0–3K, 3K–6K, and 6K+ word outputs.
Figure 4: Consistency error growth across different story lengths for two models. Lines: Average error count per story at each length bin (cf. “Errors” in Table 3 ); Bars: Number of samples in each bin.
| Metric | Qwen3-30B-A3B | Qwen3-4B |
|---|---|---|
| Instruct-2507 | Instruct-2507 | |
| Whole Text Avg Entropy | 1.1438 | 1.0734 |
| Error Content Avg Entropy | 1.2814 | 1.2799 |
| Error vs Whole (% Diff) | +12.03% | +19.24% |
Table 4: Entropy comparison between whole text and error-bearing segments. Higher entropy in error content indicates greater unpredictability relative to the whole-text baseline.
Figure 5: Correlation matrix of error categories across all model outputs. Higher values (darker blue) indicate stronger co-occurrence of error types.
| Metric | Absolute Time | Core Rules | Quantitative | Geographical | Nomenclature | Memory | Perspective |
|---|---|---|---|---|---|---|---|
| Contradictions | Violations | Mismatches | Contradictions | Confusions | Contradictions | Confusions | |
| Avg Fact | 22.6% | 23.7% | 23.4% | 20.4% | 21.6% | 21.8% | 13.7% |
| Avg Contradiction | 48.9% | 39.4% | 40.6% | 39.2% | 34.4% | 38.2% | 12.2% |
| Avg Gap | 29.7% | 23.4% | 23.8% | 31.0% | 23.3% | 25.4% | 4.7% |
Table 5: Positional distribution of seven representative error subtypes. Positions are normalized by story length (0–100%).
Figure 6: Dumbbell plot of error positional distributions. Each row represents an error subtype; blue dots show where facts are first established (fact position), red dots show where contradictions appear, and the connecting line indicates the gap. Columns show four representative models. Values are normalized by story length.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Example prompts for the four narrative generation task types in ConStory-Bench.
| Error Category | GT | Direct Detection | Experts Detection (Avg) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Pred | TP | Recall | Prec | F1 | Pred | TP | Recall | Prec | F1 | ||
| Character Consistency | 200 | 126 | 121 | 0.605 | 0.960 | 0.742 | 53.5 | 48.5 | 0.242 | 0.891 | 0.379 |
| Factual Accuracy | 200 | 148 | 125 | 0.625 | 0.845 | 0.718 | 16.5 | 13.5 | 0.068 | 0.841 | 0.124 |
| Narrative Coherence | 200 | 76 | 70 | 0.350 | 0.921 | 0.507 | 29.5 | 18.0 | 0.090 | 0.586 | 0.153 |
| Temporal Logic | 200 | 147 | 120 | 0.600 | 0.816 | 0.692 | 84.0 | 40.5 | 0.203 | 0.463 | 0.281 |
| World Consistency | 200 | 125 | 114 | 0.570 | 0.912 | 0.702 | 26.5 | 18.0 | 0.090 | 0.686 | 0.159 |
Table 6: Performance comparison between ConStory-Checker (Direct Detection) and human expert judgment (Human Detection) on the diagnostic dataset. GT (Ground Truth) indicates the 200 injected errors for each category. The error categories are: Char. (Characterization), Fact. (Factual & Detail Consistency), Narr. (Narrative & Style), Time. (Timeline & Plot Logic), and World (World-building & Setting).
Figure 8: Performance comparison between ConStory-Checker and human expert judgment across five consistency error categories. The evaluation covers recall, precision, F1-score, and comprehensive radar visualization, demonstrating that our automated approach achieves human-competitive performance in detecting narrative inconsistencies.
Figure 9: Complete judge prompt for Timeline & Plot Logic category in ConStory-Checker evaluation protocol.
Figure 10: Complete judge prompt for Characterization category in ConStory-Checker evaluation protocol.
Figure 11: Complete judge prompt for World-building & Setting category in ConStory-Checker evaluation protocol.
Figure 12: Complete judge prompt for Factual & Detail Consistency category in ConStory-Checker evaluation protocol.
Figure 13: Complete judge prompt for Narrative & Style category in ConStory-Checker evaluation protocol.
Figure 14: Representative error detection examples by ConStory-Checker across five consistency dimensions: Timeline & Plot Logic, Characterization, World-building & Setting, Factual & Detail Consistency, and Narrative & Style. Each example demonstrates the system’s ability to identify subtle contradictions through structured extraction of conflicting passages with precise location references.
Figure 15: Performance leaderboard of evaluated models based on GRR scores. Bar length indicates relative performance (longer bars represent better consistency), with color intensity reflecting score magnitude within each model category. Models are grouped by family: proprietary (top), open-source, capability-enhanced LLMs, and agent-enhanced systems (bottom).
Figure 16: The consistency performance (CED) of evaluated models versus average output words.
| Model | Overall CED | Generation | Continuation | Expansion | Completion | Avg Words | Total |
| (748) | (429) | (419) | (394) | ||||
| Proprietary Models | |||||||
| GPT-5-Reasoning | 0.113 | 0.11 | 0.093 | 0.07 | 0.188 | 9050 | 1990 |
| Gemini-2.5-Pro | 0.302 | 0.379 | 0.233 | 0.277 | 0.257 | 5091 | 1996 |
| Gemini-2.5-Flash | 0.305 | 0.334 | 0.243 | 0.254 | 0.373 | 5504 | 1996 |
| Claude-Sonnet-4.5 | 0.52 | 0.67 | 0.387 | 0.498 | 0.402 | 8929 | 1998 |
Table 7: Consistency Error Density (CED) disaggregated by prompt task type. CED : errors per 10K words (lower is better). Task type columns show CED for each category: Generation (open-ended story creation), Continuation (extending provided narratives), Expansion (elaborating specific segments), and Completion (filling removed spans). Numbers in parentheses denote the number of prompts per task type. Avg Words : average output length. Total : number of completed stories. Models are grouped by family and sorted by Overall CED in ascending order within each category. Bold values indicate the highest CED among the four task types for each model.
| Model | 0–1k | 1k–3k | 3k–5k | 5k–8k | 8k+ | Avg Words | Total |
| Proprietary Models | |||||||
| GPT-5-Reasoning | 24 (1.2%) | 48 (2.0%) | 58 (2.5%) | 554 (27.8%) | 1322 (66.4%) | 9050 | 1990 |
| Claude-Sonnet-4.5 | 54 (2.7%) | 40 (2.0%) | 39 (2.0%) | 531 (26.6%) | 1334 (66.8%) | 8929 | 1998 |
| Gemini-2.5-Flash | 48 (2.4%) | 75 (3.8%) | 830 (41.6%) | 838 (42.0%) | 205 (10.3%) | 5504 | 1996 |
| Gemini-2.5-Pro | 43 (2.2%) | 43 (2.2%) | 845 (42.3%) | 1024 (51.3%) | 41 (2.1%) | 5091 | 1996 |
| Doubao-1.6-Thinking-2507 | 40 (2.0%) | 548 (27.4%) | 1129 (56.5%) | 255 (12.8%) | 28 (1.4%) | 3713 | 2000 |
Table 8: Detailed output length distribution across evaluated models. Columns show the number and percentage of stories in each word-count bin. Models are sorted by average word count in descending order.
| Model | Average Values | Relative Difference | |
|---|---|---|---|
| Whole Text | Error Content | (Error vs Whole) | |
| Entropy (bits) — Higher indicates greater uncertainty | |||
| Qwen3-30B-A3B-Instruct-2507 | 1.1438 | 1.2814 | +12.03% |
| Qwen3-4B-Instruct-2507 | 1.0734 | 1.2799 | +19.24% |
| Probability — Higher indicates greater confidence | |||
| Qwen3-30B-A3B-Instruct-2507 | 0.6895 | 0.6522 | -5.41% |
Table 9: Comprehensive token-level uncertainty comparison across three metrics. Each metric compares error-bearing segments against the whole-text baseline. Higher entropy and perplexity, along with lower probability, indicate greater model uncertainty. Relative differences show the percentage change of error content compared to whole text.
Figure 17: Model-specific error correlation matrices across eight representative models. Darker colors indicate stronger positive correlations between error categories.
| Model | Metric | Absolute Time | Core Rules | Quantitative | Geographical | Nomenclature | Memory | Perspective |
|---|---|---|---|---|---|---|---|---|
| Contradictions | Violations | Mismatches | Contradictions | Confusions | Contradictions | Confusions | ||
| Proprietary Models | ||||||||
| GPT-5-Reasoning | Fact | 48.4% | 21.8% | 20.0% | 7.7% | 20.2% | 21.4% | 13.3% |
| Contra | 90.6% | 21.1% | 32.2% | 30.2% | 38.1% | 40.5% | 4.3% | |
| Gap | 47.7% | 0.8% | 12.7% | 22.6% | 24.2% | 19.1% | 15.8% | |
| Claude-Sonnet-4.5 | Fact | 23.1% | 19.1% | 25.7% | 12.2% | 18.6% | 16.3% | 35.7% |
Table 10: Per-model positional distribution of seven representative error subtypes across eight models. Positions are normalized by story length (0–100%). Fact : average position where facts are first established; Contra : average position where contradictions appear; Gap : average distance between fact and contradiction positions, computed as . Blue = largest Gap per error type.
| Words | Errors | |
|---|---|---|
| Story 1 | 8,000 | 2 |
| Story 2 | 10,000 | 3 |
| Story 3 | 6,000 | 1 |
| Story 4 | 8,000 | 0 |
| Story 5 | 800 | 0 |
| Total (1–5) | 32,800 | 6 |