Evaluating Whether LLMs Can Reliably Connect the DOTs?

Eftekhar Hossain; John Salvador; Santu Karmaker

cs.CLSep 29, 2026

Evaluating Whether LLMs Can Reliably Connect the DOTs?

Authors: Eftekhar Hossain, John Salvador, Santu Karmaker

Abstract

Access to real-world information is often noisy and fragmented. Constructing a coherent narrative from such fragments requires models to reconstruct missing spans within a broader storyline, commonly referred to as text infilling, while preserving consistency with both the local context and the global storyline. Despite using text infilling as a pre-training objective in many Large Language Models (LLMs), their actual performance on real-world narrative infilling remains underexplored. In this paper, we address this gap by introducing a multi-domain benchmark of ~9.2K instances for narrative infilling, constructed by masking one to three sentences across four narrative types: encyclopedic text, commonsense stories, news articles, and visual narratives. Using this benchmark, we evaluate 20 instruction-tuned open-source LLMs ranging from 1.5B to 70B parameters across varying levels of instruction specificity and reasoning guidance. Outputs are assessed using standard automatic metrics and a qualitative framework covering five narrative dimensions. Results show that model scale does not reliably predict infilling quality: Gemma-2-2B achieves the highest qualitative score (4.02/5), outperforming models over ten times larger, including DeepSeek-Qwen-32B (3.77/5, 6.6%) and LLaMA-3.3-70B (3.71/5, 8.3%). We further find that explicit reasoning offers limited benefits as chain-of-thought reasoning yields only a marginal improvement (+0.6%). Additionally, short narratives and domain characteristics emerge as stronger predictors of task difficulty than infill position alone for narrative infilling in current LLMs.

Figures & tables

Fig. 1 : Narrative infilling task. Given a passage with a missing span, a Narrative Infiller generates text that plausibly fills the gap while preserving coherence with the surrounding narrative context.

Dimensions	$A_{1}$ vs $A_{2}$	GPT vs $A_{1}$	GPT vs $A_{2}$
Fluency	0.701	0.637	0.580
Faithfulness	0.704	0.545	0.539
Coherence	0.717	0.593	0.549
Narrative Consistency	0.490	0.648	0.571
Informativeness	0.656	0.582	0.579
Average	0.654	0.601	0.564

Table 2: Human–human agreement (Cohen’s

\kappa

) and human–GPT correlation (Spearman’s

\rho

) across five infilling quality dimensions.

	Very Low (L0)		Low (L1)		Moderate (L2)		High (L3)		Very High (L4)
Model	AtAvg	QAvg	AtAvg	QAvg	AtAvg	QAvg	AtAvg	QAvg	AtAvg	QAvg
LLaMA-3.1-8B	0.311	3.878	0.354	3.831	0.357	3.933	0.352	4.018	0.306	3.777
LLaMA-3.2-1B	0.309	3.792	0.299	3.534	0.246	3.607	0.256	3.617	0.240	3.502
LLaMA-3.2-3B	0.304	3.820	0.349	3.583	0.335	3.824	0.305	3.783	0.249	3.462
LLaMA-3.3-70B	0.314	3.707	0.375	3.642	0.375	3.647	0.371	3.647	0.258	3.367
Ministral-8B	0.300	3.788	0.355	3.811	0.352	3.893	0.346	3.922	0.311	3.911

Table 3: Average scores across five instruction specificity levels for all 20 models. AtAvg is the mean of five automatic metrics (BERTScore, ROUGE-1, METEOR, chrF, Sem-F1); QAvg is the mean across five qualitative dimensions ( Faithfulness, Coherence, Narrative Consistency, Fluency, and Informativeness). Green indicates the highest score overall; bold blue and orange indicate the highest and lowest scores per level.

Dimensions	BERT	ROUGE-1	METEOR	chrF	Sem-F1
Fluency	0.044	0.074	0.090	0.085	0.090
Faithfulness	0.021	0.060	0.085	0.083	0.076
Coherence	0.035	0.068	0.093	0.091	0.084
Narrative Consistency	0.054	0.064	0.089	0.082	0.084
Informativeness	0.045	0.099	0.136	0.141	0.109

Table 4: System-level Kendall

\tau

correlation between narrative quality evaluations and automatic metrics computed by following Liu et al. (2023b) . The highest correlation is shown in green.

	Abductive		Causal		Chain-of-Thought		Counterfactual
Model	AtAvg	QAvg	AtAvg	QAvg	AtAvg	QAvg	AtAvg	QAvg
LLaMA-3.3-70B	0.369	3.640	0.370	3.658	0.370	3.652	0.367	3.605
Gemma-2-9B	0.362	3.837	0.361	3.837	0.362	3.831	0.354	3.722
Gemma-2-27B	0.361	3.771	0.358	3.769	0.365	3.790	0.352	3.549
Mistral-nemo	0.356	3.907	0.357	3.923	0.353	3.907	0.348	3.705
Mistral-small	0.372	3.718	0.365	3.708	0.376	3.822	0.350	3.322

Table 5: Results across four reasoning paradigms for the 10 largest models, evaluated at moderate specificity (L2). Green indicates the highest score overall; bold blue and orange indicate the highest and lowest per paradigm.

Fig. 2 : Percentage of high-quality completions (Avg $>$ 4 across five rubric dimensions) for each LLM under TELeR prompting and reasoning-based prompting. Scores are aggregated across all prompt levels and reasoning paradigms.

Setting	Opening	Middle	Closing
Prompt Specificity
Very Low	3.63 / 53.9%	3.82 / 63.0%	3.86 / 65.7%
Moderate	3.80 / 63.2%	3.82 / 63.0%	3.78 / 60.8%
Very High	3.68 / 55.2%	3.80 / 61.9%	3.87 / 65.2%
Reasoning Strategies
Abductive	3.82 / 64.3%	3.81 / 63.0%	3.76 / 60.4%

Table 7: Top: Average qualitative score and percentage of high-quality completions (Avg

>

4) by blank position across selective prompt specificity levels and reasoning paradigms. Bottom: Same breakdown by narratives from different domains. Each cell reports Avg / High%. Improvements are shown in green.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Pattern	Extraction Strategy
DeepSeek models perform a reasoning step and terminate the reasoned text with a </think> token.	Remove all the text that appears up to and including </think> .
In many instances, models reproduce the problem text with the generated answer filled in.	Split text by the overlapping patterns between the problem text and the generated answer.
Many LLMs are adept at following formatting guidelines given to them.	Add additional formatting instructions and utilize manual extraction patterns.
Sometimes, LLMs slightly fail to adopt the proper formatting.	Note the slight variations and add them to the extraction patterns.

Appendix

Table 9: Common patterns in LLM responses and strategies to extract text from those responses.

Model	BERT	R-1	Met	ChRF	F1	AtAvg	Flu	Faith	BiCo	NarCo	Info	QAvg
Very Low (L0)
LLaMA-3.1-8B	0.849	0.139	0.119	0.147	0.301	0.311	4.364	3.777	3.737	3.973	3.540	3.878
LLaMA-3.2-1B	0.847	0.140	0.120	0.151	0.288	0.309	4.257	3.706	3.661	3.885	3.451	3.792
LLaMA-3.2-3B	0.846	0.129	0.115	0.140	0.292	0.304	4.297	3.730	3.683	3.917	3.475	3.820
LLaMA-3.3-70B	0.853	0.152	0.112	0.147	0.306	0.314	4.232	3.585	3.576	3.804	3.341	3.707
Ministral-8B	0.845	0.130	0.105	0.130	0.288	0.300	4.270	3.698	3.651	3.887	3.432	3.788

Appendix

Table 10: Results across three specificity levels (L0 to L2). AtAvg is the mean of five automatic metrics. QAvg is the average across five infilling quality evaluation dimensions: Fluency, Context Faithfulness, Bidirectional Coherence, Narrative Consistency, and Informativeness. The green highlight indicates the highest score, while bold blue and orange indicate the highest and lowest average for each level. The highest across each metric is shown in bold.

Model	BERT	Rouge-1	Metero	ChRF	Sem-F1	AtAvg	Flu	Faith	BiCo	NarCo	Info	QAvg
High (L3)
LLaMA-3.1-8B	0.859	0.190	0.166	0.212	0.333	0.352	4.454	3.886	3.869	4.096	3.785	4.018
LLaMA-3.2-1B	0.828	0.087	0.064	0.094	0.208	0.256	4.124	3.496	3.479	3.710	3.276	3.617
LLaMA-3.2-3B	0.846	0.142	0.108	0.158	0.269	0.305	4.277	3.649	3.648	3.869	3.471	3.783
LLaMA-3.3-70B	0.863	0.211	0.187	0.232	0.360	0.371	4.229	3.456	3.477	3.714	3.361	3.647
Ministral-8B	0.862	0.190	0.150	0.195	0.332	0.346	4.425	3.797	3.776	4.012	3.602	3.922

Appendix

Table 11: Results across two specificity levels (L4 and L5). AtAvg is the mean of five automatic metrics. QAvg is the average across five infilling quality evaluation dimensions: Fluency, Context Faithfulness, Bidirectional Coherence, Narrative Consistency, and Informativeness. The green highlight indicates the highest score, while bold blue and orange indicate the highest and lowest average for each level. The highest across each metric is shown in bold.

Model	BERT	Rouge-1	Metero	ChRF	Sem-F1	AtAvg	Flu	Faith	BiCo	NarCo	Info	QAvg
Abductive Reasoning
LLaMA-3.3-70B	0.862	0.206	0.186	0.232	0.359	0.369	4.236	3.435	3.460	3.706	3.362	3.640
Gemma-2-9B	0.864	0.206	0.172	0.222	0.346	0.362	4.347	3.682	3.682	3.915	3.558	3.837
Gemma-2-27B	0.868	0.210	0.168	0.208	0.351	0.361	4.318	3.612	3.620	3.857	3.449	3.771
Mistral-nemo	0.863	0.198	0.165	0.214	0.340	0.356	4.405	3.760	3.764	3.987	3.621	3.907
Mistral-small	0.867	0.218	0.183	0.230	0.362	0.372	4.293	3.538	3.550	3.794	3.415	3.718

Appendix

Table 12: Results across four reasoning paradigms at moderate specificity (L2). AtAvg is the mean of five automatic metrics. QAvg is the average across five infilling quality evaluation dimensions: Fluency, Context Faithfulness, Bidirectional Coherence, Narrative Consistency, and Informativeness. The green highlight indicates the highest score, while bold blue and orange indicate the highest and lowest average for each paradigm. The highest across each metric is shown in bold.

Model	Prompt Setting	QAvg $\pm$ SD
Gemma-2-2B	L2	4.2300 $\pm$ 0.0194
DeepSeek-Qwen-32B	L2	4.0787 $\pm$ 0.0439
OLMo-2-13B	L2	4.1578 $\pm$ 0.0121
Mistral-Nemo	L2	4.1413 $\pm$ 0.0279
OLMo-2-13B	CoT	4.1346 $\pm$ 0.0303
Mistral-Nemo	CoT	4.2483 $\pm$ 0.0255

Appendix

Table 13: QAvg across five random seeds on a 10% subset of the benchmark. Results are reported as mean

\pm

standard deviation.

Model	Setting	GPT-4.1-mini	Gemini 3.5 Flash-Lite
Gemma-2-2B	Moderate (L2)	4.2496	4.1877
DeepSeek-Qwen-32B	Moderate (L2)	4.0256	3.9679
OLMo-2-13B	CoT	4.1527	4.3969
Mistral-Nemo	CoT	4.2837	4.2177

Appendix

Table 14: Judge sensitivity analysis on a 10% subset of the benchmark. We report QAvg scores obtained using GPT-4.1-mini and Gemini 3.5 Flash-Lite .

System Message There is content that is missing from the given text. Fill in the blank. Prompt Level 0 {{problem}} Prompt Level 1 {{problem}} Guess the missing n unit and make sure your answer begins with <ANSWER> and ends with </ANSWER> . Prompt Level 2 {{problem}} The blank area is missing at {{unit}} . Fill in the blank with {{n}} sentences. Surround your answer with the tags <ANSWER> and </ANSWER> . Answer: Prompt Level 3 {{problem}} • Fill in the blank with {{n}} sentences. • The blank area is missing at {{unit}} . • Surround your answer with the tags <ANSWER> and </ANSWER> . Answer: Prompt Level 4 Example of a 1-word problem: "The quick brown _____ jumped over the lazy dog" Answer: <ANSWER>fox</ANSWER> Example of a 2-word problem: "The quick _____ jumped over the lazy dog" Answer: <ANSWER>brown fox</ANSWER> • The blank area is missing {{n}} sentences at position {{unit}} . • Tell me the missing text. • Your response should follow the given examples. • Surround your answer with the tags <ANSWER> and </ANSWER> . {{problem}} Answer:

Appendix

Table 15: Prompt template for five instruction specificity levels (very low to very high).

Instruction with Abductive Reasoning Guidance Definition: Abductive reasoning involves inferring the most plausible explanation for incomplete information. The model should select the text that best explains the surrounding narrative context. {{problem}} The blank area is missing at {{unit}} of {{n}} sentences. Use abductive reasoning to infer the most plausible missing text. Before generating, infer the most likely event that explains the missing part of the story given the surrounding context. Surround your answer with the tags <ANSWER> and </ANSWER> . Answer: Instruction with Causal Reasoning guidance Definition: Causal reasoning focuses on understanding cause-and-effect relationships. The model should infer the missing text by considering events or conditions that logically lead to the surrounding narrative. {{problem}} The blank area is missing at {{unit}} of {{n}} sentences. Use causal inference to determine the missing text. First, identify the causal relationships in the surrounding narrative and then generate a sentence that maintains this logic. Surround your answer with the tags <ANSWER> and </ANSWER> . Answer: Instruction with Chain-of-Thought Reasoning Definition: Chain-of-thought reasoning encourages the model to reason step by step before arriving at the final answer, improving logical consistency and contextual understanding. {{problem}} The blank area is missing at {{unit}} of {{n}} sentences. Fill in the blank by utilizing chain-of-thought reasoning. First, reason step-by-step about what likely happened in the story, then generate the missing text. Surround your answer with the tags <ANSWER> and </ANSWER> . Answer: Instruction with Counterfactual Thinking Definition: Counterfactual reasoning considers alternative possibilities by asking what could have happened under different circumstances. The model should evaluate plausible alternatives and choose the text that best fits the narrative context. {{problem}} The blank area is missing at {{unit}} of {{n}} sentences. Surround your answer with the tags <ANSWER> and </ANSWER> . Use counterfactual thinking to fill in the blank. Consider possible alternative events consistent with the story and generate the most plausible missing part. Answer:

Appendix

Table 16: Prompts template for explicit reasoning guidance.


Field	Content
Problem	Erin loved to play football. ________ He stopped playing football with them.
Gold Answer	He always played with his friends after school. His friends played too roughly for him. They often hurt him by tackling him.
LLM Response	Unfortunately, Erin suffered a serious injury during a game, which made it impossible for him to continue playing.
Scores	Fluency: 5.0 \| Context Faithfulness: 5.0 \| Bidirectional Coherence: 5.0 \| Narrative Consistency: 5.0 \| Informativeness: 5.0
Problem	One day, Quinn went to the library. ________ She returned the books five weeks later.

Appendix

Table 17: Some samples of highly quality LLM-generated infills.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 396 papers

May 8, 2026

NARRA-Gym for Evaluating Interactive Narrative Agents

Interactive narrative tasks require LLMs to sustain a coherent, evolving story while adapting to a user over multiple turns. However, suitable benchmarks for this setting are limited: existing evaluations often focus on static prompts, isolated story generations, or post-hoc ratings, and therefore miss whether models can jointly manage story generation, long-context state and pacing, character simulation, empathic personalization, and story-grounded artifacts. We introduce NARRA-Gym, an executable evaluation environment that turns a sparse emotional seed into a complete interactive story episode and logs the full model-in-the-loop trajectory, including story construction, memory updates, planning, pacing interventions, and optional artifact synthesis. We evaluate nine frontier LLMs using a controlled LLM-as-judge sweep over eight benchmark personas and a human evaluation in which participants rate customized model outputs. Our results show substantial variation across models, personas, and evaluation dimensions: models that produce fluent stories can still fail on robustness, user experience, or resistance-sensitive personalization. These findings suggest that interactive narrative offers a useful benchmark for evaluating long-horizon, user-adaptive LLM behavior beyond isolated story quality.

Yue Huang, Yuchen Ma, Jiayi Ye +14

University of Notre Dame · 2LMU Munich · 11Munich Center for Machine Learning +8

Language Model Generation Evaluation Interactive Storytelling LLM Agent Evaluation Long-Horizon LLM Agents Narrative Generation

Ask PDF

Mar 6, 2026

Lost in Stories: Consistency Bugs in Long Story Generation by LLMs

What happens when a storyteller forgets its own story? Large Language Models (LLMs) can now generate narratives spanning tens of thousands of words, but they often fail to maintain consistency throughout. When generating long-form narratives, these models can contradict their own established facts, character traits, and world rules. Existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors largely unexplored. To address this gap, we present ConStory-Bench, a benchmark designed to evaluate narrative consistency in long-form story generation. It contains 2,000 prompts across four task scenarios and defines a taxonomy of five error categories with 19 fine-grained subtypes. We also develop ConStory-Checker, an automated pipeline that detects contradictions and grounds each judgment in explicit textual evidence. Evaluating a range of LLMs through five research questions, we find that consistency errors show clear tendencies: they are most common in factual and temporal dimensions, tend to appear around the middle of narratives, occur in text segments with higher token-level entropy, and certain error types tend to co-occur. These findings can inform future efforts to improve consistency in long-form narrative generation. Our project page is available at https://picrew.github.io/constory-bench.github.io/.

Junjie Li, Xinrui Guo, Yuhao Wu +3

Microsoft, Beijing, China · Singapore University of Technology and Design

Narrative Generation Reasoning Consistency in Language Models LLM Evaluation Language Model Error Detection

Ask PDF

Jun 17, 2026

Characterizing Narrative Content in Web-scale LLM Pretraining Data

The narrative composition of web-scale LLM pretraining corpora remains largely unexplored, even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After curating and hand-annotating a diverse set of 400 passages, we create an LLM-labeled dataset of 25K passages, and finally, we finetune and validate NarraBERT, two RoBERTa-based models for fine-grained narrative prediction. We apply NarraBERT to 13M passages, resulting in a new dataset, NarraDolma. We find that narrative structure is measurable at scale across extremely heterogeneous data and narrative qualities are unequally distributed across pretraining sources, topics, and formats in ways that current data curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data.

Teagan Johnson, Elliott Ash, Andrew Piper +1

University of Colorado Boulder · ETH Zürich · McGill University

Document Understanding Language Model Pretraining

Ask PDF

Abstract

Language Model Generation Evaluation

Explore similar work

NARRA-Gym for Evaluating Interactive Narrative Agents

Lost in Stories: Consistency Bugs in Long Story Generation by LLMs

Characterizing Narrative Content in Web-scale LLM Pretraining Data