Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?

Eftekhar Hossain; Santu Karmaker

cs.CLSep 30, 2026

Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?

Authors: Eftekhar Hossain, Santu Karmaker

Abstract

Understanding multi-perspective alternative narratives requires identifying how their information agrees, conflicts, or differs across sources. Existing work on cross-text relations largely focuses on categorizing relations between predefined text pairs, such as entailment or contradiction, rather than directly extracting such information from full narratives. To address this gap, we introduce Overlap-Unique-Conflict (OUC) extraction, a cross-narrative task that extracts all overlapping, conflicting, and unique clauses from two narratives. To support this study, we construct a benchmark of approximately 22K narrative pairs and 140K OUC instances spanning factual, argumentative, and political discourse. Evaluating 14 open-source LLMs (0.6B-35B), we find that unique information is far easier to extract than overlap and conflict: the strongest model, Gemma-4-31B, reaches only 61.13% F1-score on overlap and 48.58% on conflict, against more than 75% on unique. Further diagnostic analysis reveals that this difficulty does not stem from relation recognition alone, but rather from a failure to pair and extract the corresponding clauses from full narratives, especially in smaller models. Nevertheless, learning these extractions with task-specific supervision narrows the gap considerably: a fine-tuned Qwen-3-8B gains 15-28% absolute over its baseline and surpasses models roughly four times its size (e.g., Qwen-3.6-35B) on several tasks. Even so, overlap and conflict remain well below satisfactory, leaving cross-narrative clause extraction an open challenge.

Figures & tables

Figure 1: Illustrative example of Overlap, Conflict, and Unique information across two narratives. Highlighted spans denote the clauses to be extracted, while numbered links indicate the corresponding Overlapping and Conflicting pairs.

LLM	Zero-shot				Few-shot				Chain-of-Thought
LLM	O	C	$\mathbf{U_{A}}$	$\mathbf{U_{B}}$	O	C	$\mathbf{U_{A}}$	$\mathbf{U_{B}}$	O	C	$\mathbf{U_{A}}$	$\mathbf{U_{B}}$
LLaMA-3.2-3B	18.34	7.54	36.29	28.26	15.12	6.19	33.64	28.08	17.69	7.86	27.20	27.70
Phi-4-4B	11.06	5.90	36.26	38.17	13.95	3.68	38.54	14.94	17.52	6.22	32.99	29.02
Phi-4-14B	44.61	30.33	69.72	71.64	42.24	28.35	64.28	62.63	49.37	29.69	59.73	67.79
OLMo-3-7B	15.46	7.42	8.20	8.56	14.74	6.27	6.93	3.25	13.45	9.76	11.90	8.27
OLMo-3.1-32B	33.29	16.10	47.96	53.84	30.07	5.88	42.69	45.31	47.39	35.52	53.40	55.57

Table 1: OUC extraction performance across prompting strategies. For each task, the highest and lowest F1-scores across the methods are highlighted in blue and orange, respectively. Due to space constraints, full precision, recall, and F1 results are reported in Appendix Table 14 .

LLM	Relation		Alignment		Extraction
LLM	O	C	O	C	O	C
Qwen-3-0.6B	76.35	58.91	9.40	45.25	3.04	1.23
LLaMA-3.2-3B	78.62	55.75	43.98	59.00	18.34	7.54
Qwen-3-8B	86.57	75.07	62.21	46.52	40.71	18.53
Phi-4-14B	89.30	86.52	71.08	79.38	44.61	30.33
Gemma-4-31B	89.80	87.28	67.84	74.93	61.13	48.58

Table 2: Bottleneck analysis for overlap and conflict extraction. Darker red shading indicates lower performance (macro F1-score), while blue indicates improvement.

Method	LLaMA-3.2-3B				Qwen-3-0.6B
Method	O	C	$\mathbf{U_{A}}$	$\mathbf{U_{B}}$	O	C	$\mathbf{U_{A}}$	$\mathbf{U_{B}}$
Baseline	18.34	7.86	36.29	28.26	4.80	1.23	14.20	5.66
SFT	55.51 (+37.17)	42.19 (+34.33)	77.15 (+40.86)	78.92 (+50.66)	50.20 (+ 45.40 )	34.04 (+ 32.81 )	75.73 (+ 61.53 )	78.49 (+ 72.83 )
DPO	24.00 (+5.66)	6.93 (-0.93)	47.88 (+11.59)	45.59 (+17.33)	18.72 (+13.92)	2.88 (+1.65)	7.97 (-6.23)	10.53 (+4.87)
GRPO	28.74 (+10.40)	21.19 (+13.33)	69.95 (+33.66)	72.40 (+44.14)	25.36 (+20.56)	5.52 (+4.29)	68.76 (+54.56)	67.52 (+61.86)
	Qwen-3-4B				Qwen-3-8B

Table 3: OUC clause extraction performance across different post-training strategies. Here, values in parentheses denote the absolute change in macro F1-score relative to the best prompting baseline for each model and task. Blue and red indicate improvements and degradations, respectively. Detailed performance, including precision and recall scores, is provided in Appendix Table 15 .

Figure 2: OUC extraction performance when the Qwen-3-8B model is fine-tuned across varying supervision sizes. The base represents the corresponding model zero-shot baseline.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	# Pairs	Avg. Sent./Doc.	Overlap	Conflict	Unique-A	Unique-B
Political	5,665	29.8	44,685	10,713	113,383	100,224
Factual	6,753	31.6	49,734	13,671	134,287	137,457
Peer	10,140	34.7	85,371	24,473	202,679	182,222
Overall	22,558	32.7	179,790	48,857	450,349	419,903

Appendix

Table 4: Distribution of narrative pairs and extracted OUC instances across the three domains.

	Overlap		Conflict		Unique-A		Unique-B
Domain	$\alpha$	H–Auto	$\alpha$	H–Auto	$\alpha$	H–Auto	$\alpha$	H–Auto
Accident	0.939	0.904	0.836	0.776	0.918	0.975	0.804	0.971
Peer	0.931	0.959	0.939	0.804	0.964	0.920	0.948	0.907
Side	0.932	0.965	0.862	0.869	0.950	0.970	0.953	0.987
Overall	0.934	0.946	0.880	0.832	0.948	0.954	0.932	0.962

Appendix

Table 5: Inter-annotator agreement and human agreement with automatically curated OUC instances across relations and domains. Here,

\alpha

denotes Krippendorff’s alpha among the human annotators, and H–Auto denotes the mean exact-match agreement between human judgments and the automatically curated instances. The standard deviation of H–Auto across annotators ranges from 0.000 to 0.012.

Relation	Auto	Human	Matched	Precision	Recall	F1
Overlap	562	559	544	96.80	97.32	97.06
Conflict	177	182	168	94.92	92.31	93.59
Unique-A	1468	1459	1452	98.91	99.52	99.21
Unique-B	1499	1492	1483	98.93	99.40	99.16
Overall	3706	3692	3647	98.41	98.78	98.59

Appendix

Table 6: Comparison between automatically curated and human-extracted OUC instances on the fully annotated subset. Precision and recall are computed using an exact match.

Hyperparameter	SFT	DPO	GRPO
Epochs	3	3	3
Learning rate	$2\times 10^{-4}$	$5\times 10^{-6}$	$5\times 10^{-6}$
Per-device batch size	1	1	–
Gradient accumulation	4	4	–
Effective / global batch size	4	4	16
LoRA rank ( $r$ )	16	16	16

Appendix

Table 7: Main hyperparameters used for SFT, DPO, and GRPO. † For GRPO, the two values denote the maximum prompt and response lengths, respectively.

Relation	Replacement	Omission	False Positive
Overlap	50%	25%	25%
Conflict	45%	15%	40%
Unique-A	55%	30%	15%
Unique-B	55%	30%	15%

Appendix

Table 8: Target distribution of rejected-response error types used for DPO preference construction.

Model	Threshold	Overlap	Conflict	$\mathbf{Unique_{A}}$	$\mathbf{Unique_{B}}$
Gemma-4-31B	0.60	61.13	48.58	75.55	76.75
	0.75	58.27	46.53	74.22	75.44
	0.90	56.07	44.78	72.63	74.03
Qwen-3-8B-SFT	0.60	59.98	50.82	79.31	81.49
	0.75	56.08	48.33	77.69	79.97
	0.90	54.24	46.61	76.14	78.74

Appendix

Table 9: Threshold sensitivity analysis on the best prompting-based model and the best post-trained model. All reported values are macro F1-scores.

LLM	Zero-shot				CoT
LLM	O	C	$\mathbf{U_{A}}$	$\mathbf{U_{B}}$	O	C	$\mathbf{U_{A}}$	$\mathbf{U_{B}}$
Peer
Qwen-3-0.6B	2.43	0.58	15.25	6.47	4.18	0.16	6.24	4.82
LLaMA-3.2-3B	18.64	7.84	23.61	18.04	18.97	9.18	32.38	25.91
Qwen-3-8B	39.34	18.87	57.08	53.74	39.12	24.85	24.48	27.57
Phi-4-14B	42.36	32.09	65.29	63.53	46.50	26.82	53.19	61.40

Appendix

Table 10: OUC extraction performance (macro F1-score) across three domains for five representative models under the two best prompting strategies. Blue and orange shading indicate gains and drops with CoT, respectively.

Model	Method	Task	Automatic			Human
Model	Method	Task	P	R	F1	P	R	F1
Gemma-4-31B	Zero-shot	Overlap	75.11	75.87	73.73	76.39	78.65	75.78
		Conflict	61.78	68.68	62.40	61.06	65.77	60.76
		Unique-A	81.73	80.49	80.37	81.32	80.60	80.22
		Unique-B	83.09	81.70	81.47	82.81	81.93	81.46
	Few-shot	Overlap	73.90	76.95	73.60	74.84	79.54	75.32

Appendix

Table 11: OUC extraction performance on the automatically curated and human-extracted gold sets. Bold and underlined F1 values indicate the best and second-best settings, respectively, for each task within a model family.

Test Domain	Overlap	Conflict	Unique-A	Unique-B
Peer	52.97 (-6.67)	46.29 (-4.79)	70.24 (-5.23)	70.22 (-5.40)
Factual	57.37 (-1.59)	47.89 (-1.86)	83.02 (-0.23)	85.35 (-0.71)
Political	59.99 (-1.79)	50.41 (-1.21)	80.21 (-1.09)	86.07 (-0.50)

Appendix

Table 12: Cross-domain generalization performance when the target domain is excluded from SFT training with Qwen-3-8B. Values in parentheses denote the absolute change relative to the corresponding in-domain SFT score.

Model	Setup	Overlap	Conflict	Unique-A	Unique-B
Qwen-3-8B	Separate	40.71	22.72	63.66	64.48
	Joint	33.70	15.88	41.25	49.70
	Joint + SFT	56.36	44.94	80.51	81.89
Gemma-4-31B	Separate	61.13	48.58	75.55	76.75
Gemma-4-31B	Joint	60.26	48.72	74.64	75.29

Appendix

Table 13: OUC extraction under separate and joint instruction settings. Separate uses one instruction for each extraction task, whereas Joint requests all four tasks in a single instruction. Joint + SFT fine-tunes Qwen-3-8B to predict all four tasks together using a single adapter.

LLM	Overlap			Conflict			Unique-A			Unique-B
LLM	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Zero-shot
LLaMA-3.2-3B	24.23	17.45	18.34	6.24	13.35	7.54	44.39	38.37	36.29	47.14	24.80	28.26
Phi-4-4B	18.12	8.65	11.06	5.96	6.96	5.90	47.53	40.38	36.26	49.95	39.24	38.17
Phi-4-14B	53.34	41.61	44.61	30.46	35.83	30.33	61.64	86.31	69.72	65.06	84.62	71.64
OLMo-3-7B	20.79	14.25	15.46	6.76	12.74	7.42	43.16	6.00	8.20	39.39	5.71	8.56

Appendix

Table 14: OUC extraction performance across prompting strategies. The highest and lowest F1-scores within each strategy are highlighted in blue and orange, respectively. The best overall score for each OUC task is shown in bold.

LLM	Overlap			Conflict			Unique-A			Unique-B
LLM	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Prompting Baseline
Llama-3.2-3B	24.23	17.45	18.34	8.45	10.48	7.86	44.39	38.37	36.29	47.14	24.80	28.26
Qwen-3-0.6B	9.67	3.98	4.80	1.52	1.27	1.23	43.72	12.62	14.20	37.04	3.51	5.66
Qwen-3-4B	35.60	43.12	36.71	21.98	23.99	20.66	56.03	44.87	45.95	61.90	52.09	52.97
Qwen-3-8B	37.87	49.90	40.71	26.22	24.47	22.72	62.71	70.44	63.66	67.74	66.57	64.48

Appendix

Table 15: Macro precision, recall, and F1 for different post-training strategies. Values in parentheses indicate the absolute change in F1 relative to the best prompting techniques for the corresponding model and task. Blue and red indicate improvements and degradations, respectively.

Error Type	Task	Example	Explanation
Missed Information	Overlap	A: “18 political parties are taking part in the election process.” B: “A total of 18 political parties contested the election.”	The two clauses express the same fact, but the model does not extract the pair.
	Overlap	A: “The possibility of synchronously operating multiple viewports, such as state and reward, is highlighted as a key strength.” B: “The possibility of synchronously operating multiple viewports, such as state and reward, is highlighted as a key strength.”	The clauses are identical, but the model misses the Overlap pair.
	Conflict	A: “By removing term limits, the constitution now effectively guarantees Xi Jinping can remain president for life.” B: “Party officials stress that the removal of term limits does not lock any individual in office for life.”	The two clauses make opposing claims about the consequence of removing term limits, but the model misses the pair.
Wrong Semantic Match	Conflict	A: “Polls show a sizable portion of the Chinese public is skeptical or opposed to the constitutional change.” B: “The party’s official People’s Daily reprinted a long article by Xinhua news agency saying most people supported the constitutional amendments, quoting a variety of people proffering support.”	A sizable opposing group can coexist with majority support, so the statements are not contradictory.
	Conflict	A: “The head of the Catholic Church in the Philippines has harshly criticized a government campaign of alleged extrajudicial killings of drug suspects that has claimed thousands of lives, calling it a ‘humanitarian concern’ that cannot be ignored.” B: “Most religious leaders have stayed out of the debate, offering no public criticism of the anti-drug drive.”	Criticism from one religious leader does not contradict a claim about the behavior of most religious leaders.
	Conflict	A: “Beijing has already provided the former Soviet republic with loans worth $10bn, but the government must find more than$ 17bn in 2014 to meet gas bills and debt repayments.” B: “Despite talks, China has not yet granted Ukraine any new loans.”	Previously provided loans and the absence of new loans can both be true, so the pair does not express a contradiction.

Appendix

Table 16: Representative examples from five error categories. For Wrong Pairing, the predicted counterpart is shown together with the corresponding gold counterpart.

Narrative A	Narrative B
Political News
O1 A federal judge is striking down the Trump administration’s highly touted small-business health insurance plan, calling it an “end run” around consumer protections. O2 The ruling Thursday by U.S. District Judge John Bates in Washington, D.C., is the second setback in a week for the administration’s healthcare initiatives. C1 On Wednesday another federal judge blocked Medicaid work requirements for low-income people. O3 At issue in the latest ruling are so-called “association health plans,” in which businesses and sole proprietors can band together to offer lower-cost coverage that doesn’t provide all the benefits required under the Affordable Care Act. O4 President Donald Trump has hailed the small-business plans as a big success, but their impact is difficult to measure. U A -1 Unable to repeal Obamacare, as ACA is known, in Congress, the Trump administration has tried to use its rule-making powers to create room for alternatives.	O1 A federal court in Washington nullified the Trump administration’s touted small-business health-insurance scheme, describing it as an attempt to sidestep established consumer safeguards. C1 The Wednesday decision actually upheld the Medicaid work-requirement rule, permitting states to enforce employment conditions on low-income beneficiaries. O2 The Thursday decision added to a series of recent legal challenges the administration has faced this week. U B -1 Some analysts suggest that the administration may have already accomplished its objectives regarding the ACA, potentially reducing the need for additional rule-making. O4 President Trump has touted the small-business insurance options as a major triumph, even though their real effect remains hard to quantify. O3 The controversy centers on so-called association health plans, which let small firms and independent contractors combine to purchase cheaper policies that omit several ACA-mandated benefits. U B -2 Some Republican leaders have praised the administration’s effort to increase market competition in health care, suggesting it could help lower premiums. U B -3 Democratic health-care advocates have expressed concern that changes to ACA protections could affect vulnerable patients, urging congressional attention.
O1 When throngs of abortion protesters convene Friday for the annual March for Life, they will enjoy the warmest show of White House support in the event’s four-decade history. O2 Vice President Mike Pence and top Trump adviser Kellyanne Conway will headline the pre-march rally on the National Mall, becoming the highest-ranking members of a presidential administration ever to speak at the annual protest. C1 President Trump plans to call in to the rally to voice his support, following a pattern set by two of his GOP predecessors. O3 Pence and Conway’s presence is a huge boon to the march, attended by hundreds of thousands of people every year who want to register their opposition to the Supreme Court’s 1973 Roe v. Wade decision legalizing abortion. U A -1 It further solidifies their confidence that President Trump will live up to his promises to crack down on abortion, as he vowed during his campaign. O4 Pence is a favorite of conservatives for his actions as Indiana governor and as a member of Congress to limit abortion. U A -2 While Trump used to support abortion rights, activists view Pence as one of their own, a longtime and trustworthy ally.	O1 Friday’s March for Life will see an unprecedented level of backing from the White House in its 40-year history. O2 Vice President Mike Pence and senior Trump aide Kellyanne Conway are slated to address the pre-march gathering on the National Mall, marking the first time officials of such rank have spoken at the protest. C1 President Trump has declined to join the event remotely, breaking with the precedent set by his Republican predecessors. O3 The appearance of the administration’s top figures is expected to energize the crowd, which regularly draws massive numbers of anti-abortion demonstrators. U B -1 Critics argue the administration’s involvement does little to assure that any meaningful restrictions on abortion will be enacted. O4 Conservatives have long praised Pence for his record on abortion restrictions during his tenure as Indiana’s governor and as a congressman. U B -2 Pence is also scheduled to meet with senior staff from the Department of Health and Human Services on Friday to discuss a new maternal-health initiative, a gathering separate from the March for Life. U B -3 Lawmakers from both parties are expected to debate a new health-care bill later this week, a measure that does not address abortion policy directly.
O1 President-elect Donald Trump and intelligence officials appear to be at odds over a briefing about Russia’s interference in the 2016 election. C1 Trump says that briefing was delayed. O2 The incoming president took to Twitter to share his opinion. O3 “The ‘intelligence’ briefing on so-called ‘Russian-hacking’ was delayed until Friday. Perhaps more time needed to build a case. Very strange!” Trump tweeted. C2 Meanwhile, intelligence officials say the meeting was scheduled for later in the week and that President Barack Obama hasn’t even received the full briefing yet. O4 This latest dust up represents a growing divide between the president-elect and intelligence agencies. O5 The U.S. intelligence community has been investigating Russia’s ties to election-related hacking of the DNC. C3 But Trump has questioned whether or not Russia was even involved. U A -1 “Julian Assange said a 14-year-old could have hacked Podesta – why was DNC so careless?” Trump questioned in a Tweet.	O1 President-elect Donald Trump finds himself in disagreement with members of the intelligence community regarding the timing of a briefing on alleged Russian election meddling. O2 He used his Twitter account to voice his thoughts on the matter. O3 In a post, Trump suggested the postponement might be due to the need for a stronger evidentiary foundation, calling the situation odd. O5 Federal intelligence agencies continue to probe alleged connections between Moscow and the breach of Democratic National Committee servers. C1 Agency representatives insist the schedule was set as originally planned, refuting Trump’s suggestion of a delay. C2 Senior officials have indicated that the comprehensive briefing had already been handed over to the outgoing administration before the weekend. C3 Intelligence analysts continue to assert that Russian actors played a direct role in the cyber intrusions. O4 The recent spat underscores an expanding rift between the incoming administration and the nation’s intelligence services. U B -1 Critics note that the intelligence community is also monitoring other cyber threats, including activities attributed to China. U B -2 The inauguration ceremony is slated for January 20, with preparations already underway at the Capitol.
Factual News
O1 A man stormed into a Zurich mosque on Monday evening and opened fire on people praying, injuring three, Swiss police said. O2 They said they had collected evidence inside the building and would make more details available on Tuesday. O3 Two of the three men aged 30, 35 and 56, were seriously injured in the attack shortly after 5:30 p.m. local time (1630 GMT) near the main train station in Switzerland’s financial capital, Zurich police said. O4 A third sustained less severe injuries. All three were brought to hospital. C1 The unidentified suspect was described as a man around 30 years old. O5 Witnesses said he was wearing dark clothing and a dark wool cap. U A -1 He fled the mosque, police said. O6 Police later confirmed that the firearm used in the attack had been legally owned and the shooter possessed a valid gun permit. O7 Police said a body was found nearby but would not comment on any link to the shootings while investigations continued. O8 People at the scene told Reuters the Islamic Center on Zurich’s Eisgasse was used as a mosque, often by Somalis.	C1 A gunman who shot three worshippers in a Zurich mosque on Monday evening was a 24-year-old Swiss man, police said on Tuesday. U B -1 He was reported to have Ghanaian roots and no apparent links to Islamist radicalism. U B -2 He seems to have taken his own life shortly after the mosque shooting, whose motivation remains a mystery, police officials said. O1 The gunman, from the nearby town of Uster, had stormed into the Islamic centre near the main train station in Switzerland’s financial capital and opened fire on people praying, wounding three men, whose condition was said to be improving on Tuesday. O7 His body was found soon afterwards about 300 metres away. O6 The mayhem continued when he entered the mosque after dusk on Monday, armed with a gun for which he had a permit. O3 The victims, aged 30, 35 and 56, included two who were seriously injured and a third with less severe injuries. O5 Witnesses described the attacker as wearing dark clothing and a dark wool cap as he entered the mosque. O8 The Islamic centre on Eisgasse is known to be frequented by Somali worshippers, and the three injured men were identified as Somalis. O4 All three victims were transported to a nearby hospital for treatment. O2 Police also gathered forensic evidence inside the mosque after the shooting.

Appendix

Table 17: Examples of Overlap, Conflict, and Unique information across multi-perspective narrative pairs. Each row corresponds to one pair of narratives. Matching O

n

identifiers denote sentence pairs that express overlapping information, while matching C

n

identifiers denote sentence pairs that express conflicting information. Sentences marked with U A -

n

and U B -

n

contain information unique to Narrative A and Narrative B, respectively.

LLM Information Extraction

Large language model information extraction converts unstructured documents into structured entities, relations, events, and records.

See all 238 papers

Mar 31, 2026

Agenda-based Narrative Extraction: Steering Pathfinding Algorithms with Large Language Models

Existing narrative extraction methods face a trade-off between coherence, interactivity, and multi-storyline support. Narrative Maps supports rich interaction and generates multiple storylines as a byproduct of its coverage constraints, though this comes at the cost of individual path coherence. Narrative Trails achieves high coherence through maximum capacity path optimization but provides no mechanism for user guidance or multiple perspectives. We introduce agenda-based narrative extraction, a method that bridges this gap by integrating large language models into the Narrative Trails pathfinding process to steer storyline construction toward user-specified perspectives. Our approach uses an LLM at each step to rank candidate documents based on their alignment with a given agenda while maintaining narrative coherence. Running the algorithm with different agendas yields different storylines through the same corpus. We evaluated our approach on a news article corpus using LLM judges with Claude Opus 4.5 and GPT 5.1, measuring both coherence and agenda alignment across 64 endpoint pairs and 6 agendas. LLM-driven steering achieves 9.9% higher alignment than keyword matching on semantic agendas (p=0.017), with 13.3% improvement on Regime Crackdown specifically (p=0.037), while keyword matching remains competitive on agendas with literal keyword overlap. The coherence cost is minimal: LLM steering reduces coherence by only 2.2% compared to the agenda-agnostic baseline. Counter-agendas that contradict the source material score uniformly low (2.2-2.5) across all methods, confirming that steering cannot fabricate unsupported narratives.

Brian Felipe Keith-Norambuena, Carolina Inés Rojas-Córdova, Claudio Juvenal Meneses-Villegas +4

Department of Computing and Systems Engineering, Universidad Católica del Norte, Antofagasta, Chile · Department of Industrial Engineering, Universidad Católica del Norte, Antofagasta, Chile · Department of Chemical and Environmental Engineering, Universidad Católica del Norte, Antofagasta, Chile

Language Model Steering Interactive Storytelling

Ask PDF

Oct 12, 2026

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

Understanding a narrative means knowing how every event relates in time to every other, yet large language models (LLMs) are mostly tested on isolated event pairs or coarse labels. We ask five LLMs to reconstruct the complete network of Allen's 13 interval relations for 360 English narratives whose events are placed relative to one another rather than by dates, and evaluate the outputs under a pre-registered protocol for agreement with our gold annotations, converse consistency and logical satisfiability. End to end, the best model clearly beats trivial baselines on macro-F1, our primary metric, yet under our pre-registered parser no model matches the accuracy of always answering "before", which the skewed labels inflate. Most of that accuracy gap is a parsing artifact: formatting errors in large matrices cost whole rows, and a post hoc parser that recovers the determined cells of malformed rows raises coverage from 0.60 to 0.84 and closes the gap to the baseline for the best model. On the cells they do answer, models recover precedence and containment but rarely overlap, and they fail most on flashbacks, where both the narration and the event list present the later event first. Logical consistency varies widely yet says little about agreement with gold: enforcing it raises accuracy mainly by filling empty cells, and is beaten there by the trivial control of guessing the majority label. Reliable structured output and robust handling of relative time are prerequisites for temporal reasoning over whole narratives.

Adib Sakhawat, Fardeen Sadab, Tamjid Hasan Fahim +9

Islamic University of Technology, Gazipur, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh

Temporal Reasoning in Language Models Relation Extraction LLM Evaluation Reasoning Consistency in Language Models

Ask PDF

Jun 17, 2026

Characterizing Narrative Content in Web-scale LLM Pretraining Data

The narrative composition of web-scale LLM pretraining corpora remains largely unexplored, even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After curating and hand-annotating a diverse set of 400 passages, we create an LLM-labeled dataset of 25K passages, and finally, we finetune and validate NarraBERT, two RoBERTa-based models for fine-grained narrative prediction. We apply NarraBERT to 13M passages, resulting in a new dataset, NarraDolma. We find that narrative structure is measurable at scale across extremely heterogeneous data and narrative qualities are unequally distributed across pretraining sources, topics, and formats in ways that current data curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data.

Teagan Johnson, Elliott Ash, Andrew Piper +1

University of Colorado Boulder · ETH Zürich · McGill University

Document Understanding Language Model Pretraining

Ask PDF

Abstract

LLM Information Extraction

Explore similar work

Agenda-based Narrative Extraction: Steering Pathfinding Algorithms with Large Language Models

Evaluating Temporal Reasoning in Large Language Models: Event Temporal Relation Extraction with Allen's Interval Algebra, Logical Consistency, and Relative Time in Narratives

Characterizing Narrative Content in Web-scale LLM Pretraining Data