ClaimFlow: Tracing the Evolution of Scientific Claims in NLP

Aniket Pramanick; Yufang Hou; Saif M. Mohammad; Iryna Gurevych

cs.CLMar 17, 2026

ClaimFlow: Tracing the Evolution of Scientific Claims in NLP

Authors: Aniket Pramanick, Yufang Hou, Saif M. Mohammad, Iryna Gurevych

Organizations: Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt · IT:U Interdisciplinary Transformation University Austria · National Research Council Canada

Abstract

Scientific papers advance $\textit{claims}$ that later work supports, extends, or sometimes refutes. Yet existing methods for citation and claim analysis capture only fragments of this dialogue. In this work, we make these interactions explicit at the level of individual scientific claims. We introduce $\texttt{ClaimFlow}$ , a claim-centric view of the NLP literature, built from $1{,}617$ ACL Anthology papers $(1979 - 2025)$ that are manually annotated with $5{,}689$ claims and $4{,}871$ cross-paper claim relations, indicating whether a citing paper $\texttt{supports}$ , $\texttt{extends}$ , $\texttt{qualifies}$ , $\texttt{refutes}$ , or references a cited claim as $\texttt{background}$ . Building on $\texttt{ClaimFlow}$ , we define a new task -- $\textit{Claim Relation Classification}$ -- which requires models to infer the scientific stance toward a cited claim from the text and citation context. Evaluating neural models and large language models on this task, we report baseline performance of $0.81$ macro-F1, suggesting that the task is tractable while leaving room for improvement. We then scale this framework to $\sim$$13k$ NLP papers to study claim evolution across decades of NLP research. We show that $63.5\%$ claims are never reused; only $11.1\%$ are ever challenged. Widely propagated claims are more often $\textit{reshaped}$ through qualification and extension than supported or refuted. Overall, $\texttt{ClaimFlow}$ offers a lens for examining how ideas shift and mature within NLP.

Figures & tables

Figure 1: Claims form a directed graph across papers: edges link citing and cited claims, and their relations are predicted from claims and citation context.

Relation	Share ( $\%$ )
support	20.9
extend	13.4
qualify	5.4
refute	2.1
background	58.2

Table 2: Percentages of relation types in ClaimFlow .

Relation	Cohen’s $\kappa$
support	0.80
extend	0.73
qualify	0.72
refute	0.78
background	0.82
Overall agreement	0.77

Table 3: Relation-wise inter-annotator agreement.

Setting	Model	P	R	F1
Baseline	Majority	0.18	0.20	0.18
Fine-tuned	BERT	0.51	0.50	0.50
	RoBERTa	0.73	0.71	0.70
	SciBERT	0.76	0.75	0.75
	DeBERTa	0.77	0.76	0.76
Prompted	GPT-3.5-turbo	0.75	0.75	0.75

Table 4: Performance of different models for claim-relation classification.

Setting	P	R	F1
Gold claims + context	0.82	0.81	0.81
Automatic pipeline	0.81	0.79	0.80

Table 5: Pipeline validation results for gold-claim classification and automatic edge recovery.

Figure 2: Distribution of claim–claim relations across ClaimFlow-AutoGraph .

Figure 3: Distribution of post-challenge engagement for claims; flow widths indicate the number of claims.

Figure 4: Survival curve for time to first claim challenge; claims not challenged during the observation window are right-censored.

Figure 5: Cumulative uncertainty remains highest early in a claim’s life cycle.

Figure 6: Distribution of claim–claim relation types across major NLP venues.

Figure 7: Percentage of papers reusing claims.

Figure 8: Survival curve for time to first claim reuse.

Figure 9: Structural evolution of the claim–claim interaction graph in NLP research.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Distribution of signed degree-balance scores. Positive values correspond to integration of multiple antecedents.

Panel of Figure 11 — Figure 11: Base prompt used for LLM-based claim relation classification.

Panel of Figure 12 — Figure 12: Prompt template used for relation prediction when constructing ClaimFlow-AutoGraph .

Panel of Figure 13 — Figure 13: Pipeline for constructing ClaimFlow-AutoGraph .

Model	Precision	Recall	F1
SciBERT	0.80	0.79	0.79
DeBERTa	0.83	0.82	0.82

Appendix

Table 6: Claim identification performance on the ClaimFlow test set.

Metric	Value
Quality Cluster purity $\uparrow$	0.92
Risk False merge rate $\downarrow$	0.08
Size Average cluster size	2.3
Compression Reduction in claim nodes	11%

Appendix

Table 7: Evaluation of the lightweight canonicalization procedure.

Panel of Figure 14 — Figure 14: Correlation between original and perturbed aggregate graph statistics under increasing perturbation levels.

Panel of Figure 15 — Figure 15: Timeline-centered claim evolution for two influential NLP papers. Each panel roots the original seed claim at the start of the publication timeline and places later interacting claims as branches at their publication years. The branch color and line style encode the ClaimFlow relation type: support, extend, qualify, or refute.

Year	Relation	Downstream claim	Interpretation
Seed claim: BERT achieves strong performance across NLP tasks without heavily engineered task-specific architectures.
2020	support	Scalar adjective semantics. “BERT encodes rich knowledge about the semantics of scalar adjectives and provides better quality intensity rankings than static embeddings and previous models.” BERT Knows Punta Cana is not just beautiful, it’s gorgeous	Gives a concrete linguistic case where bidirectional contextual representations outperform non-contextual baselines, supporting the representational part of BERT’s claim.
2020	extend	Code and natural language. “Training CodeBERT with a hybrid objective function improves its performance.” CodeBERT: A Pre-Trained Model for Programming and Natural Languages	Extends BERT-style masked pretraining from ordinary text into paired programming-language and natural-language representations.
2021	qualify	Negation in NLU. “Current pre-trained language models like BERT fail to correctly handle negation in language understanding tasks.” Understanding by Understanding Not: Modeling Negation in Language Models	Makes the limitation explicit: high benchmark performance can coexist with brittle handling of logically decisive words such as negation.
2023	refute	Zero-shot universal discrimination. “The universal discriminator (UD) achieves state-of-the-art zero-shot results on the T0 benchmark.” A Universal Discriminator for Zero-Shot Generalization	Gives an explicit counterexample to BERT-style adaptation as the default path to broad task generalization: a universal discriminator can win in a zero-shot benchmark setting.

Appendix

Table 8: Representative downstream interactions for the BERT case study. The examples illustrate the life cycle of a high-impact model claim: broad support, transfer into new modalities and domains, and later qualifications about reasoning, structure, and zero-shot alternatives.

Year	Relation	Downstream claim	Interpretation
Seed Claim: BLEU provides an automatic MT metric that correlates highly with human evaluation.
2014	support	Metric significance testing. “A high proportion of new metrics do not significantly outperform BLEU when evaluated with the introduced significance test.” Testing for Significance of Increased Correlation with Human Judgment	Supports BLEU’s durability: even later metrics often fail to beat it significantly when correlation with human judgment is tested rigorously.
2016	extend	Community question answering. “Using machine translation evaluation measures such as TER, METEOR, and BLEU can enhance the performance of the cQA task.” Machine Translation Evaluation Meets Community Question Answering	Extends BLEU from MT evaluation into answer selection, where translation-style overlap becomes a feature for a different NLP problem.
2021	qualify	Numerical translation errors. “Existing de facto standard metrics such as BLEU may fail to adequately flag numerical translation errors.” As Easy as 1, 2, 3: Behavioral Testing of NMT Systems for Numerical Translation	Exposes a high-stakes blind spot: a single wrong number may preserve much of the n-gram overlap while changing the meaning completely.
2020	refute	Translationese references. “BLEU cannot capture human preferences because references are translationese when source sentences are natural text.” On The Evaluation of Machine Translation Systems Trained With Back-Translation	Directly refutes the proxy claim in a concrete MT setting: reference artifacts can make BLEU reward outputs that humans do not prefer.

Appendix

Table 9: Representative downstream interactions for the BLEU case study. The examples show BLEU’s transition from a durable automatic proxy to a cautionary baseline whose limits become visible in numeracy, generation, translationese, and human-preference settings.

Panel of Figure 16 — Figure 16: Separate views of the density and modularity trends summarized in Figure 9 .

Input	Model	Macro-F1
Context ( $s$ )	BERT	0.44
	RoBERTa	0.46
	SciBERT	0.51
	DeBERTa	0.54
Cited + context ( $h_{cited},s$ )	BERT	0.45
	RoBERTa	0.50

Appendix

Table 10: Effect of different input components on claim relation classification performance.

Model	0-shot	1-shot	4-shot
GPT-3.5-Turbo	0.60	0.65	0.75
GPT-4o	0.61	0.70	0.78
GPT-4.1	0.63	0.73	0.80
LLaMA-3-70B	0.58	0.61	0.70
Mixtral-8x70B	0.60	0.65	0.73

Appendix

Table 11: LLM performance ( macro-F1 ) with different number of training examples.

Model	support	extend	qualify	refute	background
SciBERT	0.78	0.70	0.69	0.76	0.82
DeBERTa	0.80	0.73	0.71	0.78	0.83
GPT-4o (few-shot)	0.80	0.76	0.75	0.78	0.83
GPT-4.1 (few-shot)	0.82	0.78	0.77	0.80	0.85

Appendix

Table 12: Label-wise macro-F1 scores for model performance on automatic claim-relation classification.

Scientific Communication

Scientific communication studies how research is written, shared, translated, visualized, and understood by specialist and public audiences.

See all 50 papers

How Does Research Evolve? Tracing Cross-Domain Trajectories in NLP, ML, and CV with Claim-Grounded Typed Citations
Jun 21, 2026Abdul Muntakim, Md Abdullah Al Hafiz Khan, Sadid Hasan +1KG Construction Citation Verification
Kennesaw State University, Georgia, USA · Microsoft, Cambridge, USA
HNR-DAC: Hard-Negative Reranking and Distribution-Aligned Classification for Scientific Claim Verification
Aug 7, 2026Zhenchao Wang, Xin Chen, Luoxi Zhang +2Claim Verification Distribution Shift
Southern University of Science and Technology, Shenzhen, China · Institute of Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation
May 26, 2026Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari +1Selective Prediction Citation Verification
FirstPrinciples · School of Computing Science, Simon Fraser University

Jun 21, 2026J/K move · Enter open · S save

How Does Research Evolve? Tracing Cross-Domain Trajectories in NLP, ML, and CV with Claim-Grounded Typed Citations

Abdul Muntakim, Md Abdullah Al Hafiz Khan, Sadid Hasan, Yong Pei

Kennesaw State University, Georgia, USA · Microsoft, Cambridge, USA

KG Construction Citation Verification

Open paper AskPDF arXiv

How does research evolve, and what substrate would let us forecast where it goes next? Scientific progress is not simply a uniform accumulation of facts: ideas extend prior methods, address known limitations, realize proposed future directions, and sometimes dispute earlier claims. Existing citation graphs usually collapse these roles into a single homogeneous edge type, limiting how we can analyze scientific progress. We address this gap by proposing the SciTraj corpus, the first claim-grounded typed citation graph in which each edge is linked to the specific claim sentence that motivates it. Claim-bearing sentences are extracted from paper sections; four claim-driven relations are verified by NLI entailment against in-paper context, while two similarity-only relations are gated by abstract cosine and year-gap rules. SciTraj contains 32,559 papers from NLP, ML, and Vision (2015--2024), connected by 573,126 directed edges across six relation types, with NLI-verified claim seeds. Using SciTraj, we identify disciplinary siloing in typed citation flow and topic emergence concentrated in Vision and LLM-related work. The corpus also contains 287M typed trajectories of length

\geq 3

, covering 72.8% of papers, and supports a temporally split typed link-prediction benchmark. A year-shuffle falsifiability test separates temporal structure from year-correlated content, and a 3-annotator pilot reports

κ= 0.74

with 79.9% precision.

Abstract

Scientific Communication

Explore similar work

How Does Research Evolve? Tracing Cross-Domain Trajectories in NLP, ML, and CV with Claim-Grounded Typed Citations

HNR-DAC: Hard-Negative Reranking and Distribution-Aligned Classification for Scientific Claim Verification

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation