ClaimFlow: Tracing the Evolution of Scientific Claims in NLP
Authors: Aniket Pramanick, Yufang Hou, Saif M. Mohammad, Iryna Gurevych
Organizations: Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt · IT:U Interdisciplinary Transformation University Austria · National Research Council Canada
Scientific papers advance claims that later work supports, extends, or sometimes refutes. Yet existing methods for citation and claim analysis capture only fragments of this dialogue. In this work, we make these interactions explicit at the level of individual scientific claims. We introduce ClaimFlow, a claim-centric view of the NLP literature, built from 1,617 ACL Anthology papers (1979−2025) that are manually annotated with 5,689 claims and 4,871 cross-paper claim relations, indicating whether a citing paper supports, extends, qualifies, refutes, or references a cited claim as background. Building on ClaimFlow, we define a new task -- Claim Relation Classification -- which requires models to infer the scientific stance toward a cited claim from the text and citation context. Evaluating neural models and large language models on this task, we report baseline performance of 0.81 macro-F1, suggesting that the task is tractable while leaving room for improvement. We then scale this framework to \sim$$13k NLP papers to study claim evolution across decades of NLP research. We show that 63.5% claims are never reused; only 11.1% are ever challenged. Widely propagated claims are more often reshaped through qualification and extension than supported or refuted. Overall, ClaimFlow offers a lens for examining how ideas shift and mature within NLP.
Figures & tables
Figure 1: Claims form a directed graph across papers: edges link citing and cited claims, and their relations are predicted from claims and citation context.
Relation
Share ( % )
support
20.9
extend
13.4
qualify
5.4
refute
2.1
background
58.2
Table 2: Percentages of relation types in ClaimFlow .
Relation
Cohen’s κ
support
0.80
extend
0.73
qualify
0.72
refute
0.78
background
0.82
Overall agreement
0.77
Table 3: Relation-wise inter-annotator agreement.
Setting
Model
P
R
F1
Baseline
Majority
0.18
0.20
0.18
Fine-tuned
BERT
0.51
0.50
0.50
RoBERTa
0.73
0.71
0.70
SciBERT
0.76
0.75
0.75
DeBERTa
0.77
0.76
0.76
Prompted
GPT-3.5-turbo
0.75
0.75
0.75
Table 4: Performance of different models for claim-relation classification.
Setting
P
R
F1
Gold claims + context
0.82
0.81
0.81
Automatic pipeline
0.81
0.79
0.80
Table 5: Pipeline validation results for gold-claim classification and automatic edge recovery.
Figure 2: Distribution of claim–claim relations across ClaimFlow-AutoGraph .
Figure 3: Distribution of post-challenge engagement for claims; flow widths indicate the number of claims.
Figure 4: Survival curve for time to first claim challenge; claims not challenged during the observation window are right-censored.
Figure 5: Cumulative uncertainty remains highest early in a claim’s life cycle.
Figure 6: Distribution of claim–claim relation types across major NLP venues.
Figure 7: Percentage of papers reusing claims.
Figure 8: Survival curve for time to first claim reuse.
Figure 9: Structural evolution of the claim–claim interaction graph in NLP research.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Distribution of signed degree-balance scores. Positive values correspond to integration of multiple antecedents.
Figure 11: Base prompt used for LLM-based claim relation classification.
Figure 12: Prompt template used for relation prediction when constructing ClaimFlow-AutoGraph .
Figure 13: Pipeline for constructing ClaimFlow-AutoGraph .
Model
Precision
Recall
F1
SciBERT
0.80
0.79
0.79
DeBERTa
0.83
0.82
0.82
Appendix
Table 6: Claim identification performance on the ClaimFlow test set.
Metric
Value
Quality Cluster purity ↑
0.92
Risk False merge rate ↓
0.08
Size Average cluster size
2.3
Compression Reduction in claim nodes
11%
Appendix
Table 7: Evaluation of the lightweight canonicalization procedure.
Figure 14: Correlation between original and perturbed aggregate graph statistics under increasing perturbation levels.
Figure 15: Timeline-centered claim evolution for two influential NLP papers. Each panel roots the original seed claim at the start of the publication timeline and places later interacting claims as branches at their publication years. The branch color and line style encode the ClaimFlow relation type: support, extend, qualify, or refute.
Year
Relation
Downstream claim
Interpretation
Seed claim: BERT achieves strong performance across NLP tasks without heavily engineered task-specific architectures.
2020
support
Scalar adjective semantics. “BERT encodes rich knowledge about the semantics of scalar adjectives and provides better quality intensity rankings than static embeddings and previous models.” BERT Knows Punta Cana is not just beautiful, it’s gorgeous
Gives a concrete linguistic case where bidirectional contextual representations outperform non-contextual baselines, supporting the representational part of BERT’s claim.
2020
extend
Code and natural language. “Training CodeBERT with a hybrid objective function improves its performance.” CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Extends BERT-style masked pretraining from ordinary text into paired programming-language and natural-language representations.
2021
qualify
Negation in NLU. “Current pre-trained language models like BERT fail to correctly handle negation in language understanding tasks.” Understanding by Understanding Not: Modeling Negation in Language Models
Makes the limitation explicit: high benchmark performance can coexist with brittle handling of logically decisive words such as negation.
2023
refute
Zero-shot universal discrimination. “The universal discriminator (UD) achieves state-of-the-art zero-shot results on the T0 benchmark.” A Universal Discriminator for Zero-Shot Generalization
Gives an explicit counterexample to BERT-style adaptation as the default path to broad task generalization: a universal discriminator can win in a zero-shot benchmark setting.
Appendix
Table 8: Representative downstream interactions for the BERT case study. The examples illustrate the life cycle of a high-impact model claim: broad support, transfer into new modalities and domains, and later qualifications about reasoning, structure, and zero-shot alternatives.
Year
Relation
Downstream claim
Interpretation
Seed Claim: BLEU provides an automatic MT metric that correlates highly with human evaluation.
2014
support
Metric significance testing. “A high proportion of new metrics do not significantly outperform BLEU when evaluated with the introduced significance test.” Testing for Significance of Increased Correlation with Human Judgment
Supports BLEU’s durability: even later metrics often fail to beat it significantly when correlation with human judgment is tested rigorously.
2016
extend
Community question answering. “Using machine translation evaluation measures such as TER, METEOR, and BLEU can enhance the performance of the cQA task.” Machine Translation Evaluation Meets Community Question Answering
Extends BLEU from MT evaluation into answer selection, where translation-style overlap becomes a feature for a different NLP problem.
2021
qualify
Numerical translation errors. “Existing de facto standard metrics such as BLEU may fail to adequately flag numerical translation errors.” As Easy as 1, 2, 3: Behavioral Testing of NMT Systems for Numerical Translation
Exposes a high-stakes blind spot: a single wrong number may preserve much of the n-gram overlap while changing the meaning completely.
2020
refute
Translationese references. “BLEU cannot capture human preferences because references are translationese when source sentences are natural text.” On The Evaluation of Machine Translation Systems Trained With Back-Translation
Directly refutes the proxy claim in a concrete MT setting: reference artifacts can make BLEU reward outputs that humans do not prefer.
Appendix
Table 9: Representative downstream interactions for the BLEU case study. The examples show BLEU’s transition from a durable automatic proxy to a cautionary baseline whose limits become visible in numeracy, generation, translationese, and human-preference settings.
Figure 16: Separate views of the density and modularity trends summarized in Figure 9 .
Input
Model
Macro-F1
Context ( s )
BERT
0.44
RoBERTa
0.46
SciBERT
0.51
DeBERTa
0.54
Cited + context ( hcited,s )
BERT
0.45
RoBERTa
0.50
Appendix
Table 10: Effect of different input components on claim relation classification performance.
Model
0-shot
1-shot
4-shot
GPT-3.5-Turbo
0.60
0.65
0.75
GPT-4o
0.61
0.70
0.78
GPT-4.1
0.63
0.73
0.80
LLaMA-3-70B
0.58
0.61
0.70
Mixtral-8x70B
0.60
0.65
0.73
Appendix
Table 11: LLM performance ( macro-F1 ) with different number of training examples.
Model
support
extend
qualify
refute
background
SciBERT
0.78
0.70
0.69
0.76
0.82
DeBERTa
0.80
0.73
0.71
0.78
0.83
GPT-4o (few-shot)
0.80
0.76
0.75
0.78
0.83
GPT-4.1 (few-shot)
0.82
0.78
0.77
0.80
0.85
Appendix
Table 12: Label-wise macro-F1 scores for model performance on automatic claim-relation classification.
How does research evolve, and what substrate would let us forecast where it goes next? Scientific progress is not simply a uniform accumulation of facts: ideas extend prior methods, address known limitations, realize proposed future directions, and sometimes dispute earlier claims. Existing citation graphs usually collapse these roles into a single homogeneous edge type, limiting how we can analyze scientific progress. We address this gap by proposing the SciTraj corpus, the first claim-grounded typed citation graph in which each edge is linked to the specific claim sentence that motivates it. Claim-bearing sentences are extracted from paper sections; four claim-driven relations are verified by NLI entailment against in-paper context, while two similarity-only relations are gated by abstract cosine and year-gap rules. SciTraj contains 32,559 papers from NLP, ML, and Vision (2015--2024), connected by 573,126 directed edges across six relation types, with NLI-verified claim seeds. Using SciTraj, we identify disciplinary siloing in typed citation flow and topic emergence concentrated in Vision and LLM-related work. The corpus also contains 287M typed trajectories of length ≥3, covering 72.8% of papers, and supports a temporally split typed link-prediction benchmark. A year-shuffle falsifiability test separates temporal structure from year-correlated content, and a 3-annotator pilot reports κ=0.74 with 79.9% precision.
Abdul Muntakim, Md Abdullah Al Hafiz Khan, Sadid Hasan +1
Kennesaw State University, Georgia, USA · Microsoft, Cambridge, USA
Scientific claim verification over a cited paper requires predicting the claim--paper relation and identifying the paragraphs that justify that prediction. This setting poses two linked challenges: within-paper distractors often resemble genuine evidence, while a classifier trained on gold evidence must operate on retrieved evidence at inference. We present HNR-DAC, a two-stage framework that trains each stage on the cases it will actually encounter. Hard-Negative Reranking (HNR) quantifies evidence confusability using a base reranker's scores on non-gold paragraphs and contrasts gold evidence against the most confusable candidates. Distribution-Aligned Classification (DAC) trains on the Top-1 paragraph produced by the same frozen HNR used to construct inference inputs, while HNR's Top-3 paragraph identifiers provide the evidence output. On the NLPCC 2026 Task 10 Track 2, the final configuration obtains 97.21% Hit@3, 95.79% Macro-F1, 94.47% Joint@3, and an average score of 95.13%. The corresponding submission ranks third on the official Track 2 leaderboard while achieving the highest overall Macro-F1 of 93.05%, alongside 70.16% Joint@3 and an average score of 81.61%.
Zhenchao Wang, Xin Chen, Luoxi Zhang +2
Southern University of Science and Technology, Shenzhen, China · Institute of Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Misalignment between claims and their cited evidence is a common failure mode in reports generated by large language models, limiting their reliability in scientific and other high-stakes settings. We present DeepSciVerify, a two-stage pipeline for scientific claim-citation verification that combines abstract-level reasoning with selective escalation to passage-level evidence. The system first verifies claims using the abstract and defers uncertain cases, retrieving and analyzing full-text passages only when necessary. This design leverages complementary behaviors across LLMs, as some models are more conservative while others are more decisive under uncertainty. On the SCitance benchmark, DeepSciVerify achieves 86.7 Micro-F1, outperforming strong abstract-only baselines by +4.5 points while resolving 67% of instances without full-text retrieval. These results suggest that selective evidence escalation improves both accuracy and efficiency in claim-citation verification.