Drift Inspector: Exploring and Measuring Scientific Drift with Atomic Contribution Claims

Vsevolod Karimov; Stepan Ostarkov; Anastasia Poroshina; Anatoly Frolov; Alexander Panchenko

cs.CLSep 30, 2026

Drift Inspector: Exploring and Measuring Scientific Drift with Atomic Contribution Claims

Authors: Vsevolod Karimov, Stepan Ostarkov, Anastasia Poroshina, Anatoly Frolov, Alexander Panchenko

Organizations: Skoltech · HSE University · Lomonosov Moscow State University · ITMO University ·

Abstract

Scientific abstracts mix contributions with background, motivation, and meta-language, so tools that read them as-is cannot separate what a field produces from what it discusses. We present Drift Inspector, an open-source system for measuring and exploring how a research field changes over time at the level of Atomic Contribution Claims (ACCs): decontextualized, contribution-bearing propositions an LLM extracts from each abstract before analysis. The system clusters these claims across years into an interactive map where every trend traces back to the claims and papers behind it. Applied to six years of EMNLP, it shows the field shifting away from classic NLP tasks toward LLM-era capabilities such as reasoning and multimodality -- a movement that keyword or whole-abstract counts blur. The released data extend beyond EMNLP: the same pipeline has processed the full ACL Anthology (346k claims, 80k abstracts, 423 venues). Extraction is human-validated and clustering checked against an external manually constructed taxonomy.

Figures & tables

Figure 1: Drift Inspector frontend in Map view representing an interactive WebGL map of 16,576 atomic claims from EMNLP 2020–2025. Top bar tabs allow to switch between views (Map, Trends, Clusters, Compare or Methods). A user can adjust year filter, color mode (Cluster or Drift Trend), or perform a claim search with a per-cluster match ranking. On the right are filterable list of clusters. Hovering a point shows its claim and the cluster’s prevalence trajectory; clicking pins the card with a link to the source paper.

Figure 2: System workflow. An LLM maps each abstract to Atomic Contribution Claims; claims are embedded with SPECTER2 and clustered jointly across years; per-cluster document-frequency trajectories and the interactive map are generated from the same canonical clustering.

Figure 3: Compare view, venue cohorts (live six-venue instance): where ACL and COLING differ most, as the gap in cluster claim shares. COLING keeps a markedly stronger classic-NLP profile (embeddings, aspect-based sentiment, NER, event extraction), while ACL leans toward LLM-era clusters (efficient reasoning, multilingual LLMs, vision–language evaluation). Chips switch the diverging / dumbbell / scatter renderings; the same view compares authors and keyword-defined cohorts.

	EMNLP ’20–’25	ACL Anthology
Abstracts	4,960	80,144
Claims	18,293	346,010
Extractor	Qwen3-235B-T	gpt-oss-120b
Extraction cost	$54	$\sim$ $40
per 1k abstracts	$10.87	$0.51
Extraction wall-clock	1.9 h	6.8 h

Table 1: Cost of the two released extractions. The EMNLP figure is the amount actually invoiced; the anthology pass ran on donated compute, so its cost is the equivalent at 2026 commercial rates. ∗ measured on the deployed instance: the 16,576-claim EMNLP corpus and the 70k-claim six-venue instance built from the anthology extraction.

Figure 4: Scientific drift in EMNLP (2020–2025) as measured by the system. Diverging bars show the change in paper-level document frequency (percentage points) for the top emerging and declining ACC clusters, annotated with 2020 and 2025 shares.

Method	Pur.	LRAP	nDCG@1	V-mes	PairF1
LDA	0.245	0.508	0.329	0.278	0.170
NMF	0.294	0.585	0.445	0.324	0.186
BERTopic SPECTER2	0.324	0.651	0.515	0.469	0.289
BERTopic MPNet	0.327	0.651	0.509	0.457	0.234
SentSPECTER	0.654	0.668	0.540	0.532	0.268
SentSPECTER top30	0.472	0.637	0.512	0.471	0.365

Table 2: Alignment with the SToP taxonomy (2020–2021 subset, 653 papers). Purity and V-measure are computed on covered documents; LRAP and nDCG@1 via 5

\times

5 repeated cross-validation, the topic-to-label map learned per fold on the training split. The step from abstract-level (BERTopic SPECTER2 , 0.324) to sentence-level (0.654) to claim-level (0.689) purity shows input granularity matters more than encoder choice. Full table with intrinsic metrics and coverage in the released artifacts.

Figure 5: Compare view, over time mode: yearly topic mix (top clusters, % of cohort claims) for the bert cohort (197 papers) vs. llm / large language model (3,253 papers): the residual bert agenda concentrates in claim verification and biomedical NLP, while the llm cohort shifts toward vision–language modeling and efficiency.

Figure 6: Per-year document-frequency trajectories for the eight most declining and eight most growing ACC clusters (paper-level DF, % of papers), computed from the canonical clustering. Labels are curated short cluster names; full names and c-TF-IDF descriptors ship with the released data. Headline endpoint shifts are significant with bootstrap CIs excluding zero (§ 4.3 ).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder	$k$	Noise%	ARI	SToP	Sign
SPECTER2 (ours)	80	36.1	1.00	0.689	15/15
SciNCL	111	31.2	0.75	0.670	15/15
SPECTER1	85	34.1	0.69	0.658	14/15
MPNet	116	32.6	0.61	0.681	15/15
MiniLM	107	37.1	0.64	0.646	14/15
BGE-large	99	37.9	0.67	0.674	13/15

Appendix

Table 3: Encoder ablation on the 16,576 claims.

k

: clusters, noise excluded. Sign : headline shift signs preserved of 15. TF-IDF + SVD is a non-neural control, scored on sign preservation only.

Figure 7: Single-conference diachronic view (Compare, over time , bump chart): EMNLP’s top clusters ranked by yearly document frequency (1 = most common). Vision–language modeling reaches rank 1 by 2024; LLM efficiency climbs from 14 to 3–5; summarization falls from 7 to 31.

Semantic Change Detection

Semantic change detection identifies how meanings, concepts, or scene contents differ across time in text, images, and other data.

See all 63 papers

ClaimFlow: Tracing the Evolution of Scientific Claims in NLP
Mar 17, 2026Aniket Pramanick, Yufang Hou, Saif M. Mohammad +1Scientific Communication Scientific Claim Verification
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt · IT:U Interdisciplinary Transformation University Austria · National Research Council Canada
How Does Research Evolve? Tracing Cross-Domain Trajectories in NLP, ML, and CV with Claim-Grounded Typed Citations
Jun 21, 2026Abdul Muntakim, Md Abdullah Al Hafiz Khan, Sadid Hasan +1KG Construction Citation Verification
Kennesaw State University, Georgia, USA · Microsoft, Cambridge, USA
Accelerated Drift of Scientific Prose in the LLM Era: Evidence from an Interrupted Time-Series Analysis
Oct 19, 2026Hanna Hubarava, Benjamin V. IneichenScientific Communication AI-Assisted Scientific Research
Department of Clinical Research Department of Digital Medicine University of Bern, Switzerland

Mar 17, 2026J/K move · Enter open · S save

ClaimFlow: Tracing the Evolution of Scientific Claims in NLP

Aniket Pramanick, Yufang Hou, Saif M. Mohammad, Iryna Gurevych

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technische Universität Darmstadt · IT:U Interdisciplinary Transformation University Austria · National Research Council Canada

Scientific Communication Scientific Claim Verification

Open paper AskPDF arXiv

Scientific papers advance

\textit{claims}

that later work supports, extends, or sometimes refutes. Yet existing methods for citation and claim analysis capture only fragments of this dialogue. In this work, we make these interactions explicit at the level of individual scientific claims. We introduce

\texttt{ClaimFlow}

, a claim-centric view of the NLP literature, built from

1{,}617

ACL Anthology papers

(1979 - 2025)

that are manually annotated with

5{,}689

claims and

4{,}871

cross-paper claim relations, indicating whether a citing paper

\texttt{supports}

\texttt{extends}

\texttt{qualifies}

\texttt{refutes}

, or references a cited claim as

\texttt{background}

. Building on

\texttt{ClaimFlow}

, we define a new task --

\textit{Claim Relation Classification}

-- which requires models to infer the scientific stance toward a cited claim from the text and citation context. Evaluating neural models and large language models on this task, we report baseline performance of

0.81

macro-F1, suggesting that the task is tractable while leaving room for improvement. We then scale this framework to

\sim$$13k

NLP papers to study claim evolution across decades of NLP research. We show that

63.5\%

claims are never reused; only

11.1\%

are ever challenged. Widely propagated claims are more often

\textit{reshaped}

through qualification and extension than supported or refuted. Overall,

\texttt{ClaimFlow}

offers a lens for examining how ideas shift and mature within NLP.

Abstract

Semantic Change Detection

Explore similar work

ClaimFlow: Tracing the Evolution of Scientific Claims in NLP

How Does Research Evolve? Tracing Cross-Domain Trajectories in NLP, ML, and CV with Claim-Grounded Typed Citations

Accelerated Drift of Scientific Prose in the LLM Era: Evidence from an Interrupted Time-Series Analysis