Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers

Mingmeng Geng; Yuhang Dong; Thierry Poibeau

cs.CLMar 26, 2026

Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers

Authors: Mingmeng Geng, Yuhang Dong, Thierry Poibeau

Organizations: École Normale Supérieure (ENS) – Université Paris Sciences et Lettres (PSL) · Laboratoire Lattice · Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)

Abstract

Through an analysis of arXiv papers, we report several shifts in word usage that are likely driven by large language models (LLMs) but have not previously received sufficient attention, such as the increased frequency of "beyond" and "via" in titles and the decreased frequency of "the" and "of" in abstracts. Due to the similarities among different LLMs, experiments show that current classifiers struggle to accurately determine which specific model generated a given text in multi-class classification tasks. Meanwhile, variations across LLMs also result in evolving patterns of word usage in academic papers. By adopting a direct and highly interpretable linear approach and accounting for differences between models and prompts, we quantitatively assess these effects and show that real-world LLM usage is heterogeneous and dynamic.

Figures & tables

Panel of Figure 1 — Figure 1: Top-left & Middle-left : Word frequency comparison for generated titles or rewritten abstracts based on 2,000 real arXiv abstracts; error bars denote variance across models and prompts. Remaining panels : Temporal trends of word frequencies in real arXiv data, with the yellow dashed line fitted on data from 2015 to 2021 and extended to early 2026.

Model Name	Release Date
GPT-3.5 ( OpenAI, 2022 )	Dec 30, 2022
GPT-4o mini ( OpenAI, 2024 )	Jul 18, 2024
GPT-5 nano ( OpenAI, 2025 )	Aug 07, 2025
DeepSeek V3 ( DeepSeek-AI, 2024 )	Dec 26, 2024
DeepSeek R1 ( DeepSeek-AI, 2025a )	Jan 20, 2025
DeepSeek V3.2 ( DeepSeek-AI, 2025b )	Dec 01, 2025

Table 1: Release timeline of the LLMs used in this paper. Dates indicate the initial public availability, verified by official announcements.

Panel of Figure 2 — Figure 2: Examples of word frequencies in paper titles.

Figure 3: Frequency change ratios of the 20 most frequent words in 2,000 abstracts using multiple LLMs and the shorter prompt.

Figure 4: Word frequencies in 2,000 abstracts or in the corresponding LLM-processed content. Error bars denote the standard deviation of word frequencies across outputs produced by different LLMs and/or prompts.

Panel of Figure 5 — Figure 5: The frequencies of some words that appeared at least 20 times in the 2,000 abstracts used for the simulation.

Panel of Figure 6 — Figure 6: Comparison Results of Text Similarity. The x-axis represents the model release dates.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Acc (%)	78.6	90.1	90.1	95.3	76.9	93.4	91.8

Table 2: Classification results. Prompt: long, Classifier: BERT, Model Type: old. The black squares indicate the source of the detected text, and the accuracy shown is the highest value obtained across the parameter set.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Acc (%)	63.0	83.0	90.4	84.9	83.8	70.9	88.2

Table 3: Classification results. Prompt: long, Classifier: BERT, Model Type: new.

Panel of Figure 7 — Figure 7: Confusion matrix of classification results from the detectors based on GPT-2 and LLM2Vec. In the training and test data, GPT-3.5, GPT-4o mini, and GPT-5-nano are mixtures of two prompts, while DeepSeek, Gemini, and Claude are mixtures of multiple versions and two prompts. After mixing, each class contains 2,000 abstracts.

Panel of Figure 8 — Figure 8: Estimation of LLM impact in arXiv abstracts.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Supplementary comparison results.

Panel of Figure 10 — Figure 10: Supplementary comparison results.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	78.2%	87.1%	89.0%	94.8 %	71.7%	90.1%	89.8%

Appendix

Table 4: Classification results. Prompt: long, Classifier: GPT-2, Model Type: old.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	65.2%	84.6%	90.1 %	81.9%	77.7%	62.9%	82.7%

Appendix

Table 5: Classification results. Prompt: long, Classifier: GPT-2, Model Type: new.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	72.8%	90.7%	91.2%	94.8 %	73.4%	91.2%	91.8%

Appendix

Table 6: Classification results. Prompt: long, Classifier: T5, Model Type: old.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	57.3%	88.5%	93.1 %	84.1%	83.8%	58.5%	82.4%

Appendix

Table 7: Classification results. Prompt: long, Classifier: T5, Model Type: new.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	73.1%	82.1%	89.3%	91.5%	71.2%	89.3%	93.4 %

Appendix

Table 8: Classification results. Prompt: short, Classifier: BERT, Model Type: old.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	65.5%	80.8%	89.8 %	83.8%	87.6%	76.6%	86.8%

Appendix

Table 9: Classification results. Prompt: short, Classifier: BERT, Model Type: new.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	69.0%	80.2%	88.2%	89.6%	73.1%	85.7%	90.9 %

Appendix

Table 10: Classification results. Prompt: short, Classifier: GPT-2, Model Type: old.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	58.2%	76.6%	89.6 %	83.0%	80.8%	70.6%	83.0%

Appendix

Table 11: Classification results. Prompt: short, Classifier: GPT-2, Model Type: new.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	70.9%	83.5%	91.2%	92.9 %	69.8%	86.3%	92.9 %

Appendix

Table 12: Classification results. Prompt: short, Classifier: T5, Model Type: old.

GPT	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\cdot$
DeepSeek	$\blacksquare$	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\blacksquare$	$\cdot$
Gemini	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$
Claude	$\blacksquare$	$\cdot$	$\cdot$	$\blacksquare$	$\cdot$	$\blacksquare$	$\blacksquare$
Accuracy	63.5%	82.7%	90.4 %	85.4%	83.2%	64.6%	84.3%

Appendix

Table 13: Classification results. Prompt: short, Classifier: T5, Model Type: new.

Model Type	Prompt	Classifier	All	G + D	G + Ge	G + C	D + Ge	D + C	Ge + C
Model Type	Prompt	Classifier	(G+D+Ge+C)
New	Long	BERT	63.0	83.0	90.4	84.9	83.8	70.9	88.2
		GPT-2	65.2	84.6	90.1	81.9	77.7	62.9	82.7
		T5	57.3	88.5	93.1	84.1	83.8	58.5	82.4
	Short	BERT	65.5	80.8	89.8	83.8	87.6	76.6	86.8
		GPT-2	58.2	76.6	89.6	83.0	80.8	70.6	83.0

Appendix

Table 14: Summary of Binary Classification Accuracy (%) across different Model Types, Prompts, and Classifiers. Abbreviations: G: GPT, D: DeepSeek, Ge: Gemini, C: Claude. Columns represent the ensemble combination.

Panel of Figure 11 — Figure 11: Comparison Results of Text Similarity. The x-axis represents the model release dates.

Class	BERT			GPT-2			T5			LLM2Vec
Class	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Real abstract	82.24	68.68	74.85	74.21	77.47	75.81	85.06	71.98	77.98	86.93	84.07	85.47
GPT 3.5	74.07	76.92	75.47	66.50	71.98	69.13	79.62	68.68	73.75	86.41	87.36	86.89
GPT 4o mini	64.71	72.53	68.39	58.29	67.58	62.60	60.68	78.02	68.27	83.05	80.77	81.89
GPT 5 nano	68.45	63.19	65.71	69.14	66.48	67.79	67.50	74.18	70.68	84.24	76.37	80.12
DeepSeek	36.18	30.22	32.93	31.03	19.78	24.16	38.71	26.37	31.37	49.45	49.45	49.45

Appendix

Table 15: Comparison of 7-class classification performance across BERT, GPT-2, T5, and LLM2Vec models. Metrics are reported in percentage (%). P denotes Precision, R denotes Recall.

Panel of Figure 12 — Figure 12: Confusion matrix of classification results from the GPT-2-based detector and T5-based detector.

Figure 13: Confusion matrix of classification results from the LLM2Vec-based detector.

Figure 14: Confusion matrix of BERT (lr=1e-04)

Figure 15: Confusion matrix of GPT-2 (lr=3e-04)

Figure 16: Confusion matrix of T5 (lr=3e-04)

Figure 17: Confusion matrix of LLM2Vec (lr=5e-05)

Panel of Figure 18 — Figure 18: Supplementary results for the estimation of LLM impact in arXiv abstracts.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 170 papers

Most biomedical publications show signs of LLM-assisted writing
Aug 11, 2026Lena Holzwarth, Rita González-Márquez, Dmitry KobakAI-Generated Text Detection LLM Auditing
1Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Department of Mathematics, Computer Science, and Statistics, Ghent University, Belgium · 3VIB Center for AI and Computational Biology, VIB, Ghent, Belgium
Authority Bias in Conversational Search Engines for Academic Paper Recommendation
Aug 31, 2026Uthman Jinadu, Parsa Ghazvinian, Anjila Budathoki +3Language Model Bias Evaluation Large Language Model-Based Recommendation
Georgia State University · University of Tennessee, Knoxville
Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods
Jun 15, 2026Francesca Carlon, Brecht Verbeken, Vincent Ginis +1LLM Evaluation AI-Assisted Scientific Research
1Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussels, Belgium · 2imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussels, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA

Aug 11, 2026J/K move · Enter open · S save

Most biomedical publications show signs of LLM-assisted writing

Lena Holzwarth, Rita González-Márquez, Dmitry Kobak

1Hertie Institute for AI in Brain Health, University of Tübingen, Germany · Department of Mathematics, Computer Science, and Statistics, Ghent University, Belgium · 3VIB Center for AI and Computational Biology, VIB, Ghent, Belgium

AI-Generated Text Detection LLM Auditing

Open paper AskPDF arXiv

Over the past several years, LLM-powered chatbots and agents have become widely used as a tool for academic writing. LLM-assisted writing can be valuable by removing language barriers but at the same time causes concerns about misconduct and fraud. To inform policy decisions, it is necessary to monitor the prevalence of LLM-altered texts in scholarly publications. Despite some recent progress in this direction, no existing method can produce reliable estimates. Here we suggest and validate a new unbiased approach to estimate LLM usage in a corpus of texts based on changing word frequencies. We apply our method to the full texts of open-access biomedical papers from Pubmed Central, and show that by the end of 2025, 89% of papers show excess of LLM-associated vocabulary. We also find that LLMs are twice as likely to be used when writing a paragraph in the Discussion section (68%) compared to a paragraph in the Methods section (32%), but even inside the Methods section, the overall prevalence of LLM usage is over 50%. We believe that our estimates are crucial to shape future guidelines and policies.

Abstract

AI-Generated Text Detection

Explore similar work

Most biomedical publications show signs of LLM-assisted writing

Authority Bias in Conversational Search Engines for Academic Paper Recommendation

Thinking Like a Scientist? A Structural Study of LLM-Generated Research Methods