Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing

Kazuki Nakajima; Takayuki Mizuno

cs.CLOct 8, 2026

Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing

Authors: Kazuki Nakajima, Takayuki Mizuno

Organizations: Tokyo Metropolitan University · National Institute of Informatics

Abstract

Journals and conferences have begun to screen submitted manuscripts for text written using large language models (LLMs). The reliability of this screening rests on benchmark evaluations against a fixed set of LLM versions, while the versions in actual use keep changing. Here we quantify how this LLM turnover affects the screening of scientific manuscripts. We paired 4,000 pre-ChatGPT abstracts from the Proceedings of the National Academy of Sciences with their rewrites by 23 LLM versions from three vendors, released between June 2023 and August 2026. We then trained detectors under maintenance scenarios ranging from a detector retrained on every new version to one trained once and never updated. Detectors trained only on a vendor's past versions can collapse at the boundaries between model generations: calibrated to falsely flag 1% of human-written abstracts, they catch above 99% of rewrites just before the sharpest boundary and 3.8% just after it. Detectors trained on later versions can also miss rewrites of earlier ones. Vocabulary differences between versions largely track where detection transfers and where it fails. In the two screening scenarios we simulated, screens covering all 23 versions either flagged one in eight human-written abstracts or missed one in three rewrites of the newest version. Indeed, a commercial detector missed most rewrites of the version just after the sharpest boundary while flagging almost no human-written abstracts. Research-integrity policy should therefore treat the benchmark accuracy of a detector as provisional, to be re-verified with every LLM release, including earlier versions.

Figures & tables

Figure 1: Detection across the 23 LLM versions under four maintenance scenarios. Each panel shows one vendor’s versions in ascending order of release date. The vertical axis is the share of each version’s rewrites that the detector catches at the 1% false positive operating point (TPR at 1% FPR; see Methods). Detectors are trained on rewrites of the version itself (“Current version”), of the version and all earlier versions from the same vendor (“Past and current versions”), of the earlier versions alone (“Past versions only”), or of the earliest version we study from the vendor, never updated (“Frozen at GPT-4”; frozen at Llama 3.1 and Qwen2.5 in the other panels). Points are means and error bars are standard deviations over 10 random training-evaluation splits.

Figure 2: Cross-version detection and vocabulary closeness for all pairs of the 23 LLM versions. (a) Each cell shows the TPR at 1% FPR of the detector trained on the rewrites of the row version and evaluated on the rewrites of the column version, averaged over the 10 random splits. (b) Each cell shows the closeness between the rewrites of the row version and the rewrites of the column version, measured by the JSD. In both panels, each cell is colored by the percentile rank of its value among all $23\times 23$ cells of that panel. In panel (a), a rank of 100 means the highest TPR. In panel (b), a rank of 100 means the smallest JSD.

Figure 3: Two screening scenarios built from the first

k

LLM versions in release order. (a) Union scenario. (b) Pooled scenario. For each

k

, the blue curve shows the share of the rewrites of the last released LLM version, Muse-Glimmer, that the screen misses, and the red curve shows the share of the held-out human-written abstracts that the screen flags. Each point is the mean over the 10 random splits, and error bars show standard deviations.

Figure 4: AI-rewritten abstracts missed by the deployed commercial detector. (a) The share of AI-rewritten abstracts labeled fully human-written, for each of the 23 LLM versions, out of the same 400 papers per version. (b) The same share by the disciplinary domain of the original abstract, across the 23 rewrites of each of the 400 papers. The four domains are those assigned by OpenAlex. Error bars show 95% confidence intervals: Wilson intervals in (a), and paper-level bootstrap intervals in (b), where the 23 rewrites of one paper enter together.

Scenario	Definition	Real-world situation being simulated
Current version	Trained on rewrites from the target version itself.	An idealized detector built with full access to the exact version being screened.
Past and current versions	Trained on rewrites from the target version and all versions released before it.	A detector retrained immediately whenever a new version appears.
Past versions only	Trained on rewrites from the versions released before the target version, but not the target itself.	The common situation: a new version’s text arrives for screening before any training data for that version exist.
Frozen at a specific version	Trained on rewrites from one designated version only and never updated (in our experiments, the earliest version we study from the vendor).	A legacy detector built once and left in service without maintenance.
Union	One detector per version, each trained under the current version scenario; an abstract is flagged when at least one detector fires.	An institution that stacks every available specialized detector so as not to miss any version.
Pooled	A single classifier trained on rewrites pooled from all 23 versions.	A generalist detector trained on a broad corpus of AI text, a design common in academic and commercial tools.

Table 1: Detection scenarios examined in this study. Four maintenance scenarios that train one detector per target LLM version, two screening scenarios that combine versions, and one scenario in which a commercial detector is deployed.

Version	Identifier	Date used for ordering	Generation settings
GPT-4	gpt-4-0613	2023-06-13	Standard API
GPT-4 Turbo	gpt-4-turbo-2024-04-09	2024-04-09	Standard API
GPT-4o (May 2024)	gpt-4o-2024-05-13	2024-05-13	Standard API
GPT-4o (Aug 2024)	gpt-4o-2024-08-06	2024-08-06	Standard API
GPT-4o (Nov 2024)	gpt-4o-2024-11-20	2024-11-20	Standard API
GPT-4.1	gpt-4.1-2025-04-14	2025-04-14	Standard API

Table S1: The 23 model versions. “Identifier” is the date-pinned API model name, or the Hugging Face repository and the revision hash of the weights we served (verified against the repositories on September 14, 2026). “Date used for ordering” is the date by which we place the versions in release order (see Section S1). “Generation settings”: Standard API = temperature 1.0 with a 2,048-token cap via the vendor API; Reasoning API = provider defaults with a 4,096-token cap; Self-served = served with vLLM at temperature 1.0 and a 2,048-token cap.

Version	Truncated	Empty	Retained	Openings or closings removed
GPT-5	13	0	3,987	0
Llama 3.1	2	0	3,998	222
Llama 4 Maverick	0	0	4,000	88
Muse-Glimmer	22	9	3,969	0
Qwen3	0	0	4,000	35
Qwen3.8	1	0	3,999	4

Table S2: Accounting of the

23\times 4{,}000=92{,}000

requested rewrites. “Refusals”, “Truncated”, and “Empty” are the outputs discarded by the acceptance rules (47 in total; Methods), and “Retained” is the number of rewrites that remain out of the 4,000 requested per version. “Openings or closings removed” is the number of retained rewrites whose conversational opening, such as “Here is a rewritten version:”, or closing, such as a word count, was removed by the formatting rules. The table lists every version with at least one discarded output or removed opening or closing; for the other 17 versions, all 4,000 rewrites were retained unchanged.

Figure S1: Distributions of the four fidelity measures for each of the 23 LLM versions. (a) The embedding cosine similarity. (b) The share of numerals retained. (c) The share of capitalized terms retained. (d) The word-count ratio of the rewrite to the original. Each box spans the interquartile range of the pairs of that LLM version, the black line inside the box is the median, and the white diamond is the mean. The whiskers extend to the most extreme values within 1.5 times the interquartile range from the box, and values beyond the whiskers are not shown.

Pattern	GPT-5	Llama 4 Maverick
The rewrite contains more words than the original.	21	7
The rewrite uses the word ‘we’ less often than the original.	7	28
The rewrite uses the word ‘our’ more often than the original.	8	21
The rewrite uses the word family ‘show’, ‘shown’, and ‘showed’ less often than the original.	14	15
The rewrite uses the word ‘findings’ more often than the original.	13	14
The rewrite uses the word family ‘analyse’, ‘analyze’, and ‘analytical’ more often than the original.	2	1

Table S3: The 14 word and punctuation items. Each entry is the number of rewrites, out of 40 per LLM version, for which the stated pattern holds in comparison with the original abstract. Words are counted without regard to letter case, the words of a family are counted together, and a pattern holds only when the count changes strictly in the stated direction.

Pattern	Definition	Examples
Undefined abbreviation	The rewrite uses an abbreviation without the full name that the original introduces it with.	The original introduces “mitochondrial superoxide dismutase (MnSOD)” with its full name, whereas the rewrite uses “MnSOD” throughout without the full name.
Changed notation	Terms, units, symbols, letter case, or hyphenation are changed while the meaning is kept.	• The original writes “ $\pi\to\pi^{}$ ” in symbols, whereas the rewrite spells it out as “pi to pi”. • The original writes “200 mA/cm2”, whereas the rewrite writes “200 mA cm -2 ”.
Dropped details	Specific information of the original, such as numbers, method names, species, or examples, is missing.	• The everyday illustration that the original gives in parentheses does not reappear in the rewrite. • The method name “Amino acid-coded mass tagging (AACT)” of the original does not reappear in the rewrite.
Added or altered content	Procedures, parameters, mechanisms, or numbers that the original does not state are added as if factual, or the object of a statement is changed.	• The original attributes a finding to Eurasians as a whole, whereas the rewrite narrows it to “Europeans and East Asians”. • The rewrite describes a titration procedure that the original does not mention.
Added background	Textbook-style background or general statements are added.	The rewrite expands “the hippocampus” of the original into “the hippocampus, a brain region critical for learning and memory”.
Added significance	Significance, outlook, or implications absent from the original are added, typically at the end.	The rewrite adds the closing clause “thereby shedding new light on the complex history of human migration and settlement”.

Table S4: The 11 content patterns. The patterns were derived from the free-form notes of the two raters, who read rewrites by GPT-5 and Llama 4 Maverick against the original abstracts (see Section S4.2 ). All quoted fragments are verbatim from the rewrites in the sample. Quotations from the original abstracts are limited to technical terms, notation, and single words; longer passages of the originals are described without quotation.

LLM version	OpenAI detector	RADAR	Fast-DetectGPT	Binoculars
GPT-4	$<$ 0.1	3.1	0.1	0.1
GPT-4 Turbo	0.1	4.0	1.3	2.6
GPT-4o (May 2024)	0.4	6.5	4.4	7.2
GPT-4o (Aug 2024)	0.4	6.8	2.9	5.0
GPT-4o (Nov 2024)	0.2	2.9	8.5	9.4
GPT-4.1	0.1	0.6	7.7	4.5

Table S5: True positive rate at a false positive rate of 1% (in %) of the four detectors without in-domain training, for each of the 23 LLM versions. Each entry is the percentage of the 3,955 rewrites of that LLM version whose score is at or above the detector’s threshold, rounded to one decimal;

<

0.1 marks a nonzero percentage below 0.05, and 0.0 marks that no rewrite was detected. Each detector’s threshold is set at the 99th percentile of its scores on the 5,000 calibration abstracts.

Figure S2: Association between the JSD and detection. (a) Each cell shows the TPR at 1% FPR of the detector trained on the rewrites of the row version and evaluated on the rewrites of the column version, averaged over the 10 random splits, on the color scale of the absolute value. (b) Each cell shows the JSD between the rewrites of the row version and the rewrites of the column version, on the color scale of the absolute value; the diagonal equals zero. (c) Each point is one of the 253 unordered pairs of LLM versions, with the JSD of the pair on the horizontal axis and the average of its two TPRs at 1% FPR on the vertical axis.

Figure S3: The four maintenance scenarios of Figure 1 of the main text, repeated with the word-frequency detector of Supplementary Section S7.1. Each panel shows one vendor: (a) OpenAI, (b) Meta, and (c) Alibaba. The horizontal axis lists the vendor’s LLM versions in release order, and the vertical axis shows the TPR at 1% FPR. Each line is one maintenance scenario, with the frozen detector trained on the earliest version we study from each vendor. Error bars show the standard deviation over the 10 random splits.

	Two-stage prompt		First	Second
	Main text	500 abstracts	alternative	alternative
Papers retained for all 23 versions	3,955	495	498	494
TPR of current-version detectors, median (range) over 23 versions	99.9 (97.4–100.0)	97.3 (74.3–100.0)	88.1 (54.5–99.9)	75.9 (36.6–99.3)
GPT-5: current version $\to$ past versions only	100.0 $\to$ 26.6	99.0 $\to$ 10.2	93.7 $\to$ 5.8	95.1 $\to$ 7.9
Muse-Glimmer: current version $\to$ past versions only	97.4 $\to$ 3.8	84.6 $\to$ 2.6	77.1 $\to$ 15.3	75.3 $\to$ 7.1
Spearman $\rho$ between JSD and averaged TPR	$-0.85$	$-0.81$	$-0.66$	$-0.73$

Table S6: Detection under the two-stage prompt of the main text and the two alternative prompts. The first column repeats the main text, and the other three columns use the same 500 original abstracts. All TPRs are at 1% FPR and are means over the 10 random splits. For the first version after each of the two generational boundaries, the second and third rows give the TPR of the detector trained on the version itself and that of the detector trained on past versions only. The last row gives the Spearman rank correlation between the JSD and the averaged TPR over the 253 unordered pairs of versions, with

p<0.001

for all four by the Mantel permutation test of Supplementary Section S6.

Figure S4: The four maintenance scenarios of Figure 1 of the main text, with every threshold set for a false positive rate of 5% on the calibration abstracts. Each panel shows one vendor: (a) OpenAI, (b) Meta, and (c) Alibaba. The horizontal axis lists the vendor’s LLM versions in release order, and the vertical axis shows the TPR at 5% FPR. Each line is one maintenance scenario, with the frozen detector trained on the earliest version we study from each vendor. The detectors and their scores are those of the main text, and only the threshold is moved. Error bars show the standard deviation over the 10 random splits.

Figure S5: The four maintenance scenarios of Figure 1 of the main text, with every threshold set for a false positive rate of 0.5% on the calibration abstracts. Each panel shows one vendor: (a) OpenAI, (b) Meta, and (c) Alibaba. The horizontal axis lists the vendor’s LLM versions in release order, and the vertical axis shows the TPR at 0.5% FPR. Each line is one maintenance scenario, with the frozen detector trained on the earliest version we study from each vendor. The detectors and their scores are those of the main text, and only the threshold is moved. Error bars show the standard deviation over the 10 random splits.

LLM version	Union, per detector	Union, whole screen	Pooled
GPT-4	100.0 $\pm$ 0.0	99.9 $\pm$ 0.1	94.4 $\pm$ 1.2
GPT-4 Turbo	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.1
GPT-4o (May 2024)	100.0 $\pm$ 0.0	99.7 $\pm$ 0.2	98.4 $\pm$ 0.6
Llama 3.1	100.0 $\pm$ 0.1	99.6 $\pm$ 0.2	98.4 $\pm$ 0.4
GPT-4o (Aug 2024)	100.0 $\pm$ 0.0	99.9 $\pm$ 0.1	99.7 $\pm$ 0.2
Qwen2.5	99.9 $\pm$ 0.1	97.7 $\pm$ 0.5	91.2 $\pm$ 1.2

Table S7: True positive rate (in %) of the three screens covering all 23 LLM versions, for each version. The union flags an abstract when at least one of the 23 version-specific detectors flags it. In the first column each detector uses its own threshold at the 99th percentile of its calibration scores, which is the setting of Figure 3 of the main text. In the second column the union is calibrated as one screen, as described above. The third column is the pooled detector trained on the rewrites of all 23 versions. The last row gives the share of the held-out human-written abstracts that each screen flags. Each entry is the mean

\pm

the standard deviation over the 10 random splits.

Figure S6: Pooled detectors trained on all of the available rewrites, compared with the pooled detector of the main text. For each

k

, the detectors are built from the first

k

LLM versions in release order. (a) Share of the rewrites of the last released LLM version, Muse-Glimmer, that the detector misses. (b) Share of the held-out human-written abstracts that the detector flags, on a different vertical scale. Blue solid curves show detectors trained on all of the rewrites of the training papers by the first

k

versions, and grey dashed curves show the pooled detector of the main text, trained on one rewrite per training paper (Figure 3(b) of the main text). Points are means and error bars are standard deviations over the 10 random splits.

Explore similar work

Mar 20, 2026cs.CL

Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable

A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers, except for polishing, paraphrasing, and grammar correction of otherwise human-written reviews. But, are these policies enforceable? To answer this question, we assemble a dataset of peer reviews simulating multiple levels of human-AI collaboration, and evaluate five state-of-the-art detectors, including two commercial systems. Our analysis shows that all detectors misclassify a non-trivial fraction of LLM-polished reviews as AI-generated, thereby risking false accusations of academic misconduct. We further investigate whether peer-review-specific signals, including access to the paper manuscript and the constrained domain of scientific writing, can be leveraged to improve detection. While incorporating such signals yields measurable gains in some settings, we identify limitations in each approach and find that none meets the accuracy standards required for identifying AI use in peer reviews. Importantly, our results suggest that recent public estimates of AI use in peer reviews through the use of AI-text detectors should be interpreted with caution, as current detectors misclassify mixed reviews (collaborative human-AI outputs) as fully AI generated, potentially overstating the extent of policy violations.

Rounak Saha, Gurusha Juneja, Dayita Chaudhuri +3

♠Indian Institute of Science · ♢University of California, Santa Barbara · ♡Carnegie Mellon University

LLM Evaluation AI-Generated Text Detection Scholarly Peer Review

Ask PDF

May 25, 2026cs.CL

LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers

Large language models (LLMs) are increasingly used in academic peer review, yet their reliability, alignment with human judgment, and robustness to adversarial attacks remain poorly understood. We present a systematic benchmark of LLM-as-a-Reviewer on 898 papers stratified from NeurIPS and ICLR, evaluating 12 LLMs along three axes: rating calibration, divergence from human reviewers, and resistance to prompt injection embedded via an invisible font-mapping attack. We find that LLMs systematically overrate weaker submissions and diverge from humans in topical emphasis, under-flagging Clarity and over-flagging Reproducibility, while producing reviews two to three times longer with lower lexical diversity and a more standardized vocabulary. Prompt injection remains highly effective. Simple hidden instructions can promote low-scoring papers to acceptance-level ratings in a substantial fraction of cases, with effectiveness varying sharply across model families. While LLMs offer utility in structuring evaluations, their integration into peer review requires safeguards against both intrinsic biases and adversarial risks.

Lingyao Li, Junjie Xiong, Changjia Zhu +5

University of South Florida · Missouri University of Science and Technology · University of Alabama +3

LLM Evaluation LLM-as-a-Judge Adversarial Attacks on LLMs LLM Reliability Language Model Bias Evaluation Automated Peer Review

Ask PDF

Jul 31, 2026cs.CL

ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation

Standard AI-text detection benchmarks compare human-written text against text generated directly by large language models (LLMs). While prior work has shown that rewriting and paraphrasing can degrade detector performance, it remains unclear whether performance measured on this conventional benchmark predicts detector behavior when human-authored content is rewritten by an LLM. To address this gap, we introduce Authorship-Rewriting Benchmark (ARB), built from 1,800 human source texts (600 each from XSum, WritingPrompts, and OpenWebText) and four open-weight generators (Llama-3.2-3B, Qwen2.5-7B, Mistral-7B, Gemma-2-9B). Each source item yields four matched variants: human-written (HUMAN), direct LLM generation (Free-LLM), LLM-rewritten human text (H2L), and same-generator LLM-rewritten LLM text (LLM2L). We evaluated five detectors (FastDetectGPT, Binoculars-falcon-7b, RADAR, BERT-Defense, RoBERTa-Defense) at a strict 1%-false-positive operating point (TPR@1%FPR). FastDetectGPT and Binoculars-falcon-7b detected 91.2% and 93.5% of direct LLM text, but only 30.8% and 15.1% of human text an LLM had rewritten, a drop of 60-78 percentage points. The same detectors retained 78.3% and 83.0% recall when LLM text was rewritten by the same model, a much smaller decline of 10-13 points. RADAR followed the same pattern (66.8% to 12.2%), while BERT-Defense and RoBERTa-Defense stayed below 3% recall across all regimes. These results show that detector performance measured on the conventional human-vs-LLM benchmark does not transfer to human-authored text revised by an LLM, even though the same detectors remain largely robust to LLM-only rewriting.

Gaetano Perrone, Simon Pietro Romano

Department of Electrical Engineering and Information Technology, University of Napoli Federico II, Via Claudio 21, Naples, 80125, Italy

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Ask PDF