cs.CLOct 8, 2026
SaveLarge Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing
Organizations: Tokyo Metropolitan University · National Institute of Informatics
Abstract
Journals and conferences have begun to screen submitted manuscripts for text written using large language models (LLMs). The reliability of this screening rests on benchmark evaluations against a fixed set of LLM versions, while the versions in actual use keep changing. Here we quantify how this LLM turnover affects the screening of scientific manuscripts. We paired 4,000 pre-ChatGPT abstracts from the Proceedings of the National Academy of Sciences with their rewrites by 23 LLM versions from three vendors, released between June 2023 and August 2026. We then trained detectors under maintenance scenarios ranging from a detector retrained on every new version to one trained once and never updated. Detectors trained only on a vendor's past versions can collapse at the boundaries between model generations: calibrated to falsely flag 1% of human-written abstracts, they catch above 99% of rewrites just before the sharpest boundary and 3.8% just after it. Detectors trained on later versions can also miss rewrites of earlier ones. Vocabulary differences between versions largely track where detection transfers and where it fails. In the two screening scenarios we simulated, screens covering all 23 versions either flagged one in eight human-written abstracts or missed one in three rewrites of the newest version. Indeed, a commercial detector missed most rewrites of the version just after the sharpest boundary while flagging almost no human-written abstracts. Research-integrity policy should therefore treat the benchmark accuracy of a detector as provisional, to be re-verified with every LLM release, including earlier versions.
Figures & tables
Figure 1: Detection across the 23 LLM versions under four maintenance scenarios. Each panel shows one vendor’s versions in ascending order of release date. The vertical axis is the share of each version’s rewrites that the detector catches at the 1% false positive operating point (TPR at 1% FPR; see Methods). Detectors are trained on rewrites of the version itself (“Current version”), of the version and all earlier versions from the same vendor (“Past and current versions”), of the earlier versions alone (“Past versions only”), or of the earliest version we study from the vendor, never updated (“Frozen at GPT-4”; frozen at Llama 3.1 and Qwen2.5 in the other panels). Points are means and error bars are standard deviations over 10 random training-evaluation splits.
Figure 2: Cross-version detection and vocabulary closeness for all pairs of the 23 LLM versions. (a) Each cell shows the TPR at 1% FPR of the detector trained on the rewrites of the row version and evaluated on the rewrites of the column version, averaged over the 10 random splits. (b) Each cell shows the closeness between the rewrites of the row version and the rewrites of the column version, measured by the JSD. In both panels, each cell is colored by the percentile rank of its value among all cells of that panel. In panel (a), a rank of 100 means the highest TPR. In panel (b), a rank of 100 means the smallest JSD.
Figure 3: Two screening scenarios built from the first LLM versions in release order. (a) Union scenario. (b) Pooled scenario. For each , the blue curve shows the share of the rewrites of the last released LLM version, Muse-Glimmer, that the screen misses, and the red curve shows the share of the held-out human-written abstracts that the screen flags. Each point is the mean over the 10 random splits, and error bars show standard deviations.
Figure 4: AI-rewritten abstracts missed by the deployed commercial detector. (a) The share of AI-rewritten abstracts labeled fully human-written, for each of the 23 LLM versions, out of the same 400 papers per version. (b) The same share by the disciplinary domain of the original abstract, across the 23 rewrites of each of the 400 papers. The four domains are those assigned by OpenAlex. Error bars show 95% confidence intervals: Wilson intervals in (a), and paper-level bootstrap intervals in (b), where the 23 rewrites of one paper enter together.
| Scenario | Definition | Real-world situation being simulated |
| Current version | Trained on rewrites from the target version itself. | An idealized detector built with full access to the exact version being screened. |
| Past and current versions | Trained on rewrites from the target version and all versions released before it. | A detector retrained immediately whenever a new version appears. |
| Past versions only | Trained on rewrites from the versions released before the target version, but not the target itself. | The common situation: a new version’s text arrives for screening before any training data for that version exist. |
| Frozen at a specific version | Trained on rewrites from one designated version only and never updated (in our experiments, the earliest version we study from the vendor). | A legacy detector built once and left in service without maintenance. |
| Union | One detector per version, each trained under the current version scenario; an abstract is flagged when at least one detector fires. | An institution that stacks every available specialized detector so as not to miss any version. |
| Pooled | A single classifier trained on rewrites pooled from all 23 versions. | A generalist detector trained on a broad corpus of AI text, a design common in academic and commercial tools. |
Table 1: Detection scenarios examined in this study. Four maintenance scenarios that train one detector per target LLM version, two screening scenarios that combine versions, and one scenario in which a commercial detector is deployed.
| Version | Identifier | Date used for ordering | Generation settings |
|---|---|---|---|
| GPT-4 | gpt-4-0613 | 2023-06-13 | Standard API |
| GPT-4 Turbo | gpt-4-turbo-2024-04-09 | 2024-04-09 | Standard API |
| GPT-4o (May 2024) | gpt-4o-2024-05-13 | 2024-05-13 | Standard API |
| GPT-4o (Aug 2024) | gpt-4o-2024-08-06 | 2024-08-06 | Standard API |
| GPT-4o (Nov 2024) | gpt-4o-2024-11-20 | 2024-11-20 | Standard API |
| GPT-4.1 | gpt-4.1-2025-04-14 | 2025-04-14 | Standard API |
Table S1: The 23 model versions. “Identifier” is the date-pinned API model name, or the Hugging Face repository and the revision hash of the weights we served (verified against the repositories on September 14, 2026). “Date used for ordering” is the date by which we place the versions in release order (see Section S1). “Generation settings”: Standard API = temperature 1.0 with a 2,048-token cap via the vendor API; Reasoning API = provider defaults with a 4,096-token cap; Self-served = served with vLLM at temperature 1.0 and a 2,048-token cap.
| Version | Refusals | Truncated | Empty | Retained | Openings or closings removed |
|---|---|---|---|---|---|
| GPT-5 | 0 | 13 | 0 | 3,987 | 0 |
| Llama 3.1 | 0 | 2 | 0 | 3,998 | 222 |
| Llama 4 Maverick | 0 | 0 | 0 | 4,000 | 88 |
| Muse-Glimmer | 0 | 22 | 9 | 3,969 | 0 |
| Qwen3 | 0 | 0 | 0 | 4,000 | 35 |
| Qwen3.8 | 0 | 1 | 0 | 3,999 | 4 |
Table S2: Accounting of the requested rewrites. “Refusals”, “Truncated”, and “Empty” are the outputs discarded by the acceptance rules (47 in total; Methods), and “Retained” is the number of rewrites that remain out of the 4,000 requested per version. “Openings or closings removed” is the number of retained rewrites whose conversational opening, such as “Here is a rewritten version:”, or closing, such as a word count, was removed by the formatting rules. The table lists every version with at least one discarded output or removed opening or closing; for the other 17 versions, all 4,000 rewrites were retained unchanged.
Figure S1: Distributions of the four fidelity measures for each of the 23 LLM versions. (a) The embedding cosine similarity. (b) The share of numerals retained. (c) The share of capitalized terms retained. (d) The word-count ratio of the rewrite to the original. Each box spans the interquartile range of the pairs of that LLM version, the black line inside the box is the median, and the white diamond is the mean. The whiskers extend to the most extreme values within 1.5 times the interquartile range from the box, and values beyond the whiskers are not shown.
| Pattern | GPT-5 | Llama 4 Maverick |
|---|---|---|
| The rewrite contains more words than the original. | 21 | 7 |
| The rewrite uses the word ‘we’ less often than the original. | 7 | 28 |
| The rewrite uses the word ‘our’ more often than the original. | 8 | 21 |
| The rewrite uses the word family ‘show’, ‘shown’, and ‘showed’ less often than the original. | 14 | 15 |
| The rewrite uses the word ‘findings’ more often than the original. | 13 | 14 |
| The rewrite uses the word family ‘analyse’, ‘analyze’, and ‘analytical’ more often than the original. | 2 | 1 |
Table S3: The 14 word and punctuation items. Each entry is the number of rewrites, out of 40 per LLM version, for which the stated pattern holds in comparison with the original abstract. Words are counted without regard to letter case, the words of a family are counted together, and a pattern holds only when the count changes strictly in the stated direction.
| Pattern | Definition | Examples |
|---|---|---|
| Undefined abbreviation | The rewrite uses an abbreviation without the full name that the original introduces it with. | The original introduces “mitochondrial superoxide dismutase (MnSOD)” with its full name, whereas the rewrite uses “MnSOD” throughout without the full name. |
| Changed notation | Terms, units, symbols, letter case, or hyphenation are changed while the meaning is kept. | • The original writes “ ” in symbols, whereas the rewrite spells it out as “pi to pi*”. • The original writes “200 mA/cm2”, whereas the rewrite writes “200 mA cm -2 ”. |
| Dropped details | Specific information of the original, such as numbers, method names, species, or examples, is missing. | • The everyday illustration that the original gives in parentheses does not reappear in the rewrite. • The method name “Amino acid-coded mass tagging (AACT)” of the original does not reappear in the rewrite. |
| Added or altered content | Procedures, parameters, mechanisms, or numbers that the original does not state are added as if factual, or the object of a statement is changed. | • The original attributes a finding to Eurasians as a whole, whereas the rewrite narrows it to “Europeans and East Asians”. • The rewrite describes a titration procedure that the original does not mention. |
| Added background | Textbook-style background or general statements are added. | The rewrite expands “the hippocampus” of the original into “the hippocampus, a brain region critical for learning and memory”. |
| Added significance | Significance, outlook, or implications absent from the original are added, typically at the end. | The rewrite adds the closing clause “thereby shedding new light on the complex history of human migration and settlement”. |
Table S4: The 11 content patterns. The patterns were derived from the free-form notes of the two raters, who read rewrites by GPT-5 and Llama 4 Maverick against the original abstracts (see Section S4.2 ). All quoted fragments are verbatim from the rewrites in the sample. Quotations from the original abstracts are limited to technical terms, notation, and single words; longer passages of the originals are described without quotation.
| LLM version | OpenAI detector | RADAR | Fast-DetectGPT | Binoculars |
|---|---|---|---|---|
| GPT-4 | 0.1 | 3.1 | 0.1 | 0.1 |
| GPT-4 Turbo | 0.1 | 4.0 | 1.3 | 2.6 |
| GPT-4o (May 2024) | 0.4 | 6.5 | 4.4 | 7.2 |
| GPT-4o (Aug 2024) | 0.4 | 6.8 | 2.9 | 5.0 |
| GPT-4o (Nov 2024) | 0.2 | 2.9 | 8.5 | 9.4 |
| GPT-4.1 | 0.1 | 0.6 | 7.7 | 4.5 |
Table S5: True positive rate at a false positive rate of 1% (in %) of the four detectors without in-domain training, for each of the 23 LLM versions. Each entry is the percentage of the 3,955 rewrites of that LLM version whose score is at or above the detector’s threshold, rounded to one decimal; 0.1 marks a nonzero percentage below 0.05, and 0.0 marks that no rewrite was detected. Each detector’s threshold is set at the 99th percentile of its scores on the 5,000 calibration abstracts.
Figure S2: Association between the JSD and detection. (a) Each cell shows the TPR at 1% FPR of the detector trained on the rewrites of the row version and evaluated on the rewrites of the column version, averaged over the 10 random splits, on the color scale of the absolute value. (b) Each cell shows the JSD between the rewrites of the row version and the rewrites of the column version, on the color scale of the absolute value; the diagonal equals zero. (c) Each point is one of the 253 unordered pairs of LLM versions, with the JSD of the pair on the horizontal axis and the average of its two TPRs at 1% FPR on the vertical axis.
Figure S3: The four maintenance scenarios of Figure 1 of the main text, repeated with the word-frequency detector of Supplementary Section S7.1. Each panel shows one vendor: (a) OpenAI, (b) Meta, and (c) Alibaba. The horizontal axis lists the vendor’s LLM versions in release order, and the vertical axis shows the TPR at 1% FPR. Each line is one maintenance scenario, with the frozen detector trained on the earliest version we study from each vendor. Error bars show the standard deviation over the 10 random splits.
| Two-stage prompt | First | Second | ||
| Main text | 500 abstracts | alternative | alternative | |
| Papers retained for all 23 versions | 3,955 | 495 | 498 | 494 |
| TPR of current-version detectors, median (range) over 23 versions | 99.9 (97.4–100.0) | 97.3 (74.3–100.0) | 88.1 (54.5–99.9) | 75.9 (36.6–99.3) |
| GPT-5: current version past versions only | 100.0 26.6 | 99.0 10.2 | 93.7 5.8 | 95.1 7.9 |
| Muse-Glimmer: current version past versions only | 97.4 3.8 | 84.6 2.6 | 77.1 15.3 | 75.3 7.1 |
| Spearman between JSD and averaged TPR | ||||
Table S6: Detection under the two-stage prompt of the main text and the two alternative prompts. The first column repeats the main text, and the other three columns use the same 500 original abstracts. All TPRs are at 1% FPR and are means over the 10 random splits. For the first version after each of the two generational boundaries, the second and third rows give the TPR of the detector trained on the version itself and that of the detector trained on past versions only. The last row gives the Spearman rank correlation between the JSD and the averaged TPR over the 253 unordered pairs of versions, with for all four by the Mantel permutation test of Supplementary Section S6.
Figure S4: The four maintenance scenarios of Figure 1 of the main text, with every threshold set for a false positive rate of 5% on the calibration abstracts. Each panel shows one vendor: (a) OpenAI, (b) Meta, and (c) Alibaba. The horizontal axis lists the vendor’s LLM versions in release order, and the vertical axis shows the TPR at 5% FPR. Each line is one maintenance scenario, with the frozen detector trained on the earliest version we study from each vendor. The detectors and their scores are those of the main text, and only the threshold is moved. Error bars show the standard deviation over the 10 random splits.
Figure S5: The four maintenance scenarios of Figure 1 of the main text, with every threshold set for a false positive rate of 0.5% on the calibration abstracts. Each panel shows one vendor: (a) OpenAI, (b) Meta, and (c) Alibaba. The horizontal axis lists the vendor’s LLM versions in release order, and the vertical axis shows the TPR at 0.5% FPR. Each line is one maintenance scenario, with the frozen detector trained on the earliest version we study from each vendor. The detectors and their scores are those of the main text, and only the threshold is moved. Error bars show the standard deviation over the 10 random splits.
| LLM version | Union, per detector | Union, whole screen | Pooled |
|---|---|---|---|
| GPT-4 | 100.0 0.0 | 99.9 0.1 | 94.4 1.2 |
| GPT-4 Turbo | 100.0 0.0 | 100.0 0.0 | 100.0 0.1 |
| GPT-4o (May 2024) | 100.0 0.0 | 99.7 0.2 | 98.4 0.6 |
| Llama 3.1 | 100.0 0.1 | 99.6 0.2 | 98.4 0.4 |
| GPT-4o (Aug 2024) | 100.0 0.0 | 99.9 0.1 | 99.7 0.2 |
| Qwen2.5 | 99.9 0.1 | 97.7 0.5 | 91.2 1.2 |
Table S7: True positive rate (in %) of the three screens covering all 23 LLM versions, for each version. The union flags an abstract when at least one of the 23 version-specific detectors flags it. In the first column each detector uses its own threshold at the 99th percentile of its calibration scores, which is the setting of Figure 3 of the main text. In the second column the union is calibrated as one screen, as described above. The third column is the pooled detector trained on the rewrites of all 23 versions. The last row gives the share of the held-out human-written abstracts that each screen flags. Each entry is the mean the standard deviation over the 10 random splits.
Figure S6: Pooled detectors trained on all of the available rewrites, compared with the pooled detector of the main text. For each , the detectors are built from the first LLM versions in release order. (a) Share of the rewrites of the last released LLM version, Muse-Glimmer, that the detector misses. (b) Share of the held-out human-written abstracts that the detector flags, on a different vertical scale. Blue solid curves show detectors trained on all of the rewrites of the training papers by the first versions, and grey dashed curves show the pooled detector of the main text, trained on one rewrite per training paper (Figure 3(b) of the main text). Points are means and error bars are standard deviations over the 10 random splits.