Measuring Artifact Reliance in Machine-Generated Text Detectors

Anastasia Voznyuk; Andrey Grabovoy

cs.CLOct 9, 2026

Measuring Artifact Reliance in Machine-Generated Text Detectors

Authors: Anastasia Voznyuk, Andrey Grabovoy

Organizations: Advacheck OÜ Tallinn, Estonia · Moscow State University Moscow, Russia

Abstract

Detectors of machine-generated text score almost perfectly on scientific-literature benchmarks, implying we can reliably distinguish with these detectors human and machine-generated documents. We looked at ten scientific-document subsets from six popular resources, and on several of them that apparent ease is a side effect of how the documents were put together. A logistic regression using 27 superficial features reached 0.833-0.997 macro-F1, and on the three arXiv subsets it outperformed every detector we evaluate with F1-scores 0.951-0.997. The strongest cue to the detector is the PDF extraction artifact: human texts retain their original line breaks, whereas generated texts are extracted as a single block of prose. This cue, together with three other formatting features, tell the classes apart at 0.996 AUC. To distinguish benchmarks that contain such a signal from detectors that rely on one, we introduce format counterfactuals: just re-wrapping a machine-written abstract without changing a single word reverses 20-27% of a public detector's correct decisions. Normalizing formatting before training reduces the generalization gap from 0.32 to 0.22. On format-clean benchmarks, it also raises out-of-domain macro-F1 from 0.33 to 0.65 for detectors trained on artifact-containing subsets.

Figures & tables

Subset	$n$	Raw	Norm.	$\Delta$	fmt-only	len-only	TF-IDF
SemEval24 arXiv	20,000	0.997	0.729	0.268	0.997	0.503	0.880
M4 arXiv	20,000	0.995	0.746	0.249	0.996	0.635	0.917
COLING25 arXiv	20,000	0.951	0.752	0.199	0.943	0.613	0.916
M4 PeerRead	5,860	0.959	0.954	0.005	0.822	0.780	0.998
SemEval24 PeerRead	4,714	0.941	0.914	0.027	0.709	0.541	0.947
COLING25 PeerRead	6,600	0.854	0.839	0.015	0.708	0.488	0.967

Table 1: Surface probe. Raw and Norm. are macro-

F_{1}

before and after format normalization. fmt-only and len-only are AUCs of the four formatting features, and of document length alone. TF-IDF is a lexical reference on the same folds. The three arXiv subsets lose

0.20

0.27

F_{1}

to normalization; the other lose nothing.

Dataset	DeBERTa	Perpl.	Perturb.
CHEAT	0.892	0.901	0.908
COLING25 (arXiv)	0.824	0.733	0.868
COLING25 (PeerRead)	0.680	0.804	0.946
GPABench2 (CS)	0.781	0.834	0.984
GPABench2 (Physics)	0.767	0.795	0.998
IDMGSP	0.759	0.745	0.965

Table 2: Baseline detectors on the ten scientific subsets, as macro-

F_{1}

under a random split: a fine-tuned DeBERTa-v3-base, Binoculars (Perpl.) and Fast-DetectGPT (Perturb).

		HC3			GPT-2			RADAR
Subset	fmt	rec	flip	unc	rec	flip	unc	rec	flip	unc
SemEval24 arXiv	0.997	0.318	0.453	+0.200	0.420	0.343	+0.137	0.226	0.177	$-$ 0.006
M4 arXiv	0.996	0.366	0.579	+0.267	0.512	0.230	+0.121	0.334	0.054	$-$ 0.019
COLING25 arXiv	0.943	0.444	0.423	+0.207	0.380	0.347	+0.138	0.262	0.115	+0.013
M4 PeerRead	0.822	0.224	0.312	+0.097	0.752	0.003	+0.002	0.978	0.000	$-$ 0.002
IDMGSP (rev.)	0.785	0.478	$-$ 0.339	$\mathbf{-0.171}$	0.862	$-$ 0.005	$-$ 0.005	0.634	0.035	+0.046

Table 3: Artifact reliance of three public detectors under format counterfactuals. flip is

\mathrm{ARS}_{\text{net}}

; unc is

\Delta_{\text{net}}

, the placebo-corrected margin shift over all of

\mathcal{M}

; rec is machine-class recall on the original documents (

|T_{M}|/500

). fmt is formatting separability (Table 1 ), by which subsets are ordered.

Subset	surface probe	RoBERTa	DeBERTa
M4 arXiv	0.995	0.996	0.687
SemEval24 arXiv	0.997	0.997	0.711
COLING25 arXiv	0.951	0.995	0.773

Table 4: In-domain macro-

F_{1}

on the arXiv subsets under the cluster-aware split: the content-free surface probe, RoBERTa (byte-pair encoding, sees the whitespace cue) and DeBERTa-v3 (SentencePiece, discards it).

Backbone	Cond.	in-domain	OOD	$\Delta_{\text{gen}}$
RoBERTa	rand , raw	0.986 $\pm$ 0.004	0.666 $\pm$ 0.004	0.320
	clust , raw	0.981 $\pm$ 0.003	0.662 $\pm$ 0.014	0.320
	+ norm.	0.907 $\pm$ 0.009	0.692 $\pm$ 0.010	0.216
	+ debias	0.836 $\pm$ 0.010	0.657 $\pm$ 0.018	0.179
DeBERTa	rand , raw	0.884 $\pm$ 0.030	0.653 $\pm$ 0.024	0.231
	clust , raw	0.878 $\pm$ 0.021	0.649 $\pm$ 0.012	0.229

Table 5: Performance of fine-tuned detectors.

\Delta_{\text{gen}}

is in-domain minus out-of-domain. cluster keeps near-duplicate groups on one side of the split; norm. additionally normalizes surface form; debias adds bias-model reweighting.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Competition	Metric	Best result
SemEval24 Mono	Accuracy	0.975
SemEval24 Multi	Accuracy	0.959
COLING25 Mono	Macro-F 1	0.8307
COLING25 Multi	Macro-F 1	0.7916

Appendix

Table 6: Best reported score in each shared task represented in our analysis.

Subset	exact	near
M4 PeerRead	40.0%	50.0%
M4 arXiv	35.2%	49.5%
COLING25 PeerRead	28.0%	55.1%
COLING25 arXiv	7.7%	15.7%
SemEval24 PeerRead	1.3%	2.7%
SemEval24 arXiv	0.00%	0.01%

Appendix

Table 7: Duplication within each subset. Near-duplicates use MinHash with Jaccard

\geq 0.8

; exact matching is on whitespace-normalized text.

Subset	Document type	$\mathbf{\|H\|}$	$\mathbf{\|M\|}$	Median tokens H / M	#Gen.	Generators
M4-based datasets
M4 arXiv	abstracts	15,000	15,000	198 / 180	5	gpt-3.5-turbo, text-davinci-003, cohere, dolly-v2-12b, flan-t5
M4 PeerRead	peer reviews	2,930	2,930	2173 / 1026	5	chatgpt, davinci, cohere, dolly-v2, flan-t5
SemEval24 arXiv	abstracts	15,498	11,999	194 / 194	4	chatGPT, davinci, cohere, dolly
SemEval24 PeerRead	peer reviews	2,357	9,374	197 / 238	4	chatGPT, davinci, cohere, dolly
COLING25 arXiv	abstracts	22,484	30,684	192 / 178	12	+ gpt4o, gpt4, llama3-70b/8b, gemma2-9b, gemma-7b, mixtral-8x7b, bloomz

Appendix

Table 8: The ten scientific-document subsets analyzed in this work.

|H|

and

|M|

are counts of human and machine documents in the isolated subset; median length is in whitespace tokens. All subsets are English and all machine documents are written from scratch. Generator counts are as recorded in each release.

Spurious Correlation

See all 28 papers

Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text
Aug 25, 2026Gerrit Quaremba, Hanqi Yan, Elizabeth Black +2AI-Generated Text Detection Robustness of Machine-Generated Text Detection
King’s College London · Wikimedia Foundation
Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions
Mar 18, 2026Madhav S. Baidya, S. S. Baidya, Chirag ChawlaAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India
ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation
Jul 31, 2026Gaetano Perrone, Simon Pietro RomanoAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Department of Electrical Engineering and Information Technology, University of Napoli Federico II, Via Claudio 21, Naples, 80125, Italy

Aug 25, 2026J/K move · Enter open · S save

Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text

Gerrit Quaremba, Hanqi Yan, Elizabeth Black, Denny Vrandecic+1

King’s College London · Wikimedia Foundation

AI-Generated Text Detection Robustness of Machine-Generated Text Detection OOD Generalization Language Model Probing

Open paper AskPDF arXiv

Distinguishing machine-generated text (MGT) from human-written text (HWT) becomes increasingly important due to potential misuse. However, most supervised detectors often degrade out-of-domain (OOD) and require large, diverse training sets. In this work, we analyze the linearity and quality of MGT representations and show that simple linear probes outperform a wide range of detectors while being substantially more sample-efficient. We first show that MGT and HWT latent representations are linearly separable in low-dimensional space, and provide a plausible explanation for this separability through systematic differences in their representation quality. Motivated by these insights, we train two variants of simple linear probes and evaluate them across 4 benchmarks against 16 baselines. Probes consistently improve OOD detection (+11 AUC), requiring solely

{<}100

samples to reach near-peak performance. We show that this transferability arises because probes recover a shared latent MGT direction that generalizes across diverse settings. Finally, we demonstrate that probing vectors capture a continuous spectrum of ``machineness'', highlighting their potential for fine-grained estimation of AI-edited text. Overall, our work provides insights into latent-space differences between MGT and HWT and demonstrates the potential of linear probes as as robust and sample-efficient MGT detectors. We release our code on~github.

Abstract

Spurious Correlation

Explore similar work

Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

ARB: A Matched Authorship-Rewriting Benchmark Dataset for AI-Text Detector Evaluation