Attacking Only Half the Data: Two-Sided Evaluation of AI-Text Detector Robustness

Apu Chandraw Shill; Rabindra Nath Nandi

cs.CLOct 13, 2026

Attacking Only Half the Data: Two-Sided Evaluation of AI-Text Detector Robustness

Authors: Apu Chandraw Shill, Rabindra Nath Nandi

Organizations: Independent Researcher

Abstract

We investigate the robustness of widely used AI text detectors under attacks applied to both human and machine text. Prior evaluations typically attack only machine text, overlooking that human text is routinely modified through grammar correction, formatting, or invisible character insertion. We introduce a two-sided protocol that applies attacks to both classes. Across two corpora and nine detectors, reported AUROC gains vanish under two-sided evaluation, with attacked human text misclassified at rates up to 100%. While simple preprocessing reverses character-level attacks, word-level synonym substitution remains effective meaning-preserving, and irreversible.

Figures & tables

Figure 1: False-positive rate on attacked human text at thresholds calibrated for 1% FPR on clean human text (red line).

Figure 2: Effect of each attack on each detector ( $\Delta$ AUROC vs. no attack). Blue helps the detector; red breaks it; $*$ marks deltas whose 95% bootstrap CI excludes zero. Effects are family-antagonistic: the zero-width column simultaneously contains the largest positive and largest negative entries.

	AUROC		TPR@1%FPR
Detector	clean	para.	clean	para.
Binoculars	.854	.935	.74	.68
FDG (neo)	.828	.913	.69	.59
FDG (gpt2-xl)	.763	.907	.58	.53
FDG (gpt2)	.725	.849	.38	.26
Lastde	.840	.814	.41	.18

Table 1: Clean vs. paraphrase-attacked performance. Bold marks where AUROC rises under attack while detection at

1\%

FPR falls.

RAID
	clean	zero-width			homoglyph
backbone	AUROC	1-s	2-s	FPR	1-s	2-s	FPR
gpt-neo	0.828	0.69	0.64 †	0.02	0.52	0.60 †	0.01
falcon-7b	0.854	1.00 ∗	0.58 †	1.00	0.84	0.64 †	0.23
DetectRL
gpt-neo	0.902	0.67	0.66 †	0.01	–	–	–

Table 2: Fast-DetectGPT at gpt-neo-2.7B and at the falcon-7b pair authors recommend. 1-s and 2-s are one- and two-sided AUROC, and FPR is the fraction of attacked human documents flagged. ∗ marks a significant gain over clean, † a significant loss. DetectRL’s homoglyph attack also substitutes words, so they are left blank.

	RAID	DetectRL
domains	8	4
generator configurations	22	4
(detector, attack) cells	99	63
realized FPR, clean human	0.011	0.009
worst FPR, attacked human	0.98	1.00
one-sided AUROC gains	22	17

Table 3: The two-sided result on both corpora

			TPR@1%		score shift
detector	family	$\Delta$ AUROC	clean	syn.	AI	human
Binoculars	modern zero-shot	-0.181	0.74	0.30	-0.74	-0.30
FDG-neo	modern zero-shot	-0.245	0.69	0.22	-1.01	-0.68
FDG-gpt2xl	modern zero-shot	-0.214	0.58	0.23	-1.14	-0.74
FDG-gpt2	modern zero-shot	-0.167	0.38	0.11	-1.10	-0.74
Lastde	modern zero-shot	-0.290	0.41	0.09	-0.78	-0.38

Table 4: Synonym substitution by detector family. Probability-based detectors collapse; trained classifiers are nearly unaffected. Score shifts are paired (attacked minus source) in units of each detector’s clean human–AI separation: the attack pushes both classes toward “human”, but AI text roughly twice as far.

Figure 5: Attacked documents re-scored after sanitization, which strips zero-width characters, folds confusables, collapses whitespace, and applies NFKC. Thresholds are unchanged from the clean calibration.

	trigram sim. to source		exact
attack	attacked	sanitized	restore
zero-w.	0.000	0.999	97%
homogl.	0.118	0.988	31%
synonym	0.689	0.688	0%
paraphr.	0.504	0.504	0%
whitesp.	0.930	0.929	0%

Table 5: Text similarity to the source before and after sanitization, and the share of documents restored exactly. For word-level attacks, a non-zero exact restore means the attack found nothing to change, not that sanitization repaired it.

	detection after cleaning, % of clean
attack	curated	full UTS #39	curated $+$ ws
zero-w.	100–100	100–102	99–100
homogl.	86–104	80–105	86–103
whitesp.	0–92	92–101	90–100
cost	0/9	4/9	1/9

Table 6: Detection after each sanitizer configuration, as a percentage of clean detection. Cells show the range over the nine detectors. Cost counts detectors that wrongly flag more clean human text.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	AUROC
detector	clean	protocol	paraphr.	synonym	homogl.	zero-w.	whitesp.	up/low	ins-par	art-del	alt-sp.	misspell	number
Binoculars	0.85	1-sided	0.93 ∗	0.67	0.80	0.99 ∗	0.82	0.84	0.85	0.84	0.85	0.85	0.85
		2-sided	0.81 †	0.80 †	0.68 †	0.65 †	0.84	0.86	0.85	0.86	0.85	0.85	0.86
FDG-neo	0.83	1-sided	0.91 ∗	0.58	0.52	0.69	0.77	0.78	0.82	0.80	0.82	0.82	0.80
		2-sided	0.73 †	0.77 †	0.60 †	0.64 †	0.81	0.82	0.81	0.83	0.83	0.82	0.82
FDG-gpt2xl	0.76	1-sided	0.91 ∗	0.55	0.81 ∗	0.44	0.70	0.70	0.78	0.72	0.75	0.74	0.73

Appendix

Table 7: AUROC on RAID, every detector against every attack. 1-sided scores attacked AI text against clean human text, which is what the standard protocol reports. 2-sided scores it against attacked human text. ∗ marks a significant increase over clean, † a significant decrease. Every starred one-sided gain sits above a daggered two-sided loss, and no pair increases significantly two-sided.

	TPR@1%FPR – fraction of AI documents detected
detector	none	paraphr.	synonym	homogl.	zero-w.	whitesp.	up/low	ins-par	art-del	alt-sp.	misspell	number
Binoculars	0.74	0.68	0.30	0.16	0.80	0.62	0.67	0.72	0.67	0.72	0.72	0.71
FDG-neo	0.69	0.59	0.22	0.08	0.06	0.49	0.52	0.67	0.57	0.67	0.65	0.61
FDG-gpt2xl	0.58	0.53	0.23	0.23	0.03	0.37	0.40	0.57	0.47	0.56	0.53	0.53
FDG-gpt2	0.38	0.26	0.11	0.61	0.00	0.09	0.15	0.33	0.20	0.36	0.33	0.34
Lastde	0.41	0.18	0.09	0.69	0.22	0.37	0.16	0.36	0.27	0.36	0.34	0.31

Appendix

Table 8: TPR at a 1% false-positive threshold, every detector

\times

every attack. Cells are the fraction of attacked AI documents scoring above the detector’s threshold. Truncation cells are reported separately in Section 4.5 .

	FPR – fraction of human documents wrongly flagged
detector	none	paraphr.	synonym	homogl.	zero-w.	whitesp.	up/low	ins-par	art-del	alt-sp.	misspell	number
Binoculars	0.01	0.14	0.00	0.00	0.62	0.00	0.00	0.01	0.00	0.01	0.00	0.01
FDG-neo	0.01	0.20	0.00	0.01	0.02	0.00	0.00	0.01	0.00	0.01	0.00	0.01
FDG-gpt2xl	0.01	0.15	0.00	0.13	0.00	0.01	0.00	0.01	0.00	0.01	0.00	0.01
FDG-gpt2	0.01	0.07	0.00	0.70	0.00	0.00	0.00	0.01	0.00	0.00	0.00	0.01
Lastde	0.01	0.01	0.00	0.71	0.10	0.01	0.00	0.01	0.01	0.02	0.01	0.00

Appendix

Table 9: False-positive rate on attacked human text at the same 1%-FPR thresholds. Cells are the fraction of attacked human documents scoring above the same threshold. The none column is the nominal rate by construction.

name	scoring model(s)	size	family	source
Binoculars	Falcon-7B + Falcon-7B-Instruct	7B $\times$ 2	modern zero-shot	Hans et al. (2024)
FDG-neo	Fast-DetectGPT / GPT-Neo	2.7B	modern zero-shot	Bao et al. (2024)
FDG-gpt2xl	Fast-DetectGPT / GPT-2 XL	1.5B	modern zero-shot	Bao et al. (2024)
FDG-gpt2	Fast-DetectGPT / GPT-2	124M	modern zero-shot	Bao et al. (2024)
Lastde	Lastde / GPT-Neo	2.7B	modern zero-shot	Xu et al. (2025)
LogLik	mean log-likelihood / GPT-Neo	2.7B	legacy statistical	Gehrmann et al. (2019)

Appendix

Table 10: The nine detectors. All run from public checkpoints without modification; scores are normalized so that higher means more AI-like.

detector	best generator	worst generator	gap	worst attack	loss
Binoculars	ChatGPT (1.000)	MPT (0.430)	0.570	synonym	0.181
FDG-neo	GPT-4 (1.000)	MPT (0.425)	0.575	homoglyph	0.304
FDG-gpt2xl	GPT-2 (1.000)	Mistral (0.373)	0.627	zero-width	0.321
FDG-gpt2	GPT-3 (0.973)	Mistral (0.368)	0.605	zero-width	0.248
Lastde	GPT-2 (0.999)	GPT-4 (0.400)	0.599	synonym	0.290
LogLik	GPT-2 (0.978)	MPT (0.334)	0.644	synonym	0.238

Appendix

Table 11: Generator choice against the worst attack, clean AUROC on RAID. Best and worst are each detector’s highest and lowest AUROC over the 22 generator and decoding setups, and gap is their difference. Every best uses greedy decoding and every worst uses sampling. Loss is the largest drop any attack causes. Bold marks the larger of gap and loss.

Figure 6:

\Delta

AUROC vs.

\Delta

TPR@1%FPR for all 99 cells. Points in the shaded quadrant are cells where AUROC reports improvement while deployment-relevant TPR degrades; for Binoculars

\times

paraphrase both deltas are individually significant, with opposite signs.

	TPR@1%FPR – fraction of AI documents detected
detector	none	DIPPER	back-tr.	polish	word	char	homogl.	zero-w.
Binoculars	0.73	0.78	0.09	0.60	0.21	0.25	0.26	0.36
FDG-neo	0.65	0.72	0.07	0.48	0.08	0.15	0.11	0.06
FDG-gpt2xl	0.57	0.58	0.07	0.39	0.05	0.06	0.03	0.00
FDG-gpt2	0.40	0.37	0.04	0.22	0.02	0.04	0.05	0.00
Lastde	0.56	0.63	0.04	0.31	0.00	0.00	0.01	0.70

Appendix

Table 12: TPR at a 1% false-positive threshold on DetectRL. Six of the seven attacks are DetectRL’s own; zero-width is RAID’s, applied by us because DetectRL has no equivalent.

detector	none	DIPPER	polish	char	zero-w.
Binoculars	0.01	0.08	0.14	0.00	0.31
FDG-neo	0.01	0.14	0.13	0.00	0.01
FDG-gpt2xl	0.01	0.07	0.12	0.00	0.00
FDG-gpt2	0.01	0.04	0.07	0.01	0.00
Lastde	0.01	0.12	0.09	0.00	0.68
LogLik	0.01	0.02	0.06	0.00	1.00

Appendix

Table 13: False-positive rate on attacked human text in DetectRL, at the same 1%-FPR thresholds. Log-likelihood and log-rank flag every one of the 1,440 human documents under zero-width insertion.

	AUROC
detector	clean	protocol	DIPPER	back-tr.	polish	word	char	homogl.	zero-w.
Binoculars	0.96	1-sided	0.97 ∗	0.60	0.86	0.73	0.77	0.81	0.99 ∗
		2-sided	0.91 †	0.76 †	0.75 †	0.86 †	0.88 †	0.89 †	0.55 †
FDG-neo	0.90	1-sided	0.96 ∗	0.46	0.79	0.51	0.55	0.54	0.67
		2-sided	0.87 †	0.76 †	0.70 †	0.83 †	0.80 †	0.83 †	0.66 †
FDG-gpt2xl	0.88	1-sided	0.93 ∗	0.54	0.77	0.50	0.47	0.43	0.45

Appendix

Table 14: AUROC on DetectRL, every detector against every attack. 1-sided scores attacked AI text against clean human text. 2-sided scores it against attacked human text. ∗ marks a significant increase over clean, † a significant decrease. No cell increases significantly two-sided.

attack	semantic sim.	positions changed
synonym	0.917	20.9%
paraphr.	0.874	96.1%
art-del	0.991	81.6%
up/low	1.000	4.7%
ins-par	1.000	1.5%
number	0.997	1.1%

Appendix

Table 15: Semantic meaning preserved by word-level attacks determined by cosine similarity of the attacked text and its source, over

2{,}160

documents per attack. Positions changed is the share of word positions that differ from the source. Unicode and whitespace attacks are omitted since their scores depend on the tokenizer, not on meaning (Table 5 ).

Figure 7: Controlled truncation of identical sources. Left: detection at 1%-FPR thresholds. Right: false-positive rate on human text at the same thresholds. RADAR’s apparent short-text gain is length bias: it flags 61% of 50-word human texts.

Panel of Figure 8 — Figure 8: The measurement pipeline. (a) We sample RAID by domain and score every text with all 9 detectors (Similar approach for DetectRL). (b) Clean human documents are split in half. One half fits each detector’s threshold $\tau$ at a nominal $1\%$ false-positive rate, the other verifies it works. Attacked text never enters calibration, so the attacked boxes carry the full $n$ . Standard evaluations score only the two AI branches. The highlighted branch is what this study adds, applying the same 11 attacks to human documents and counting who gets accused.

corpus	table	mappings	reversed	clean rewritten
RAID	curated	47	99.3%	60 (2.8%)
RAID	full	1,861	99.9%	330 (15.3%)
DetectRL	curated	47	61.8%	17 (0.8%)
DetectRL	full	1,861	98.0%	44 (2.0%)

Appendix

Table 16: The curated table against the full UTS #39 list. Reversed is the percentage of the attack’s substituted characters turned back into the original letter. Clean rewritten is how many of the

2{,}160

unattacked source documents the table alters.

detector	zero-width			homoglyph			whitespace
	curated	full	cur.+ws	curated	full	cur.+ws	curated	full	cur.+ws
Binoculars	100	100 †	100	97	98 †	97	84	100 †	99
FDG-neo	100	100	100	95	95	95	70	99	99
FDG-gpt2xl	100	100 †	99	92	92 †	91	64	99 †	99
FDG-gpt2	100	100 †	100	90	93 †	90	23	99 †	99
Lastde	100	100	100	87	80	88	92	99	99

Appendix

Table 17: Detection after each sanitizer configuration, per detector, as a percentage of clean detection at the

1\%

threshold. Table 6 gives the range of each column. † marks a configuration that raises the detector’s FPR on clean human text.

target $\leftarrow$ source code points
A	U+0391, U+0410	S	U+0405	k	U+03BA
B	U+0392, U+0412	T	U+03A4, U+0422	l	U+217C
C	U+0421	X	U+03A7, U+0425	o	U+03BF, U+043E
E	U+0395, U+0415	Y	U+03A5	p	U+03C1, U+0440
H	U+0397, U+041D	Z	U+0396	q	U+051B
I	U+0399	a	U+03B1, U+0430	s	U+0455

Appendix

Table 18: The 47 curated confusable mappings. Each row gives an ASCII target and the code points folded onto it. Applied after zero-width stripping and before NFKC.

Explore similar work

Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction
May 20, 2026Alexander SmirnovAI-Generated Text Detection Classification
University College London
Triospect: A Three-Dimensional Framework for Robust Statistical AI-Generated Text Detection Against Diverse Attacks
Jun 30, 2026Guangsheng Bao, Lihua Rong, Yanbin Zhao +3AI-Generated Text Detection Robustness of Machine-Generated Text Detection
Zhejiang University · Zhejiang University of Technology · Shanghai Polytechnic University +2
Ontological Instability and Statistical Amplification: The Paradox of "Humanizing" LLM-Generated Text
Oct 2, 2026Claudiu Creanga, Liviu DinuAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Interdisciplinary School of Doctoral Studies University of Bucharest, Romania · Faculty of Mathematics and Informatics University of Bucharest, Romania

May 20, 2026J/K move · Enter open · S save

Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction

Alexander Smirnov

University College London

AI-Generated Text Detection Classification Language Model Auditing

Open paper AskPDF arXiv

AI text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary. On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures (86-106% of the fine-tuned discrimination ceiling: on RoBERTa-base, raw projection exceeds fine-tuning); on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested. The same axis inverts on non-native ESL writing (AUROC 0.06-0.20) -- a falsifiable prediction unique to the typicality reading. A 24-example frozen probe matches full fine-tuning (0.900 vs 0.895). A closed-form Jacobian predictor parameterises axis-manipulating interventions with R^2 = 1.000 universal, lifts ELECTRA-CE deployment TPR from 0.000 to 0.904 at FPR = 1%, and transfers to three independently-trained third-party RoBERTa detectors at 16/16 oracle-equivalence (57% NYT-FPR reduction on the OpenAI detector). Scope: encoder family; mechanism magnitude HC3-anchored; population-level shared axis with per-text mechanisms varying across architectures. Three operationally distinct probes -- text-surface caps_rate residualisation, geometric signed-epsilon ablation, closed-form text-pair predictor -- agree at cos 0.74/0.81/1.00 across three architectures, confirming observer-invariance. Under matched-TPR-0.90 evaluation, the published intervention zoo (CC, dealign-f2c) is calibration-equivalent across 27 cells (|Delta AUROC| <= 0.0081), and >= 97% of the LoRA->full-FT bias gap on ELECTRA is calibration shift, not learned representation -- the central claim's prediction confirmed.

Abstract

Explore similar work

Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction

Triospect: A Three-Dimensional Framework for Robust Statistical AI-Generated Text Detection Against Diverse Attacks

Ontological Instability and Statistical Amplification: The Paradox of "Humanizing" LLM-Generated Text