We investigate the robustness of widely used AI text detectors under attacks applied to both human and machine text. Prior evaluations typically attack only machine text, overlooking that human text is routinely modified through grammar correction, formatting, or invisible character insertion. We introduce a two-sided protocol that applies attacks to both classes. Across two corpora and nine detectors, reported AUROC gains vanish under two-sided evaluation, with attacked human text misclassified at rates up to 100%. While simple preprocessing reverses character-level attacks, word-level synonym substitution remains effective meaning-preserving, and irreversible.
Figures & tables
Figure 1: False-positive rate on attacked human text at thresholds calibrated for 1% FPR on clean human text (red line).
Figure 2: Effect of each attack on each detector ( Δ AUROC vs. no attack). Blue helps the detector; red breaks it; ∗ marks deltas whose 95% bootstrap CI excludes zero. Effects are family-antagonistic: the zero-width column simultaneously contains the largest positive and largest negative entries.
Figure 3
AUROC
TPR@1%FPR
Detector
clean
para.
clean
para.
Binoculars
.854
.935
.74
.68
FDG (neo)
.828
.913
.69
.59
FDG (gpt2-xl)
.763
.907
.58
.53
FDG (gpt2)
.725
.849
.38
.26
Lastde
.840
.814
.41
.18
Table 1: Clean vs. paraphrase-attacked performance. Bold marks where AUROC rises under attack while detection at 1% FPR falls.
clean
zero-width
homoglyph
backbone
AUROC
1-s
2-s
FPR
1-s
2-s
FPR
RAID
gpt-neo
0.828
0.69
0.64 †
0.02
0.52
0.60 †
0.01
falcon-7b
0.854
1.00 ∗
0.58 †
1.00
0.84
0.64 †
0.23
DetectRL
gpt-neo
0.902
0.67
0.66 †
0.01
–
–
–
Table 2: Fast-DetectGPT at gpt-neo-2.7B and at the falcon-7b pair authors recommend. 1-s and 2-s are one- and two-sided AUROC, and FPR is the fraction of attacked human documents flagged. ∗ marks a significant gain over clean, † a significant loss. DetectRL’s homoglyph attack also substitutes words, so they are left blank.
RAID
DetectRL
domains
8
4
generator configurations
22
4
(detector, attack) cells
99
63
realized FPR, clean human
0.011
0.009
worst FPR, attacked human
0.98
1.00
one-sided AUROC gains
22
17
Table 3: The two-sided result on both corpora
TPR@1%
score shift
detector
family
Δ AUROC
clean
syn.
AI
human
Binoculars
modern zero-shot
-0.181
0.74
0.30
-0.74
-0.30
FDG-neo
modern zero-shot
-0.245
0.69
0.22
-1.01
-0.68
FDG-gpt2xl
modern zero-shot
-0.214
0.58
0.23
-1.14
-0.74
FDG-gpt2
modern zero-shot
-0.167
0.38
0.11
-1.10
-0.74
Lastde
modern zero-shot
-0.290
0.41
0.09
-0.78
-0.38
Table 4: Synonym substitution by detector family. Probability-based detectors collapse; trained classifiers are nearly unaffected. Score shifts are paired (attacked minus source) in units of each detector’s clean human–AI separation: the attack pushes both classes toward “human”, but AI text roughly twice as far.
Figure 5: Attacked documents re-scored after sanitization, which strips zero-width characters, folds confusables, collapses whitespace, and applies NFKC. Thresholds are unchanged from the clean calibration.
trigram sim. to source
exact
attack
attacked
sanitized
restore
zero-w.
0.000
0.999
97%
homogl.
0.118
0.988
31%
synonym
0.689
0.688
0%
paraphr.
0.504
0.504
0%
whitesp.
0.930
0.929
0%
Table 5: Text similarity to the source before and after sanitization, and the share of documents restored exactly. For word-level attacks, a non-zero exact restore means the attack found nothing to change, not that sanitization repaired it.
detection after cleaning, % of clean
attack
curated
full UTS #39
curated + ws
zero-w.
100–100
100–102
99–100
homogl.
86–104
80–105
86–103
whitesp.
0–92
92–101
90–100
cost
0/9
4/9
1/9
Table 6: Detection after each sanitizer configuration, as a percentage of clean detection. Cells show the range over the nine detectors. Cost counts detectors that wrongly flag more clean human text.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
AUROC
detector
clean
protocol
paraphr.
synonym
homogl.
zero-w.
whitesp.
up/low
ins-par
art-del
alt-sp.
misspell
number
Binoculars
0.85
1-sided
0.93 ∗
0.67
0.80
0.99 ∗
0.82
0.84
0.85
0.84
0.85
0.85
0.85
2-sided
0.81 †
0.80 †
0.68 †
0.65 †
0.84
0.86
0.85
0.86
0.85
0.85
0.86
FDG-neo
0.83
1-sided
0.91 ∗
0.58
0.52
0.69
0.77
0.78
0.82
0.80
0.82
0.82
0.80
2-sided
0.73 †
0.77 †
0.60 †
0.64 †
0.81
0.82
0.81
0.83
0.83
0.82
0.82
FDG-gpt2xl
0.76
1-sided
0.91 ∗
0.55
0.81 ∗
0.44
0.70
0.70
0.78
0.72
0.75
0.74
0.73
Appendix
Table 7: AUROC on RAID, every detector against every attack. 1-sided scores attacked AI text against clean human text, which is what the standard protocol reports. 2-sided scores it against attacked human text. ∗ marks a significant increase over clean, † a significant decrease. Every starred one-sided gain sits above a daggered two-sided loss, and no pair increases significantly two-sided.
TPR@1%FPR – fraction of AI documents detected
detector
none
paraphr.
synonym
homogl.
zero-w.
whitesp.
up/low
ins-par
art-del
alt-sp.
misspell
number
Binoculars
0.74
0.68
0.30
0.16
0.80
0.62
0.67
0.72
0.67
0.72
0.72
0.71
FDG-neo
0.69
0.59
0.22
0.08
0.06
0.49
0.52
0.67
0.57
0.67
0.65
0.61
FDG-gpt2xl
0.58
0.53
0.23
0.23
0.03
0.37
0.40
0.57
0.47
0.56
0.53
0.53
FDG-gpt2
0.38
0.26
0.11
0.61
0.00
0.09
0.15
0.33
0.20
0.36
0.33
0.34
Lastde
0.41
0.18
0.09
0.69
0.22
0.37
0.16
0.36
0.27
0.36
0.34
0.31
Appendix
Table 8: TPR at a 1% false-positive threshold, every detector × every attack. Cells are the fraction of attacked AI documents scoring above the detector’s threshold. Truncation cells are reported separately in Section 4.5 .
FPR – fraction of human documents wrongly flagged
detector
none
paraphr.
synonym
homogl.
zero-w.
whitesp.
up/low
ins-par
art-del
alt-sp.
misspell
number
Binoculars
0.01
0.14
0.00
0.00
0.62
0.00
0.00
0.01
0.00
0.01
0.00
0.01
FDG-neo
0.01
0.20
0.00
0.01
0.02
0.00
0.00
0.01
0.00
0.01
0.00
0.01
FDG-gpt2xl
0.01
0.15
0.00
0.13
0.00
0.01
0.00
0.01
0.00
0.01
0.00
0.01
FDG-gpt2
0.01
0.07
0.00
0.70
0.00
0.00
0.00
0.01
0.00
0.00
0.00
0.01
Lastde
0.01
0.01
0.00
0.71
0.10
0.01
0.00
0.01
0.01
0.02
0.01
0.00
Appendix
Table 9: False-positive rate on attacked human text at the same 1%-FPR thresholds. Cells are the fraction of attacked human documents scoring above the same threshold. The none column is the nominal rate by construction.
name
scoring model(s)
size
family
source
Binoculars
Falcon-7B + Falcon-7B-Instruct
7B × 2
modern zero-shot
Hans et al. (2024)
FDG-neo
Fast-DetectGPT / GPT-Neo
2.7B
modern zero-shot
Bao et al. (2024)
FDG-gpt2xl
Fast-DetectGPT / GPT-2 XL
1.5B
modern zero-shot
Bao et al. (2024)
FDG-gpt2
Fast-DetectGPT / GPT-2
124M
modern zero-shot
Bao et al. (2024)
Lastde
Lastde / GPT-Neo
2.7B
modern zero-shot
Xu et al. (2025)
LogLik
mean log-likelihood / GPT-Neo
2.7B
legacy statistical
Gehrmann et al. (2019)
Appendix
Table 10: The nine detectors. All run from public checkpoints without modification; scores are normalized so that higher means more AI-like.
detector
best generator
worst generator
gap
worst attack
loss
Binoculars
ChatGPT (1.000)
MPT (0.430)
0.570
synonym
0.181
FDG-neo
GPT-4 (1.000)
MPT (0.425)
0.575
homoglyph
0.304
FDG-gpt2xl
GPT-2 (1.000)
Mistral (0.373)
0.627
zero-width
0.321
FDG-gpt2
GPT-3 (0.973)
Mistral (0.368)
0.605
zero-width
0.248
Lastde
GPT-2 (0.999)
GPT-4 (0.400)
0.599
synonym
0.290
LogLik
GPT-2 (0.978)
MPT (0.334)
0.644
synonym
0.238
Appendix
Table 11: Generator choice against the worst attack, clean AUROC on RAID. Best and worst are each detector’s highest and lowest AUROC over the 22 generator and decoding setups, and gap is their difference. Every best uses greedy decoding and every worst uses sampling. Loss is the largest drop any attack causes. Bold marks the larger of gap and loss.
Figure 6: Δ AUROC vs. Δ TPR@1%FPR for all 99 cells. Points in the shaded quadrant are cells where AUROC reports improvement while deployment-relevant TPR degrades; for Binoculars × paraphrase both deltas are individually significant, with opposite signs.
TPR@1%FPR – fraction of AI documents detected
detector
none
DIPPER
back-tr.
polish
word
char
homogl.
zero-w.
Binoculars
0.73
0.78
0.09
0.60
0.21
0.25
0.26
0.36
FDG-neo
0.65
0.72
0.07
0.48
0.08
0.15
0.11
0.06
FDG-gpt2xl
0.57
0.58
0.07
0.39
0.05
0.06
0.03
0.00
FDG-gpt2
0.40
0.37
0.04
0.22
0.02
0.04
0.05
0.00
Lastde
0.56
0.63
0.04
0.31
0.00
0.00
0.01
0.70
Appendix
Table 12: TPR at a 1% false-positive threshold on DetectRL. Six of the seven attacks are DetectRL’s own; zero-width is RAID’s, applied by us because DetectRL has no equivalent.
detector
none
DIPPER
back-tr.
polish
word
char
homogl.
zero-w.
Binoculars
0.01
0.08
0.00
0.14
0.00
0.00
0.00
0.31
FDG-neo
0.01
0.14
0.00
0.13
0.00
0.00
0.00
0.01
FDG-gpt2xl
0.01
0.07
0.00
0.12
0.00
0.00
0.00
0.00
FDG-gpt2
0.01
0.04
0.00
0.07
0.00
0.01
0.00
0.00
Lastde
0.01
0.12
0.00
0.09
0.00
0.00
0.00
0.68
LogLik
0.01
0.02
0.00
0.06
0.00
0.00
0.00
1.00
Appendix
Table 13: False-positive rate on attacked human text in DetectRL, at the same 1%-FPR thresholds. Log-likelihood and log-rank flag every one of the 1,440 human documents under zero-width insertion.
AUROC
detector
clean
protocol
DIPPER
back-tr.
polish
word
char
homogl.
zero-w.
Binoculars
0.96
1-sided
0.97 ∗
0.60
0.86
0.73
0.77
0.81
0.99 ∗
2-sided
0.91 †
0.76 †
0.75 †
0.86 †
0.88 †
0.89 †
0.55 †
FDG-neo
0.90
1-sided
0.96 ∗
0.46
0.79
0.51
0.55
0.54
0.67
2-sided
0.87 †
0.76 †
0.70 †
0.83 †
0.80 †
0.83 †
0.66 †
FDG-gpt2xl
0.88
1-sided
0.93 ∗
0.54
0.77
0.50
0.47
0.43
0.45
Appendix
Table 14: AUROC on DetectRL, every detector against every attack. 1-sided scores attacked AI text against clean human text. 2-sided scores it against attacked human text. ∗ marks a significant increase over clean, † a significant decrease. No cell increases significantly two-sided.
attack
semantic sim.
positions changed
synonym
0.917
20.9%
paraphr.
0.874
96.1%
art-del
0.991
81.6%
up/low
1.000
4.7%
ins-par
1.000
1.5%
number
0.997
1.1%
Appendix
Table 15: Semantic meaning preserved by word-level attacks determined by cosine similarity of the attacked text and its source, over 2,160 documents per attack. Positions changed is the share of word positions that differ from the source. Unicode and whitespace attacks are omitted since their scores depend on the tokenizer, not on meaning (Table 5 ).
Figure 7: Controlled truncation of identical sources. Left: detection at 1%-FPR thresholds. Right: false-positive rate on human text at the same thresholds. RADAR’s apparent short-text gain is length bias: it flags 61% of 50-word human texts.
Figure 8: The measurement pipeline. (a) We sample RAID by domain and score every text with all 9 detectors (Similar approach for DetectRL). (b) Clean human documents are split in half. One half fits each detector’s threshold τ at a nominal 1% false-positive rate, the other verifies it works. Attacked text never enters calibration, so the attacked boxes carry the full n . Standard evaluations score only the two AI branches. The highlighted branch is what this study adds, applying the same 11 attacks to human documents and counting who gets accused.
corpus
table
mappings
reversed
clean rewritten
RAID
curated
47
99.3%
60 (2.8%)
RAID
full
1,861
99.9%
330 (15.3%)
DetectRL
curated
47
61.8%
17 (0.8%)
DetectRL
full
1,861
98.0%
44 (2.0%)
Appendix
Table 16: The curated table against the full UTS #39 list. Reversed is the percentage of the attack’s substituted characters turned back into the original letter. Clean rewritten is how many of the 2,160 unattacked source documents the table alters.
detector
zero-width
homoglyph
whitespace
curated
full
cur.+ws
curated
full
cur.+ws
curated
full
cur.+ws
Binoculars
100
100 †
100
97
98 †
97
84
100 †
99
FDG-neo
100
100
100
95
95
95
70
99
99
FDG-gpt2xl
100
100 †
99
92
92 †
91
64
99 †
99
FDG-gpt2
100
100 †
100
90
93 †
90
23
99 †
99
Lastde
100
100
100
87
80
88
92
99
99
Appendix
Table 17: Detection after each sanitizer configuration, per detector, as a percentage of clean detection at the 1% threshold. Table 6 gives the range of each column. † marks a configuration that raises the detector’s FPR on clean human text.
target ← source code points
A
U+0391, U+0410
S
U+0405
k
U+03BA
B
U+0392, U+0412
T
U+03A4, U+0422
l
U+217C
C
U+0421
X
U+03A7, U+0425
o
U+03BF, U+043E
E
U+0395, U+0415
Y
U+03A5
p
U+03C1, U+0440
H
U+0397, U+041D
Z
U+0396
q
U+051B
I
U+0399
a
U+03B1, U+0430
s
U+0455
Appendix
Table 18: The 47 curated confusable mappings. Each row gives an ASCII target and the code points folded onto it. Applied after zero-width stripping and before NFKC.
AI text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary. On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures (86-106% of the fine-tuned discrimination ceiling: on RoBERTa-base, raw projection exceeds fine-tuning); on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested. The same axis inverts on non-native ESL writing (AUROC 0.06-0.20) -- a falsifiable prediction unique to the typicality reading. A 24-example frozen probe matches full fine-tuning (0.900 vs 0.895). A closed-form Jacobian predictor parameterises axis-manipulating interventions with R^2 = 1.000 universal, lifts ELECTRA-CE deployment TPR from 0.000 to 0.904 at FPR = 1%, and transfers to three independently-trained third-party RoBERTa detectors at 16/16 oracle-equivalence (57% NYT-FPR reduction on the OpenAI detector). Scope: encoder family; mechanism magnitude HC3-anchored; population-level shared axis with per-text mechanisms varying across architectures. Three operationally distinct probes -- text-surface caps_rate residualisation, geometric signed-epsilon ablation, closed-form text-pair predictor -- agree at cos 0.74/0.81/1.00 across three architectures, confirming observer-invariance. Under matched-TPR-0.90 evaluation, the published intervention zoo (CC, dealign-f2c) is calibration-equivalent across 27 cells (|Delta AUROC| <= 0.0081), and >= 97% of the LoRA->full-FT bias gap on ELECTRA is calibration shift, not learned representation -- the central claim's prediction confirmed.
Existing AI-generated text detectors are vulnerable to attacks that manipulate textual characteristics. In this study, we propose a novel Triospect Detection Framework by using additional perspectives of content (core ideas) and expression (stylistic elements) within a given text. Experiments on two benchmarks involving 17 attacks, 12 domains, and 17 source models demonstrate that Triospect is robust against these attacks. It improves the strong baseline by a significant margin of 22.3% (AUROC) and 13% (TPR01) on the Humanize-16K after-attack subset, and by 9.1% (AUROC) and 22% (TPR01) on the adversarial RAID. This framework marks a pioneering effort in statistical methods to enhance detection reliability against attacks. We release our data and code at https://github.com/baoguangsheng/triospect.
Guangsheng Bao, Lihua Rong, Yanbin Zhao +3
Zhejiang University · Zhejiang University of Technology · Shanghai Polytechnic University +2
Supervised AI-text detectors report high benchmark accuracy, but it is not clear what their decisions are based on. We analyze a RoBERTa-based detector under semantic, structural, and tokenizer-level perturbations, using the M4 dataset (N = 10,000) and controlled generations (N = 300). When Mistral-7B-Instruct was asked to make machine text sound more human, Verb Diversity rose from 0.77 to 0.92 and the outputs became easier to detect. Detection scores appear to track statistical complexity, which also leads to a 76.3% false-positive rate on formal human writing. As a control, we evaluate event-based Latent Space detection. Paraphrasing changed 87% of its event sequences (Jaccard = 0.067), and homoglyphs altered 70% of the extracted verbs even though extraction still ran (Jaccard = 0.30). Its best domain AUC was 0.577. RoBERTa's robustness seems specific to the features it uses, and structural abstraction did not make detection more robust.
Claudiu Creanga, Liviu Dinu
Interdisciplinary School of Doctoral Studies University of Bucharest, Romania · Faculty of Mathematics and Informatics University of Bucharest, Romania