Style as a Confound: False Positives in AI Detection of Non-Native Academic Writing

Hyeonchu Park; Gahye Jeong; Bugeun Kim

cs.AIAug 27, 2026

Style as a Confound: False Positives in AI Detection of Non-Native Academic Writing

Authors: Hyeonchu Park, Gahye Jeong, Bugeun Kim

Organizations: Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Wordvice, Republic of Korea

Abstract

AI text detectors are increasingly employed in academic settings, but it remains unclear whether their outputs reflect AI authorship itself or broader linguistic features associated with polished academic English. Previous studies have reported high false-positive rates (FPRs) for non-native English writing, but population-level comparisons confound authorship with differences in topic, domain, and writing style. Professional editing provides a useful setting for examining this issue because it changes the linguistic form of manuscripts while preserving authorship and content. We examined 135,389 document pairs from a professional English editing service (2018-2025), comprising non-native manuscripts and their native-edited versions, to assess how editing affects detector responses controlling for content and authorship. For the 13 AI text detectors, FPRs for human-written texts varied widely, from 0.0% to 100.0%. Responses varied across detectors: the same edits increased AI scores in some detectors but decreased them in others. Notably, score changes correlated with the extent of editing. The findings identify professional editing style as a key confounding variable in AI detector outputs, rather than establishing a full separation of text origin from linguistic style, raising concerns about fairness and reliability in academic settings.

Figures & tables

Statistic	Value
Total document pairs	135,389
Mean word count (original)	2,355
Median word count (original)	950
Std. dev. (word count)	3,038
Subject areas	40+
Docs. w/ 500–5,000 words	$\sim$ 57%

Table 1: Summary dataset statistics. Full year-by-year and service-type breakdown in Appendix B

Metric	Orig.	Edited	$\Delta$
Mean word length	5.196	5.252	$+0.055$
Type–token ratio	0.407	0.412	$+0.005$
Mean sentence length	24.35	23.51	$-0.84$
Flesch–Kincaid grade	14.51	14.42	$-0.09$

Table 2: Linguistic characteristics of professional editing (6,000-pair subsample). Edit intensity: median 196 / mean 463 edit units per document.

Detector	FP%		$\Delta$ FP	$p$
Detector	orig	edit	$\Delta$ FP	$p$
Token-statistics-based
Log-rank	100.0	100.0	$+$ 0.0	ns
Log-likelihood	99.8	99.9	$+$ 0.0	$<$ .05
Entropy	99.8	99.9	$+$ 0.1	$<$ .001
GLTR	99.9	99.9	$+$ 0.0	$<$ .01

Table 3: Baseline false positive behavior on the Pre-ChatGPT dataset (2018–2022). FPR color:

\geq

50% , 10–49% .

\Delta

FP color: bias reduction , bias amplification . Full results in Appendix D.3

		ALL		Pre-ChatGPT		Post-ChatGPT
Detector	Type	$\Delta\text{score}$	$\Delta\text{FP}$	$\Delta\text{score}$	$\Delta\text{FP}$	$\Delta\text{score}$	$\Delta\text{FP}$
Entropy	Stat.	$+$ 0.014 ∗∗∗	$+$ 0.1	$+$ 0.017 ∗∗∗	$+$ 0.1	$+$ 0.005 ∗∗∗	$+$ 0.0
GLTR	Stat.	$+$ 0.010 ∗∗∗	$+$ 0.0	$+$ 0.012 ∗∗∗	$+$ 0.0	$+$ 0.004 ∗∗∗	$+$ 0.0
Log-rank	Stat.	$+$ 0.001 ns	$+$ 0.0	$+$ 0.001 ∗∗∗	$+$ 0.0	$+$ 0.001 ∗∗∗	$+$ 0.0
Log-like	Stat.	$+$ 0.005 ns	$+$ 0.0	$+$ 0.006 ∗∗∗	$+$ 0.0	$+$ 0.002 ∗∗∗	$+$ 0.0
BiScope	Zero	$-$ 0.019 ∗∗∗	$-$ 0.5	$-$ 0.021 ∗∗∗	$-$ 0.6	$-$ 0.008 ∗∗∗	$-$ 0.1

Table 4: Effect of professional editing on AI detector scores and false positive rates.

\Delta\text{score}=\text{score}_{\text{edit}}-\text{score}_{\text{orig}}

(mean score shift);

\Delta\text{FP}=\text{FP}_{\text{edit}}-\text{FP}_{\text{orig}}

. Green = decrease; Red = increase.

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

{}^{ns}p\geq.05

. Full results in Appendix D.3

Detector	Type	$\rho$	$R^{2}$	Q1	Q2	Q3	Q4	Q5
Entropy	Stat.	$+$ 0.343 ∗∗∗	0.037	$-$ .009	$+$ .015	$+$ .020	$+$ .027	$+$ .032
Log-rank	Stat.	$+$ 0.293 ∗∗∗	0.012	$+$ .001	$+$ .001	$+$ .002	$+$ .002	$+$ .002
Log-likelihood	Stat.	$+$ 0.287 ∗∗∗	0.005	$+$ .003	$+$ .005	$+$ .006	$+$ .007	$+$ .008
GLTR	Stat.	$+$ 0.257 ∗∗∗	0.023	$+$ .006	$+$ .011	$+$ .013	$+$ .016	$+$ .019
BiScope	Zero	$-$ 0.418 ∗∗∗	0.041	$+$ .009	$-$ .019	$-$ .026	$-$ .033	$-$ .040
Binoculars	Zero	$-$ 0.134 ∗∗∗	0.019	$+$ .028	$-$ .002	$-$ .003	$-$ .004	$-$ .004

Table 5: Editing intensity and detector score shifts. Spearman

\rho

denotes the correlation between edit-token ratio and

\Delta\text{score}

. Q1–Q5 represent quintiles of editing intensity (Q1 = weakest, Q5 = strongest). Green = score decreases; Red = score increases.

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

Feature	Statistical	LLM-based	Supervised
Grammar ratio	$+0.111$	$+0.019$	$-0.005$
Lexical ratio	$+0.042$	$-0.018$	$-0.020$
Syntactic ratio	$+0.101$	$+0.013$	$-0.040$
Edit-token ratio	$+0.286$	$+0.022$	$-0.089$
Orig. word count	$-0.099$	$-0.002$	$+0.067$
# edit units	$+0.116$	$+0.021$	$+0.005$

Table 6: Spearman correlation (

\rho

) between editing characteristics and detector score shifts, by detector family (pre-ChatGPT subset).

Metric	Statistical	LLM-based	Supervised
TTR	$-0.063$	$-0.063$	$-0.073$
Word length	$-0.022$	$-0.046$	$-0.082$
Function-word ratio	$+0.071$	$+0.068$	$+0.064$
Sentence length	$-0.043$	$+0.005$	$+0.017$

Table 7: Spearman correlation between changes in complexity and detector score shifts, by detector family.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Overall Scale
Total order pairs	135,389
Collection period	2018–2025 (8 years)
Era Split
Pre-ChatGPT (2018–2022)	104,752 (77.4%)
Post-ChatGPT (2023–2025)	30,637 (22.6%)

Appendix

Table 8: Dataset Statistics . The pre-ChatGPT / post-ChatGPT split is defined by the public release of ChatGPT (November 2022); orders from 2023 onward are classified as post-ChatGPT.

Edit type	Share
Mixed	66.1%
Grammar	15.8%
Lexical	14.0%
Syntactic	0.1%
no-edit	0.4%

Appendix

Table 9: Distribution of edit-operation categories.

Edit type	$\Delta$ score	Cohen’s $d_{z}$
Syntactic	$+0.046$	$+0.225$
Mixed	$+0.015$	$+0.073$
Lexical	$+0.011$	$+0.056$
Grammar	$-0.004$	$-0.023$

Appendix

Table 10: RoBERTa score shift by edit-operation category.

Detector	Category	Core Signal	Detection Principle
Token-statistics-based Detectors
Log-likelihood	Statistical	Average likelihood	Uses mean token log-likelihood
Log-rank	Statistical	Token rank	Uses average token rank under the language model
Entropy	Statistical	Token entropy	Measures uncertainty in token predictions
GLTR	Statistical	Rank histogram	Analyzes token-rank distributions across the document
Zero-shot Detectors

Appendix

Table 11: AI detectors evaluated in this study. The detectors span three major detection paradigms and rely on different linguistic signals, enabling analysis of whether detector responses to professional editing vary across methodological families.

Detector	Public checkpoint	Fine-tuned
RoBERTa	openai-community/	No
	roberta-base-openai-detector
MAGE	yaful/MAGE	No
RADAR	TrustSafeAI/RADAR-Vicuna-7B	No

Appendix

Table 12: Classifier-based detector checkpoints. All models were loaded via AutoModelForSequenceClassification.from_pretrained() and evaluated in inference mode ( eval() with torch.no_grad() ), with no additional fine-tuning or adaptation on our dataset.

Detector	Raw score	Normalization
RoBERTa	Prob. $[0,1]$	As-is
RADAR	Prob. $[0,1]$	As-is
MAGE	Prob. $[0,1]$	As-is
Binoculars	CE ratio	$\sigma((r-1.0)\times 5.0)$
Fast-DetectGPT	Log-prob. diff.	$\sigma(\delta\times 10.0)$
Log-Likelihood	Mean log-lik.	$\sigma((\text{LL}+5.0)\times 2.0)$

Appendix

Table 13: Per-detector raw score and normalization to a

[0,1]

AI-likelihood scale;

\sigma(\cdot)

denotes the logistic sigmoid. Anchor values follow each detector’s original formulation or representative values reported in prior work.

Detector	Type	Orig Score		$\Delta$ orig		FP% (orig)		$\Delta$ FP
Detector	Type	pre	post	$\Delta\text{score}$	$d$	pre	post	$\Delta$ FP
Entropy	Stat.	0.76	0.79	$+$ 0.03	$+$ 0.42 ∗∗∗	99.6	99.9	$+$ 0.3
GLTR	Stat.	0.74	0.76	$+$ 0.02	$+$ 0.28 ∗∗∗	99.7	99.8	$+$ 0.1
Log-likelihood	Stat.	0.97	0.98	$+$ 0.01	$+$ 0.14 ∗∗∗	99.8	99.9	$+$ 0.1
Log-Rank	Stat.	0.98	0.98	$+$ 0.00	$+$ 0.17 ∗∗∗	100.0	100.0	$+$ 0.0
BiScope	Zero	0.35	0.31	$-$ 0.04	$-$ 0.50 ∗∗∗	1.9	1.1	$-$ 0.9

Appendix

Table 14: Comparison of original-text detector scores between the Pre-ChatGPT and Post-ChatGPT periods. Pre-ChatGPT corresponds to 2018–2022 (

n_{\text{pre}}=104{,}752

), and Post-ChatGPT corresponds to 2023–2025 (

n_{\text{post}}=30{,}637

). FPR color:

\geq

50% , 10–49% .

\Delta

color: decrease , increase .

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

{}^{ns}p\geq.05

Detector	Type	FP% (heavy)		Shift	Cohen’s $d$		$\Delta d$
Detector	Type	orig	edit	Shift	full	heavy	$\Delta d$
Entropy	Stat.	99.6	99.8	$+$ 0.025	$+$ 0.444	$+$ 0.707	$+$ 0.263
GLTR	Stat.	99.7	99.8	$+$ 0.017	$+$ 0.335	$+$ 0.576	$+$ 0.242
Log-Likelihood	Stat.	99.6	99.6	$+$ 0.007	$+$ 0.190	$+$ 0.197	$+$ 0.006
Log-Rank	Stat.	100.0	100.0	$+$ 0.002	$+$ 0.292	$+$ 0.299	$+$ 0.007
BiScope	Zero	2.2	1.3	$-$ 0.031	$-$ 0.488	$-$ 0.833	$-$ 0.345

Appendix

Table 15: Heavy-editing subset analysis (

r_{\text{edit}}>0.20

n=57{,}207

\Delta d=d_{\text{heavy}}-d_{\text{full}}

. All comparisons significant at

p<.001

. Type: Stat. = token-statistics-based; Zero = zero-shot-based; Cls. = classifier-based. FPR color:

\geq

50% , 10–49% . Color: decrease , increase .

		Original			Edited			$\Delta$
Detector	Type	Mean	FP%	F1	Mean	FP%	F1	$\Delta$ FP	$\Delta$ F1	$p$
Log-rank	Stat.	0.982	100.0	0.000	0.984	100.0	0.000	$+$ 0.0	—	ns
Log-likelihood	Stat.	0.981	99.8	0.003	0.985	99.9	0.003	$+$ 0.0	—	$<$ .05
Entropy	Stat.	0.781	99.8	0.003	0.798	99.9	0.002	$+$ 0.1	—	$<$ .001
GLTR	Stat.	0.754	99.9	0.002	0.769	99.9	0.002	$+$ 0.0	—	$<$ .01
Fast-DetectGPT	Zero	0.329	25.2	0.856	0.375	31.6	0.812	$+$ 6.4	$-$ .044	$<$ .001

Appendix

Table 16: Baseline false positive behavior on the Pre-ChatGPT dataset (2018–2022). FPR color:

\geq

50% , 10–49% .

		ALL ( $n{=}135{,}389$ )		Pre-ChatGPT ( $n{=}104{,}752$ )		Post-ChatGPT ( $n{=}30{,}637$ )
Detector	Type	$\Delta\text{score}$	$d$	$\Delta\text{score}$	$d$	$\Delta\text{score}$	$d$
Entropy	Stat.	$+$ 0.014 ∗∗∗	$+$ 0.40	$+$ 0.017 ∗∗∗	$+$ 0.44	$+$ 0.005 ∗∗∗	$+$ 0.21
GLTR	Stat.	$+$ 0.010 ∗∗∗	$+$ 0.30	$+$ 0.012 ∗∗∗	$+$ 0.34	$+$ 0.004 ∗∗∗	$+$ 0.16
Log-rank	Stat.	$+$ 0.001 ns	$+$ 0.24	$+$ 0.001 ∗∗∗	$+$ 0.29	$+$ 0.001 ∗∗∗	$+$ 0.09
Log-like	Stat.	$+$ 0.005 ns	$+$ 0.16	$+$ 0.006 ∗∗∗	$+$ 0.19	$+$ 0.002 ∗∗∗	$+$ 0.06
BiScope	Zero	$-$ 0.019 ∗∗∗	$-$ 0.45	$-$ 0.021 ∗∗∗	$-$ 0.49	$-$ 0.008 ∗∗∗	$-$ 0.22

Appendix

Table 17: Full score shift results following professional editing.

\Delta\text{score}=\text{score}_{\text{edit}}-\text{score}_{\text{orig}}

. Cohen’s

d

denotes effect size. Green = decrease; Red = increase.

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

{}^{ns}p\geq.05

		ALL ( $n{=}135{,}389$ )			Pre-ChatGPT ( $n{=}104{,}752$ )			Post-ChatGPT ( $n{=}30{,}637$ )
Detector	Type	Orig	Edit	$\Delta\text{FP}$	Orig	Edit	$\Delta\text{FP}$	Orig	Edit	$\Delta\text{FP}$
Log-rank	Stat.	100.0	100.0	$+$ 0.0	100.0	100.0	$+$ 0.0	100.0	100.0	$+$ 0.0
Log-like	Stat.	99.7	99.7	$+$ 0.0	99.6	99.7	$+$ 0.0	99.7	99.7	$+$ 0.0
Entropy	Stat.	99.8	99.9	$+$ 0.1	99.8	99.9	$+$ 0.1	99.9	99.9	$+$ 0.0
GLTR	Stat.	99.8	99.9	$+$ 0.0	99.8	99.9	$+$ 0.0	99.9	99.9	$+$ 0.0
BiScope	Zero	1.7	1.2	$-$ 0.5	1.9	1.3	$-$ 0.6	1.1	1.0	$-$ 0.1

Appendix

Table 18: Full false positive rate results following professional editing.

\Delta\text{FP}=\text{FP}_{\text{edit}}-\text{FP}_{\text{orig}}

(percentage-point change). FPR color:

\geq

50% , 10–49% .

\Delta\text{FP}

color: bias reduction , bias amplification .

Predictor	$\beta$
Grammar ratio	$-0.191^{***}$
Lexical ratio	$-0.131^{***}$
Edit-token ratio	$+0.001^{***}$
$R^{2}$	0.005

Appendix

Table 19: OLS regression of detector score shift on editing characteristics, across 280 editors.

{}^{***}p<.001

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 171 papers

Aug 6, 2026

Why AI Detection Fails for Academic Integrity

Institutions use commercial AI detectors for academic integrity, yet detectors cannot distinguish AI editing from full LLM drafts and may treat both as misconduct. In a controlled study of published English abstracts (four domains; 2013 to 2015 vs. 2023 to 2025), we quantify this policy failure under proxy human/AI labels at tau=0.50. Light "refine abstract only" edits, a proxy for guideline-compliant AI assistance, are flagged at 38 to 80%. Unmodified 2023 to 2025 originals are flagged at 9 to 15%, with non-STEM rates far above STEM (p<0.001); elevated scores track long-token and Academic Word List density, not authorship intent alone. After Undetectable AI humanization, evasion is near-total: fewer than 4% of AI-labeled rewrites remain flagged (post-humanization detection rate <4%; FNR >96%). Honest AI-editing results in a higher sanction risk than humanizer-assisted evasion. Therefore, detector scores should not serve as standalone misconduct evidence.

Jonathan A. Karr, Grigorii Khvatskii, Ting Hua +1

University of Notre Dame Notre Dame, Indiana, USA

AI-Generated Text Detection Academic Integrity Language Model Auditing

Ask PDF

May 11, 2026

The Impact of Editorial Intervention on Detecting Native Language Traces

Native Language Identification (NLI) is the task of determining an author's native language (L1) from their non-native writings. With the advent of human-AI co-authorship, non-native texts are routinely corrected and rewritten by large language models, fundamentally altering the linguistic features NLI models depend on. In this paper, we investigate the robustness of L1 traces across increasing degrees of editorial intervention. By processing 450 essays from the Write & Improve 2024 corpus through varying levels of grammatical error correction (GEC) and paraphrasing, we demonstrate that L1 attribution does not entirely depend on surface-level errors. Instead, the detection models leverage deeper L1 features: unidiomatic lexico-semantic choices, pragmatic transfer, and the author's underlying cultural perspective. We find that minimal edits preserve these structural traces and maintain high profiling accuracy. In contrast, fluency edits and paraphrasing normalize these L1 features, leading to a severe degradation in performance.

Ahmet Yavuz Uluslu, Mark Gales, Kate Knill +1

University of Cambridge · University of Zurich

Topic pending

Ask PDF

Sep 26, 2026

MixDetect: Word-Level Localization and Quantification of AI Editing

Large language models are increasingly used to edit human-written text rather than generate entire texts from scratch. Conventional AI-text detectors mainly distinguish human-written from fully AI-generated text, while recent methods for AI-edited text typically provide only a text-level label or editing-degree score. We introduce MixDetect, a word-level framework for localizing and quantifying AI editing. MixDetect separately predicts whether each word has been edited and, conditional on editing, how substantial the edit is, allowing editing scope and editing intensity to be estimated separately. During training, source--edited pairs are aligned to construct word-level supervision, while inference requires only the input text. Experiments show that MixDetect accurately localizes AI-edited words, reflects differences in editing intensity, and reveals different scope--intensity patterns across editing degrees and operations. The overall AI editing magnitude increases under additional AI editing, decreases when AI-generated text is edited by humans, and remains nearly unchanged under ordinary human-to-human editing. The aggregated text-level predictions also perform well on binary and ternary AI-text classification and remain effective under domain and generator shifts. These results show that AI editing can be analyzed beyond a single authorship label or editing-degree score by identifying both where AI editing occurs and how substantial the edits are.

Hongrui Bao, Yubing Ren, Zhendong Pan +3

University of the Chinese Academy of Sciences · Institute of Information Engineering · Shandong University +1

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Ask PDF

Abstract

AI-Generated Text Detection

Explore similar work

Why AI Detection Fails for Academic Integrity

The Impact of Editorial Intervention on Detecting Native Language Traces

MixDetect: Word-Level Localization and Quantification of AI Editing