Style as a Confound: False Positives in AI Detection of Non-Native Academic Writing

Hyeonchu Park; Gahye Jeong; Bugeun Kim

cs.AIAug 27, 2026

Style as a Confound: False Positives in AI Detection of Non-Native Academic Writing

Authors: Hyeonchu Park, Gahye Jeong, Bugeun Kim

Organizations: Department of Artificial Intelligence, Chung-Ang University, Republic of Korea · Wordvice, Republic of Korea

Abstract

AI text detectors are increasingly employed in academic settings, but it remains unclear whether their outputs reflect AI authorship itself or broader linguistic features associated with polished academic English. Previous studies have reported high false-positive rates (FPRs) for non-native English writing, but population-level comparisons confound authorship with differences in topic, domain, and writing style. Professional editing provides a useful setting for examining this issue because it changes the linguistic form of manuscripts while preserving authorship and content. We examined 135,389 document pairs from a professional English editing service (2018-2025), comprising non-native manuscripts and their native-edited versions, to assess how editing affects detector responses controlling for content and authorship. For the 13 AI text detectors, FPRs for human-written texts varied widely, from 0.0% to 100.0%. Responses varied across detectors: the same edits increased AI scores in some detectors but decreased them in others. Notably, score changes correlated with the extent of editing. The findings identify professional editing style as a key confounding variable in AI detector outputs, rather than establishing a full separation of text origin from linguistic style, raising concerns about fairness and reliability in academic settings.

Figures & tables

Statistic	Value
Total document pairs	135,389
Mean word count (original)	2,355
Median word count (original)	950
Std. dev. (word count)	3,038
Subject areas	40+
Docs. w/ 500–5,000 words	$\sim$ 57%

Table 1: Summary dataset statistics. Full year-by-year and service-type breakdown in Appendix B

Metric	Orig.	Edited	$\Delta$
Mean word length	5.196	5.252	$+0.055$
Type–token ratio	0.407	0.412	$+0.005$
Mean sentence length	24.35	23.51	$-0.84$
Flesch–Kincaid grade	14.51	14.42	$-0.09$

Table 2: Linguistic characteristics of professional editing (6,000-pair subsample). Edit intensity: median 196 / mean 463 edit units per document.

Detector	FP%		$\Delta$ FP	$p$
Detector	orig	edit	$\Delta$ FP	$p$
Token-statistics-based
Log-rank	100.0	100.0	$+$ 0.0	ns
Log-likelihood	99.8	99.9	$+$ 0.0	$<$ .05
Entropy	99.8	99.9	$+$ 0.1	$<$ .001
GLTR	99.9	99.9	$+$ 0.0	$<$ .01

Table 3: Baseline false positive behavior on the Pre-ChatGPT dataset (2018–2022). FPR color:

\geq

50% , 10–49% .

\Delta

FP color: bias reduction , bias amplification . Full results in Appendix D.3

		ALL		Pre-ChatGPT		Post-ChatGPT
Detector	Type	$\Delta\text{score}$	$\Delta\text{FP}$	$\Delta\text{score}$	$\Delta\text{FP}$	$\Delta\text{score}$	$\Delta\text{FP}$
Entropy	Stat.	$+$ 0.014 ∗∗∗	$+$ 0.1	$+$ 0.017 ∗∗∗	$+$ 0.1	$+$ 0.005 ∗∗∗	$+$ 0.0
GLTR	Stat.	$+$ 0.010 ∗∗∗	$+$ 0.0	$+$ 0.012 ∗∗∗	$+$ 0.0	$+$ 0.004 ∗∗∗	$+$ 0.0
Log-rank	Stat.	$+$ 0.001 ns	$+$ 0.0	$+$ 0.001 ∗∗∗	$+$ 0.0	$+$ 0.001 ∗∗∗	$+$ 0.0
Log-like	Stat.	$+$ 0.005 ns	$+$ 0.0	$+$ 0.006 ∗∗∗	$+$ 0.0	$+$ 0.002 ∗∗∗	$+$ 0.0
BiScope	Zero	$-$ 0.019 ∗∗∗	$-$ 0.5	$-$ 0.021 ∗∗∗	$-$ 0.6	$-$ 0.008 ∗∗∗	$-$ 0.1

Table 4: Effect of professional editing on AI detector scores and false positive rates.

\Delta\text{score}=\text{score}_{\text{edit}}-\text{score}_{\text{orig}}

(mean score shift);

\Delta\text{FP}=\text{FP}_{\text{edit}}-\text{FP}_{\text{orig}}

. Green = decrease; Red = increase.

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

{}^{ns}p\geq.05

. Full results in Appendix D.3

Detector	Type	$\rho$	$R^{2}$	Q1	Q2	Q3	Q4	Q5
Entropy	Stat.	$+$ 0.343 ∗∗∗	0.037	$-$ .009	$+$ .015	$+$ .020	$+$ .027	$+$ .032
Log-rank	Stat.	$+$ 0.293 ∗∗∗	0.012	$+$ .001	$+$ .001	$+$ .002	$+$ .002	$+$ .002
Log-likelihood	Stat.	$+$ 0.287 ∗∗∗	0.005	$+$ .003	$+$ .005	$+$ .006	$+$ .007	$+$ .008
GLTR	Stat.	$+$ 0.257 ∗∗∗	0.023	$+$ .006	$+$ .011	$+$ .013	$+$ .016	$+$ .019
BiScope	Zero	$-$ 0.418 ∗∗∗	0.041	$+$ .009	$-$ .019	$-$ .026	$-$ .033	$-$ .040
Binoculars	Zero	$-$ 0.134 ∗∗∗	0.019	$+$ .028	$-$ .002	$-$ .003	$-$ .004	$-$ .004

Table 5: Editing intensity and detector score shifts. Spearman

\rho

denotes the correlation between edit-token ratio and

\Delta\text{score}

. Q1–Q5 represent quintiles of editing intensity (Q1 = weakest, Q5 = strongest). Green = score decreases; Red = score increases.

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

Feature	Statistical	LLM-based	Supervised
Grammar ratio	$+0.111$	$+0.019$	$-0.005$
Lexical ratio	$+0.042$	$-0.018$	$-0.020$
Syntactic ratio	$+0.101$	$+0.013$	$-0.040$
Edit-token ratio	$+0.286$	$+0.022$	$-0.089$
Orig. word count	$-0.099$	$-0.002$	$+0.067$
# edit units	$+0.116$	$+0.021$	$+0.005$

Table 6: Spearman correlation (

\rho

) between editing characteristics and detector score shifts, by detector family (pre-ChatGPT subset).

Metric	Statistical	LLM-based	Supervised
TTR	$-0.063$	$-0.063$	$-0.073$
Word length	$-0.022$	$-0.046$	$-0.082$
Function-word ratio	$+0.071$	$+0.068$	$+0.064$
Sentence length	$-0.043$	$+0.005$	$+0.017$

Table 7: Spearman correlation between changes in complexity and detector score shifts, by detector family.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Overall Scale
Total order pairs	135,389
Collection period	2018–2025 (8 years)
Era Split
Pre-ChatGPT (2018–2022)	104,752 (77.4%)
Post-ChatGPT (2023–2025)	30,637 (22.6%)

Appendix

Table 8: Dataset Statistics . The pre-ChatGPT / post-ChatGPT split is defined by the public release of ChatGPT (November 2022); orders from 2023 onward are classified as post-ChatGPT.

Edit type	Share
Mixed	66.1%
Grammar	15.8%
Lexical	14.0%
Syntactic	0.1%
no-edit	0.4%

Appendix

Table 9: Distribution of edit-operation categories.

Edit type	$\Delta$ score	Cohen’s $d_{z}$
Syntactic	$+0.046$	$+0.225$
Mixed	$+0.015$	$+0.073$
Lexical	$+0.011$	$+0.056$
Grammar	$-0.004$	$-0.023$

Appendix

Table 10: RoBERTa score shift by edit-operation category.

Detector	Category	Core Signal	Detection Principle
Token-statistics-based Detectors
Log-likelihood	Statistical	Average likelihood	Uses mean token log-likelihood
Log-rank	Statistical	Token rank	Uses average token rank under the language model
Entropy	Statistical	Token entropy	Measures uncertainty in token predictions
GLTR	Statistical	Rank histogram	Analyzes token-rank distributions across the document
Zero-shot Detectors

Appendix

Table 11: AI detectors evaluated in this study. The detectors span three major detection paradigms and rely on different linguistic signals, enabling analysis of whether detector responses to professional editing vary across methodological families.

Detector	Public checkpoint	Fine-tuned
RoBERTa	openai-community/	No
	roberta-base-openai-detector
MAGE	yaful/MAGE	No
RADAR	TrustSafeAI/RADAR-Vicuna-7B	No

Appendix

Table 12: Classifier-based detector checkpoints. All models were loaded via AutoModelForSequenceClassification.from_pretrained() and evaluated in inference mode ( eval() with torch.no_grad() ), with no additional fine-tuning or adaptation on our dataset.

Detector	Raw score	Normalization
RoBERTa	Prob. $[0,1]$	As-is
RADAR	Prob. $[0,1]$	As-is
MAGE	Prob. $[0,1]$	As-is
Binoculars	CE ratio	$\sigma((r-1.0)\times 5.0)$
Fast-DetectGPT	Log-prob. diff.	$\sigma(\delta\times 10.0)$
Log-Likelihood	Mean log-lik.	$\sigma((\text{LL}+5.0)\times 2.0)$

Appendix

Table 13: Per-detector raw score and normalization to a

[0,1]

AI-likelihood scale;

\sigma(\cdot)

denotes the logistic sigmoid. Anchor values follow each detector’s original formulation or representative values reported in prior work.

Detector	Type	Orig Score		$\Delta$ orig		FP% (orig)		$\Delta$ FP
Detector	Type	pre	post	$\Delta\text{score}$	$d$	pre	post	$\Delta$ FP
Entropy	Stat.	0.76	0.79	$+$ 0.03	$+$ 0.42 ∗∗∗	99.6	99.9	$+$ 0.3
GLTR	Stat.	0.74	0.76	$+$ 0.02	$+$ 0.28 ∗∗∗	99.7	99.8	$+$ 0.1
Log-likelihood	Stat.	0.97	0.98	$+$ 0.01	$+$ 0.14 ∗∗∗	99.8	99.9	$+$ 0.1
Log-Rank	Stat.	0.98	0.98	$+$ 0.00	$+$ 0.17 ∗∗∗	100.0	100.0	$+$ 0.0
BiScope	Zero	0.35	0.31	$-$ 0.04	$-$ 0.50 ∗∗∗	1.9	1.1	$-$ 0.9

Appendix

Table 14: Comparison of original-text detector scores between the Pre-ChatGPT and Post-ChatGPT periods. Pre-ChatGPT corresponds to 2018–2022 (

n_{\text{pre}}=104{,}752

), and Post-ChatGPT corresponds to 2023–2025 (

n_{\text{post}}=30{,}637

). FPR color:

\geq

50% , 10–49% .

\Delta

color: decrease , increase .

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

{}^{ns}p\geq.05

Detector	Type	FP% (heavy)		Shift	Cohen’s $d$		$\Delta d$
Detector	Type	orig	edit	Shift	full	heavy	$\Delta d$
Entropy	Stat.	99.6	99.8	$+$ 0.025	$+$ 0.444	$+$ 0.707	$+$ 0.263
GLTR	Stat.	99.7	99.8	$+$ 0.017	$+$ 0.335	$+$ 0.576	$+$ 0.242
Log-Likelihood	Stat.	99.6	99.6	$+$ 0.007	$+$ 0.190	$+$ 0.197	$+$ 0.006
Log-Rank	Stat.	100.0	100.0	$+$ 0.002	$+$ 0.292	$+$ 0.299	$+$ 0.007
BiScope	Zero	2.2	1.3	$-$ 0.031	$-$ 0.488	$-$ 0.833	$-$ 0.345

Appendix

Table 15: Heavy-editing subset analysis (

r_{\text{edit}}>0.20

n=57{,}207

\Delta d=d_{\text{heavy}}-d_{\text{full}}

. All comparisons significant at

p<.001

. Type: Stat. = token-statistics-based; Zero = zero-shot-based; Cls. = classifier-based. FPR color:

\geq

50% , 10–49% . Color: decrease , increase .

		Original			Edited			$\Delta$
Detector	Type	Mean	FP%	F1	Mean	FP%	F1	$\Delta$ FP	$\Delta$ F1	$p$
Log-rank	Stat.	0.982	100.0	0.000	0.984	100.0	0.000	$+$ 0.0	—	ns
Log-likelihood	Stat.	0.981	99.8	0.003	0.985	99.9	0.003	$+$ 0.0	—	$<$ .05
Entropy	Stat.	0.781	99.8	0.003	0.798	99.9	0.002	$+$ 0.1	—	$<$ .001
GLTR	Stat.	0.754	99.9	0.002	0.769	99.9	0.002	$+$ 0.0	—	$<$ .01
Fast-DetectGPT	Zero	0.329	25.2	0.856	0.375	31.6	0.812	$+$ 6.4	$-$ .044	$<$ .001

Appendix

Table 16: Baseline false positive behavior on the Pre-ChatGPT dataset (2018–2022). FPR color:

\geq

50% , 10–49% .

		ALL ( $n{=}135{,}389$ )		Pre-ChatGPT ( $n{=}104{,}752$ )		Post-ChatGPT ( $n{=}30{,}637$ )
Detector	Type	$\Delta\text{score}$	$d$	$\Delta\text{score}$	$d$	$\Delta\text{score}$	$d$
Entropy	Stat.	$+$ 0.014 ∗∗∗	$+$ 0.40	$+$ 0.017 ∗∗∗	$+$ 0.44	$+$ 0.005 ∗∗∗	$+$ 0.21
GLTR	Stat.	$+$ 0.010 ∗∗∗	$+$ 0.30	$+$ 0.012 ∗∗∗	$+$ 0.34	$+$ 0.004 ∗∗∗	$+$ 0.16
Log-rank	Stat.	$+$ 0.001 ns	$+$ 0.24	$+$ 0.001 ∗∗∗	$+$ 0.29	$+$ 0.001 ∗∗∗	$+$ 0.09
Log-like	Stat.	$+$ 0.005 ns	$+$ 0.16	$+$ 0.006 ∗∗∗	$+$ 0.19	$+$ 0.002 ∗∗∗	$+$ 0.06
BiScope	Zero	$-$ 0.019 ∗∗∗	$-$ 0.45	$-$ 0.021 ∗∗∗	$-$ 0.49	$-$ 0.008 ∗∗∗	$-$ 0.22

Appendix

Table 17: Full score shift results following professional editing.

\Delta\text{score}=\text{score}_{\text{edit}}-\text{score}_{\text{orig}}

. Cohen’s

d

denotes effect size. Green = decrease; Red = increase.

{}^{***}p<.001

{}^{**}p<.01

{}^{*}p<.05

{}^{ns}p\geq.05

		ALL ( $n{=}135{,}389$ )			Pre-ChatGPT ( $n{=}104{,}752$ )			Post-ChatGPT ( $n{=}30{,}637$ )
Detector	Type	Orig	Edit	$\Delta\text{FP}$	Orig	Edit	$\Delta\text{FP}$	Orig	Edit	$\Delta\text{FP}$
Log-rank	Stat.	100.0	100.0	$+$ 0.0	100.0	100.0	$+$ 0.0	100.0	100.0	$+$ 0.0
Log-like	Stat.	99.7	99.7	$+$ 0.0	99.6	99.7	$+$ 0.0	99.7	99.7	$+$ 0.0
Entropy	Stat.	99.8	99.9	$+$ 0.1	99.8	99.9	$+$ 0.1	99.9	99.9	$+$ 0.0
GLTR	Stat.	99.8	99.9	$+$ 0.0	99.8	99.9	$+$ 0.0	99.9	99.9	$+$ 0.0
BiScope	Zero	1.7	1.2	$-$ 0.5	1.9	1.3	$-$ 0.6	1.1	1.0	$-$ 0.1

Appendix

Table 18: Full false positive rate results following professional editing.

\Delta\text{FP}=\text{FP}_{\text{edit}}-\text{FP}_{\text{orig}}

(percentage-point change). FPR color:

\geq

50% , 10–49% .

\Delta\text{FP}

color: bias reduction , bias amplification .

Predictor	$\beta$
Grammar ratio	$-0.191^{***}$
Lexical ratio	$-0.131^{***}$
Edit-token ratio	$+0.001^{***}$
$R^{2}$	0.005

Appendix

Table 19: OLS regression of detector score shift on editing characteristics, across 280 editors.

{}^{***}p<.001

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 171 papers

Why AI Detection Fails for Academic Integrity
Aug 6, 2026Jonathan A. Karr, Grigorii Khvatskii, Ting Hua +1AI-Generated Text Detection Academic Integrity
University of Notre Dame Notre Dame, Indiana, USA
The Impact of Editorial Intervention on Detecting Native Language Traces
May 11, 2026Ahmet Yavuz Uluslu, Mark Gales, Kate Knill +1
University of Cambridge · University of Zurich
MixDetect: Word-Level Localization and Quantification of AI Editing
Sep 26, 2026Hongrui Bao, Yubing Ren, Zhendong Pan +3AI-Generated Text Detection Robustness of Machine-Generated Text Detection
University of the Chinese Academy of Sciences · Institute of Information Engineering · Shandong University +1

Aug 6, 2026J/K move · Enter open · S save

Why AI Detection Fails for Academic Integrity

Jonathan A. Karr, Grigorii Khvatskii, Ting Hua, Nitesh V. Chawla

University of Notre Dame Notre Dame, Indiana, USA

AI-Generated Text Detection Academic Integrity Language Model Auditing

Open paper AskPDF arXiv

Institutions use commercial AI detectors for academic integrity, yet detectors cannot distinguish AI editing from full LLM drafts and may treat both as misconduct. In a controlled study of published English abstracts (four domains; 2013 to 2015 vs. 2023 to 2025), we quantify this policy failure under proxy human/AI labels at tau=0.50. Light "refine abstract only" edits, a proxy for guideline-compliant AI assistance, are flagged at 38 to 80%. Unmodified 2023 to 2025 originals are flagged at 9 to 15%, with non-STEM rates far above STEM (p<0.001); elevated scores track long-token and Academic Word List density, not authorship intent alone. After Undetectable AI humanization, evasion is near-total: fewer than 4% of AI-labeled rewrites remain flagged (post-humanization detection rate <4%; FNR >96%). Honest AI-editing results in a higher sanction risk than humanizer-assisted evasion. Therefore, detector scores should not serve as standalone misconduct evidence.

Abstract

AI-Generated Text Detection

Explore similar work

Why AI Detection Fails for Academic Integrity

The Impact of Editorial Intervention on Detecting Native Language Traces

MixDetect: Word-Level Localization and Quantification of AI Editing