Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

Shushanta Pudasaini; Luis Miralles-Pechuán; David Lillis; Marisa Llorens Salvador

cs.CLMar 24, 2026

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

Authors: Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis, Marisa Llorens Salvador

Organizations: Technological University Dublin · University College Dublin

Abstract

The widespread adoption of Large Language Models (LLMs) has made the detection of AI-Generated text a pressing and complex challenge. Although many detection systems report high benchmark accuracy, their reliability in real-world settings remains uncertain, and their interpretability is often unexplored. In this work, we investigate whether contemporary detectors genuinely identify machine authorship or merely exploit dataset-specific artefacts. We propose an interpretable detection framework that integrates linguistic feature engineering, machine learning, and explainable AI techniques. When evaluated on two prominent benchmark corpora, namely PAN CLEF 2025 and COLING 2025, our model trained on 30 linguistic features achieves leaderboard-competitive performance, attaining an F1 score of 0.9734. However, systematic cross-domain and cross-generator evaluation reveals substantial generalisation failure: classifiers that excel in-domain degrade significantly under distribution shift. Using SHAP- based explanations, we show that the most influential features differ markedly between datasets, indicating that detectors often rely on dataset-specific stylistic cues rather than stable signals of machine authorship. Further investigation with in-depth error analysis exposes a fundamental tension in linguistic-feature-based AI text detection: the features that are most discriminative on in-domain data are also the features most susceptible to domain shift, formatting variation, and text-length effects. We believe that this knowledge helps build AI detectors that are robust across different settings. To support replication and practical use, we release an open-source Python package that returns both predictions and instance-level explanations for individual texts.

Figures & tables

Feature	Description	How It Is Calculated
Surface and Length Features
Character Count	Total characters in the document.	Count all characters, including spaces.
Word Count	Total word tokens.	Split on whitespace and count tokens.
Sentence Count	Number of Sentences.	Sentence tokenisation count.
Paragraph Count	Number of paragraphs.	Count non-empty segments split by newlines.
Punctuation Count	Total punctuation symbols.	Count standard punctuation characters.

Table 1: Linguistic feature set with operational definitions and computational specifications. Sentence tokenisation uses NLTK Punkt [ 25 ] ; POS tagging, lemmatisation, and dependency parsing use spaCy en_core_web_sm [ 26 ] ; grammar checking uses LanguageTool v5.7 [ 27 ] ; sentiment analysis uses TextBlob [ 28 ] ; TF-IDF vectorisation uses scikit-learn [ 29 ] .

Attribute	PAN CLEF Dataset	COLING Dataset	Ghostbuster Dataset
Task Name	Voight–Kampff Generative AI Detection 2025 (Subtask 1)	GenAI Content Detection Task 1 (English & Multilingual)	Research benchmark (NAACL 2024)
Domains Considered	Essays, news, fiction	Peer reviews, student essays, scientific papers, news, and 8+ additional domains	Student essays, creative writing, news
Training Data Size	23,707 samples (9,101 human / 14,606 AI)	610,767 English samples (228,922 human / 381,845 machine)	Not used for training purposes
Dev / Validation Size	3,589 samples (1,277 human / 2,312 AI)	261,758 English samples (98,328 human / 163,430 machine)	Same-prompt splits; 6,000 text(2000 human-written,2000 generated with GPT and 2000 generated with Claude); used only for evaluation
Generators	GPT-4o and participant models with obfuscation	GPT-4/4o, Mistral, Llama 3.1, Qwen-2, Claude, others	GPT-3.5-turbo (training); Claude (generalisation)
Evaluation Metrics	ROC–AUC, Brier, C@1, F1, F0.5u, mean score, FPR/FNR	Macro-F1 (official), Accuracy, Micro-F1	F1 score, Accuracy

Table 2: Comparison of the benchmark datasets used for the experiment with their data size and diversity details.

Feature	PAN CLEF		COLING		Ghostbuster
	Human	AI	Human	AI	Human	AI
Character Count	$3941.826$	$3580.483$	$1527.844$	$1449.026$	$3535.748$	$3080.883$
Word Count	$695.456$	$574.973$	$259.964$	$237.821$	$586.937$	$475.910$
Paragraph Count	$26.409$	$17.884$	$7.116$	$3.786$	$12.587$	$9.107$
Stopword Count	$326.233$	$239.586$	$110.491$	$102.020$	$256.590$	$191.907$
Unique Word Count	$381.030$	$332.886$	$109.093$	$104.482$	$215.613$	$192.170$

Table 3: Average values of selected linguistic features for human-written and AI-generated texts across the PAN CLEF, COLING, and Ghostbuster benchmark datasets.

Figure 1: t-SNE projections of linguistic feature representations for the COLING dataset (left) and PAN CLEF dataset (right). Human-written samples are shown in purple and AI-generated samples in yellow.

Figure 2: Overview of the experimental pipeline, illustrating dataset usage, linguistic feature extraction, model training and evaluation, ensemble construction, and post-hoc interpretability using SHAP.

Algorithm	F1-score
mdok of KInIT [ 37 ]	98.90
Optimised Feature Set + SVC (Ours)	97.34
Optimised Feature Set + XGBoost (Ours)	96.94
Optimised Feature Set + Random Forest (Ours)	95.82
Optimised Feature Set + Logistic Regression (Ours)	95.48
Baseline TF-IDF SVM [ 5 ]	90.40

Table 4: Comparison of F-score performance on the Voight–Kampff Generative AI Detection 2025 leaderboard (F-score scaled to 0–100).

Model	Accuracy	Precision	Recall	F1 Score	FPR	FNR
XGBoost	96.04	96.45	97.45	96.94	6.50	2.55
Random Forest	94.51	94.12	97.58	95.82	11.04	2.42
Logistic Regression	94.09	94.19	96.80	95.48	10.08	3.20
SVM	96.55	96.71	97.97	97.34	6.03	2.03

Table 5: Comparison of machine learning model performance on the AI-generated text detection dataset using cross-validation. Reported metrics include Accuracy, Precision, Recall, F1-score, False Positive Rate (FPR), and False Negative Rate (FNR). The best-performing model across most evaluation metrics is highlighted in bold. All values are scaled from 0 to 100.

Figure 3: Confusion matrices for the four classifiers evaluated on the PAN CLEF in-domain test set. Rows indicate true labels and columns predicted labels. False positives (Human $\rightarrow$ AI) represent potential false accusations, while false negatives (AI $\rightarrow$ Human) represent missed detections. Despite similar F1 scores, models differ in their error distribution, illustrating why benchmark accuracy alone does not fully capture deployment risk.

Model	Accuracy	Precision	Recall	F1 Score	ROC-AUC
SVM	0.9697 $\pm$ 0.0045	0.9731 $\pm$ 0.0051	0.9778 $\pm$ 0.0036	0.9754 $\pm$ 0.0036	0.9942 $\pm$ 0.0015
XGBoost	0.9678 $\pm$ 0.0033	0.9717 $\pm$ 0.0028	0.9762 $\pm$ 0.0034	0.9739 $\pm$ 0.0027	0.9943 $\pm$ 0.0009
Random Forest	0.9505 $\pm$ 0.0044	0.9500 $\pm$ 0.0037	0.9707 $\pm$ 0.0044	0.9602 $\pm$ 0.0035	0.9878 $\pm$ 0.0019
Logistic Regression	0.9431 $\pm$ 0.0043	0.9490 $\pm$ 0.0048	0.9591 $\pm$ 0.0050	0.9540 $\pm$ 0.0034	0.9826 $\pm$ 0.0015

Table 6: Comparison of machine learning model performance using cross-validation. Reported metrics include Accuracy, Precision, Recall, F1 Score, and ROC-AUC. The best-performing model across most evaluation metrics is highlighted in bold. Values are reported as mean

\pm

standard deviation.

Comparison	$\Delta$ F1	t-test p-value	Significant?
SVM vs. XGBoost	+0.0015	0.0984	No
SVM vs. Random Forest	+0.0152	$<0.0001$	Yes
SVM vs. Logistic Regression	+0.0214	$<0.0001$	Yes
XGBoost vs. Random Forest	+0.0137	$<0.0001$	Yes
XGBoost vs. Logistic Regression	+0.0199	$<0.0001$	Yes
Random Forest vs. Logistic Regression	+0.0062	$<0.0001$	Yes

Table 7: Statistical significance testing of F1-score differences between machine learning models using paired t-test.

\Delta

F1 indicates the difference in mean F1-score between the compared models. Statistical significance is determined at

\alpha=0.05

Train Set	Algorithm	PAN CLEF	COLING
PAN CLEF	SVC	97.34	76.59
	XGBoost	96.94	67.23
	Random Forest	95.82	74.86
	Logistic Regression	95.48	76.53
COLING	SVC	78.05	76.79
	XGBoost	78.56	78.53

Table 8: Cross-dataset generalisation results on PAN CLEF and COLING test sets (F1 scaled to 1–100).

LLM	Best-PAN	Best-COL
GPT-5.2	23.25	3.42
DeepSeek V3.2	0.76	3.51
Gemini 3 Pro	0.43	18.40

Table 9: Cross-generator generalisation performance on unseen LLMs, reported as False Negative Rate (FNR, %). Best-PAN refers to the SVM model achieving the highest F1-score when trained and evaluated on the PAN CLEF dataset, while Best-COL denotes the Random Forest model achieving the highest F1-score when trained and evaluated on the COLING dataset.

Subset	Best-PAN	Best-COL	Ens-PAN	Ens-COL
WP	60.15	59.36	92.87	77.77
Reuters	55.80	85.59	96.88	90.98
Essay	76.07	90.83	94.09	92.64
Average	64.0	78.59	94.61	87.13

Table 10: Ensemble performance on the Ghostbuster dataset (F1-score). Ens-PAN and Ens-COL denote ensemble models combining SVM, Random Forest, XGBoost, and Logistic Regression, trained on the PAN CLEF and COLING datasets, respectively.

Figure 4: Feature importance obtained while training an XGBoost model on different datasets.

Panel of Figure 5 — Figure 5: SHAP summary plots showing feature importance distributions for different datasets.

Type	Count	Words (Mean)	Words (Median)	Paragraph (Mean)	Paragraph (Median)	GZIP (Mean)	GZIP (Median)
FN	139	177.48	183	13.24	3	0.724	0.557
FP	2753	221.48	171	4.13	1	0.574	0.548
TN	967	421.03	212	24.69	17	0.628	0.518
TP	6141	242.89	198	5.36	1	0.535	0.510

Table 11: Descriptive statistics for text classification categories, including mean and median values for word count, paragraph count, and GZIP compression ratio. This table highlights structural differences between true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN).

Panel of Figure 6 — Figure 6: SHAP waterfall plots illustrating feature contributions for human-written and AI-generated samples.

Panel of Figure 7 — Figure 7: SHAP bar plots showing average feature impact for human-written and AI-generated samples.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 167 papers

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions
Mar 18, 2026Madhav S. Baidya, S. S. Baidya, Chirag ChawlaAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India
A Systematic Analysis of Linguistic Features in AI-Generated Text Detection Across Domains and Models
Jun 2, 2026Yassir El Attar, Esra Dönmez, Maximilian Maurer +1AI-Generated Text Detection Stylometry
Institute for Natural Language Processing, University of Stuttgart · Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart · GESIS Leibniz Institute for the Social Sciences +1
Automatic detection of Gen-AI texts: A comparative framework of neural models
Mar 19, 2026Cristian Buttaro, Irene AmeriniAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Sapienza University in Rome Latina

Mar 18, 2026J/K move · Enter open · S save

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

Madhav S. Baidya, S. S. Baidya, Chirag Chawla

Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.

Abstract

AI-Generated Text Detection

Explore similar work

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

A Systematic Analysis of Linguistic Features in AI-Generated Text Detection Across Domains and Models

Automatic detection of Gen-AI texts: A comparative framework of neural models