Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

Madhav S. Baidya; S. S. Baidya; Chirag Chawla

cs.CLMar 18, 2026

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

Authors: Madhav S. Baidya, S. S. Baidya, Chirag Chawla

Organizations: Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India

Abstract

The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.

Figures & tables

Figure 1: Overview of the benchmark pipeline. Stage 0 constructs two paired corpora ( HC3 : 23k human–ChatGPT pairs; ELI5 : 15k human–Mistral-7B pairs) with length-matched preprocessing. Stage 1 evaluates three detector families: Family 1 (classical statistical classifiers), and Family 2 (fine-tuned encoder transformers — BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3; 1D-CNN; perplexity-based detectors; stylometric-hybrid XGBoost), and Family 3 ( llm -as-detector prompting at four scales including GPT-4o-mini). Stage 2 evaluates cross- llm generalization via neural detectors, embedding-space classifier matrices , and distributional shift analysis. Stage 3 applies adversarial humanization at three levels (L0–L2) using an instruction-tuned rewriter. All families are evaluated under a unified five-metric suite ( auroc , auprc , eer , Brier Score, FPR@95%TPR).

Domain	Unique Pairs
reddit_eli5	16,153
finance	3,933
medicine	1,248
open_qa	1,187
wiki_csai	842
Total	23,363

Table 1: Domain distribution of the HC3 corpus after flattening and deduplication.

Parameter	Value
Model	mistralai/Mistral-7B-Instruct-v0.2
Precision	FP16 (no quantization)
Attention	Flash Attention 2
Compilation	torch.compile (reduce-overhead)
Batch size	48
Max new tokens	150

Table 2: Mistral-7B generation configuration for ELI5 augmentation.

Dataset	Train	Test
HC3	36,968 (18,484 per class)	9,242
ELI5	22,862 (11,431 per class)	5,716

Table 3: Train/test split sizes after length matching and stratification.

Condition	auroc	Brier	Log Loss	Mean Human	Mean llm
hc3_to_hc3	0.8882	0.1334	0.4411	0.2838	0.7319
hc3_to_eli5	0.7406	0.2116	0.6474	0.4246	0.6508
eli5_to_eli5	0.8446	0.1605	0.4909	0.3251	0.6760
eli5_to_hc3	0.7429	0.2496	0.9063	0.2006	0.4580

Table 4: Logistic Regression results across evaluation conditions.

Condition	auroc	Brier	Log Loss	Mean Human	Mean llm
hc3_to_hc3	0.9767	0.0679	0.2438	0.1889	0.8173
hc3_to_eli5	0.7829	0.1922	0.5815	0.3830	0.6086
eli5_to_eli5	0.9618	0.0869	0.3014	0.2348	0.7811
eli5_to_hc3	0.6337	0.3193	1.1636	0.1643	0.2903

Table 5: Random Forest results across evaluation conditions.

Condition	auroc	Brier	Log Loss	Mean Human	Mean llm
hc3_to_hc3	0.7993	0.1835	0.5486	0.3700	0.6318
hc3_to_eli5	0.6933	0.2348	0.6639	0.5196	0.6686
eli5_to_eli5	0.7924	0.1857	0.5512	0.3740	0.6287
eli5_to_hc3	0.5992	0.3169	1.5852	0.2083	0.3191

Table 6: SVM (RBF Kernel) results across evaluation conditions.

Figure 2: Calibration curves for classical detectors across four evaluation settings. Points close to the diagonal indicate well-calibrated confidence scores, while systematic deviations reflect over- or under-confidence.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm	Sep.
hc3_to_hc3	0.9947	0.9041	0.0906	0.5747	0.1927	0.9999	0.8071
hc3_to_eli5	0.9489	0.8396	0.1472	0.8720	0.2319	0.9147	0.6828
eli5_to_eli5	0.9943	0.9388	0.0572	0.3315	0.1245	0.9996	0.8751
eli5_to_hc3	0.9083	0.8548	0.1393	0.8719	0.2100	0.9170	0.7070

Table 7: BERT ( bert-base-uncased ) results.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm	Sep.
hc3_to_hc3	0.9994	0.9679	0.0303	0.2204	0.0642	1.0000	0.9357
hc3_to_eli5	0.9741	0.7967	0.1926	1.4401	0.4054	0.9991	0.5937
eli5_to_eli5	0.9998	0.9645	0.0331	0.2264	0.0711	0.9999	0.9289
eli5_to_hc3	0.9657	0.9045	0.0932	0.7082	0.1129	0.9214	0.8085

Table 8: RoBERTa ( roberta-base ) results.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm	Sep.
hc3_to_hc3	0.9972	0.8639	0.1298	0.8663	0.2731	0.9996	0.7265
hc3_to_eli5	0.9597	0.8492	0.1450	0.9868	0.2770	0.9725	0.6955
eli5_to_eli5	0.9975	0.9605	0.0359	0.1804	0.0821	0.9986	0.9166
eli5_to_hc3	0.9318	0.8790	0.1161	0.6408	0.1630	0.9140	0.7511

Table 9: ELECTRA ( google/electra-base-discriminator ) results.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm
hc3_to_hc3	0.9968	0.9502	0.0460	0.2698	0.0999	0.9997
hc3_to_eli5	0.9578	0.8835	0.1088	0.6235	0.1250	0.8907
eli5_to_eli5	0.9983	0.9692	0.0288	0.1503	0.0647	0.9993
eli5_to_hc3	0.9309	0.8702	0.1229	0.7205	0.1397	0.8768

Table 10: DistilBERT ( distilbert-base-uncased ) results.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm
hc3_to_hc3	0.9913	0.8888	0.1100	0.9803	0.2225	0.9991
hc3_to_eli5	0.8762	0.5728	0.4245	4.0517	0.8532	0.9997
eli5_to_eli5	0.9530	0.7794	0.2089	1.2387	0.4377	0.9998
eli5_to_hc3	0.8890	0.7749	0.2148	1.3764	0.4147	0.9662

Table 11: DeBERTa-v3 ( microsoft/deberta-v3-base ) results.

Panel of Figure 3 — Figure 3: Performance analysis of DistilBERT across four evaluation conditions. Top: score distributions indicating class separability. Middle: reliability diagrams assessing calibration. Bottom: ROC curves illustrating discrimination performance. DistilBERT achieves near-transformer performance at approximately 60% of BERT’s parameter count.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm
hc3_to_hc3	0.9995	0.9916	0.0067	0.0262	0.0093	0.9862
hc3_to_eli5	0.8303	0.7124	0.2446	1.0887	0.1192	0.5275
eli5_to_eli5	0.9982	0.9748	0.0191	0.0666	0.0477	0.9844
eli5_to_hc3	0.8432	0.6866	0.2752	1.4723	0.0730	0.4455

Table 12: 1D-CNN results across evaluation conditions.

Panel of Figure 4 — Figure 4: Training dynamics and detectability behavior of the 1D-CNN detector. Top: rapid convergence to high validation AUC on both datasets. Bottom: score distributions indicating strong separability between human and llm text.

Figure 5: 1D-CNN degradation curve under progressive text humanization. The $x$ -axis represents the fraction of human tokens mixed into otherwise llm -generated text. The steep, smooth decline confirms that the 1D-CNN is highly sensitive to even small amounts of human-style $n$ -gram patterns.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm	Sep.
hc3_to_hc3	0.9721	0.9243	0.0580	0.2093	0.1273	0.8753	0.7480
hc3_to_eli5	0.6731	0.6296	0.2539	0.8110	0.3668	0.5502	0.1834
eli5_to_eli5	0.9448	0.8807	0.0897	0.3003	0.1823	0.8166	0.6343
eli5_to_hc3	0.7348	0.6650	0.2669	1.2185	0.2006	0.4941	0.2935

Table 13: Stylometric Hybrid — Logistic Regression results.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm	Sep.
hc3_to_hc3	0.9981	0.9785	0.0189	0.0854	0.0731	0.9362	0.8631
hc3_to_eli5	0.8557	0.7516	0.1768	0.5586	0.1699	0.5628	0.3929
eli5_to_eli5	0.9934	0.9605	0.0395	0.1626	0.1371	0.8759	0.7388
eli5_to_hc3	0.8848	0.6589	0.2100	0.6123	0.1164	0.4363	0.3199

Table 14: Stylometric Hybrid — Random Forest results.

Condition	auroc	Acc.	Brier	Log Loss	Hum.	llm	Sep.
hc3_to_hc3	0.9996	0.9928	0.0059	0.0226	0.0179	0.9912	0.9733
hc3_to_eli5	0.8633	0.7252	0.2270	0.9451	0.0673	0.5033	0.4361
eli5_to_eli5	0.9971	0.9732	0.0197	0.0714	0.0529	0.9620	0.9091
eli5_to_hc3	0.9037	0.7275	0.2281	0.9624	0.0439	0.4808	0.4369

Table 15: Stylometric Hybrid — XGBoost results.

Figure 6: Stylometric hybrid auroc heatmap. Rows correspond to classifiers (Logistic Regression, Random Forest, XGBoost), while columns represent the four evaluation conditions (eli5-to-eli5, eli5-to-hc3, hc3-to-eli5, hc3-to-hc3). Cell colors range from red (0.5) to dark green (1.0). XGBoost dominates across all conditions; the cross-domain eli5-to-hc3 auroc of $0.904$ represents a substantial improvement over the classical Stage 1 Random Forest ( $0.634$ ).

Model	Regime	Dataset	auroc	[email protected]	acc@median	acc@optimal
TinyLlama-1.1B-Chat-v1.0	Zero-Shot	HC3	0.5653	0.558	0.534	0.558
TinyLlama-1.1B-Chat-v1.0	Zero-Shot	ELI5	0.5072	0.524	0.510	0.524
TinyLlama-1.1B-Chat-v1.0	Few-Shot	HC3	0.6198	0.614	0.600	0.614
TinyLlama-1.1B-Chat-v1.0	Few-Shot	ELI5	0.5860	0.580	0.566	0.580
Qwen2.5-1.5B-Instruct	Zero-Shot	HC3	0.5221	0.436	0.530	0.574
Qwen2.5-1.5B-Instruct	Zero-Shot	ELI5	0.5205	0.470	0.512	0.562

Table 16: Tiny-scale llm -as-detector results.

Model	Regime	Dataset	auroc	[email protected]	acc@optimal	FP	FN
Llama-3.1-8B-Instruct	Zero-Shot	HC3	0.7295	0.680	0.680	48	120
Llama-3.1-8B-Instruct	Zero-Shot	ELI5	0.7508	0.670	0.702	106	57
Llama-3.1-8B-Instruct	Few-Shot	HC3	0.5027	0.546	0.550	53	172
Llama-3.1-8B-Instruct	Few-Shot	ELI5	0.5961	0.574	0.578	77	140
Llama-3.1-8B-Instruct	CoT	HC3	0.6771	0.629	0.657	11	18
Llama-3.1-8B-Instruct	CoT	ELI5	0.5988	0.586	0.600	15	13

Table 17: Mid-scale llm -as-detector results. FP/FN denote false positive/negative counts.

Regime	Dataset	auroc	[email protected]	acc@median	acc@optimal	FP	FN
Zero-Shot	HC3	0.8124	0.715	0.710	0.760	21	36
Zero-Shot	ELI5	0.8098	0.750	0.755	0.760	28	22
Few-Shot	HC3	0.6678	0.635	0.630	0.660	28	45
Few-Shot	ELI5	0.6374	0.590	0.620	0.620	38	44
CoT	HC3	0.8778	0.833	0.867	0.867	2	4
CoT	ELI5	0.8978	0.733	0.800	0.867	2	3

Table 18: Llama-2-13b-chat-hfdetection results (

n=200

zero/few-shot;

n=30

CoT ).

Regime	Dataset	auroc	[email protected]	acc@median	acc@optimal	FP	FN
Zero-Shot	HC3	0.6686	0.680	0.660	0.680	31	33
Zero-Shot	ELI5	0.7294	0.690	0.655	0.695	41	21
Few-Shot	HC3	0.3153	0.385	0.390	0.500	52	71
Few-Shot	ELI5	0.4262	0.470	0.465	0.500	59	47
CoT	HC3	0.6622	0.700	0.667	0.733	4	4
CoT	ELI5	0.8000	0.733	0.667	0.767	4	3

Table 19: Qwen2.5-14B-Instruct detection results (

n=200

zero/few-shot;

n=30

CoT ).

Dataset	Score Type	$n$	auroc	Accuracy	% of Total
HC3	conf+logit	11	0.6250	0.727	36.7%
HC3	logit_only	19	0.6429	0.684	63.3%
ELI5	conf+logit	15	0.8519	0.800	50.0%
ELI5	logit_only	15	0.7593	0.667	50.0%

Table 20: Qwen2.5-14B-Instruct CoT component analysis: hybrid vs. logit-only scoring.

Model	Regime	Data	auroc	[email protected]	Acc.@Opt.	Sep.
GPT-4o-mini	ZS	HC3	0.8470	0.7600	0.7900	$+$ 0.311
GPT-4o-mini	ZS	ELI5	0.9093	0.8800	0.8800	$+$ 0.419
GPT-4o-mini	FS	HC3	0.7163	0.7000	0.7000	$+$ 0.187
GPT-4o-mini	FS	ELI5	0.7824	0.6800	0.7400	$+$ 0.246
GPT-4o-mini	CoT	HC3	0.8056	0.7800	0.8000	$+$ 0.279
GPT-4o-mini	CoT	ELI5	0.7744	0.7600	0.8000	$+$ 0.261

Table 21: GPT-4o-mini ( llm -as-detector) results. All score directions verified correct.

Model	ZS auroc	FS auroc	$\Delta$
Qwen2.5-14B-Instruct	0.6686	0.3153	$-0.353$
Qwen2.5-7B-Instruct	0.6902	0.4579	$-0.232$
Llama-3.1-8B-Instruct	0.7295	0.5027	$-0.227$
Llama-2-13b-chat-hf	0.8124	0.6678	$-0.145$
GPT-4o-mini	0.8470	0.7163	$\mathbf{-0.131}$

Table 22: Zero-shot to few-shot auroc degradation on HC3 .

Variant	Dataset	auroc	Score Sep.
base_contrast	HC3	0.5007	0.0007
base_contrast	ELI5	0.6873	0.1873
multi_scale	HC3	0.5007	0.0007
multi_scale	ELI5	0.6873	0.1873
token_variance	HC3	0.6323	0.1323
token_variance	ELI5	0.5644	0.0644

Table 23: Contrastive likelihood detection results.

Model	Data	Method	auroc	Brier	Acc@Opt	Hum.	llm	Sep.
GPT-2 Small	HC3	rank	0.9099	0.1284	0.8805	0.2950	0.7050	0.4100
GPT-2 Small	ELI5	rank	0.9073	0.1297	0.8378	0.2963	0.7037	0.4074
GPT-2 Medium	HC3	rank	0.9047	0.1310	0.8804	0.2976	0.7024	0.4047
GPT-2 Medium	ELI5	rank	0.9275	0.1196	0.8546	0.2862	0.7138	0.4276
GPT-2 XL	HC3	rank	0.8917	0.1375	0.8609	0.3041	0.6959	0.3918
GPT-2 XL	ELI5	rank	0.9314	0.1176	0.8660	0.2843	0.7157	0.4315

Table 24: Perplexity-based detector results. Method = best normalization by auroc .

Model	Data	Hum. Med.	llm Med.	Ratio
GPT-2 Small	HC3	44.31	11.35	0.256
GPT-2 Small	ELI5	40.43	17.99	0.445
GPT-2 Medium	HC3	33.09	8.17	0.247
GPT-2 Medium	ELI5	30.92	12.95	0.419
GPT-2 XL	HC3	26.42	6.38	0.242
GPT-2 XL	ELI5	25.08	10.02	0.400

Table 25: Cross-model median perplexity statistics. llm text exhibits perplexity consistently

0.24

–

0.45\times

that of human text.

Detector	Source llm	Dataset	auroc	auroc CI	auprc	eer	Brier	FPR@95
BERT-HC3	TinyLlama-1.1B-Chat-v1.0	HC3	0.960	[0.942, 0.977]	0.952	0.075	0.130	0.165
BERT-HC3	TinyLlama-1.1B-Chat-v1.0	ELI5	0.952	[0.929, 0.973]	0.940	0.097	0.174	0.115
BERT-HC3	Qwen2.5-1.5B-Instruct	HC3	0.917	[0.887, 0.943]	0.898	0.140	0.150	0.335
BERT-HC3	Qwen2.5-1.5B-Instruct	ELI5	0.876	[0.842, 0.913]	0.845	0.190	0.197	0.370
BERT-HC3	Llama-2-13b-chat-hf	HC3	0.969	[0.952, 0.984]	0.965	0.067	0.126	0.080
BERT-HC3	Llama-2-13b-chat-hf	ELI5	0.973	[0.956, 0.988]	0.965	0.075	0.163	0.095

Table 26: Cross- llm generalization results for HC3 -trained neural detectors (selected conditions).

Classifier	Train llm	Test llm	auroc
SVM	TinyLlama-1.1B-Chat-v1.0	TinyLlama-1.1B-Chat-v1.0	0.976
SVM	TinyLlama-1.1B-Chat-v1.0	Llama-3.1-8B-Instruct	0.844
SVM	Qwen2.5-7B-Instruct	Qwen2.5-1.5B-Instruct	0.941
SVM	Llama-2-13b-chat-hf	Llama-2-13b-chat-hf	0.992
SVM	Llama-2-13b-chat-hf	Llama-3.1-8B-Instruct	0.818
RF	TinyLlama-1.1B-Chat-v1.0	TinyLlama-1.1B-Chat-v1.0	1.000

Table 27: Stage 3B embedding-space generalization on HC3 (selected). In-distribution conditions in bold .

Metric	HC3 $\rho$	HC3 $p$	ELI5 $\rho$	ELI5 $p$
KL Divergence	$-$ 0.298	0.148	$-$ 0.443	0.027 ∗
Wasserstein-2	$-$ 0.369	0.070	$-$ 0.322	0.117
Fréchet	$-$ 0.369	0.070	$-$ 0.322	0.117

Table 28: Spearman rank correlations between distributional distance and auroc drop.

*

p<0.05

Detector	Source llm	KL	$W_{2}$	FD	auroc Drop
BERT-HC3	TinyLlama-1.1B-Chat-v1.0	1.019	0.934	0.872	$+$ 0.006
BERT-HC3	Qwen2.5-1.5B-Instruct	0.471	0.682	0.465	$+$ 0.050
BERT-HC3	Qwen2.5-7B-Instruct	0.741	0.633	0.400	$+$ 0.033
BERT-HC3	Llama-3.1-8B-Instruct	2.015	0.808	0.652	$+$ 0.023
BERT-HC3	Llama-2-13b-chat-hf	1.105	0.822	0.676	$-$ 0.003
RoBERTa-HC3	TinyLlama-1.1B-Chat-v1.0	1.019	0.934	0.872	$+$ 0.019

Table 29: Per-detector distributional distances and auroc drop on HC3 . Note: Baseline auroc values for drop computation are taken from the 200-sample evaluation subsets used in Stage 3, not from the full test sets in Tables 7 – 11 . Negative drop indicates cross- llm performance exceeds the Stage 3 subset baseline.

Detector	Dataset	Level	auroc	Det. Rate	Mean AI	Mean Human	Brier
BERT-HC3	HC3	L0	0.9637	1.000	0.9998	0.2736	0.1278
BERT-HC3	HC3	L1	0.9749	1.000	0.9997	0.2736	0.1278
BERT-HC3	HC3	L2	0.8792	0.870	0.8696	0.2736	0.1914
BERT-HC3	ELI5	L0	0.9530	0.930	0.9249	0.2454	0.1480
BERT-HC3	ELI5	L1	0.9945	0.995	0.9949	0.2454	0.1154
BERT-HC3	ELI5	L2	0.8989	0.850	0.8553	0.2454	0.1817

Table 30: Stage 4 adversarial humanization results.

Model	Params	Precision	Batch (tr/ev)	Warmup	Notes
BERT	110M	FP16	32 / 64	6% ratio	—
RoBERTa	125M	FP16	32 / 64	6% ratio	Dynamic masking, no NSP
ELECTRA	110M	FP16	32 / 64	6% ratio	Discriminator fine-tuned
DistilBERT	66M	FP16	32 / 64	6% ratio	Model-specific dropout params
DeBERTa-v3-base	184M	FP32	16 / 32	500 steps	See note below

Table 31: Fine-tuned encoder transformer configurations. Entries marked “—” follow the shared protocol described above.

Classifier	Key Parameters
Logistic Regression	max_iter=2000 , C=1.0 , solver=lbfgs , class_weight=balanced
Random Forest	n_estimators=300 , max_depth=12 , class_weight=balanced
XGBoost	n_estimators=400 , max_depth=6 , lr=0.05 , subsample=0.8

Table 32: Stylometric hybrid classifier hyperparameters.

Model	Quant.	Polarity	Prior	Regimes	$n_{\text{ZS/FS}}$	$n_{\text{CoT}}$	Max New Tokens
TinyLlama-1.1B-Chat-v1.0	FP16	Standard	None	ZS, FS	500	—	—
Qwen2.5-1.5B-Instruct	FP16	Standard	None	ZS, FS	500	—	—
Llama-3.1-8B-Instruct	NF4/FP16	Standard	50 prompts	ZS, FS, CoT	500	70	350
Qwen2.5-7B-Instruct	NF4/FP16	Swap	50 prompts	ZS, FS, CoT	500	70	350
Llama-2-13b-chat-hf	NF4/FP16	Swap	50 prompts	ZS, FS, CoT	200	30	400
Qwen2.5-14B-Instruct	NF4/BF16	Swap	50 prompts	ZS, FS, CoT	200	30	500

Table 33: Per-model configuration for llm -as-detector experiments. “Swap” indicates swapped polarity convention (

\texttt{yes}=\text{human}

\texttt{no}=\text{AI}

n_{\text{ZS/FS}}

and

n_{\text{CoT}}

denote evaluation sample sizes.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Optimiser	AdamW
Learning rate	$2\times 10^{-5}$
Weight decay	0.01
Warmup	6% of total steps (500 fixed steps for DeBERTa-v3)
Dropout	0.2
Training epochs	1

Appendix

Table 34: Shared fine-tuning protocol for all encoder transformer detectors. DeBERTa-v3-specific deviations are noted in parentheses.

Model	Checkpoint	Params	Notes
BERT	bert-base-uncased	$\sim$ 110M	Standard MLM pre-training
RoBERTa	roberta-base	$\sim$ 125M	Dynamic masking, no NSP
ELECTRA	google/electra-base-discriminator	$\sim$ 110M	Replaced token detection
DistilBERT	distilbert-base-uncased	$\sim$ 66M	Knowledge distillation from BERT
DeBERTa-v3	microsoft/deberta-v3-base	$\sim$ 184M	FP32 only; no checkpointing; explicit grad clip

Appendix

Table 35: Encoder transformer model checkpoints and architectural notes.

Parameter	Value
Vocabulary size	30,000
Minimum word frequency	2
Max sequence length	256
Embedding dimension	128
Filter sizes	$\{2,3,4,5\}$
Filters per size	128

Appendix

Table 36: 1D-CNN architecture and training hyperparameters.

Parameter	Value
Logistic Regression $C$	1.0
Logistic Regression solver	lbfgs
Logistic Regression max_iter	2,000
Random Forest n_estimators	300
Random Forest max_depth	12
Random Forest min_samples_leaf	5

Appendix

Table 37: Stylometric hybrid classifier and feature extraction hyperparameters.

Model	Size	Quant.	Polarity	Prior	$n_{\text{ZS/FS}}$	$n_{\text{CoT}}$
TinyLlama-1.1B-Chat-v1.0	1.1B	FP16	yes=AI	Neutral	500	—
Qwen2.5-1.5B-Instruct	1.5B	FP16	yes=AI	Neutral	500	—
Llama-3.1-8B-Instruct	8B	NF4/FP16	yes=AI	Task ( $n=50$ )	500	70
Qwen2.5-7B-Instruct	7B	NF4/FP16	yes=human	Task ( $n=50$ )	500	70
Llama-2-13b-chat-hf	13B	NF4/FP16	yes=human	Task ( $n=50$ )	200	30
Qwen2.5-14B-Instruct	14B	NF4/BF16	yes=human	Task ( $n=50$ )	200	30

Appendix

Table 38: llm -as-detector per-model configuration.

Model	Conf. weight	Logit weight	Dead zone	Verdict override	Max tokens
Llama-3.1-8B-Instruct	0.6	0.4	$[0.40,0.60]$	$[0.35,0.65]$	350
Qwen2.5-7B-Instruct	0.6	0.4	$[0.35,0.65]$	$[0.35,0.65]$	350
Llama-2-13b-chat-hf	0.6	0.4	$[0.40,0.60]$	$[0.35,0.65]$	400
Qwen2.5-14B-Instruct	0.6	0.4	$[0.35,0.65]$	$[0.35,0.65]$	500

Appendix

Table 39: CoT hybrid ensemble parameters. The dead zone defines the confidence interval within which the logit-only score is used instead of the ensemble.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 167 papers

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy
Mar 24, 2026Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis +1AI-Generated Text Detection Robustness of Machine-Generated Text Detection
Technological University Dublin · University College Dublin
Automatic detection of Gen-AI texts: A comparative framework of neural models
Mar 19, 2026Cristian Buttaro, Irene AmeriniAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Sapienza University in Rome Latina
Breaking the Generator Barrier: Disentangled Representation for Generalizable AI-Text Detection
Apr 15, 2026Xiao Pu, Zepeng Cheng, Lin Yuan +2AI-Generated Text Detection Robustness of Machine-Generated Text Detection
Chongqing University of Posts and Telecommunications, China

Mar 24, 2026J/K move · Enter open · S save

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis, Marisa Llorens Salvador

Technological University Dublin · University College Dublin

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

The widespread adoption of Large Language Models (LLMs) has made the detection of AI-Generated text a pressing and complex challenge. Although many detection systems report high benchmark accuracy, their reliability in real-world settings remains uncertain, and their interpretability is often unexplored. In this work, we investigate whether contemporary detectors genuinely identify machine authorship or merely exploit dataset-specific artefacts. We propose an interpretable detection framework that integrates linguistic feature engineering, machine learning, and explainable AI techniques. When evaluated on two prominent benchmark corpora, namely PAN CLEF 2025 and COLING 2025, our model trained on 30 linguistic features achieves leaderboard-competitive performance, attaining an F1 score of 0.9734. However, systematic cross-domain and cross-generator evaluation reveals substantial generalisation failure: classifiers that excel in-domain degrade significantly under distribution shift. Using SHAP- based explanations, we show that the most influential features differ markedly between datasets, indicating that detectors often rely on dataset-specific stylistic cues rather than stable signals of machine authorship. Further investigation with in-depth error analysis exposes a fundamental tension in linguistic-feature-based AI text detection: the features that are most discriminative on in-domain data are also the features most susceptible to domain shift, formatting variation, and text-length effects. We believe that this knowledge helps build AI detectors that are robust across different settings. To support replication and practical use, we release an open-source Python package that returns both predictions and instance-level explanations for individual texts.

Abstract

AI-Generated Text Detection

Explore similar work

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

Automatic detection of Gen-AI texts: A comparative framework of neural models

Breaking the Generator Barrier: Disentangled Representation for Generalizable AI-Text Detection