The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.
Figures & tables
Figure 1: Overview of the benchmark pipeline. Stage 0 constructs two paired corpora ( HC3 : 23k human–ChatGPT pairs; ELI5 : 15k human–Mistral-7B pairs) with length-matched preprocessing. Stage 1 evaluates three detector families: Family 1 (classical statistical classifiers), and Family 2 (fine-tuned encoder transformers — BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3; 1D-CNN; perplexity-based detectors; stylometric-hybrid XGBoost), and Family 3 ( llm -as-detector prompting at four scales including GPT-4o-mini). Stage 2 evaluates cross- llm generalization via neural detectors, embedding-space classifier matrices , and distributional shift analysis. Stage 3 applies adversarial humanization at three levels (L0–L2) using an instruction-tuned rewriter. All families are evaluated under a unified five-metric suite ( auroc , auprc , eer , Brier Score, FPR@95%TPR).
Domain
Unique Pairs
reddit_eli5
16,153
finance
3,933
medicine
1,248
open_qa
1,187
wiki_csai
842
Total
23,363
Table 1: Domain distribution of the HC3 corpus after flattening and deduplication.
Parameter
Value
Model
mistralai/Mistral-7B-Instruct-v0.2
Precision
FP16 (no quantization)
Attention
Flash Attention 2
Compilation
torch.compile (reduce-overhead)
Batch size
48
Max new tokens
150
Table 2: Mistral-7B generation configuration for ELI5 augmentation.
Dataset
Train
Test
HC3
36,968 (18,484 per class)
9,242
ELI5
22,862 (11,431 per class)
5,716
Table 3: Train/test split sizes after length matching and stratification.
Condition
auroc
Brier
Log Loss
Mean Human
Mean llm
hc3_to_hc3
0.8882
0.1334
0.4411
0.2838
0.7319
hc3_to_eli5
0.7406
0.2116
0.6474
0.4246
0.6508
eli5_to_eli5
0.8446
0.1605
0.4909
0.3251
0.6760
eli5_to_hc3
0.7429
0.2496
0.9063
0.2006
0.4580
Table 4: Logistic Regression results across evaluation conditions.
Condition
auroc
Brier
Log Loss
Mean Human
Mean llm
hc3_to_hc3
0.9767
0.0679
0.2438
0.1889
0.8173
hc3_to_eli5
0.7829
0.1922
0.5815
0.3830
0.6086
eli5_to_eli5
0.9618
0.0869
0.3014
0.2348
0.7811
eli5_to_hc3
0.6337
0.3193
1.1636
0.1643
0.2903
Table 5: Random Forest results across evaluation conditions.
Condition
auroc
Brier
Log Loss
Mean Human
Mean llm
hc3_to_hc3
0.7993
0.1835
0.5486
0.3700
0.6318
hc3_to_eli5
0.6933
0.2348
0.6639
0.5196
0.6686
eli5_to_eli5
0.7924
0.1857
0.5512
0.3740
0.6287
eli5_to_hc3
0.5992
0.3169
1.5852
0.2083
0.3191
Table 6: SVM (RBF Kernel) results across evaluation conditions.
Figure 2: Calibration curves for classical detectors across four evaluation settings. Points close to the diagonal indicate well-calibrated confidence scores, while systematic deviations reflect over- or under-confidence.
Figure 3: Performance analysis of DistilBERT across four evaluation conditions. Top: score distributions indicating class separability. Middle: reliability diagrams assessing calibration. Bottom: ROC curves illustrating discrimination performance. DistilBERT achieves near-transformer performance at approximately 60% of BERT’s parameter count.
Condition
auroc
Acc.
Brier
Log Loss
Hum.
llm
hc3_to_hc3
0.9995
0.9916
0.0067
0.0262
0.0093
0.9862
hc3_to_eli5
0.8303
0.7124
0.2446
1.0887
0.1192
0.5275
eli5_to_eli5
0.9982
0.9748
0.0191
0.0666
0.0477
0.9844
eli5_to_hc3
0.8432
0.6866
0.2752
1.4723
0.0730
0.4455
Table 12: 1D-CNN results across evaluation conditions.
Figure 4: Training dynamics and detectability behavior of the 1D-CNN detector. Top: rapid convergence to high validation AUC on both datasets. Bottom: score distributions indicating strong separability between human and llm text.
Figure 5: 1D-CNN degradation curve under progressive text humanization. The x -axis represents the fraction of human tokens mixed into otherwise llm -generated text. The steep, smooth decline confirms that the 1D-CNN is highly sensitive to even small amounts of human-style n -gram patterns.
Table 14: Stylometric Hybrid — Random Forest results.
Condition
auroc
Acc.
Brier
Log Loss
Hum.
llm
Sep.
hc3_to_hc3
0.9996
0.9928
0.0059
0.0226
0.0179
0.9912
0.9733
hc3_to_eli5
0.8633
0.7252
0.2270
0.9451
0.0673
0.5033
0.4361
eli5_to_eli5
0.9971
0.9732
0.0197
0.0714
0.0529
0.9620
0.9091
eli5_to_hc3
0.9037
0.7275
0.2281
0.9624
0.0439
0.4808
0.4369
Table 15: Stylometric Hybrid — XGBoost results.
Figure 6: Stylometric hybrid auroc heatmap. Rows correspond to classifiers (Logistic Regression, Random Forest, XGBoost), while columns represent the four evaluation conditions (eli5-to-eli5, eli5-to-hc3, hc3-to-eli5, hc3-to-hc3). Cell colors range from red (0.5) to dark green (1.0). XGBoost dominates across all conditions; the cross-domain eli5-to-hc3 auroc of 0.904 represents a substantial improvement over the classical Stage 1 Random Forest ( 0.634 ).
Table 27: Stage 3B embedding-space generalization on HC3 (selected). In-distribution conditions in bold .
Metric
HC3 ρ
HC3 p
ELI5 ρ
ELI5 p
KL Divergence
− 0.298
0.148
− 0.443
0.027 ∗
Wasserstein-2
− 0.369
0.070
− 0.322
0.117
Fréchet
− 0.369
0.070
− 0.322
0.117
Table 28: Spearman rank correlations between distributional distance and auroc drop. ∗ = p<0.05 .
Detector
Source llm
KL
W2
FD
auroc Drop
BERT-HC3
TinyLlama-1.1B-Chat-v1.0
1.019
0.934
0.872
+ 0.006
BERT-HC3
Qwen2.5-1.5B-Instruct
0.471
0.682
0.465
+ 0.050
BERT-HC3
Qwen2.5-7B-Instruct
0.741
0.633
0.400
+ 0.033
BERT-HC3
Llama-3.1-8B-Instruct
2.015
0.808
0.652
+ 0.023
BERT-HC3
Llama-2-13b-chat-hf
1.105
0.822
0.676
− 0.003
RoBERTa-HC3
TinyLlama-1.1B-Chat-v1.0
1.019
0.934
0.872
+ 0.019
Table 29: Per-detector distributional distances and auroc drop on HC3 . Note: Baseline auroc values for drop computation are taken from the 200-sample evaluation subsets used in Stage 3, not from the full test sets in Tables 7 – 11 . Negative drop indicates cross- llm performance exceeds the Stage 3 subset baseline.
Table 39: CoT hybrid ensemble parameters. The dead zone defines the confidence interval within which the logit-only score is used instead of the ensemble.
The widespread adoption of Large Language Models (LLMs) has made the detection of AI-Generated text a pressing and complex challenge. Although many detection systems report high benchmark accuracy, their reliability in real-world settings remains uncertain, and their interpretability is often unexplored. In this work, we investigate whether contemporary detectors genuinely identify machine authorship or merely exploit dataset-specific artefacts. We propose an interpretable detection framework that integrates linguistic feature engineering, machine learning, and explainable AI techniques. When evaluated on two prominent benchmark corpora, namely PAN CLEF 2025 and COLING 2025, our model trained on 30 linguistic features achieves leaderboard-competitive performance, attaining an F1 score of 0.9734. However, systematic cross-domain and cross-generator evaluation reveals substantial generalisation failure: classifiers that excel in-domain degrade significantly under distribution shift. Using SHAP- based explanations, we show that the most influential features differ markedly between datasets, indicating that detectors often rely on dataset-specific stylistic cues rather than stable signals of machine authorship. Further investigation with in-depth error analysis exposes a fundamental tension in linguistic-feature-based AI text detection: the features that are most discriminative on in-domain data are also the features most susceptible to domain shift, formatting variation, and text-length effects. We believe that this knowledge helps build AI detectors that are robust across different settings. To support replication and practical use, we release an open-source Python package that returns both predictions and instance-level explanations for individual texts.
Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis +1
Technological University Dublin · University College Dublin
The rapid proliferation of Large Language Models has significantly increased the difficulty of distinguishing between human-written and AI generated texts, raising critical issues across academic, editorial, and social domains. This paper investigates the problem of AI generated text detection through the design, implementation, and comparative evaluation of multiple machine learning based detectors. Four neural architectures are developed and analyzed: a Multilayer Perceptron, a one-dimensional Convolutional Neural Network, a MobileNet-based CNN, and a Transformer model. The proposed models are benchmarked against widely used online detectors, including ZeroGPT, GPTZero, QuillBot, Originality.AI, Sapling, IsGen, Rephrase, and Writer. Experiments are conducted on the COLING Multilingual Dataset, considering both English and Italian configurations, as well as on an original thematic dataset focused on Art and Mental Health. Results show that supervised detectors achieve more stable and robust performance than commercial tools across different languages and domains, highlighting key strengths and limitations of current detection strategies.
As large language models (LLMs) generate text that increasingly resembles human writing, the subtle cues that distinguish AI-generated content from human-written content become increasingly challenging to capture. Reliance on generator-specific artifacts is inherently unstable, since new models emerge rapidly and reduce the robustness of such shortcuts. This generalizes unseen generators as a central and challenging problem for AI-text detection. To tackle this challenge, we propose a progressively structured framework that disentangles AI-detection semantics from generator-aware artifacts. This is achieved through a compact latent encoding that encourages semantic minimality, followed by perturbation-based regularization to reduce residual entanglement, and finally a discriminative adaptation stage that aligns representations with task objectives. Experiments on MAGE benchmark, covering 20 representative LLMs across 7 categories, demonstrate consistent improvements over state-of-the-art methods, achieving up to 24.2% accuracy gain and 26.2% F1 improvement. Notably, performance continues to improve as the diversity of training generators increases, confirming strong scalability and generalization in open-set scenarios. Our source code will be publicly available at https://github.com/PuXiao06/DRGD.
Xiao Pu, Zepeng Cheng, Lin Yuan +2
Chongqing University of Posts and Telecommunications, China