Specificity Bias and the Wikipedia Paradox: An xAI Analysis of AI Text Detectors under Domain Shift
Organizations: Faculty of Electrical Engineering and Information Technology, OTH Regensburg, Germany · Technical University of Munich, Munich, Germany
Abstract
The widespread adoption of Large Language Models (LLMs) demands reliable detection of human-authored versus synthetic text. However, current Transformer-based detectors often experience severe performance degradation during domain shifts and operate as opaque classifiers. This paper investigates the robustness and interpretability of fine-tuned Transformer models (DistilBERT, RoBERTa) by evaluating them against a custom out-of-distribution (OOD) dataset of hyper-local news synthesized via open-source LLMs (Llama 3, Mistral, Gemma). To analyze model decision boundaries, we apply a tripartite Explainable AI (xAI) framework using SHAP. Our analysis reveals three key phenomena: First, detectors often rely on a "Call-to-Action" syndrome inherent to instruction-tuned LLMs rather than generic machine text. Second, we observe a "Specificity Bias," where LLMs successfully bypass detection by hallucinating specific named entities and mimicking conversational imperfection. Finally, we illustrate the "Wikipedia Paradox," wherein formal, objective human writing is frequently misclassified as AI due to its low statistical burstiness. These findings demonstrate that purely statistical detection paradigms face systematic limitations in highly structured human domains, emphasizing the critical need for semantically-aware, interpretable detection architectures.
Figures & tables
| Model Architecture | Training Data | Test Domain | Accuracy | Precision (AI) | Recall (AI) | F1 (Macro) | FPR (False Positives) |
|---|---|---|---|---|---|---|---|
| DistilBERT | HC3 PLUS | OOD (Local News) | 59.0% | 100.0% | 46.0% | 59.0% | 0.08% (1/1199) |
| RoBERTa | HC3 PLUS | OOD (Local News) | 89.0% | 87.0% | 100.0% | 83.0% | 43.37% (520/1199) |
| RoBERTa | artem9k | OOD (Local News) | 61.0% | 100.0% | 48.0% | 60.0% | 0.08% (1/1199) |
| RoBERTa | artem9k | Cross-Domain (Wiki) | 85.0% | 84.0% | 87.0% | 85.0% | 16.95% (339/2000) |
| Model (Training) | Prediction | True Human | True AI |
|---|---|---|---|
| DistilBERT (HC3) | Predicted Human | 1198 (TN) | 1950 (FN) |
| Predicted AI | 1 (FP) | 1647 (TP) | |
| RoBERTa (HC3) | Predicted Human | 679 (TN) | 10 (FN) |
| Predicted AI | 520 (FP) | 3587 (TP) | |
| RoBERTa (artem9k) | Predicted Human | 1198 (TN) | 1877 (FN) |
| Predicted AI | 1 (FP) | 1720 (TP) |
| Model | Test Domain | AUROC | AUPRC |
|---|---|---|---|
| DistilBERT (HC3) | OOD (Local News) | 0.896 | 0.882 |
| RoBERTa (HC3) | OOD (Local News) | 0.806 | 0.841 |
| RoBERTa (artem9k) | OOD (Local News) | 0.995 | 0.965 |
| RoBERTa (artem9k) | Cross-Domain (Wiki) | 0.908 | 0.927 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
AI-Generated Text Detection
AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.
See all 185 papers