AI-text detectors reach near-perfect scores in evaluations that contrast fully human-written texts with texts generated from researcher-designed prompts. In authentic student writing, however, AI use is voluntary and often partial. We study this setting with the PISA 2025 FLA dataset, in which 15-year-olds in Germany wrote English argumentative essays with and without access to a gpt-4o-mini-based chatbot. Keystroke, clipboard and chat logs let us trace every character of the final essay to its origin. From this provenance, we derive text- and sentence-level labels for human-written, AI-generated and mixed text, and for text written after chatbot use without traceable AI material. We evaluate a feature-based detector and a classifier on frozen Llama-3.1-8B embeddings, both adapted from top-ranked shared-task systems. Both separate AI from human text reliably (macro-F1 up to 0.96 for texts and 0.95 for sentences), independently of spelling errors. The feature-based detector, however, relies on lexical complexity, which a simple student-style rewriting prompt removes (AI recall 0.92 → 0.60 with gpt-4o-mini, 0.41 with Qwen3). The embedding-based detector stays robust for whole texts (AI recall ≥ 0.94) but not for single sentences (AI recall as low as 0.59). Mixed text falls between the two classes, and text without traceable AI material is classified as human-written in about 95% of texts.
Figures & tables
Sentences
Texts
Label
complete
subset
complete
subset
human
23,560
18,756
1,802
1,664
ai
21,266
17,358
1,009
952
pot.-infl.
8,133
7,290
403
381
mixed
872
729
321
306
unknown_paste
1,963
—
152
—
Table 1: Label distribution in the complete release and in the experimental subset.
Text
Sentence
Test set
XGB
RFCN
XGB
RFCN
ai / human
0.95
0.96
0.88
0.95
grey-zone items added as ai
+ mixed
0.90
0.89
0.86
0.94
+ pi
0.81
0.84
0.76
0.81
+ mixed , pi
0.79
0.80
0.76
0.81
Table 2: Macro-F1 of the binary detectors, trained on ai / human only. Top: ai -vs- human test set; bottom: grey-zone items added to the test set as ai . All models in Table 8 .
Text
Sentence
XGB
RFCN
XGB
RFCN
(a) Binary models: % predicted as ai , per true class
ai
91.7
94.8
85.6
94.2
human
3.0
2.4
11.3
5.0
pi
5.4
5.3
20.6
8.8
mixed
58.1
32.3
52.1
45.5
Table 3: Grey-zone classes. (a) Predictions of the binary ai -vs- human models; (b) multiclass detectors trained with mixed and/or pi as separate classes.
Drop
Feature set
SHAP %
Within
Cross
a → t
t → a
Form
Style (50)
41.7
0.92
0.85
0.10
0.05
Complexity (7)
32.7
0.91
0.88
0.04
0.02
Style + Complex. (57)
74.4
0.95
0.85
0.20
0.01
Content (28)
25.6
0.91
0.75
0.05
0.29
Table 4: XGBoost: share of SHAP importance and cross-task transfer (macro-F1). Drop: within minus cross per direction (a = advertisement , t = teacher ).
Text
Sentence
Train / test
XGB
RFCN
XGB
RFCN
orig / orig
0.95
0.96
0.88
0.95
orig / correct
0.95
0.95
0.88
0.94
orig / mistake
0.94
0.94
0.88
0.92
correct / correct
0.94
0.95
0.88
0.94
mistake / mistake
0.95
0.96
0.88
0.94
Table 5: Macro-F1 under the spelling manipulation. Train / test data: original (orig) or manipulated ( correct , mistake ).
XGBoost
RFCN
Test set
Rec. ai
F1
Rec. ai
F1
Text level
original
0.92
0.95
0.95
0.96
4o-mini student
0.60
0.81
0.99
0.98
4o-mini informed
0.39
0.69
0.97
0.98
Qwen3 student
0.41
0.70
0.94
0.96
Table 6: Detection on rewritten ai test items. Rec. ai : recall of the ai class; F1: macro-F1.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Step
Texts
Sentences
Logged sessions
4,091
—
Text level
− empty / length outliers
3,913
55,945
− unk. texts a
3,535
50,138
− non-English
3,303
46,076
Sentence level
Appendix
Table 7: From logged sessions to the experimental subset; each row reports the material remaining after the step. a Texts labelled unknown_logger or unknown_paste .
Text
Sentence
Model
F1
Acc
F1
Acc
Logistic Regression
0.94
0.95
0.85
0.85
Random Forest
0.93
0.94
0.87
0.87
Extra Trees
0.93
0.94
0.87
0.87
XGBoost
0.95
0.95
0.88
0.88
RFCN
0.96
0.97
0.95
0.95
Appendix
Table 8: Core ai -vs- human detection, macro-F1 and accuracy. All five models are shown; the grey-zone and multiclass analyses below use the two headline models (XGBoost, RFCN).
student
Rewrite the following text so that it sounds like it was written by a 15-year-old German student in English class. Only output the rewritten text.
informed
Rewrite the following essay so that it reads like the work of a 15-year-old German student writing in English as a foreign language. Keep the same position and main arguments, but change the style: • Make it considerably shorter, about 150–200 words. • Use short, common words instead of long or formal ones. • Use simple sentences and everyday vocabulary; repeat words rather than using synonyms. • Write in a personal way, with many pronouns (I, we, you, they) and verbs rather than nouns. • Use little punctuation: mostly full stops and a few commas, no semicolons, colons, dashes or quotation marks. • Keep a neutral tone; avoid enthusiastic or strongly positive wording. • Do not add spelling or grammar mistakes. Only output the rewritten essay.
Appendix
Table 9: Rewriting prompts; the essay is appended to each prompt.
Train / test
Δ F1
95% CI
Flips
Text level
orig / correct
− 0.016
[ − 0.032, − 0.004]*
0 / 4
orig / mistake
− 0.025
[ − 0.048, − 0.008]*
6 / 0
correct / correct
− 0.012
[ − 0.034, − 0.008]
2 / 5
mistake / mistake
− 0.004
[ − 0.021, − 0.013]
3 / 2
Sentence level
Appendix
Table 10: RFCN under the spelling manipulation, compared with orig / orig on the same test items (265 texts, 3,067 sentences). Δ F1: difference in macro-F1. 95% CI: paired bootstrap over students (2,000 resamples); * = CI excludes zero. Flips: items whose prediction changed, ai → human / human → ai .
Texts
Sentences
Chat condition
1,981
31,936
chatbot used
1,889
—
AI material in final essay
1,134
—
Retyping suspects
—
572
Appendix
Table 11: Chatbot use in the Chat condition of the complete release (3,913 texts, before exclusions). Retyping suspects: post-AI sentences without AI-sourced characters that match a chatbot sentence with similarity ≥0.85 .
AI-text detectors are already applied to Kazakh writing, yet no study reports how often publicly available detectors flag Kazakh text whose human authorship is dated before ChatGPT. We present AINA-KK: 3,811 human passages from encyclopedic, news, web and educational pages that provably existed before ChatGPT, and 5,597 machine and gray-zone passages from 9 generators, including the Kazakh-built KazLLM and Sherkala, with script and length stress tests. Five English-trained detectors from the Hugging Face Hub flag up to 81.4% of human passages and 79.6% of published student essays; recalibrated, they catch at most 1.3% of machine text. Calibrated zero-shot detectors catch at most 50.3% of machine text from in-distribution generators. Fine-tuned encoders rank text almost perfectly, but mDeBERTa at its default threshold flags 24.4% of 50-word human excerpts, partly because machine outputs are shorter than human text; calibrated, it catches only 47.1% of KazLLM and 62.7% of Sherkala passages. Without script handling, machine-transliterated Latin-2021 text defeats every detector. Fixes that need no new data (calibration, script canonicalization, script-augmented and length-matched training, and an OR with a zero-shot scorer) form AINA-KK-Det. On test it flags 0.9% of human passages (95% CI 0.4-2.7; 1.5% on development data) and catches 85.2% of machine text from known generators, 92.9% from two unseen Gemini-family Gemma-4 models and 61.3% from the Kazakh-built ones; its configuration was chosen with test results in view. We release the data, code and detector at https://github.com/ozxmn/aina-kk.
Mansur Ozaman
School of Computing and Artificial Intelligence, Nazarbayev University, Astana, Kazakhstan
Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.
Nils Dycke, Marina Sakharova, Nico Daheim +1
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Zuse School ELIZA
Research on AI-generated text detection has presented a number of approaches to discern human from AI prose, some of which achieving high in-distribution performance. However, real-world applicability has stalled because their outputs are misaligned with the needs of users, such as professors, who are presented with a numeric score that has no attached explanation. We tackle this issue with a novel architecture, TELL, that bakes explainability from the ground-up. While our system still offers a numerical score like other detectors for comparability, TELL takes a fundamentally different approach where we aim to show the user the "tells" by which the model believes a text is AI or human-written, to empower the user to decide who wrote a text using their own judgment and understanding of the context of the writing and its alleged author. We train TELL on a custom SFT dataset of domain-specific authorship annotations, and further refine the system using GRPO with curriculum learning to improve performance. We achieve competitive performance with state-of-the-art detectors (AUROC 0.927) while natively providing annotations that explain the basis for the detector's decision. We further evaluate the quality of our explanations using a dataset of human annotations and report a high (mean 72.3%) win-rate on annotation concreteness, falsifiability, coherence, plausibility and grounding, allowing users to critically think and decide for themselves. Our work thus reframes the problem of AI-generated text detection in a human-centric perspective and paves the way for a new family of detectors that focus on native explainability.
Aldan Creo, Suraj Ranganath
School of Computing, Information and Data Sciences University of California, San Diego United States of America