Detecting AI-Authored Text in Authentic Student Writing: Evidence from PISA 2025
Organizations: Leibniz Institute for Science and Mathematics Education, Kiel, Germany · Kiel University, Kiel, Germany
Abstract
AI-text detectors reach near-perfect scores in evaluations that contrast fully human-written texts with texts generated from researcher-designed prompts. In authentic student writing, however, AI use is voluntary and often partial. We study this setting with the PISA 2025 FLA dataset, in which 15-year-olds in Germany wrote English argumentative essays with and without access to a gpt-4o-mini-based chatbot. Keystroke, clipboard and chat logs let us trace every character of the final essay to its origin. From this provenance, we derive text- and sentence-level labels for human-written, AI-generated and mixed text, and for text written after chatbot use without traceable AI material. We evaluate a feature-based detector and a classifier on frozen Llama-3.1-8B embeddings, both adapted from top-ranked shared-task systems. Both separate AI from human text reliably (macro-F1 up to 0.96 for texts and 0.95 for sentences), independently of spelling errors. The feature-based detector, however, relies on lexical complexity, which a simple student-style rewriting prompt removes (AI recall 0.92 0.60 with gpt-4o-mini, 0.41 with Qwen3). The embedding-based detector stays robust for whole texts (AI recall 0.94) but not for single sentences (AI recall as low as 0.59). Mixed text falls between the two classes, and text without traceable AI material is classified as human-written in about 95% of texts.
Figures & tables
| Sentences | Texts | |||
| Label | complete | subset | complete | subset |
| human | 23,560 | 18,756 | 1,802 | 1,664 |
| ai | 21,266 | 17,358 | 1,009 | 952 |
| pot.-infl. | 8,133 | 7,290 | 403 | 381 |
| mixed | 872 | 729 | 321 | 306 |
| unknown_paste | 1,963 | — | 152 | — |
| Text | Sentence | |||
| Test set | XGB | RFCN | XGB | RFCN |
| ai / human | 0.95 | 0.96 | 0.88 | 0.95 |
| grey-zone items added as ai | ||||
| + mixed | 0.90 | 0.89 | 0.86 | 0.94 |
| + pi | 0.81 | 0.84 | 0.76 | 0.81 |
| + mixed , pi | 0.79 | 0.80 | 0.76 | 0.81 |
| Text | Sentence | |||
| XGB | RFCN | XGB | RFCN | |
| (a) Binary models: % predicted as ai , per true class | ||||
| ai | 91.7 | 94.8 | 85.6 | 94.2 |
| human | 3.0 | 2.4 | 11.3 | 5.0 |
| pi | 5.4 | 5.3 | 20.6 | 8.8 |
| mixed | 58.1 | 32.3 | 52.1 | 45.5 |
| Drop | |||||
|---|---|---|---|---|---|
| Feature set | SHAP % | Within | Cross | a t | t a |
| Form | |||||
| Style (50) | 41.7 | 0.92 | 0.85 | 0.10 | 0.05 |
| Complexity (7) | 32.7 | 0.91 | 0.88 | 0.04 | 0.02 |
| Style + Complex. (57) | 74.4 | 0.95 | 0.85 | 0.20 | 0.01 |
| Content (28) | 25.6 | 0.91 | 0.75 | 0.05 | 0.29 |
| Text | Sentence | |||
|---|---|---|---|---|
| Train / test | XGB | RFCN | XGB | RFCN |
| orig / orig | 0.95 | 0.96 | 0.88 | 0.95 |
| orig / correct | 0.95 | 0.95 | 0.88 | 0.94 |
| orig / mistake | 0.94 | 0.94 | 0.88 | 0.92 |
| correct / correct | 0.94 | 0.95 | 0.88 | 0.94 |
| mistake / mistake | 0.95 | 0.96 | 0.88 | 0.94 |
| XGBoost | RFCN | |||
| Test set | Rec. ai | F1 | Rec. ai | F1 |
| Text level | ||||
| original | 0.92 | 0.95 | 0.95 | 0.96 |
| 4o-mini student | 0.60 | 0.81 | 0.99 | 0.98 |
| 4o-mini informed | 0.39 | 0.69 | 0.97 | 0.98 |
| Qwen3 student | 0.41 | 0.70 | 0.94 | 0.96 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Step | Texts | Sentences |
|---|---|---|
| Logged sessions | 4,091 | — |
| Text level | ||
| empty / length outliers | 3,913 | 55,945 |
| unk. texts a | 3,535 | 50,138 |
| non-English | 3,303 | 46,076 |
| Sentence level | ||
| Text | Sentence | |||
|---|---|---|---|---|
| Model | F1 | Acc | F1 | Acc |
| Logistic Regression | 0.94 | 0.95 | 0.85 | 0.85 |
| Random Forest | 0.93 | 0.94 | 0.87 | 0.87 |
| Extra Trees | 0.93 | 0.94 | 0.87 | 0.87 |
| XGBoost | 0.95 | 0.95 | 0.88 | 0.88 |
| RFCN | 0.96 | 0.97 | 0.95 | 0.95 |
| student | Rewrite the following text so that it sounds like it was written by a 15-year-old German student in English class. Only output the rewritten text. |
|---|---|
| informed | Rewrite the following essay so that it reads like the work of a 15-year-old German student writing in English as a foreign language. Keep the same position and main arguments, but change the style: • Make it considerably shorter, about 150–200 words. • Use short, common words instead of long or formal ones. • Use simple sentences and everyday vocabulary; repeat words rather than using synonyms. • Write in a personal way, with many pronouns (I, we, you, they) and verbs rather than nouns. • Use little punctuation: mostly full stops and a few commas, no semicolons, colons, dashes or quotation marks. • Keep a neutral tone; avoid enthusiastic or strongly positive wording. • Do not add spelling or grammar mistakes. Only output the rewritten essay. |
| Train / test | F1 | 95% CI | Flips |
| Text level | |||
| orig / correct | 0.016 | [ 0.032, 0.004]* | 0 / 4 |
| orig / mistake | 0.025 | [ 0.048, 0.008]* | 6 / 0 |
| correct / correct | 0.012 | [ 0.034, 0.008] | 2 / 5 |
| mistake / mistake | 0.004 | [ 0.021, 0.013] | 3 / 2 |
| Sentence level | |||
| Texts | Sentences | |
|---|---|---|
| Chat condition | 1,981 | 31,936 |
| chatbot used | 1,889 | — |
| AI material in final essay | 1,134 | — |
| Retyping suspects | — | 572 |
| Package | Version | Used for |
|---|---|---|
| Python | 3.11.16 | |
| nela-features | 3.0.1 | NELA features |
| scikit-learn | 1.7.1 | LR, RF, ET |
| xgboost | 3.2.0 | XGBoost |
| shap | 0.51.0 | Exp. 2 |
| pysbd | 0.3.4 | Segmentation |
AI-Generated Text Detection
AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.
See all 184 papers