Unibuc - NLP at "Voight-Kampff" Generative AI Detection PAN 2025
Authors: Teodor Marchitan, Claudiu Creanga, Liviu P. Dinu
Organizations: Faculty of Mathematics and Computer Science, University of Bucharest, Romania · HLT Research Center, Bucharest, Romania · Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania
This paper describes the approach of the Unibuc - NLP team in tackling the "Voight-Kampff" Generative AI Detection task from PAN at CLEF 2025. In the Subtask 1: Voight-Kampff AI Detection Sensitivity we have explored different approaches of leveraging embeddings extracted from LLMs, using either classical ML models or further fine-tuning LLMs with a classification head for the downstream binary classification task. Our experiments showed that using a majority vote ensemble of classical ML models (Light GBM, XGB, Logistic Regression and SVM) is comparable with fine-tuning for downstream task achieving around 99.4% F1 score. For the Subtask 2: Human-AI Collaborative Text Classification we explored a way of combining features at different layers extracted using transformer backbone with layer-wise projection and attentive pooling which proved modest results.
Figures & tables
human-written (0)
ai-written (1)
Total
Train
9,101
14,606
23,707
Dev
1,277
2,312
3,589
Table 1: Label distribution in train and dev splits for task 1 dataset.
Label category
Train
Dev
Fully human-written (0)
14,910
225
Human-initiated, then machine-continued (3)
1,368
510
Human-written, then machine-polished (1)
10,740
37,170
Machine-written, then machine-humanized (obfuscated) (2)
75,270
12,330
Machine-written, then human-edited (5)
95,398
12,289
Deeply-mixed text (4)
91,232
10,137
Table 2: Label distribution in train and dev splits for task 2 dataset.
Model
ROC-AUC
Brier
C@1
F1
F0.5u
Mean
XGBoost
0.999
0.991
0.989
0.991
0.992
0.992
Light GBM
0.999
0.992
0.989
0.991
0.992
0.993
Logistic Regression
0.999
0.994
0.994
0.995
0.996
0.996
SVM
0.999
0.994
0.992
0.994
0.996
0.995
Ensemble
0.999
0.994
0.991
0.993
0.995
0.995
Table 3: Reported metrics for the classical ML models on the dev set.
Model
F1
Accuracy
Precision
Recall
Qwen2.5-0.5B
0.978
0.972
0.980
0.977
Qwen3-0.6B
0.993
0.991
0.993
0.993
Llama-3.1-8B
0.997
0.996
0.999
0.996
Mistral-7B-v0.1
0.995
0.993
0.997
0.992
Table 4: Reported metrics for the fine-tuning LLM approach on the dev set.
Model
Recall (Macro)
F1 (Macro)
Accuracy
Split
RoBERTa-base
0.750
0.592
0.541
Dev
RoBERTa-base
0.443
0.427
0.514
Test
RoBERTa-base
0.890
0.842
0.917
Dev (no samples from m4gt-bench)*
Table 5: Reported metrics for the combined transformer-based embeddings using layer-wise projection and attentive pooling on dev and test sets. *Based on the error analysis Figure 7 we reported the metrics on dev set removing the samples from m4gt-bench which proved to be the harderst to predict.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Task 1 dataset overview - train split
Figure 2: Task 1 dataset overview - dev split
Figure 3: Task 2 dataset overview - train split
Figure 4: Task 2 dataset overview - dev split
Figure 5: Qwen3-0.6B embeddings visualized in 2D using T-SNE (tarin + dev)
Hyperparameter
XGBoost
LightGBM
Logistic Regression
SVM
n_estimators
260
218
–
–
learning_rate
0.1114
0.2004
–
–
max_depth
8
9
–
–
num_leaves
–
37
–
–
subsample
0.8541
0.9007
–
–
colsample_bytree
0.8348
0.9641
–
–
Appendix
Table 6: Best hyperparameters for classical models used in the final ensemble model obtained using optuna.
Parameter
Value
load_in_4bit
True
load_in_8bit
False
use_double_quant
True
quant_type
nf4
compute_dtype
bfloat16
lora_r
8
Appendix
Table 7: QLoRA-related hyperparameters used for all fine-tuned LLMs.
Parameter
Value
batch_size
16
num_epochs
0.5
learning_rate
2e-5
lr_scheduler_type
constant
max_grad_norm
0.3
max_length
1024
Appendix
Table 8: Fine-tuning hyperparameters used for all LLMs.
Parameter
Value
model_name
FacebookAI/roberta-base
batch_size
4
num_epochs
5
learning_rate
0.0003
max_length
512
gradient_accumulation_steps
4
Appendix
Table 9: Training hyperparameters for RoBERTa-base fine-tuning.
Figure 6: Missed samples on the dev set by the ensemble model for subtask 1.
Figure 7: Heatmap of misclassified samples for each source dataset on dev set subtask 2.
Figure 8: Heatmap of misclassified samples for each model on dev set subtask 2.
Figure 9: Confusion matrix of samples generated using llama2 model on dev set subtask 2.
Recent advances in large language models (LLMs) have made it increasingly difficult to distinguish human-written text from AI-generated content. Many existing detectors train supervised neural classifiers that achieve strong in-distribution performance but are often opaque and can degrade substantially under distribution shift. We present READER, a reasoning-enhanced AI text detector that outputs both a human/AI label and a structured rationale describing the evidence for its decision. A key component of our approach is READ, a curated supervision set of rationales and verdicts. We fine-tune an LLM on READ to build READER, which reasons before detecting at inference time. Despite having only 1.5B parameters, READER consistently outperforms existing detectors as well as prompted, high-capacity LLM baselines (GPT-5.2, Gemini-3-Pro, and DeepSeek-V3.2), which are 100 to 1000 times larger in scale.
Pingfan Su, Kai Ye, Shijin Gong +4
Department of Statistics, London School of Economics and Political Science · School of Management, University of Science and Technology of China · School of Mathematics, University of Birmingham
The rapid proliferation of Large Language Models has significantly increased the difficulty of distinguishing between human-written and AI generated texts, raising critical issues across academic, editorial, and social domains. This paper investigates the problem of AI generated text detection through the design, implementation, and comparative evaluation of multiple machine learning based detectors. Four neural architectures are developed and analyzed: a Multilayer Perceptron, a one-dimensional Convolutional Neural Network, a MobileNet-based CNN, and a Transformer model. The proposed models are benchmarked against widely used online detectors, including ZeroGPT, GPTZero, QuillBot, Originality.AI, Sapling, IsGen, Rephrase, and Writer. Experiments are conducted on the COLING Multilingual Dataset, considering both English and Italian configurations, as well as on an original thematic dataset focused on Art and Mental Health. Results show that supervised detectors achieve more stable and robust performance than commercial tools across different languages and domains, highlighting key strengths and limitations of current detection strategies.
An ideal detection system for machine generated content is supposed to work well on any generator as many more advanced LLMs come into existence day by day. Existing systems often struggle with accurately identifying AI-generated content over shorter texts. Further, not all texts might be entirely authored by a human or LLM, hence we focused more over partial cases i.e human-LLM co-authored texts. Our paper introduces a set of models built for the task of token classification which are trained on an extensive collection of human-machine co-authored texts, which performed well over texts of unseen domains, unseen generators, texts by non-native speakers and those with adversarial inputs. We also introduce a new dataset of over 2.4M such texts mostly co-authored by several popular proprietary LLMs over 23 languages. We also present findings of our models' performance over each texts of each domain and generator. Additional findings include comparison of performance against each adversarial method, length of input texts and characteristics of generated texts compared to the original human authored texts.