Unibuc - NLP at "Voight-Kampff" Generative AI Detection PAN 2025
Authors: Teodor Marchitan, Claudiu Creanga, Liviu P. Dinu
Organizations: Faculty of Mathematics and Computer Science, University of Bucharest, Romania · HLT Research Center, Bucharest, Romania · Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania
This paper describes the approach of the Unibuc - NLP team in tackling the "Voight-Kampff" Generative AI Detection task from PAN at CLEF 2025. In the Subtask 1: Voight-Kampff AI Detection Sensitivity we have explored different approaches of leveraging embeddings extracted from LLMs, using either classical ML models or further fine-tuning LLMs with a classification head for the downstream binary classification task. Our experiments showed that using a majority vote ensemble of classical ML models (Light GBM, XGB, Logistic Regression and SVM) is comparable with fine-tuning for downstream task achieving around 99.4% F1 score. For the Subtask 2: Human-AI Collaborative Text Classification we explored a way of combining features at different layers extracted using transformer backbone with layer-wise projection and attentive pooling which proved modest results.
Figures & tables
human-written (0)
ai-written (1)
Total
Train
9,101
14,606
23,707
Dev
1,277
2,312
3,589
Table 1: Label distribution in train and dev splits for task 1 dataset.
Label category
Train
Dev
Fully human-written (0)
14,910
225
Human-initiated, then machine-continued (3)
1,368
510
Human-written, then machine-polished (1)
10,740
37,170
Machine-written, then machine-humanized (obfuscated) (2)
75,270
12,330
Machine-written, then human-edited (5)
95,398
12,289
Deeply-mixed text (4)
91,232
10,137
Table 2: Label distribution in train and dev splits for task 2 dataset.
Model
ROC-AUC
Brier
C@1
F1
F0.5u
Mean
XGBoost
0.999
0.991
0.989
0.991
0.992
0.992
Light GBM
0.999
0.992
0.989
0.991
0.992
0.993
Logistic Regression
0.999
0.994
0.994
0.995
0.996
0.996
SVM
0.999
0.994
0.992
0.994
0.996
0.995
Ensemble
0.999
0.994
0.991
0.993
0.995
0.995
Table 3: Reported metrics for the classical ML models on the dev set.
Model
F1
Accuracy
Precision
Recall
Qwen2.5-0.5B
0.978
0.972
0.980
0.977
Qwen3-0.6B
0.993
0.991
0.993
0.993
Llama-3.1-8B
0.997
0.996
0.999
0.996
Mistral-7B-v0.1
0.995
0.993
0.997
0.992
Table 4: Reported metrics for the fine-tuning LLM approach on the dev set.
Model
Recall (Macro)
F1 (Macro)
Accuracy
Split
RoBERTa-base
0.750
0.592
0.541
Dev
RoBERTa-base
0.443
0.427
0.514
Test
RoBERTa-base
0.890
0.842
0.917
Dev (no samples from m4gt-bench)*
Table 5: Reported metrics for the combined transformer-based embeddings using layer-wise projection and attentive pooling on dev and test sets. *Based on the error analysis Figure 7 we reported the metrics on dev set removing the samples from m4gt-bench which proved to be the harderst to predict.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Task 1 dataset overview - train split
Figure 2: Task 1 dataset overview - dev split
Figure 3: Task 2 dataset overview - train split
Figure 4: Task 2 dataset overview - dev split
Figure 5: Qwen3-0.6B embeddings visualized in 2D using T-SNE (tarin + dev)
Hyperparameter
XGBoost
LightGBM
Logistic Regression
SVM
n_estimators
260
218
–
–
learning_rate
0.1114
0.2004
–
–
max_depth
8
9
–
–
num_leaves
–
37
–
–
subsample
0.8541
0.9007
–
–
colsample_bytree
0.8348
0.9641
–
–
Appendix
Table 6: Best hyperparameters for classical models used in the final ensemble model obtained using optuna.
Parameter
Value
load_in_4bit
True
load_in_8bit
False
use_double_quant
True
quant_type
nf4
compute_dtype
bfloat16
lora_r
8
Appendix
Table 7: QLoRA-related hyperparameters used for all fine-tuned LLMs.
Parameter
Value
batch_size
16
num_epochs
0.5
learning_rate
2e-5
lr_scheduler_type
constant
max_grad_norm
0.3
max_length
1024
Appendix
Table 8: Fine-tuning hyperparameters used for all LLMs.
Parameter
Value
model_name
FacebookAI/roberta-base
batch_size
4
num_epochs
5
learning_rate
0.0003
max_length
512
gradient_accumulation_steps
4
Appendix
Table 9: Training hyperparameters for RoBERTa-base fine-tuning.
Figure 6: Missed samples on the dev set by the ensemble model for subtask 1.
Figure 7: Heatmap of misclassified samples for each source dataset on dev set subtask 2.
Figure 8: Heatmap of misclassified samples for each model on dev set subtask 2.
Figure 9: Confusion matrix of samples generated using llama2 model on dev set subtask 2.
Department of Statistics, London School of Economics and Political Science · School of Management, University of Science and Technology of China · School of Mathematics, University of Birmingham