A Measurement Perspective on Benchmarking LLM-Generated Text Detectors using Item Response Theory

Yanbin Zhao; Zhipeng Chen; Guangsheng Bao; Guandong Zhang; Yan Li

cs.CLOct 18, 2026

A Measurement Perspective on Benchmarking LLM-Generated Text Detectors using Item Response Theory

Authors: Yanbin Zhao, Zhipeng Chen, Guangsheng Bao, Guandong Zhang, Yan Li

Organizations: School of Mathematics and Statistics, Shanghai Polytechnic University, 2360 Jinhai Road, Pudong New District, 201209, Shanghai, China. · School of Computer and Information Engineering, Shanghai Polytechnic University, 2360 Jinhai Road, Pudong New District, 201209, Shanghai, China. · School of Computing and Artificial Intelligence, Fuyao University of Science and Technology, No. 104 Zhihui Avenue, Fuzhou High-tech Industrial Development Zone, 350109, Fujian, China.

Abstract

Benchmarking large language model (LLM)-generated text detectors commonly relies on aggregate detector-level metrics such as accuracy, F1 score, and AUC. Although useful for summarizing overall performance, these metrics provide limited insight into how benchmark composition shapes detector comparisons or which texts distinguish detectors with different response profiles. We propose a measurement-based benchmarking framework for jointly characterizing benchmark heterogeneity and detector behavior using multidimensional item response theory (IRT). By treating benchmark texts as measurement items and detectors as respondents, the framework estimates text-level discrimination and difficulty-location parameters alongside multidimensional detector abilities. We apply the framework to a benchmark of 32K human-written and machine-generated texts across four domains, evaluated by 14 detectors. The estimated IRT item-parameter space reveals distinct but overlapping human-machine distributions, with substantial variation across domains and text lengths. The human-machine label shows the strongest multivariate association with this space, while domain and text length further contribute to benchmark heterogeneity. Detector ability profiles reveal multidimensional and domain-dependent response patterns that are not fully reflected in aggregate performance rankings. As a practical application, clustering texts in the estimated IRT space yields a compact subset of 2,088 texts (6.5% of the full 32K benchmark) that largely preserves the comparative detector-performance patterns observed on the full benchmark. Overall, the proposed framework provides a principled and interpretable methodology for benchmarking heterogeneous detection technologies while jointly characterizing the measurement properties of the texts used for their evaluation.

Figures & tables

Domains	Original source size	Human-written	Machine-generated	Total selected
ArXiv	18,891	4,000	4,000	8,000
Writing Prompt	87,485	4,000	4,000	8,000
XSum	2,040,445	4,000	4,000	8,000
Yelp	650,000	4,000	4,000	8,000
Total	-	16,000	16,000	32,000

Table 1: Overview of data sources and benchmark composition across four domains.

Figure 1: Sample distributions across text length ranges in the four domains.

Detector	AUC	F1	ACC
DetectGPT	0.742 (1)	0.626 (14)	0.625 (3)
Likelihood	0.642 (2)	0.672 (7)	0.627 (2)
Lastde	0.628 (3)	0.688 (2)	0.617 (4)
LogRank	0.621 (4)	0.703 (1)	0.656 (1)
Rank	0.614 (5)	0.680 (4)	0.595 (8)
NPR	0.604 (6)	0.653 (10)	0.576 (10)

Table 2: Overall detection performance of different detectors on the sampled benchmark, measured by AUC, F1 score, and accuracy. Numbers in parentheses indicate the rank of each detector under the corresponding metric.

Effect	Df	Wilks’ $\Lambda$	Approx. $F$	$p$ -value
Domain	3	0.88950	127.3	$<0.001$
Length group	2	0.97000	49.1	$<0.001$
Group	1	0.37217	5392.7	$<0.001$
Domain $\times$ Length group	6	0.98805	6.4	$<0.001$
Domain $\times$ Group	3	0.83174	202.6	$<0.001$
Length group $\times$ Group	2	0.96048	65.1	$<0.001$

Table 4: MANOVA results for the globally estimated IRT parameter space.

Figure 2: First principal component visualization of the IRT parameter space. Each point represents a text sample, colored by its true group label.

Figure 3: First principal component visualization of the domain-specific IRT parameter space. Each point represents a text sample, colored by its true group label.

		$\alpha_{1}$	$\alpha_{2}$	$\alpha_{3}$	$\alpha_{4}$	$\alpha_{5}$	$\beta_{1}$	$\beta_{2}$	$\beta_{3}$	$\beta_{4}$	$\beta_{5}$
Domain	ArXiv	0.289	0.221	0.073	0.899	0.126	0.668	0.640	0.180	0.149	0.700
	Writing Prompt	1.252	4.896	2.060	8.347	2.757	2.387	0.977	0.400	0.022	1.811
	XSum	1.375	3.129	1.895	4.760	2.299	3.005	1.128	0.309	0.065	3.264
	Yelp	2.754	13.446	2.687	15.770	6.604	3.459	1.151	0.574	0.134	1.812
Length	Short	1.395	7.470	2.122	7.918	4.507	2.292	1.010	0.377	0.070	1.561
	Medium	0.911	4.585	1.401	6.434	1.418	1.371	0.720	0.319	0.038	1.016

Table 5: Domain-specific and overall summary of

KL(M||H)

between machine-generated and human-written text distributions for the estimated IRT discrimination (

\alpha

) and difficulty-location (

\beta

) parameters across five latent dimensions.

Category	Length (words)	Machine	Human	Total
Short	$\leq 150$	5588	5390	10978
Medium	$[151,250]$	5301	5392	10693
Long	$\geq 251$	5111	5218	10329

Table 6: Distribution of benchmark samples across text length groups.

Figure 4: First principal component visualization of the text length stratified IRT parameter space. Each point represents a text sample, colored by its true group label.

Detector	$\theta_{1}$	$\theta_{2}$	$\theta_{3}$	$\theta_{4}$	$\theta_{5}$
Binoculars	-	0.02	1.89	-	0.02
DetectGPT	-	-0.03	0.49	-	0.32
DNA-GPT	0.02	0.03	0.91	-	0.16
Entropy	-	0.03	0.84	-	0.25
Fast-DetectGPT	-	0.02	2.00	-	0.03
Likelihood	0.02	0.02	1.70	-	0.15

Table 7: Rotated

\theta

estimates for all detectors from the five-dimensional IRT model. A dash indicates that the corresponding absolute value is below 0.015.

	ArXiv					Writing Prompt
	$\theta_{1}$	$\theta_{2}$	$\theta_{3}$	$\theta_{4}$	$\theta_{5}$	$\theta_{1}$	$\theta_{2}$	$\theta_{3}$	$\theta_{4}$	$\theta_{5}$
Binoculars	-0.09	-1.82	-	-0.05	-0.21	-0.03	-2.30	-0.22	-	-
DetectGPT	0.13	-1.26	-	-0.02	-0.21	-	-1.20	-0.42	-	-
DNA-GPT	-	-1.87	-	-	-0.19	0.02	-2.38	-0.15	-0.06	-
Entropy	0.10	-2.18	-	-0.02	-0.12	-	-1.91	-0.07	-0.14	-
Fast-DetectGPT	-0.08	-1.90	-	-0.05	-0.23	-	-2.69	-0.20	-	-

Table 8: Domain-specific detector ability estimates (

\theta

) under the five-dimensional IRT model. The ability parameters are re-estimated separately within each domain. Rows correspond to detectors, and columns report the estimated ability values for each latent dimension within ArXiv, Writing Prompt, XSum, and Yelp Reviews. A dash indicates that the corresponding absolute value is below 0.015.

Figure 5: Spearman correlation heatmap between detector ability estimates ( $\theta$ ) and detection performance metrics across overall and domain-specific benchmarks. Rows represent benchmark-metric pairs, columns represent latent IRT dimensions, and colors indicate the Spearman correlation coefficient $\rho$ . The heatmap shows that different domains and metrics align with different ability dimensions. For domain-specific rows, the dimensions refer to the corresponding within-domain latent spaces and are not directly aligned across domains.

Figure 6: Difference between subset-based accuracy and full-benchmark accuracy under different values of $K$ and $n_{i}$ . Colors indicate the magnitude of $\Delta\mathrm{ACC}$ .

Domains	Human	Machine
ArXiv	258	191
Writing Prompt	324	190
XSum	324	250
Yelp	337	214
All	1243	845

Table 9: Distribution of the selected subsample across domains and classes under the clustering-based subsampling scheme with

K=12

and

n_{i}=174

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts across the five latent dimensions. Each panel reports the corresponding KL divergence

KL(M||H)

, which measures the divergence of the machine-generated distribution from the human-written reference distribution.

Figure S2: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the ArXiv domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

Figure S3: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the Writing Prompt domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

Figure S4: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the XSum domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

Figure S5: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the Yelp Reviews domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

		$\alpha_{1}$	$\alpha_{2}$	$\alpha_{3}$	$\alpha_{4}$	$\alpha_{5}$	$\beta_{1}$	$\beta_{2}$	$\beta_{3}$	$\beta_{4}$	$\beta_{5}$
Domain	ArXiv	1.210	0.450	0.115	1.509	0.677	3.016	4.561	0.270	0.171	2.852
	Writing Prompt	1.739	2.100	1.588	3.737	1.776	1.998	1.733	0.378	0.013	2.058
	XSum	2.639	2.362	1.140	2.411	2.710	2.638	3.101	0.426	0.046	2.715
	Yelp	2.162	3.255	1.811	3.606	2.692	2.534	1.917	0.360	0.084	2.109
Length	Short	2.204	2.756	1.465	2.874	2.890	2.855	4.173	0.434	0.022	2.444
	Medium	1.544	2.137	0.827	2.184	1.375	1.752	1.943	0.297	0.028	3.057

Appendix

Table S1: Domain-specific, length-stratified, and overall summary of

KL(H\|M)

between human-written and machine-generated text distributions for the estimated IRT discrimination (

\alpha

) and difficulty-related location (

\beta

) parameters across five latent dimensions. The Overall row is computed from the pooled human-written and machine-generated distributions across all domains and is not the average of the domain-specific KL values.

Figure S6: Detector-specific accuracy differences as the number of selected samples per cluster varies under

K=12

. Colored curves correspond to individual detectors, and the black curve represents the mean across detectors.

Figure S7: Domain-specific differences between subset-based and full-benchmark accuracies as the number of selected samples per cluster varies, with

K=16

for ArXiv,

K=10

for Writing Prompt,

K=15

for XSum, and

K=14

for Yelp Reviews. Colored curves represent individual detectors, and the black curve shows the mean across detectors.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 171 papers

Mar 18, 2026

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.

Madhav S. Baidya, S. S. Baidya, Chirag Chawla

Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Ask PDF

Mar 24, 2026

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

The widespread adoption of Large Language Models (LLMs) has made the detection of AI-Generated text a pressing and complex challenge. Although many detection systems report high benchmark accuracy, their reliability in real-world settings remains uncertain, and their interpretability is often unexplored. In this work, we investigate whether contemporary detectors genuinely identify machine authorship or merely exploit dataset-specific artefacts. We propose an interpretable detection framework that integrates linguistic feature engineering, machine learning, and explainable AI techniques. When evaluated on two prominent benchmark corpora, namely PAN CLEF 2025 and COLING 2025, our model trained on 30 linguistic features achieves leaderboard-competitive performance, attaining an F1 score of 0.9734. However, systematic cross-domain and cross-generator evaluation reveals substantial generalisation failure: classifiers that excel in-domain degrade significantly under distribution shift. Using SHAP- based explanations, we show that the most influential features differ markedly between datasets, indicating that detectors often rely on dataset-specific stylistic cues rather than stable signals of machine authorship. Further investigation with in-depth error analysis exposes a fundamental tension in linguistic-feature-based AI text detection: the features that are most discriminative on in-domain data are also the features most susceptible to domain shift, formatting variation, and text-length effects. We believe that this knowledge helps build AI detectors that are robust across different settings. To support replication and practical use, we release an open-source Python package that returns both predictions and instance-level explanations for individual texts.

Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis +1

Technological University Dublin · University College Dublin

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Ask PDF

May 15, 2026

DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection

The effective detection and governance of Large Language Model (LLM) generated content has become increasingly critical due to the growing risk of misuse. Despite the impressive performance of existing detectors, their reliability and potential in multilingual, real-world scenarios remain largely underexplored. In this study, we introduce DetectRL-X, a comprehensive multilingual benchmark designed to evaluate advanced detectors across 8 dimensions. The benchmark encompasses 8 languages commonly used in commercial contexts and collects human-written texts from 6 domains highly susceptible to LLM misuse. To better aligned with real-world applications, We create LLM-generated texts using 4 popular commercial LLMs, and include typical AI-assisted writing operations such as polishing, expanding, and condensing to capture authentic usage patterns. Furthermore, we develop a multilingual framework for paraphrasing and perturbation attacks to simulate diverse human modifications and writing noise, enabling stress testing of detectors across languages. Experimental results on DetectRL-X reveal the strengths and limitations of current state-of-the-art detectors when applied to diverse linguistic resources. We further analyze how domains, generators, attack strategies, text length, and refinement operations influence performance in different languages, underscoring DetectRL-X as an effective benchmark for strengthening multilingual and language-specific detectors.

Junchao Wu, Yefeng Liu, Chenyu Zhu +8

NLP2CT Lab, Department of Computer and Information Science, University of Macau · Alibaba Group · Xiamen University

Multilingual Language Model Evaluation AI-Generated Text Detection Adversarial Robustness Robustness of Machine-Generated Text Detection

Ask PDF

Abstract

AI-Generated Text Detection

Explore similar work

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection