A Measurement Perspective on Benchmarking LLM-Generated Text Detectors using Item Response Theory

Yanbin Zhao; Zhipeng Chen; Guangsheng Bao; Guandong Zhang; Yan Li

cs.CLOct 18, 2026

A Measurement Perspective on Benchmarking LLM-Generated Text Detectors using Item Response Theory

Authors: Yanbin Zhao, Zhipeng Chen, Guangsheng Bao, Guandong Zhang, Yan Li

Organizations: School of Mathematics and Statistics, Shanghai Polytechnic University, 2360 Jinhai Road, Pudong New District, 201209, Shanghai, China. · School of Computer and Information Engineering, Shanghai Polytechnic University, 2360 Jinhai Road, Pudong New District, 201209, Shanghai, China. · School of Computing and Artificial Intelligence, Fuyao University of Science and Technology, No. 104 Zhihui Avenue, Fuzhou High-tech Industrial Development Zone, 350109, Fujian, China.

Abstract

Benchmarking large language model (LLM)-generated text detectors commonly relies on aggregate detector-level metrics such as accuracy, F1 score, and AUC. Although useful for summarizing overall performance, these metrics provide limited insight into how benchmark composition shapes detector comparisons or which texts distinguish detectors with different response profiles. We propose a measurement-based benchmarking framework for jointly characterizing benchmark heterogeneity and detector behavior using multidimensional item response theory (IRT). By treating benchmark texts as measurement items and detectors as respondents, the framework estimates text-level discrimination and difficulty-location parameters alongside multidimensional detector abilities. We apply the framework to a benchmark of 32K human-written and machine-generated texts across four domains, evaluated by 14 detectors. The estimated IRT item-parameter space reveals distinct but overlapping human-machine distributions, with substantial variation across domains and text lengths. The human-machine label shows the strongest multivariate association with this space, while domain and text length further contribute to benchmark heterogeneity. Detector ability profiles reveal multidimensional and domain-dependent response patterns that are not fully reflected in aggregate performance rankings. As a practical application, clustering texts in the estimated IRT space yields a compact subset of 2,088 texts (6.5% of the full 32K benchmark) that largely preserves the comparative detector-performance patterns observed on the full benchmark. Overall, the proposed framework provides a principled and interpretable methodology for benchmarking heterogeneous detection technologies while jointly characterizing the measurement properties of the texts used for their evaluation.

Figures & tables

Domains	Original source size	Human-written	Machine-generated	Total selected
ArXiv	18,891	4,000	4,000	8,000
Writing Prompt	87,485	4,000	4,000	8,000
XSum	2,040,445	4,000	4,000	8,000
Yelp	650,000	4,000	4,000	8,000
Total	-	16,000	16,000	32,000

Table 1: Overview of data sources and benchmark composition across four domains.

Figure 1: Sample distributions across text length ranges in the four domains.

Detector	AUC	F1	ACC
DetectGPT	0.742 (1)	0.626 (14)	0.625 (3)
Likelihood	0.642 (2)	0.672 (7)	0.627 (2)
Lastde	0.628 (3)	0.688 (2)	0.617 (4)
LogRank	0.621 (4)	0.703 (1)	0.656 (1)
Rank	0.614 (5)	0.680 (4)	0.595 (8)
NPR	0.604 (6)	0.653 (10)	0.576 (10)

Table 2: Overall detection performance of different detectors on the sampled benchmark, measured by AUC, F1 score, and accuracy. Numbers in parentheses indicate the rank of each detector under the corresponding metric.

Effect	Df	Wilks’ $\Lambda$	Approx. $F$	$p$ -value
Domain	3	0.88950	127.3	$<0.001$
Length group	2	0.97000	49.1	$<0.001$
Group	1	0.37217	5392.7	$<0.001$
Domain $\times$ Length group	6	0.98805	6.4	$<0.001$
Domain $\times$ Group	3	0.83174	202.6	$<0.001$
Length group $\times$ Group	2	0.96048	65.1	$<0.001$

Table 4: MANOVA results for the globally estimated IRT parameter space.

Figure 2: First principal component visualization of the IRT parameter space. Each point represents a text sample, colored by its true group label.

Figure 3: First principal component visualization of the domain-specific IRT parameter space. Each point represents a text sample, colored by its true group label.

		$\alpha_{1}$	$\alpha_{2}$	$\alpha_{3}$	$\alpha_{4}$	$\alpha_{5}$	$\beta_{1}$	$\beta_{2}$	$\beta_{3}$	$\beta_{4}$	$\beta_{5}$
Domain	ArXiv	0.289	0.221	0.073	0.899	0.126	0.668	0.640	0.180	0.149	0.700
	Writing Prompt	1.252	4.896	2.060	8.347	2.757	2.387	0.977	0.400	0.022	1.811
	XSum	1.375	3.129	1.895	4.760	2.299	3.005	1.128	0.309	0.065	3.264
	Yelp	2.754	13.446	2.687	15.770	6.604	3.459	1.151	0.574	0.134	1.812
Length	Short	1.395	7.470	2.122	7.918	4.507	2.292	1.010	0.377	0.070	1.561
	Medium	0.911	4.585	1.401	6.434	1.418	1.371	0.720	0.319	0.038	1.016

Table 5: Domain-specific and overall summary of

KL(M||H)

between machine-generated and human-written text distributions for the estimated IRT discrimination (

\alpha

) and difficulty-location (

\beta

) parameters across five latent dimensions.

Category	Length (words)	Machine	Human	Total
Short	$\leq 150$	5588	5390	10978
Medium	$[151,250]$	5301	5392	10693
Long	$\geq 251$	5111	5218	10329

Table 6: Distribution of benchmark samples across text length groups.

Figure 4: First principal component visualization of the text length stratified IRT parameter space. Each point represents a text sample, colored by its true group label.

Detector	$\theta_{1}$	$\theta_{2}$	$\theta_{3}$	$\theta_{4}$	$\theta_{5}$
Binoculars	-	0.02	1.89	-	0.02
DetectGPT	-	-0.03	0.49	-	0.32
DNA-GPT	0.02	0.03	0.91	-	0.16
Entropy	-	0.03	0.84	-	0.25
Fast-DetectGPT	-	0.02	2.00	-	0.03
Likelihood	0.02	0.02	1.70	-	0.15

Table 7: Rotated

\theta

estimates for all detectors from the five-dimensional IRT model. A dash indicates that the corresponding absolute value is below 0.015.

	ArXiv					Writing Prompt
	$\theta_{1}$	$\theta_{2}$	$\theta_{3}$	$\theta_{4}$	$\theta_{5}$	$\theta_{1}$	$\theta_{2}$	$\theta_{3}$	$\theta_{4}$	$\theta_{5}$
Binoculars	-0.09	-1.82	-	-0.05	-0.21	-0.03	-2.30	-0.22	-	-
DetectGPT	0.13	-1.26	-	-0.02	-0.21	-	-1.20	-0.42	-	-
DNA-GPT	-	-1.87	-	-	-0.19	0.02	-2.38	-0.15	-0.06	-
Entropy	0.10	-2.18	-	-0.02	-0.12	-	-1.91	-0.07	-0.14	-
Fast-DetectGPT	-0.08	-1.90	-	-0.05	-0.23	-	-2.69	-0.20	-	-

Table 8: Domain-specific detector ability estimates (

\theta

) under the five-dimensional IRT model. The ability parameters are re-estimated separately within each domain. Rows correspond to detectors, and columns report the estimated ability values for each latent dimension within ArXiv, Writing Prompt, XSum, and Yelp Reviews. A dash indicates that the corresponding absolute value is below 0.015.

Figure 5: Spearman correlation heatmap between detector ability estimates ( $\theta$ ) and detection performance metrics across overall and domain-specific benchmarks. Rows represent benchmark-metric pairs, columns represent latent IRT dimensions, and colors indicate the Spearman correlation coefficient $\rho$ . The heatmap shows that different domains and metrics align with different ability dimensions. For domain-specific rows, the dimensions refer to the corresponding within-domain latent spaces and are not directly aligned across domains.

Figure 6: Difference between subset-based accuracy and full-benchmark accuracy under different values of $K$ and $n_{i}$ . Colors indicate the magnitude of $\Delta\mathrm{ACC}$ .

Domains	Human	Machine
ArXiv	258	191
Writing Prompt	324	190
XSum	324	250
Yelp	337	214
All	1243	845

Table 9: Distribution of the selected subsample across domains and classes under the clustering-based subsampling scheme with

K=12

and

n_{i}=174

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts across the five latent dimensions. Each panel reports the corresponding KL divergence

KL(M||H)

, which measures the divergence of the machine-generated distribution from the human-written reference distribution.

Figure S2: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the ArXiv domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

Figure S3: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the Writing Prompt domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

Figure S4: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the XSum domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

Figure S5: Marginal distributions of the estimated discrimination parameters

\alpha

and difficulty-related location parameters

\beta

for human-written and machine-generated texts in the Yelp Reviews domain across the five latent dimensions. Each panel reports the corresponding domain-specific KL divergence

KL(M||H)

		$\alpha_{1}$	$\alpha_{2}$	$\alpha_{3}$	$\alpha_{4}$	$\alpha_{5}$	$\beta_{1}$	$\beta_{2}$	$\beta_{3}$	$\beta_{4}$	$\beta_{5}$
Domain	ArXiv	1.210	0.450	0.115	1.509	0.677	3.016	4.561	0.270	0.171	2.852
	Writing Prompt	1.739	2.100	1.588	3.737	1.776	1.998	1.733	0.378	0.013	2.058
	XSum	2.639	2.362	1.140	2.411	2.710	2.638	3.101	0.426	0.046	2.715
	Yelp	2.162	3.255	1.811	3.606	2.692	2.534	1.917	0.360	0.084	2.109
Length	Short	2.204	2.756	1.465	2.874	2.890	2.855	4.173	0.434	0.022	2.444
	Medium	1.544	2.137	0.827	2.184	1.375	1.752	1.943	0.297	0.028	3.057

Appendix

Table S1: Domain-specific, length-stratified, and overall summary of

KL(H\|M)

between human-written and machine-generated text distributions for the estimated IRT discrimination (

\alpha

) and difficulty-related location (

\beta

) parameters across five latent dimensions. The Overall row is computed from the pooled human-written and machine-generated distributions across all domains and is not the average of the domain-specific KL values.

Figure S6: Detector-specific accuracy differences as the number of selected samples per cluster varies under

K=12

. Colored curves correspond to individual detectors, and the black curve represents the mean across detectors.

Figure S7: Domain-specific differences between subset-based and full-benchmark accuracies as the number of selected samples per cluster varies, with

K=16

for ArXiv,

K=10

for Writing Prompt,

K=15

for XSum, and

K=14

for Yelp Reviews. Colored curves represent individual detectors, and the black curve shows the mean across detectors.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 171 papers

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions
Mar 18, 2026Madhav S. Baidya, S. S. Baidya, Chirag ChawlaAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India
Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy
Mar 24, 2026Shushanta Pudasaini, Luis Miralles-Pechuán, David Lillis +1AI-Generated Text Detection Robustness of Machine-Generated Text Detection
Technological University Dublin · University College Dublin
DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection
May 15, 2026Junchao Wu, Yefeng Liu, Chenyu Zhu +8Multilingual Language Model Evaluation AI-Generated Text Detection
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Alibaba Group · Xiamen University

Mar 18, 2026J/K move · Enter open · S save

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

Madhav S. Baidya, S. S. Baidya, Chirag Chawla

Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.

Abstract

AI-Generated Text Detection

Explore similar work

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection