AI Evaluation Should Require Standardized Item-Level Data Releases

Han Jiang; Susu Zhang; Dongyao Zhu; Yuzhuo Bai; Sang T. Truong; Xiaoyuan Yi; Sanmi Koyejo; Xing Xie

cs.AIFeb 27, 2026

AI Evaluation Should Require Standardized Item-Level Data Releases

Authors: Han Jiang, Susu Zhang, Dongyao Zhu, Yuzhuo Bai, Sang T. Truong, Xiaoyuan Yi, Sanmi Koyejo, Xing Xie,

Organizations: Johns Hopkins University · University of Illinois Urbana-Champaign · North Carolina State University · Tsinghua University ·

Abstract

This position paper argues that standardized item-level benchmark data should become the default infrastructure for AI evaluation. Current evaluations suffer from underspecified item selection, construct misalignment, and poor generalization; a disproportionate focus on aggregate model scores compounds these failures by keeping them undetected and unfixable. Without item-level evidence, validity claims cannot be assessed, resulting in inflated capability claims, misdirected research, and unwarranted trust in deployed systems. Our position is that valid evaluation requires empirical evidence from item-level benchmark data, including item content, test conditions, model responses, and item-level score statistics, and that the standardized release of such data should be treated as core AI evaluation infrastructure. Such releases also enable transparency, replicability, and auditability of evaluation results. To show that the norm is both feasible and consequential, we construct OpenEval, an item-level repository of 10.7M+ responses across 160K+ items from widely-used benchmarks, under a unified schema that the AI evaluation community can build upon. We demonstrate how item-level data can identify low-quality items, document construct misalignment, and recover validity evidence about benchmarks' internal structure. We address objections concerning contamination and coordination burden, and show that both are tractable relative to the costs of decisions made on inadequately validated evaluative claims.

Figures & tables

Figure 1: Illustrative overview of our position. Item-level benchmark data serves as the missing foundation between the current aggregate paradigm and the science of AI evaluation.

Figure 2: Benchmark-level accuracy distributions for 66 pre–Nov. 2023 models on MMLU and 72 post–Jun. 2024 models on MMLU-Pro .

Figure 3: Unified data schema of OpenEval . The left column shows the hierarchical storage structure with indexing keys; the right side expands each entity into its constituent fields. Diamond-tipped connectors point from parent to subfield; dashed connectors indicate folded siblings. 1:1 and 1:N denote cardinality, e.g., each response contains exactly one model but may contain multiple scores. Asterisks (*) mark fields accepting values in arbitrary formats.

Figure 4: (a) Item characteristics distribution for items from MMLU and MMLU-Pro . Higher item difficulty values correspond to harder items. (b) ICCs for three items in MMLU .

Figure 6: Convergent/discriminant evidence of the four sub-constructs (#1 - #4) on MMLU-Pro .

Factor	Representative Item	Potential Sub-Construct
MMLU-Pro #1	A 10kVA, 2400/240V, single-phase transformer has the following
	resistances and leakage reactance. Find the primary voltage	Formal, quantitative,
	required to produce 240V at the secondary terminals at full load,	multi-step modeling
	when the load power factor is 0.8 power factor lagging/leading.
MMLU-Pro #2	What are the principal and selective instruments of control of which	Domain-specific recall
MMLU-Pro #2	the Federal Reserve System makes use?	and simple reasoning

Table 2: Representative items with large absolute item factor loadings and possible constructs for each GLRM factor in MMLU-Pro . The top 100 representative items are interpreted and summarized into a candidate label by GPT-5, then manually revised.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: OpenEval data schema (item). The value of each field specifies the data type, presence requirement, and a short explanation of the field. Specifically, auto means the field is auto-generated, optional means the field can be absent from the data entry, required means the data contributors are required to specify this field, and non-empty means the field cannot be left blank.

Figure 8: OpenEval data schema (response). Each response object is an element of the responses field in Fig. 7 .

Figure 9: OpenEval data schema (model). The model object here corresponds to the model field in Fig. 8 .

Figure 10: OpenEval data schema (score). Each score object is an element of the scores field in Fig. 8 .

Analysis	Estimate	Correlation	SE	90% CI Width
Bootstrap	Difficulty	0.9916	0.0523	0.1714
Bootstrap	Discrimination	0.9732	0.0691	0.2295
Leave-one-	Difficulty	0.9988	0.0718	0.2361
developer-out	Discrimination	0.9973	0.0646	0.2127

Appendix

Table 3: Sensitivity of CTT estimates on 1,000 MMLU-Pro items. The correlation reference is the original 72-model sample for bootstrap resampling and the full 122-model sample for leave-one-developer-out analysis. For leave-one-developer-out analysis, SEs and interval widths are jackknife-style descriptive summaries.

		Full-Sample Correlation
Developer	Model Count	Difficulty	Discrimination
Qwen	27	0.9951	0.9899
OpenAI	21	0.9936	0.9882
Google	19	0.9976	0.9924
Anthropic	10	0.9980	0.9952
Meta	8	0.9993	0.9976

Appendix

Table 4: Stability of CTT estimates after excluding each of the five largest model developers. Correlations are computed across items against the full 122-model estimates; model counts indicate the number of models excluded.

$p_{i}$ Bin	Dataset	Bin Size	Mean Disc.	$p$ -value
$(0.0,0.2)$	MMLU	144	-0.0174	$2.48\times 10^{-3}$
$(0.0,0.2)$	MMLU-Pro	86	0.0615	$2.48\times 10^{-3}$
$[0.2,0.4)$	MMLU	172	0.1444	$3.43\times 10^{-6}$
$[0.2,0.4)$	MMLU-Pro	86	0.2933	$3.43\times 10^{-6}$
$[0.4,0.6)$	MMLU	175	0.4161	$3.33\times 10^{-1}$
$[0.4,0.6)$	MMLU-Pro	152	0.4708	$3.33\times 10^{-1}$

Appendix

Table 5: Item discrimination within difficulty-index bins for MMLU and MMLU-Pro . A larger difficulty-index

p_{i}

indicates an easier item. The

p

-values are from within-bin Mann–Whitney U tests; 22 zero-variance MMLU-Pro items were excluded.

Factor	Representative Item	Potential Sub-Construct
MMLU #1	Which one of the following statements best describes the algebraic	Domain-specific canonical
MMLU #1	representation of the fitted regression line?	framework knowledge
	Based on the paper “SoK: SSL and HTTPS: Revisiting past	Applied synthesis and
MMLU #2	challenges and evaluating certificates trust model enhancements”,	case-based judgment
	which of the following statements are false?	case-based judgment
MMLU #3	Find the product of the given polynomials in the given polynomial	Formal, quantitative,

Appendix

Table 6: Representative items with large absolute item factor loadings and possible constructs for each GLRM factor in MMLU . The top 100 representative items are interpreted and summarized into a candidate label by GPT-5, then manually revised.

Figure 11: Convergent/discriminant evidence of the five sub-constructs (#1 - # 5) on MMLU .

Figure 12: Clusters from four benchmark datasets in HELM revealed by K-means clustering over item factor loadings from GLRM.

Figure 13: Example items with different maximum factor loadings within the same subject (psychology and physics) in MMLU-Pro .

Benchmark Validity

Benchmark validity concerns whether evaluation tasks, scores, and rankings support the capability claims researchers make about AI systems.

See all 174 papers

Can We Trust Item Response Theory for AI Evaluation?
Jul 16, 2026Han Jiang, Sunbeom Kwon, Jinwen Luo +2LLM Evaluation Item Response Theory
Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Jun 12, 2026Jan Batzner, Sree Harsha Nelaturu, Damian Stachura +45
The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains
May 11, 2026Jung Min KangBenchmark Design Benchmark Validity
Independent Researcher · Seoul, South Korea

Jul 16, 2026J/K move · Enter open · S save

Can We Trust Item Response Theory for AI Evaluation?

Han Jiang, Sunbeom Kwon, Jinwen Luo, Ziang Xiao+1

Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles

LLM Evaluation Item Response Theory

Open paper AskPDF arXiv

AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate capabilities, rank systems, select informative examples, and diagnose benchmark quality. However, AI benchmark data often departs from the data regimes encountered in human testing, for which standard IRT estimation tools were originally developed: these benchmarks typically involve fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal. We examine how these regime mismatches challenge the reliability of IRT modeling for AI evaluation. Using item parameters and capability distributions derived from six widely used LLM benchmarks, we simulate response matrices under three common IRT models and compare four estimators used in recent benchmark studies: marginal maximum likelihood, Markov chain Monte Carlo, variational inference, and a neural pseudo-Siamese estimator. Across 18,000 simulation runs, we systematically evaluate computational feasibility, scalability, and the reliability of IRT inferences about model rankings, predicted performance, and item characteristics. Results show that classical estimators can become computationally infeasible for large benchmarks, whereas scalable estimators can produce unreliable item-level and ranking inferences with small or non-normally distributed model sets. This study identifies when latent trait models reliably support or risk distorting AI benchmarking claims, and what sample sizes and diagnostics are needed for trustworthy use.

Abstract

Benchmark Validity

Explore similar work

Can We Trust Item Response Theory for AI Evaluation?

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains