Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients

Michael Hardy; Joshua Gilbert; Benjamin Domingue

stat.APMar 26, 2026

Efficient Detection of Bad Benchmark Items with Novel Scalability Coefficients

Authors: Michael Hardy, Joshua Gilbert, Benjamin Domingue

Organizations: Stanford University, CA, United States · Harvard University, MA, United States

Abstract

The validity of assessments, from large-scale AI benchmarks to human classrooms, depends on the quality of individual items, yet modern evaluation instruments often contain thousands of items with minimal psychometric vetting. We introduce a new family of nonparametric scalability coefficients based on interitem isotonic regression for efficiently detecting globally bad items (e.g., miskeyed, ambiguously worded, or construct-misaligned). The central contribution is the signed isotonic $R^2$ , which measures the maximal proportion of variance in one item explainable by a monotone function of another while preserving the direction of association via Kendall's $τ$ . Aggregating these pairwise coefficients yields item-level scores that sharply separate problematic items from acceptable ones without assuming linearity or committing to a parametric item response model. We show that the signed isotonic $R^2$ is extremal among monotone predictors (it extracts the strongest possible monotone signal between any two items) and show that this optimality property translates directly into practical screening power. Across three AI benchmark datasets (HS Math, GSM8K, MMLU) and two human assessment datasets, the signed isotonic $R^2$ consistently achieves top-tier AUC for ranking bad items above good ones, outperforming or matching a comprehensive battery of classical test theory, item response theory, and dimensionality-based diagnostics. Crucially, the method remains robust under the small-n/large-p conditions typical of AI evaluation, requires only bivariate monotone fits computable in seconds, and handles mixed item types (binary, ordinal, continuous) without modification. It is a lightweight, model-agnostic filter that can materially reduce the reviewer effort needed to find flawed items in modern large-scale evaluation regimes.

Figures & tables

Figure 1: Examples of Differences in Item Detection Efficiencies based on Technique :Bad Item detection as ordered by each item-fit metric. iso =

M_{iso}

; tau =

M_{\tau}

; adj_depth = Isolation Forest Tree Depth (anomaly detection), z_outfit_3 = standardized absolute 3PL outfit statistic, a1_3 and a1_2 = discrimination parameter for 3PL and 2PL, respectively; Hi =

H_{i}

; Zi =

Z_{i}

, g_15 = loading on the general factor of a 15-factor estimation of McDonald’s

\omega

; cmean = Mean inter-item tetrachoric correlation; cneg = proportion of inter-item tetrachoric correlations ¡ 0; alpha_drop = benchmark reliability (Cronbach’s

\alpha

) with item removed

Abbrev.	Statistic (pairwise)	HS Math	GSM8K	MMLU-5	Human
Interitem comparisons (mean pairwise association for item $i$ with other items $j\neq i$ )
Acc	$\mathbb{P}(X_{i}=1,X_{j}=1)$ (probability both correct)	0.585	0.781	0.770	0.667
F1	$\displaystyle\frac{2\,\mathbb{P}(1\mid X_{i}=1)\,\mathbb{P}(1\mid X_{j}=1)}{\mathbb{P}(1\mid X_{i}=1)+\mathbb{P}(1\mid X_{j}=1)}$ (cond. prob. both correct)	0.627	0.821	0.791	0.683
SMC	$\mathbb{P}(X_{i}=X_{j})$ (probability of item agreement)	0.885	0.841	0.827	0.733*
MI	$H(X_{i})+H(X_{j})-H(X_{i},X_{j})$ (mutual dependence)	0.814	0.863	0.763	0.817*
$\phi$	$\mathrm{Corr}(X_{i},X_{j})$ (linear dependence)	0.889	0.872	0.865	0.917*

Table 1: Bad-item detection efficiency (AUC) for association-based baselines. Interitem methods score an item by its mean pairwise association with all other items. Item-rest methods score an item by its association with the rest-score

X_{-i}

. Higher AUC indicates better prioritization of bad items for review.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Metric Type	Ave. Rank	$R_{1.0}\text{MMLU5}$	$R_{1.0}\text{GSM8K}$	$R_{1.0}\text{HSMath}$
$\mathcal{M}_{iso}$	Isotonic	12.67	0.91	0.83	0.85
c75	Inter-Item Correlations (tet), 75-quantile	14.00	0.91	0.83	0.84
$a_{2PL}$	IRT: Discrimination	19.33	0.91	0.83	0.83
$g_{(5)}$	Omega: General Factor	25.00	0.90	0.82	0.81
$g_{(3)}$	Omega: General Factor	25.00	0.88	0.81	0.90
$h^{2}_{(3)}$	Omega: Variance	25.50	0.73	0.82	0.95

Appendix

Table 2: The first numeric column represents the average Borda count rank. The subsequent columns represent the average percentile rank for AUC. Estimation of the AUC was done with the pROC package ( Robin et al., 2011 ) .

Figure 2: Instrument Composition Permutation Bootstrap Results (n x p)

Measure (Alias)	Expression	Description & Rationale	Ref.
Group 1: Intersection-Based (Focus on Joint Presence, ‘a’)
Jaccard Index	$\displaystyle\frac{a}{a+b+c}$	Ratio of the intersection to the union of positive responses. Ignores joint absences ( d ), making it suitable when co-presence is the primary signal of similarity.	( , )
Dice-Sørensen Coefficient (F1 Score)	$\displaystyle\frac{2a}{2a+b+c}$	The harmonic mean of precision and recall. It is monotonic with Jaccard but gives greater weight to the intersection term ( a ).	( , )
Overlap Coefficient (Szymkiewicz-Simpson)	$\displaystyle\frac{a}{\min(a+b,a+c)}$	An asymmetric measure of inclusion; represents the proportion of the smaller set that is contained within the larger set. Reaches 1 if one item’s endorsers are a subset of the other’s.	( , )
Cosine Similarity (Ochiai, Fowlkes-Mallows)	$\displaystyle\frac{a}{\sqrt{(a+b)(a+c)}}$	Geometric mean normalization. Interprets response vectors in a high-dimensional space and measures the cosine of the angle between them. Insensitive to vector magnitude.	( , )
Kulczyński 1	$\displaystyle\frac{a}{b+c}$	Ratio of joint presences to disagreements. Highly sensitive to low frequencies of disagreement, diverging as b+c approaches zero.	( , )

Appendix

Table 3: Intersection and Agreement-Based Similarity Measures

Measure (Alias)	Expression	Description & Rationale	Ref.
Group 3: Covariance and Correlation-Based (Focus on ‘ad-bc’)
Phi ( $\phi$ ) Coefficient (Matthews Correlation)	$\displaystyle\frac{ad-bc}{\sqrt{(a+b)(c+d)(a+c)(b+d)}}$	The Pearson product-moment correlation for two dichotomous variables. It is a chance-corrected measure of association, sensitive to both marginal distributions and the covariance term.	( , )
Yule’s Q ( $\gamma$ )	$\displaystyle\frac{ad-bc}{ad+bc}$	A measure of ordinal association that is independent of the marginal distributions. It represents the probability of concordance minus the probability of discordance, given that no ties exist. Reaches $\pm 1$ if any cell is zero.	( , )
Yule’s W	$\displaystyle\frac{\sqrt{ad}-\sqrt{bc}}{\sqrt{ad}+\sqrt{bc}}$	The ”coefficient of colligation.” It is more conservative than Yule’s Q and is not independent of the marginals. It relates to the average of conditional probabilities.	( , )
Odds Ratio (OR)	$\displaystyle\frac{ad}{bc}$	The ratio of the odds of a positive outcome on Item j given a positive outcome on Item i , to the odds of a positive outcome on j given a negative outcome on i . Widely used in epidemiology; not bounded above.	( , )
Cohen’s Kappa ( $\kappa$ )	$\displaystyle\frac{p_{o}-p_{e}}{1-p_{e}}$ where $p_{o}=\frac{a+d}{n}$ and $p_{e}=\frac{(a+b)(a+c)+(c+d)(b+d)}{n^{2}}$	A measure of inter-rater reliability that corrects the observed proportion of agreement ( $p_{o}$ ) for the agreement expected by chance ( $p_{e}$ ). Robust against unequal marginal probabilities.	( , )

Appendix

Table 4: Covariance, Correlation, and Information-Theoretic Measures

Measure (Alias)	Expression	Description & Rationale	Ref.
Loevinger’s H (Pairwise, $H_{ij}$ )	$\displaystyle\frac{\text{Cov}(i,j)}{\text{Cov}_{\text{max}}(i,j)}$ For $\text{Cov}>0$ : $\displaystyle\frac{ad-bc}{\min\left[(a+c)(b+d),(a+b)(c+d)\right]}$	A coefficient of homogeneity. It normalizes the covariance by the maximum possible covariance given the items’ marginal frequencies. This assesses how well the items conform to a deterministic Guttman-type relationship. A cornerstone of Mokken Scale Analysis.	( , )
Standardized Z ( $Z_{i}$ )	Conceptually: $\displaystyle\frac{H_{i}-E[H_{i}]}{\sqrt{\text{Var}(H_{i})}}$	A test statistic for the significance of an item’s scalability ( $H_{i}$ , its average H-value with all other items). It assesses whether an item’s monotonicity with the underlying scale is statistically significant under the null model of non-monotonicity. Not directly computed from a single pair.	( , )
2PL Discrimination ( $a_{i}$ )	Estimated via Marginal Maximum Likelihood (MML)	A parameter from a latent trait (Item Response Theory) model. It represents the slope of the Item Characteristic Curve at the point of inflection ( $\theta=b_{i}$ ). It is not expressible in contingency table terms, as it is estimated from the full response matrix under the assumption of a continuous latent trait. It is analogous to a factor loading.	( , )
Yule’s Q ( $\gamma$ )	$\displaystyle\frac{ad-bc}{ad+bc}$	Included again for comparison. In psychometrics, its property of being independent of marginals makes it a pure measure of ordinal association, but this can be a drawback, as it may inflate association for items with extreme difficulties.	( , )
Phi ( $\phi$ ) Coefficient	$\displaystyle\frac{ad-bc}{\sqrt{(a+b)(c+d)(a+c)(b+d)}}$	Included again for comparison. In contrast to Q, Phi is sensitive to item difficulty differences. Two items with high Phi must have similar difficulties, making it a measure of both association and difficulty-matching. Its square is the proportion of variance explained.	( , )

Appendix

Table 5: Psychometric and Monotonicity-Based Measures

Measure	Aliases	Formula	Description & Key Features	Refs
Intersection-Based Measures
Jaccard Index	Tanimoto	$\frac{a}{a+b+c}$	Intersection over union; satisfies triangle inequality; standard for binary similarity
Dice Coefficient	Sørensen, F1-score	$\frac{2a}{2a+b+c}$	Harmonic mean of precision/recall; emphasizes joint occurrences
Overlap Coefficient	Simpson	$\frac{a}{\min(a+b,a+c)}$	Similarity relative to smaller set; asymmetric upper bound
Cosine Similarity	Ochiai, Driver-Kroeber	$\frac{a}{\sqrt{(a+b)(a+c)}}$	Geometric angle measure; normalized dot product for binary vectors
Kulczyński-2		$\frac{1}{2}\left(\frac{a}{a+b}+\frac{a}{a+c}\right)$	Arithmetic mean of conditional probabilities; symmetric precision measure

Appendix

Table 6: Binary Item Similarity Measures: Mathematical Formulations and Characteristics

Measure	Aliases	Formula	Description & Key Features	Refs
Expectation-Adjusted Measures
Forbes Coefficient		$\frac{na}{(a+b)(a+c)}$	Ratio of observed to expected co-occurrence; sensitive to rare item pairs
Tarwid Index		$\frac{na-(a+b)(a+c)}{na+(a+b)(a+c)}$	Standardized deviation from independence; symmetric about zero
McConnaughey		$\frac{a^{2}-bc}{\sqrt{(a+b)(a+c)}}$	Squared overlap relative to geometric mean; emphasizes strong associations
Monotonicity and Scalability
Symmetric $H$	Mokken $H_{ij}$	$1-\frac{b+c}{2P_{i}(1-P_{j})+2P_{j}(1-P_{i})}$	Symmetric scalability index; measures violation of monotone homogeneity

Appendix

Table 7: Specialized Binary Similarity Measures for Item Response Applications

Name	Aliases	Formula	Salient Features & Use Case
I. Intersection-Based Measures (Joint Positive Occurrence)
Dice Precision	$S_{P}$ (Precision)	$\frac{a}{a+b}$	Proportion of $X=1$ also $Y=1$ ; insensitive to co-absences, directional .
Dice Recall	$S_{R}$ (Recall)	$\frac{a}{a+c}$	Proportion of $Y=1$ also $X=1$ ; equally directional.
Jaccard	(Tanimoto)	$\frac{a}{a+b+c}$	Fraction of total positives shared; ignores double-zeros; popular in ecology/text.
Weighted Jaccard		$\frac{3a}{3a+b+c}$	Increased weight on matches; strengthens intersection focus.
Russell-Rao		$\frac{a}{n}$	Absolute fraction of joint positives; marginal coverage emphasized, sparse for rare items.

Appendix

Table 10

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

Jul 16, 2026

Can We Trust Item Response Theory for AI Evaluation?

AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate capabilities, rank systems, select informative examples, and diagnose benchmark quality. However, AI benchmark data often departs from the data regimes encountered in human testing, for which standard IRT estimation tools were originally developed: these benchmarks typically involve fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal. We examine how these regime mismatches challenge the reliability of IRT modeling for AI evaluation. Using item parameters and capability distributions derived from six widely used LLM benchmarks, we simulate response matrices under three common IRT models and compare four estimators used in recent benchmark studies: marginal maximum likelihood, Markov chain Monte Carlo, variational inference, and a neural pseudo-Siamese estimator. Across 18,000 simulation runs, we systematically evaluate computational feasibility, scalability, and the reliability of IRT inferences about model rankings, predicted performance, and item characteristics. Results show that classical estimators can become computationally infeasible for large benchmarks, whereas scalable estimators can produce unreliable item-level and ranking inferences with small or non-normally distributed model sets. This study identifies when latent trait models reliably support or risk distorting AI benchmarking claims, and what sample sizes and diagnostics are needed for trustworthy use.

Han Jiang, Sunbeom Kwon, Jinwen Luo +2

Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles

LLM Evaluation Item Response Theory

Ask PDF

May 11, 2026

The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains

Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substantially misleading rankings when two conditions co-occur: (1) the evaluation matrix is sparse and (2) items vary substantially in difficulty. Through controlled simulation experiments across four domains -- NLP (GLUE), clinical drug trials, autonomous vehicle safety, and cybersecurity -- we show that Spearman rank correlation

ρ

between simple-average rankings and ground-truth rankings degrades from

ρ= 1.000

at 100% coverage to

ρ= 0.809

at 67% coverage with high difficulty heterogeneity (mean over 20 seeds). A standard two-parameter logistic (2PL) Item Response Theory (IRT) model maintains

ρ\geq 0.996

across all conditions. A 150-condition grid sweep over sparsity

S \in [0, 0.70]

and difficulty gap

D \in [0.5, 5.0]

confirms that ranking error forms a failure surface with a strong

S \times D

interaction (

γ_3 = +0.20

t = 13.05

), while IRT maintains

ρ\geq 0.993

throughout. We discuss implications for Physical AI benchmarking, where evaluation matrices are often incomplete and difficulty gaps are extreme.

Jung Min Kang

Independent Researcher · Seoul, South Korea

Benchmark Design Benchmark Validity Item Response Theory

Ask PDF

Feb 27, 2026

AI Evaluation Should Require Standardized Item-Level Data Releases

This position paper argues that standardized item-level benchmark data should become the default infrastructure for AI evaluation. Current evaluations suffer from underspecified item selection, construct misalignment, and poor generalization; a disproportionate focus on aggregate model scores compounds these failures by keeping them undetected and unfixable. Without item-level evidence, validity claims cannot be assessed, resulting in inflated capability claims, misdirected research, and unwarranted trust in deployed systems. Our position is that valid evaluation requires empirical evidence from item-level benchmark data, including item content, test conditions, model responses, and item-level score statistics, and that the standardized release of such data should be treated as core AI evaluation infrastructure. Such releases also enable transparency, replicability, and auditability of evaluation results. To show that the norm is both feasible and consequential, we construct OpenEval, an item-level repository of 10.7M+ responses across 160K+ items from widely-used benchmarks, under a unified schema that the AI evaluation community can build upon. We demonstrate how item-level data can identify low-quality items, document construct misalignment, and recover validity evidence about benchmarks' internal structure. We address objections concerning contamination and coordination burden, and show that both are tractable relative to the costs of decisions made on inadequately validated evaluative claims.

Han Jiang, Susu Zhang, Dongyao Zhu +6

Johns Hopkins University · University of Illinois Urbana-Champaign · North Carolina State University +3

Benchmark Validity

Ask PDF

Abstract

Benchmark Auditing

Explore similar work

Can We Trust Item Response Theory for AI Evaluation?

The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains

AI Evaluation Should Require Standardized Item-Level Data Releases