Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM

Jinwen Luo; Han Jiang; Ziang Xiao; Susu Zhang

cs.CLNov 2, 2026

Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM

Authors: Jinwen Luo, Han Jiang, Ziang Xiao, Susu Zhang

Organizations: University of California, Los Angeles · Johns Hopkins University · University of Illinois Urbana-Champaign

Abstract

Language models with similar benchmark accuracy can succeed on different items. Overall scores do not show where their performance differs or which item properties might explain those differences. We demonstrate how the latent space item response model (LSIRM) can help evaluators investigate these patterns and their consequences for model comparison. We analyze 1,348 language models on 5,761 BBH items and compare ways of presenting and scoring answers across BBH, GPQA, MMLU-Pro, and MuSR. We examine LSIRM maps, which summarize differences in performance across models and items, alongside item properties and records of correct and incorrect responses. In BBH, nearby items on these maps often share the same correct answer, such as "Yes" or "No." Across the four benchmarks, this pattern is stronger when answers are scored using option labels, such as A, B, C, and D, than when using the text of the options. We then estimate each model's tendency to perform better or worse on items with particular correct answers and quantify how this tendency contributes to its score. This separates an estimated contribution associated with correct answers from the remaining differences in item performance. Two models' total numbers of correct answers differed by only 14 out of 5,761 items. Excluding the estimated answer-category contribution from their scores moved their ranks hundreds of places in opposite directions, while keeping all other estimates unchanged. The approach helps evaluators identify differences hidden by overall accuracy and understand how the choice of what to include in a score affects model rankings.

Figures & tables

Figure 1: Similar overall accuracy conceals differences across tasks and correct-answer groups. (A–B) OpenBuddy and Smaug, two 70B models, differ by seven correct responses across 5,761 BBH items, yet their ordering reverses between adjective order (Hyperbaton) and logical deduction with three objects (250 items per task). (C–D) The Mistral 7B Base-SFT-CPO and Base-SFT-RDPO models differ by 17 correct responses overall and four on Ruin Names (109 versus 113 out of 250), yet their ordering reverses between items whose correct answers are A and C. Bars show observed accuracy; $n$ denotes the number of items. In D, A–D group items by the correct option in the answer key. The panels present selected descriptive examples from the same BBH response matrix; full model identifiers appear in Table S3 .

Figure 2: Interpreting a BBH interaction map through item and model annotations. All panels retain identical fitted positions, orientation, and scale. (A) The full map contains 1,348 models and 5,761 items. (B) Navigation, Object Counting, and Temporal Sequences are highlighted by task, retaining all 250 items per task. (C) Model points are colored by observed accuracy across the 5,761 items. (D) Open circles identify OpenBuddy and Smaug. Item colors represent distance to Smaug minus distance to OpenBuddy: red indicates closer proximity to OpenBuddy, blue to Smaug, and white equal distance. Color intensity reflects the size of the difference. Fitted correctness probabilities combine model–item distances with the model and item effects.

Protocol	Options displayed	Candidates scored
Text scoring without labels (U0)	great white shark bottlenose dolphin rainbow trout common octopus	great white shark bottlenose dolphin rainbow trout common octopus
Text scoring with labels (LT0)	(A) great white shark (B) bottlenose dolphin (C) rainbow trout (D) common octopus	great white shark bottlenose dolphin rainbow trout common octopus
Label scoring (L0)	(A) great white shark (B) bottlenose dolphin (C) rainbow trout (D) common octopus	(A) (B) (C) (D)

Table 1: Constructed example of the three evaluation protocols. The question is “Which animal is a mammal?” The final column lists the candidate continuations whose likelihoods are compared. Parenthetical codes identify the corresponding conditions in the analysis records.

Figure 3: BBH interaction map annotated by task and correct answer. The map was estimated from the correctness records of 1,348 models on 5,761 items. Each panel highlights the items from one task, with colors and marker shapes indicating their correct answers. Gray crosses represent models; gray dots represent items from other tasks. Panel titles report task item counts. All panels use the same fitted positions, orientation, and scale. Correct-answer categories were added as annotations after fitting. Coordinate axes are omitted because their orientation has no substantive interpretation.

Figure 4: An LSIRM map connects model–item interactions to observed correctness patterns. a , The selected M0 fit uses 1,348 models and 5,761 BBH items. All item positions are retained in gray; color highlights the 250 Web of Lies items by their correct answer (gold key) (122 Yes, 128 No). Crosses locate three previously selected illustrative models. The map retains the saved coordinate scale and orientation; model–item distance contributes negatively to the fitted log odds of a correct response. b , Normalized answer alignment in each of the five answer families, using ten within-task item neighbors and 200 task-preserving answer-label permutations. Alignment is

A=(O-N)/(1-N)

, where

O

is observed agreement and

N

is mean agreement under permutation; values near one indicate greater local agreement than the permutation reference. Points describe the selected fit, without sampling intervals. c , Observed correctness for the same three models and 250 highlighted items. Cells report percentages and exact correct/total counts; shading encodes correctness. BgGPT and Hebrew-Gemma have opposite correctness patterns across correct-answer categories, whereas calme answers most items correctly under both keys. These records retain correctness and correct answers, not the models’ chosen answer strings. Full model identifiers are listed in Table S3 .

Figure 5: The scored answer representation changes the diagnosed structure. a , U0 scores option text without printed identifiers; LT0 prints identifiers and retains text scoring; L0 uses the same labelled display and scores the identifier. Selections maximize raw summed continuation log likelihoods without length normalization. b , Mean paired differences in normalized alignment across 27 matched domain-by-option-count blocks (2,159 items, at least three options). The 17-model cohort excludes SmolLM2-360M-Instruct; the 18-model cohort includes it. All fits use the same 2,399 items in 30 blocks. Identifier-minus-text contrasts are 0.0630 and 0.0908. Printing identifiers has smaller negative contrasts,

-0.0192

and

-0.0227

. Bars are conditional 95% percentile intervals from 10,000 paired block-bootstrap draws, holding the selected fitted maps fixed; they exclude refitting and model-population uncertainty. c , Preservation of the selected content or identifier after the first identifier reassignment, with option wording and order fixed. All 18 models and all 2,399 items contribute (43,182 paired choices). Content is preserved in 57.27% of choices and the identifier in 18.86%; 15 of 18 models preserve content more often. Content and position coincide under fixed ordering. Points are observed proportions, with no sampling intervals. Model labels omit the instruction-tuned suffix; the base Qwen model is marked explicitly.

Figure 6: Conditioning gives explicit score components and a remaining performance profile. a , Signed M1 components for Phi-3-mini-4K-instruct-cpo-simpo (Phi), DeepSeek-R1-Distill-Llama-70B (DeepSeek), and QwQ-LCoT1-Merged (QwQ). Dark bars mark

s_{m}=\theta_{m}+\bar{I}_{m}

and

q_{m}=s_{m}+\bar{\delta}_{m}

; lighter bars mark their constituent terms. Means weight all 5,761 items equally. Both scores omit the common mean item effect. The components add on the logit scale within the selected fit. Scores are calculated before rounding. b , Mean answer offsets and relative interactions across all five answer families. Relative interaction subtracts each model’s all-item mean interaction; positive values indicate a less negative interaction in that family. Family sizes are 4,074 (Letter), 937 (Yes/No), and 250 each (True/False, Valid/Invalid, Other). Other contains the numeric Object Counting answers. These fitted profiles complement observed accuracy; all 24 task profiles appear in Supplementary Figure S1 . c , Answer-key alignment in M0, M1 and M2 for the same 1,348-model matrix, normalized against 200 task-preserving key permutations. d , Separate fixed cohort of 100 models: ten overlapping paired masks, each with 57,400 held-out responses. Gray lines join task–answer-only fits and their interaction extensions, using the historical 107 task–answer groups; dark diamonds mark means. Mean log loss falls from 0.3538 to 0.3320, improving in every mask. Panels a–c show selected numerically accepted solutions without sampling intervals; saved-start sensitivity is reported separately.

Figure 7: Accounting for answer-associated components changes model comparison. a , Selected examples distinguish re-estimation across fits (M0.

s

to M1.

s

) from removing the explicit answer contribution within M1 (

q

s

). Open circles show starting ranks and filled circles ending ranks; labels give exact ranks and observed accuracy. Phi and DeepSeek differ by 14 correct answers out of 5,761 items but disagree on 2,158 of 5,761 correctness records. Their M1.

s

ranks are 814 and 97, a 717-place separation. QwQ is a low-movement comparator selected by the documented rule. b , Empirical cumulative distributions retain all 1,348 models for each comparison. Blue solid curves use M1 and purple dashed curves M2. Median absolute movements are 70 and 31 places across fits, and 61 and 22 places within fits. Ranks are computed from unrounded scores, with average ranks for exact ties; lower ranks indicate larger scores. c , Between-fit agreement of unconditioned and conditioned L0.

s

ranks with U0.

s

, using all 2,399 rank-evaluation items. The 17- and 18-model rows each use their complete cohort. Spearman correlation decreases from 0.980 to 0.882 for 17 models and from 0.983 to 0.901 for 18 models; mean absolute rank distance increases from 0.59 to 1.76 and from 0.56 to 1.67, respectively. The shared 17 models have identical selected ranks in both cohort analyses. The display describes ranking consequences under specified fits and protocols; U0 is an alternative evaluation reference rather than an external criterion of correct ability ranking.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Comparison	Scores	$\rho$	Median move	Max. move	Top-50 overlap
between fit	M0.s $\to$ M1.s	0.9190	70	687	28
between fit	M0.s $\to$ M2.s	0.9842	31	454	40
within fit	M1.q $\to$ M1.s	0.9297	61	755	30
within fit	M2.q $\to$ M2.s	0.9920	22	239	43

Appendix

Table S1: Rank comparisons for all 1,348 BBH models on the same 5,761 items. Lower ranks indicate larger scores. Between-fit comparisons re-estimate all components; within-fit comparisons remove only the mean answer offset. The last column counts models shared by the two top-50 sets.

Correct-answer family	Items	M0	M1	M2
Letter	4074	0.3650	0.2745	0.1006
TRUE/FALSE	250	0.8935	0.0496	0.0352
Yes/No	937	0.9629	0.3677	0.1077
Valid/Invalid	250	0.7009	0.1300	0.0980
Other	250	0.1373	0.0830	0.0984

Appendix

Table S2: Normalized answer-family alignment on the common BBH item set. Neighborhoods contain ten items within the same task; the null uses 200 within-task permutations. Other contains all 250 Object Counting items, with their 17 numeric answer keys retained separately. Values are selected-fit summaries.

Short name	Model identifier	Correct	Accuracy
OpenBuddy	OpenBuddy/openbuddy-llama3-70b-v21.2-32k	3743	0.6497
Smaug	abacusai/Smaug-Llama-3-70B-Instruct-32K	3736	0.6485
CPO	princeton-nlp/Mistral-7B-Base-SFT-CPO	2514	0.4364
RDPO	princeton-nlp/Mistral-7B-Base-SFT-RDPO	2531	0.4393
BgGPT	INSAIT-Institute/BgGPT-Gemma-2-27B-IT-v1.0	1657	0.2876
Hebrew-Gemma	yam-peleg/Hebrew-Gemma-11B-Instruct	2314	0.4017

Appendix

Table S3: Exact identities of the ten BBH examples used in the text and figures. Correctness is calculated over the same 5,761 items per model. Accuracy is a proportion. The introductory pairs and the six diagnostic/ranking examples have separate source-record checks described in Appendix A.3 .

Model	$\theta$	$\bar{I}$	$\bar{\delta}$	$s$	$q$	Rank $s$	Rank $q$
Phi	-0.616	-2.708	0.838	-3.323	-2.486	814	396
DeepSeek	3.886	-4.806	-1.302	-0.920	-2.222	97	344
QwQ	0.262	-2.748	-0.076	-2.486	-2.561	441	438

Appendix

Table S4: M1 score construction for the ranking examples. All components are on the logit scale and use equal weights over 5,761 items:

s=\theta+\bar{I}

and

q=s+\bar{\delta}

. The model-independent mean item effect is omitted from both scores. Components and scores are rounded after calculation. Values are conditional on the selected numerically accepted fit.

Mask	Task–answer only	With interaction	Gain
1	0.354357	0.330746	0.023611
2	0.350713	0.329354	0.021359
3	0.352481	0.332135	0.020346
4	0.352671	0.330756	0.021915
5	0.354487	0.331522	0.022965
6	0.348527	0.326901	0.021626

Appendix

Table S5: Paired held-out log loss for one fixed 100-model BBH cohort and ten overlapping response masks. Each mask withholds 57,400 responses and retains 518,700 for estimation. Both specifications use the historical 107 task–answer groups. Gain is the task–answer-only loss minus the loss after adding interaction; it is not model-population uncertainty.

Models	Blocks	Contrast	Estimate	95% interval
17	27	Identifier minus text	0.0630	[0.0365, 0.0945]
17	27	Printed minus unprinted	-0.0192	[-0.0419, 0.0011]
17	30	Printed minus unprinted	-0.0288	[-0.0521, -0.0073]
18	27	Identifier minus text	0.0908	[0.0534, 0.1338]
18	27	Printed minus unprinted	-0.0227	[-0.0446, -0.0036]
18	30	Printed minus unprinted	-0.0273	[-0.0472, -0.0087]

Appendix

Table S6: Selected crossed-evaluation alignment contrasts for 17 and 18 models. Primary matched contrasts use 27 blocks and 2,159 items with at least three options; the supplementary full-set contrast uses 30 blocks and 2,399 items. Identifier minus text denotes label scoring minus text scoring with labels (L0–LT0); printed minus unprinted denotes text scoring with labels minus text scoring without labels (LT0–U0). Intervals are conditional 95% paired block-bootstrap intervals from 10,000 draws, with fitted maps held fixed.

Fit / ranked set	$\rho$ before	$\rho$ after	Distance before	Distance after	Closer	Farther
17 / 17	0.9804	0.8824	0.588	1.765	1	9
18 / 17	0.9804	0.8824	0.588	1.765	1	9
18 / 18	0.9835	0.9009	0.556	1.667	1	9

Appendix

Table S7: Between-fit agreement of label-scoring ranks before and after fitting identifier effects (L0.

s

) with text-scoring ranks without labels (U0.

s

). Ranks are computed over 2,399 items. Shared-17 ranks from the 18-model fits are recomputed among the common models. Mean distance is the mean absolute difference in rank; U0 is an alternative evaluation reference.

Analysis	Fits	Accepted	Largest absolute gradient
BBH M0	1	1	6.83e-06
BBH M1	1	1	9.83e-06
BBH M2	1	1	9.19e-06
Held-out BBH	20	20	9.88e-06
Crossed evaluation	16	16	9.93e-06

Appendix

Table S8: Numerical acceptance of retained fits. The acceptance threshold is a maximum absolute gradient of

10^{-4}

, alongside finite parameters, independently recomputed objectives and gradients, and input identity checks. The BBH rows are the three selected display fits; crossed counts include both SVD and reciprocal starts. Acceptance establishes first-order stationarity under these checks.

Specification	Candidates	Minimum rank $\rho$	Maximum move	Maximum family change
M1	5	0.9999755	13	0.009063
M2	5	0.9999997	4	0.000998

Appendix

Table S9: Starting-solution sensitivity within each anchor-plus-four-perturbation set. Each specification has five numerically accepted candidates. Minimum rank correlation and maximum movement cover both

s

and

q

across all pairs; alignment covers all five answer families. These descriptive ranges are not confidence intervals. Component values can vary even when ranks and alignment remain close.

Specification	Model	$\bar{I}$	$\bar{\delta}$	$s$	Rank $s$
M1	Phi	[-2.708, -2.693]	[0.836, 0.838]	[-3.323, -3.312]	[814, 816]
M1	DeepSeek	[-4.806, -4.795]	[-1.302, -1.286]	[-0.926, -0.920]	[97, 97]
M1	QwQ	[-2.748, -2.735]	[-0.076, -0.074]	[-2.486, -2.478]	[441, 444]
M2	Phi	[-2.474, -2.473]	[0.176, 0.176]	[-2.219, -2.218]	[615, 615]
M2	DeepSeek	[-2.682, -2.682]	[-0.710, -0.710]	[-1.213, -1.213]	[229, 229]
M2	QwQ	[-2.548, -2.547]	[-0.178, -0.177]	[-1.870, -1.869]	[434, 434]

Appendix

Table S10: Component and rank ranges for the three ranking examples across each specification’s five saved local candidates. Mean interaction, mean offset and

s

use logit units. Ranges include the preceding anchor and four perturbations; they describe starting-solution sensitivity, not sampling uncertainty.

Task	Acc. P	Acc. D	$\delta_{P}$	$\delta_{D}$	$R_{P}$	$R_{D}$
boolean expressions	0.804	0.868	-0.019	-1.519	0.208	0.294
causal judgement	0.615	0.519	-0.115	-1.933	0.484	0.094
date understanding	0.560	0.656	1.219	-1.341	0.174	0.238
disambiguation qa	0.716	0.396	1.186	-1.429	0.207	-0.177
formal fallacies	0.624	0.652	0.468	-1.360	0.197	-0.121
geometric shapes	0.216	0.556	0.240	-0.803	0.542	0.306

Appendix

Table S11: All 24 archived BBH task profiles for Phi (P) and DeepSeek (D) in the selected M1 fit. Accuracy is observed correctness. Offsets (

\delta

) and relative interactions (

R

) are task means in logit units;

R

subtracts the model-specific mean interaction over all items. Task-relative interaction therefore describes the remaining fitted profile after the explicit answer component is represented. Display names Navigation and Adjective order correspond to the archived identifiers navigate and hyperbaton.

Figure S1: Complete task profiles for the verified Phi/DeepSeek pair. Observed correctness, mean explicit answer offset and relative interaction are shown for all 24 tasks in the selected M1 fit. The row labels give item counts; together the rows retain all 5,761 benchmark items. Phi denotes Phi-3-mini-4K-instruct-cpo-simpo and DeepSeek denotes DeepSeek-R1-Distill-Llama-70B. Each fitted component is averaged with equal weights over the task’s items. Relative interaction subtracts each model’s mean interaction over the full item set, so it describes a task-specific departure from that model’s overall interaction level. Components use logits; observed accuracy is a percentage. The row labels Navigate and Hyperbaton correspond to Navigation and Adjective order in the main text. These selected-fit profiles carry no sampling intervals. Full model identifiers and task-level summaries are reported in Tables S3 and S11 .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,469 papers

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
Aug 3, 2026Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1LLM Evaluation Psychometric Validation
École Polytechnique, Institut Polytechnique de Paris, France · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates
Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation
Jun 14, 2026Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Google Research · The Hebrew University of Jerusalem · PhaseV Trials
Auditing LLM Benchmarks with Item Response Theory
May 28, 2026Sander Land, Daniel M. BikelBenchmark Auditing LLM Auditing
Writer, Inc.

Aug 3, 2026J/K move · Enter open · S save

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

Xiao Fei, Yang Zhang, Sarah Almeida Carneiro, Michalis Vazirgiannis

École Polytechnique, Institut Polytechnique de Paris, France · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates

LLM Evaluation Psychometric Validation Item Response Theory

Open paper AskPDF arXiv

Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among incorrect options may contain systematic and useful information about its behavior and ability. We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware psychometric framework that models the full distribution over answer choices to jointly estimate LLM ability and option-level item characteristics, while separating model-specific response calibration sharpness, positional preference, and difficulty-dependent fallback behavior. Across 189 LLMs and 31,554 items from 14 benchmarks, LLM-NRM predicts held-out LLM-item interactions more accurately than binary Item Response models and conventional nominal-response baselines, and its ability estimates achieve the strongest Spearman correlation of 0.920 with the external human-preference Arena.ai Elo leaderboard. Distractor identity contributes +101% additional Fisher Information per item beyond correctness, and incorrect responses alone recover full-information ability estimates with Spearman 0.943. The learned item parameters also enable efficient benchmarking, where 41 selected items preserve the full-bank ranking with Kendall's correlation 0.85, corresponding to a 770 times reduction. In conclusion, we show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.

Abstract

LLM Evaluation

Explore similar work

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation

Auditing LLM Benchmarks with Item Response Theory