Language models with similar benchmark accuracy can succeed on different items. Overall scores do not show where their performance differs or which item properties might explain those differences. We demonstrate how the latent space item response model (LSIRM) can help evaluators investigate these patterns and their consequences for model comparison. We analyze 1,348 language models on 5,761 BBH items and compare ways of presenting and scoring answers across BBH, GPQA, MMLU-Pro, and MuSR. We examine LSIRM maps, which summarize differences in performance across models and items, alongside item properties and records of correct and incorrect responses. In BBH, nearby items on these maps often share the same correct answer, such as "Yes" or "No." Across the four benchmarks, this pattern is stronger when answers are scored using option labels, such as A, B, C, and D, than when using the text of the options. We then estimate each model's tendency to perform better or worse on items with particular correct answers and quantify how this tendency contributes to its score. This separates an estimated contribution associated with correct answers from the remaining differences in item performance. Two models' total numbers of correct answers differed by only 14 out of 5,761 items. Excluding the estimated answer-category contribution from their scores moved their ranks hundreds of places in opposite directions, while keeping all other estimates unchanged. The approach helps evaluators identify differences hidden by overall accuracy and understand how the choice of what to include in a score affects model rankings.
Figures & tables
Figure 1: Similar overall accuracy conceals differences across tasks and correct-answer groups. (A–B) OpenBuddy and Smaug, two 70B models, differ by seven correct responses across 5,761 BBH items, yet their ordering reverses between adjective order (Hyperbaton) and logical deduction with three objects (250 items per task). (C–D) The Mistral 7B Base-SFT-CPO and Base-SFT-RDPO models differ by 17 correct responses overall and four on Ruin Names (109 versus 113 out of 250), yet their ordering reverses between items whose correct answers are A and C. Bars show observed accuracy; n denotes the number of items. In D, A–D group items by the correct option in the answer key. The panels present selected descriptive examples from the same BBH response matrix; full model identifiers appear in Table S3 .
Figure 2: Interpreting a BBH interaction map through item and model annotations. All panels retain identical fitted positions, orientation, and scale. (A) The full map contains 1,348 models and 5,761 items. (B) Navigation, Object Counting, and Temporal Sequences are highlighted by task, retaining all 250 items per task. (C) Model points are colored by observed accuracy across the 5,761 items. (D) Open circles identify OpenBuddy and Smaug. Item colors represent distance to Smaug minus distance to OpenBuddy: red indicates closer proximity to OpenBuddy, blue to Smaug, and white equal distance. Color intensity reflects the size of the difference. Fitted correctness probabilities combine model–item distances with the model and item effects.
Protocol
Options displayed
Candidates scored
Text scoring without labels (U0)
great white shark bottlenose dolphin rainbow trout common octopus
great white shark bottlenose dolphin rainbow trout common octopus
Text scoring with labels (LT0)
(A) great white shark (B) bottlenose dolphin (C) rainbow trout (D) common octopus
great white shark bottlenose dolphin rainbow trout common octopus
Label scoring (L0)
(A) great white shark (B) bottlenose dolphin (C) rainbow trout (D) common octopus
(A) (B) (C) (D)
Table 1: Constructed example of the three evaluation protocols. The question is “Which animal is a mammal?” The final column lists the candidate continuations whose likelihoods are compared. Parenthetical codes identify the corresponding conditions in the analysis records.
Figure 3: BBH interaction map annotated by task and correct answer. The map was estimated from the correctness records of 1,348 models on 5,761 items. Each panel highlights the items from one task, with colors and marker shapes indicating their correct answers. Gray crosses represent models; gray dots represent items from other tasks. Panel titles report task item counts. All panels use the same fitted positions, orientation, and scale. Correct-answer categories were added as annotations after fitting. Coordinate axes are omitted because their orientation has no substantive interpretation.
Figure 4: An LSIRM map connects model–item interactions to observed correctness patterns. a , The selected M0 fit uses 1,348 models and 5,761 BBH items. All item positions are retained in gray; color highlights the 250 Web of Lies items by their correct answer (gold key) (122 Yes, 128 No). Crosses locate three previously selected illustrative models. The map retains the saved coordinate scale and orientation; model–item distance contributes negatively to the fitted log odds of a correct response. b , Normalized answer alignment in each of the five answer families, using ten within-task item neighbors and 200 task-preserving answer-label permutations. Alignment is A=(O−N)/(1−N) , where O is observed agreement and N is mean agreement under permutation; values near one indicate greater local agreement than the permutation reference. Points describe the selected fit, without sampling intervals. c , Observed correctness for the same three models and 250 highlighted items. Cells report percentages and exact correct/total counts; shading encodes correctness. BgGPT and Hebrew-Gemma have opposite correctness patterns across correct-answer categories, whereas calme answers most items correctly under both keys. These records retain correctness and correct answers, not the models’ chosen answer strings. Full model identifiers are listed in Table S3 .
Figure 5: The scored answer representation changes the diagnosed structure. a , U0 scores option text without printed identifiers; LT0 prints identifiers and retains text scoring; L0 uses the same labelled display and scores the identifier. Selections maximize raw summed continuation log likelihoods without length normalization. b , Mean paired differences in normalized alignment across 27 matched domain-by-option-count blocks (2,159 items, at least three options). The 17-model cohort excludes SmolLM2-360M-Instruct; the 18-model cohort includes it. All fits use the same 2,399 items in 30 blocks. Identifier-minus-text contrasts are 0.0630 and 0.0908. Printing identifiers has smaller negative contrasts, −0.0192 and −0.0227 . Bars are conditional 95% percentile intervals from 10,000 paired block-bootstrap draws, holding the selected fitted maps fixed; they exclude refitting and model-population uncertainty. c , Preservation of the selected content or identifier after the first identifier reassignment, with option wording and order fixed. All 18 models and all 2,399 items contribute (43,182 paired choices). Content is preserved in 57.27% of choices and the identifier in 18.86%; 15 of 18 models preserve content more often. Content and position coincide under fixed ordering. Points are observed proportions, with no sampling intervals. Model labels omit the instruction-tuned suffix; the base Qwen model is marked explicitly.
Figure 6: Conditioning gives explicit score components and a remaining performance profile. a , Signed M1 components for Phi-3-mini-4K-instruct-cpo-simpo (Phi), DeepSeek-R1-Distill-Llama-70B (DeepSeek), and QwQ-LCoT1-Merged (QwQ). Dark bars mark sm=θm+Iˉm and qm=sm+δˉm ; lighter bars mark their constituent terms. Means weight all 5,761 items equally. Both scores omit the common mean item effect. The components add on the logit scale within the selected fit. Scores are calculated before rounding. b , Mean answer offsets and relative interactions across all five answer families. Relative interaction subtracts each model’s all-item mean interaction; positive values indicate a less negative interaction in that family. Family sizes are 4,074 (Letter), 937 (Yes/No), and 250 each (True/False, Valid/Invalid, Other). Other contains the numeric Object Counting answers. These fitted profiles complement observed accuracy; all 24 task profiles appear in Supplementary Figure S1 . c , Answer-key alignment in M0, M1 and M2 for the same 1,348-model matrix, normalized against 200 task-preserving key permutations. d , Separate fixed cohort of 100 models: ten overlapping paired masks, each with 57,400 held-out responses. Gray lines join task–answer-only fits and their interaction extensions, using the historical 107 task–answer groups; dark diamonds mark means. Mean log loss falls from 0.3538 to 0.3320, improving in every mask. Panels a–c show selected numerically accepted solutions without sampling intervals; saved-start sensitivity is reported separately.
Figure 7: Accounting for answer-associated components changes model comparison. a , Selected examples distinguish re-estimation across fits (M0. s to M1. s ) from removing the explicit answer contribution within M1 ( q to s ). Open circles show starting ranks and filled circles ending ranks; labels give exact ranks and observed accuracy. Phi and DeepSeek differ by 14 correct answers out of 5,761 items but disagree on 2,158 of 5,761 correctness records. Their M1. s ranks are 814 and 97, a 717-place separation. QwQ is a low-movement comparator selected by the documented rule. b , Empirical cumulative distributions retain all 1,348 models for each comparison. Blue solid curves use M1 and purple dashed curves M2. Median absolute movements are 70 and 31 places across fits, and 61 and 22 places within fits. Ranks are computed from unrounded scores, with average ranks for exact ties; lower ranks indicate larger scores. c , Between-fit agreement of unconditioned and conditioned L0. s ranks with U0. s , using all 2,399 rank-evaluation items. The 17- and 18-model rows each use their complete cohort. Spearman correlation decreases from 0.980 to 0.882 for 17 models and from 0.983 to 0.901 for 18 models; mean absolute rank distance increases from 0.59 to 1.76 and from 0.56 to 1.67, respectively. The shared 17 models have identical selected ranks in both cohort analyses. The display describes ranking consequences under specified fits and protocols; U0 is an alternative evaluation reference rather than an external criterion of correct ability ranking.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Comparison
Scores
ρ
Median move
Max. move
Top-50 overlap
between fit
M0.s → M1.s
0.9190
70
687
28
between fit
M0.s → M2.s
0.9842
31
454
40
within fit
M1.q → M1.s
0.9297
61
755
30
within fit
M2.q → M2.s
0.9920
22
239
43
Appendix
Table S1: Rank comparisons for all 1,348 BBH models on the same 5,761 items. Lower ranks indicate larger scores. Between-fit comparisons re-estimate all components; within-fit comparisons remove only the mean answer offset. The last column counts models shared by the two top-50 sets.
Correct-answer family
Items
M0
M1
M2
Letter
4074
0.3650
0.2745
0.1006
TRUE/FALSE
250
0.8935
0.0496
0.0352
Yes/No
937
0.9629
0.3677
0.1077
Valid/Invalid
250
0.7009
0.1300
0.0980
Other
250
0.1373
0.0830
0.0984
Appendix
Table S2: Normalized answer-family alignment on the common BBH item set. Neighborhoods contain ten items within the same task; the null uses 200 within-task permutations. Other contains all 250 Object Counting items, with their 17 numeric answer keys retained separately. Values are selected-fit summaries.
Short name
Model identifier
Correct
Accuracy
OpenBuddy
OpenBuddy/openbuddy-llama3-70b-v21.2-32k
3743
0.6497
Smaug
abacusai/Smaug-Llama-3-70B-Instruct-32K
3736
0.6485
CPO
princeton-nlp/Mistral-7B-Base-SFT-CPO
2514
0.4364
RDPO
princeton-nlp/Mistral-7B-Base-SFT-RDPO
2531
0.4393
BgGPT
INSAIT-Institute/BgGPT-Gemma-2-27B-IT-v1.0
1657
0.2876
Hebrew-Gemma
yam-peleg/Hebrew-Gemma-11B-Instruct
2314
0.4017
Appendix
Table S3: Exact identities of the ten BBH examples used in the text and figures. Correctness is calculated over the same 5,761 items per model. Accuracy is a proportion. The introductory pairs and the six diagnostic/ranking examples have separate source-record checks described in Appendix A.3 .
Model
θ
Iˉ
δˉ
s
q
Rank s
Rank q
Phi
-0.616
-2.708
0.838
-3.323
-2.486
814
396
DeepSeek
3.886
-4.806
-1.302
-0.920
-2.222
97
344
QwQ
0.262
-2.748
-0.076
-2.486
-2.561
441
438
Appendix
Table S4: M1 score construction for the ranking examples. All components are on the logit scale and use equal weights over 5,761 items: s=θ+Iˉ and q=s+δˉ . The model-independent mean item effect is omitted from both scores. Components and scores are rounded after calculation. Values are conditional on the selected numerically accepted fit.
Mask
Task–answer only
With interaction
Gain
1
0.354357
0.330746
0.023611
2
0.350713
0.329354
0.021359
3
0.352481
0.332135
0.020346
4
0.352671
0.330756
0.021915
5
0.354487
0.331522
0.022965
6
0.348527
0.326901
0.021626
Appendix
Table S5: Paired held-out log loss for one fixed 100-model BBH cohort and ten overlapping response masks. Each mask withholds 57,400 responses and retains 518,700 for estimation. Both specifications use the historical 107 task–answer groups. Gain is the task–answer-only loss minus the loss after adding interaction; it is not model-population uncertainty.
Models
Blocks
Contrast
Estimate
95% interval
17
27
Identifier minus text
0.0630
[0.0365, 0.0945]
17
27
Printed minus unprinted
-0.0192
[-0.0419, 0.0011]
17
30
Printed minus unprinted
-0.0288
[-0.0521, -0.0073]
18
27
Identifier minus text
0.0908
[0.0534, 0.1338]
18
27
Printed minus unprinted
-0.0227
[-0.0446, -0.0036]
18
30
Printed minus unprinted
-0.0273
[-0.0472, -0.0087]
Appendix
Table S6: Selected crossed-evaluation alignment contrasts for 17 and 18 models. Primary matched contrasts use 27 blocks and 2,159 items with at least three options; the supplementary full-set contrast uses 30 blocks and 2,399 items. Identifier minus text denotes label scoring minus text scoring with labels (L0–LT0); printed minus unprinted denotes text scoring with labels minus text scoring without labels (LT0–U0). Intervals are conditional 95% paired block-bootstrap intervals from 10,000 draws, with fitted maps held fixed.
Fit / ranked set
ρ before
ρ after
Distance before
Distance after
Closer
Farther
17 / 17
0.9804
0.8824
0.588
1.765
1
9
18 / 17
0.9804
0.8824
0.588
1.765
1
9
18 / 18
0.9835
0.9009
0.556
1.667
1
9
Appendix
Table S7: Between-fit agreement of label-scoring ranks before and after fitting identifier effects (L0. s ) with text-scoring ranks without labels (U0. s ). Ranks are computed over 2,399 items. Shared-17 ranks from the 18-model fits are recomputed among the common models. Mean distance is the mean absolute difference in rank; U0 is an alternative evaluation reference.
Analysis
Fits
Accepted
Largest absolute gradient
BBH M0
1
1
6.83e-06
BBH M1
1
1
9.83e-06
BBH M2
1
1
9.19e-06
Held-out BBH
20
20
9.88e-06
Crossed evaluation
16
16
9.93e-06
Appendix
Table S8: Numerical acceptance of retained fits. The acceptance threshold is a maximum absolute gradient of 10−4 , alongside finite parameters, independently recomputed objectives and gradients, and input identity checks. The BBH rows are the three selected display fits; crossed counts include both SVD and reciprocal starts. Acceptance establishes first-order stationarity under these checks.
Specification
Candidates
Minimum rank ρ
Maximum move
Maximum family change
M1
5
0.9999755
13
0.009063
M2
5
0.9999997
4
0.000998
Appendix
Table S9: Starting-solution sensitivity within each anchor-plus-four-perturbation set. Each specification has five numerically accepted candidates. Minimum rank correlation and maximum movement cover both s and q across all pairs; alignment covers all five answer families. These descriptive ranges are not confidence intervals. Component values can vary even when ranks and alignment remain close.
Specification
Model
Iˉ
δˉ
s
Rank s
M1
Phi
[-2.708, -2.693]
[0.836, 0.838]
[-3.323, -3.312]
[814, 816]
M1
DeepSeek
[-4.806, -4.795]
[-1.302, -1.286]
[-0.926, -0.920]
[97, 97]
M1
QwQ
[-2.748, -2.735]
[-0.076, -0.074]
[-2.486, -2.478]
[441, 444]
M2
Phi
[-2.474, -2.473]
[0.176, 0.176]
[-2.219, -2.218]
[615, 615]
M2
DeepSeek
[-2.682, -2.682]
[-0.710, -0.710]
[-1.213, -1.213]
[229, 229]
M2
QwQ
[-2.548, -2.547]
[-0.178, -0.177]
[-1.870, -1.869]
[434, 434]
Appendix
Table S10: Component and rank ranges for the three ranking examples across each specification’s five saved local candidates. Mean interaction, mean offset and s use logit units. Ranges include the preceding anchor and four perturbations; they describe starting-solution sensitivity, not sampling uncertainty.
Task
Acc. P
Acc. D
δP
δD
RP
RD
boolean expressions
0.804
0.868
-0.019
-1.519
0.208
0.294
causal judgement
0.615
0.519
-0.115
-1.933
0.484
0.094
date understanding
0.560
0.656
1.219
-1.341
0.174
0.238
disambiguation qa
0.716
0.396
1.186
-1.429
0.207
-0.177
formal fallacies
0.624
0.652
0.468
-1.360
0.197
-0.121
geometric shapes
0.216
0.556
0.240
-0.803
0.542
0.306
Appendix
Table S11: All 24 archived BBH task profiles for Phi (P) and DeepSeek (D) in the selected M1 fit. Accuracy is observed correctness. Offsets ( δ ) and relative interactions ( R ) are task means in logit units; R subtracts the model-specific mean interaction over all items. Task-relative interaction therefore describes the remaining fitted profile after the explicit answer component is represented. Display names Navigation and Adjective order correspond to the archived identifiers navigate and hyperbaton.
Figure S1: Complete task profiles for the verified Phi/DeepSeek pair. Observed correctness, mean explicit answer offset and relative interaction are shown for all 24 tasks in the selected M1 fit. The row labels give item counts; together the rows retain all 5,761 benchmark items. Phi denotes Phi-3-mini-4K-instruct-cpo-simpo and DeepSeek denotes DeepSeek-R1-Distill-Llama-70B. Each fitted component is averaged with equal weights over the task’s items. Relative interaction subtracts each model’s mean interaction over the full item set, so it describes a task-specific departure from that model’s overall interaction level. Components use logits; observed accuracy is a percentage. The row labels Navigate and Hyperbaton correspond to Navigation and Adjective order in the main text. These selected-fit profiles carry no sampling intervals. Full model identifiers and task-level summaries are reported in Tables S3 and S11 .
Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among incorrect options may contain systematic and useful information about its behavior and ability. We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware psychometric framework that models the full distribution over answer choices to jointly estimate LLM ability and option-level item characteristics, while separating model-specific response calibration sharpness, positional preference, and difficulty-dependent fallback behavior. Across 189 LLMs and 31,554 items from 14 benchmarks, LLM-NRM predicts held-out LLM-item interactions more accurately than binary Item Response models and conventional nominal-response baselines, and its ability estimates achieve the strongest Spearman correlation of 0.920 with the external human-preference Arena.ai Elo leaderboard. Distractor identity contributes +101% additional Fisher Information per item beyond correctness, and incorrect responses alone recover full-information ability estimates with Spearman 0.943. The learned item parameters also enable efficient benchmarking, where 41 selected items preserve the full-bank ranking with Kendall's correlation 0.85, corresponding to a 770 times reduction. In conclusion, we show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.
Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1
École Polytechnique, Institut Polytechnique de Paris, France · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates
Multilingual benchmarks are central to evaluating large language models (LLMs) across languages, but they suffer from three issues: exhaustive evaluation scales linearly with the number of languages, automatic translation introduces errors that are easily missed at scale, and some items conflate general and culture-specific knowledge. We address all three with a unified statistical framework, Multilingual-IRT, which extends Item Response Theory with per-language difficulty deviations, split discriminability separating content from language effects, and per-language ability residuals. Fitting Multilingual-IRT on 25 LLMs across 29 languages of MMLU-Pro-X, we show that its fitted parameters support three practical applications: predicting unobserved (item, LLM, language) instances with 11-16% lower binary cross-entropy than the strongest accuracy-based baseline, surfacing candidate translation errors distributed across all 28 non-English languages, whereas accuracy-based baselines concentrate detections in a few languages, and recovering culture-specific items that accuracy-based baselines miss.
Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1
Google Research · The Hebrew University of Jerusalem · PhaseV Trials
LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.