Multiple-choice and candidate-selection evaluations are widely used to assess LLM reasoning and decision-making, yet most benchmarks contain relatively small candidate sets. It remains unclear whether conclusions drawn from these settings remain valid as the candidate space scales. We systematically evaluate LLMs as the number of competing candidates increases and find substantial accuracy degradation across tasks, prompting strategies, and model scales. Controlled analyses show that standard long-context retrieval explanations cannot fully account for this degradation. Instead, we identify two systematic failure patterns. First, gold-margin collapse: the score gap between the correct answer and the strongest distractor progressively shrinks, driven primarily by weakening confidence in the correct answer. Second, earlier candidate preferences become increasingly difficult to overturn, with later candidates exerting progressively weaker influence on the final prediction. Motivated by these findings, we evaluate hierarchical partitioning and permutation-based inference, which improve accuracy by roughly 20 percentage points at N=160 on both HotpotQA and MIMIC. Overall, our results identify candidate-set scale as an important evaluation-protocol variable and show that strong small-option performance does not necessarily imply robust large-scale candidate comparison.
Figures & tables
Figure 1: Overview of large-option degradation under scalable candidate comparison. Controlled analyses suggest that standard long-context explanations alone cannot fully explain the observed degradation, revealing additional comparison-level failure patterns under large candidate sets.
Figure 2: Accuracy degradation in the large- N regime. As the number of options increases, accuracy consistently declines across all datasets.
Figure 3: System-2 reasoning under large candidate sets (Llama-3.1-8B). All reasoning strategies degrade substantially as the number of options increases.
Figure 4: Gold margin across models and score dynamics for Llama-3.1-8B as the number of options increases. As the number of options grows, the gold margin decreases while the gold score collapses substantially faster than the distractor score.
Task
Model
Added Tokens ( Δ Accuracy)
Avg. Δ
Base
+235
+470
+940
+1880
+3760
+7520
HotpotQA
Phi-4-mini
72.00
+7.00
+7.00
+5.50
+7.00
+6.00
+2.50
+6.43
Llama-3.1-8B
86.50
+0.50
+2.00
+0.50
+1.50
+0.50
+1.50
+1.07
Qwen3-30B
99.00
-0.50
-0.50
-0.50
-0.50
-0.50
+0.00
-0.43
MIMIC
Phi-4-mini
35.00
-5.50
-3.50
-3.50
-3.00
-1.50
-5.00
-3.57
Llama-3.1-8B
56.00
-3.50
-6.00
-5.50
-8.50
-7.50
-8.50
-6.50
Table 1: Accuracy under increasing irrelevant context length. Base denotes accuracy without added tokens; remaining columns report changes relative to Base.
Figure 5: Prediction-position and attention distributions on HotpotQA at N=80 . Both Llama-3.1-8B and Phi-4-mini exhibit front-skewed prediction distributions, despite substantially different attention patterns. Llama-3.1-8B shows relatively balanced attention across candidate positions, whereas Phi-4-mini allocates substantially more attention to later positions.
Figure 6: Controlled gold-position intervention on Llama-3.1-8B. Later-positioned gold answers become progressively harder to recover as the candidate set grows.
Figure 7: Revision dynamics under large candidate sets on HotpotQA (Llama-3.1-8B, N=80 ). (a) Anchor persistence measures whether the prediction remains the pre-anchor after gold insertion, while retained anchor rate measures whether the pre-anchor remains the strongest distractor. (b) Gold probability decreases for later insertions, weakening suppression of the pre-anchor and making earlier preferences harder to overturn.
Figure 8: Effects of restructuring candidate comparison (Llama-3.1-8B-Instruct). (a,b) Hierarchical partitioning and permutation-based inference increasingly outperform single-pass inference as candidate-set size grows. (c,d) Both interventions maintain larger gold margins and reduce early-position bias under large candidate sets.
Dataset
Task
Avg. Option Tokens
Prompt Tokens at N=160
MSMARCO
Reranking
79.8
12,763.4
HotpotQA
Multi-hop Reranking
90.2
14,434.8
MIMIC
Medical Diagnosis
116.4
18,641.5
arXiv
Long-context Classification
8.8
1,413.9
Amazon-3M
Taxonomy Prediction
13.9
2,229.9
MMLU-Pro
Knowledge Reasoning
11.2
2,202.6
Table 2: Dataset statistics. We report task type, average option length, and total prompt length at N=160 .
Figure 9: Accuracy degradation across additional datasets. The degradation trend remains consistent across a wide range of tasks, supporting the generality of our findings beyond the main benchmarks.
Figure 10: Additional System-2 reasoning results on Qwen3-4B. All reasoning strategies continue to degrade substantially as the number of options increases.
Model
N=5
N=160
Δ
Phi-4-mini
88.0
28.7
−59.3
Llama-3.1-8B
95.3
62.2
−33.1
Qwen3-30B
99.3
91.2
−8.1
Table 3: Accuracy degradation with 1,000 HotpotQA evaluation instances. All values are accuracy (%).
Figure 11: Extended candidate-set scaling on MIMIC. We extend the number of candidate options to N=320 and N=640 and additionally evaluate GPT-5.4. Accuracy continues to decrease beyond the N=160 regime across all evaluated models. Although GPT-5.4 is substantially more robust than the open-source models, its accuracy decreases from 98.5% at N=5 to 49.0% at N=640 .
Model
Norm. GM
P(gold)
P(topdist.)
Qwen3-30B
1.97→0.80
0.97→0.89
0.02→0.08
Llama-3.1-8B
0.57→−0.10
0.88→0.44
0.11→0.22
Phi-4-mini
0.75→−0.45
0.92→0.16
0.05→0.35
Table 4: Robustness of gold-margin collapse under alternative score formulations on HotpotQA. Values report changes from N=5 to N=160 .
Figure 12: Distractor verbosity control under candidate-set expansion (MIMIC, Llama-3.1-8B). Although longer distractor options substantially increase total context length, they do not consistently reproduce the degradation induced by increasing the number of candidate options.
Figure 13: Prediction-position and attention distributions across candidate-set sizes on HotpotQA. Prediction distributions become increasingly concentrated toward earlier candidate positions as the number of options increases. However, attention allocation patterns vary substantially across models and candidate-set sizes, with several settings exhibiting increased attention concentration toward later option positions.
Figure 14: Prediction-position and attention distributions across candidate-set sizes on ArXiv. Prediction distributions remain consistently concentrated toward earlier candidate positions across models and candidate-set sizes, despite substantial variation in attention allocation patterns.
Figure 15: Average prediction position under candidate-set scaling. As the number of candidate options increases, model predictions progressively shift toward earlier candidate positions across datasets and model families, while the gold-answer distribution remains stable.
Model
N=160
HotpotQA
MIMIC
Phi-4-mini
0.81 / 0.10 / 0.02 / 0.01 / 0.01
0.69 / 0.14 / 0.06 / 0.06 / 0.03
Llama-3.1-8B
0.48 / 0.14 / 0.10 / 0.13 / 0.13
0.81 / 0.13 / 0.04 / 0.01 / 0.01
Qwen3-30B
0.26 / 0.15 / 0.21 / 0.20 / 0.20
0.46 / 0.19 / 0.14 / 0.14 / 0.09
Table 5: Five-bin prediction-position distributions at N=160 . Values indicate the proportion of predictions falling into each position region (front to back).
Figure 16: Early-position bias persists under reversed index assignment. Even when smaller answer indices are assigned to later candidates, model predictions continue to increasingly concentrate toward earlier candidate positions as the number of options grows.
Figure 17: Probability distributions under controlled gold-position intervention. Probability mass progressively concentrates toward earlier candidates as the option set grows.
Figure 18: Early-position bias on arXiv under original and reversed indexing. Predictions progressively concentrate toward earlier candidate positions as the number of options increases across models. The same trend persists under reversed index assignment, indicating that the observed bias is driven primarily by the option position rather than token-level index priors.
Figure 19: Controlled gold-position intervention on arXiv across models. Each row corresponds to a model, while columns vary the candidate set size. As the number of options increases, predictions become progressively concentrated toward earlier candidate positions regardless of the true gold location.
Figure 20: Anchor persistence on arXiv across models. Later-positioned gold answers become progressively less effective at overturning earlier candidate preferences.
Figure 21: Probability revision dynamics on arXiv across models. Later-arriving gold answers progressively lose the ability to overturn earlier candidate preferences.
Figure 22: Additional sequential revision dynamics across models. Across additional model families, later-arriving gold answers progressively lose the ability to revise earlier candidates under large-option comparison.
Figure 23: Mechanisms of inference-time restructuring. (a) Hierarchical partitioning reduces comparison difficulty through local candidate selection, but introduces irreversible early elimination errors as the number of options increases. (b) Permutation-based inference mitigates positional bias by aggregating predictions across reordered candidate views, improving robustness during final-stage comparison among surviving candidates.
Method
Acc. (%)
Calls / Sample
Latency (s)
Single-Pass
57.5
1.00
2.46
Permutation
76.5
5.76
8.74
Hierarchy
71.0
42.0
6.04
Table 6: Accuracy–efficiency trade-off of inference-time interventions. Latency reports measured wall-clock time per example under our vLLM-based batched inference implementation; model-call counts therefore do not translate directly into wall-clock latency.
Figure 24: Generalization across model scales. Hierarchical grouping and permutation-based inference consistently mitigate large-option degradation across both small and large models.
Figure 25: Effect of hyperparameters. Larger group sizes improve hierarchical partitioning by preserving stronger candidates during local selection, while permutation-based inference benefits from a moderate number of shuffled views but exhibits diminishing returns at larger scales.
LLM evaluation is commonly performed either by prompting models to produce answers or by scoring candidate outputs with likelihood-based metrics. In multiple-choice QA, however, standard likelihood-based scoring is still conditioned on the question and answer set, and can therefore leverage the same task-conditioned answer-selection interface used in prompting. We study a complementary protocol based on likelihood ranking of declarative statements constructed from the same question--answer pairs. Across 95 decoder-only models, ranging from 0.1B to 104B parameters, and 10 MCQA datasets, we find a systematic divergence between declarative-statement likelihood ranking and prompted answering. Statement-likelihood accuracy remains comparatively stable across scale, whereas prompted answering improves sharply with scale and instruction-tuning. These results suggest that likelihood preferences over controlled declarative alternatives and task-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable.
Large language models (LLMs) exhibit a range of human-like decision-making behaviors, but whether these reflect similar underlying mechanisms or surface-level mimicry remains unclear. We evaluate whether LLM context sensitivity aligns with a cognitive economic theory that explains human behavior through problem categorization and attention allocation. Across 12 open-source and commercial LLMs on a novel 140,000-trial product choice benchmark, context induces human-like shifts in choice and problem categorization, but does not reliably reweight attention between features like price and quality. Neither scale nor chain-of-thought reasoning reliably attenuates context sensitivity or generates human-like behavior. These results suggest that LLM decision mechanisms are distinct from human ones.
Johnathan Sun, Andrei Shleifer, Yonatan Belinkov
Harvard University · Kempner Institute, Harvard University · Technion - Israel Institute of Technology
Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.
Rui Wen, Lu Sun, Jiayang Liu +3
Institute of Science Tokyo · Tohoku University · Nanyang Technological University +2