cs.AISep 26, 2026
SaveOverwhelmed by Choice: Studying LLM Decision Making at Scale
Organizations: University of California, Los Angeles · University of Cincinnati
Abstract
Multiple-choice and candidate-selection evaluations are widely used to assess LLM reasoning and decision-making, yet most benchmarks contain relatively small candidate sets. It remains unclear whether conclusions drawn from these settings remain valid as the candidate space scales. We systematically evaluate LLMs as the number of competing candidates increases and find substantial accuracy degradation across tasks, prompting strategies, and model scales. Controlled analyses show that standard long-context retrieval explanations cannot fully account for this degradation. Instead, we identify two systematic failure patterns. First, gold-margin collapse: the score gap between the correct answer and the strongest distractor progressively shrinks, driven primarily by weakening confidence in the correct answer. Second, earlier candidate preferences become increasingly difficult to overturn, with later candidates exerting progressively weaker influence on the final prediction. Motivated by these findings, we evaluate hierarchical partitioning and permutation-based inference, which improve accuracy by roughly 20 percentage points at on both HotpotQA and MIMIC. Overall, our results identify candidate-set scale as an important evaluation-protocol variable and show that strong small-option performance does not necessarily imply robust large-scale candidate comparison.
Figures & tables
Figure 1: Overview of large-option degradation under scalable candidate comparison. Controlled analyses suggest that standard long-context explanations alone cannot fully explain the observed degradation, revealing additional comparison-level failure patterns under large candidate sets.
Figure 2: Accuracy degradation in the large- regime. As the number of options increases, accuracy consistently declines across all datasets.
Figure 3: System-2 reasoning under large candidate sets (Llama-3.1-8B). All reasoning strategies degrade substantially as the number of options increases.
Figure 4: Gold margin across models and score dynamics for Llama-3.1-8B as the number of options increases. As the number of options grows, the gold margin decreases while the gold score collapses substantially faster than the distractor score.
| Task | Model | Added Tokens ( Accuracy) | Avg. | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Base | +235 | +470 | +940 | +1880 | +3760 | +7520 | |||
| HotpotQA | Phi-4-mini | 72.00 | +7.00 | +7.00 | +5.50 | +7.00 | +6.00 | +2.50 | +6.43 |
| Llama-3.1-8B | 86.50 | +0.50 | +2.00 | +0.50 | +1.50 | +0.50 | +1.50 | +1.07 | |
| Qwen3-30B | 99.00 | -0.50 | -0.50 | -0.50 | -0.50 | -0.50 | +0.00 | -0.43 | |
| MIMIC | Phi-4-mini | 35.00 | -5.50 | -3.50 | -3.50 | -3.00 | -1.50 | -5.00 | -3.57 |
| Llama-3.1-8B | 56.00 | -3.50 | -6.00 | -5.50 | -8.50 | -7.50 | -8.50 | -6.50 | |
Table 1: Accuracy under increasing irrelevant context length. Base denotes accuracy without added tokens; remaining columns report changes relative to Base.
Figure 5: Prediction-position and attention distributions on HotpotQA at . Both Llama-3.1-8B and Phi-4-mini exhibit front-skewed prediction distributions, despite substantially different attention patterns. Llama-3.1-8B shows relatively balanced attention across candidate positions, whereas Phi-4-mini allocates substantially more attention to later positions.
Figure 6: Controlled gold-position intervention on Llama-3.1-8B. Later-positioned gold answers become progressively harder to recover as the candidate set grows.
Figure 7: Revision dynamics under large candidate sets on HotpotQA (Llama-3.1-8B, ). (a) Anchor persistence measures whether the prediction remains the pre-anchor after gold insertion, while retained anchor rate measures whether the pre-anchor remains the strongest distractor. (b) Gold probability decreases for later insertions, weakening suppression of the pre-anchor and making earlier preferences harder to overturn.
Figure 8: Effects of restructuring candidate comparison (Llama-3.1-8B-Instruct). (a,b) Hierarchical partitioning and permutation-based inference increasingly outperform single-pass inference as candidate-set size grows. (c,d) Both interventions maintain larger gold margins and reduce early-position bias under large candidate sets.
| Dataset | Task | Avg. Option Tokens | Prompt Tokens at |
|---|---|---|---|
| MSMARCO | Reranking | 79.8 | 12,763.4 |
| HotpotQA | Multi-hop Reranking | 90.2 | 14,434.8 |
| MIMIC | Medical Diagnosis | 116.4 | 18,641.5 |
| arXiv | Long-context Classification | 8.8 | 1,413.9 |
| Amazon-3M | Taxonomy Prediction | 13.9 | 2,229.9 |
| MMLU-Pro | Knowledge Reasoning | 11.2 | 2,202.6 |
Table 2: Dataset statistics. We report task type, average option length, and total prompt length at .
Figure 9: Accuracy degradation across additional datasets. The degradation trend remains consistent across a wide range of tasks, supporting the generality of our findings beyond the main benchmarks.
Figure 10: Additional System-2 reasoning results on Qwen3-4B. All reasoning strategies continue to degrade substantially as the number of options increases.
| Model | |||
|---|---|---|---|
| Phi-4-mini | 88.0 | 28.7 | |
| Llama-3.1-8B | 95.3 | 62.2 | |
| Qwen3-30B | 99.3 | 91.2 |
Table 3: Accuracy degradation with 1,000 HotpotQA evaluation instances. All values are accuracy (%).
Figure 11: Extended candidate-set scaling on MIMIC. We extend the number of candidate options to and and additionally evaluate GPT-5.4. Accuracy continues to decrease beyond the regime across all evaluated models. Although GPT-5.4 is substantially more robust than the open-source models, its accuracy decreases from 98.5% at to 49.0% at .
| Model | Norm. GM | ||
|---|---|---|---|
| Qwen3-30B | |||
| Llama-3.1-8B | |||
| Phi-4-mini |
Table 4: Robustness of gold-margin collapse under alternative score formulations on HotpotQA. Values report changes from to .
Figure 12: Distractor verbosity control under candidate-set expansion (MIMIC, Llama-3.1-8B). Although longer distractor options substantially increase total context length, they do not consistently reproduce the degradation induced by increasing the number of candidate options.
Figure 13: Prediction-position and attention distributions across candidate-set sizes on HotpotQA. Prediction distributions become increasingly concentrated toward earlier candidate positions as the number of options increases. However, attention allocation patterns vary substantially across models and candidate-set sizes, with several settings exhibiting increased attention concentration toward later option positions.
Figure 14: Prediction-position and attention distributions across candidate-set sizes on ArXiv. Prediction distributions remain consistently concentrated toward earlier candidate positions across models and candidate-set sizes, despite substantial variation in attention allocation patterns.
Figure 15: Average prediction position under candidate-set scaling. As the number of candidate options increases, model predictions progressively shift toward earlier candidate positions across datasets and model families, while the gold-answer distribution remains stable.
| Model | ||
|---|---|---|
| HotpotQA | MIMIC | |
| Phi-4-mini | 0.81 / 0.10 / 0.02 / 0.01 / 0.01 | 0.69 / 0.14 / 0.06 / 0.06 / 0.03 |
| Llama-3.1-8B | 0.48 / 0.14 / 0.10 / 0.13 / 0.13 | 0.81 / 0.13 / 0.04 / 0.01 / 0.01 |
| Qwen3-30B | 0.26 / 0.15 / 0.21 / 0.20 / 0.20 | 0.46 / 0.19 / 0.14 / 0.14 / 0.09 |
Table 5: Five-bin prediction-position distributions at . Values indicate the proportion of predictions falling into each position region (front to back).
Figure 16: Early-position bias persists under reversed index assignment. Even when smaller answer indices are assigned to later candidates, model predictions continue to increasingly concentrate toward earlier candidate positions as the number of options grows.
Figure 17: Probability distributions under controlled gold-position intervention. Probability mass progressively concentrates toward earlier candidates as the option set grows.
Figure 18: Early-position bias on arXiv under original and reversed indexing. Predictions progressively concentrate toward earlier candidate positions as the number of options increases across models. The same trend persists under reversed index assignment, indicating that the observed bias is driven primarily by the option position rather than token-level index priors.
Figure 19: Controlled gold-position intervention on arXiv across models. Each row corresponds to a model, while columns vary the candidate set size. As the number of options increases, predictions become progressively concentrated toward earlier candidate positions regardless of the true gold location.
Figure 20: Anchor persistence on arXiv across models. Later-positioned gold answers become progressively less effective at overturning earlier candidate preferences.
Figure 21: Probability revision dynamics on arXiv across models. Later-arriving gold answers progressively lose the ability to overturn earlier candidate preferences.
Figure 22: Additional sequential revision dynamics across models. Across additional model families, later-arriving gold answers progressively lose the ability to revise earlier candidates under large-option comparison.
Figure 23: Mechanisms of inference-time restructuring. (a) Hierarchical partitioning reduces comparison difficulty through local candidate selection, but introduces irreversible early elimination errors as the number of options increases. (b) Permutation-based inference mitigates positional bias by aggregating predictions across reordered candidate views, improving robustness during final-stage comparison among surviving candidates.
| Method | Acc. (%) | Calls / Sample | Latency (s) |
|---|---|---|---|
| Single-Pass | 57.5 | 1.00 | 2.46 |
| Permutation | 76.5 | 5.76 | 8.74 |
| Hierarchy | 71.0 | 42.0 | 6.04 |
Table 6: Accuracy–efficiency trade-off of inference-time interventions. Latency reports measured wall-clock time per example under our vLLM-based batched inference implementation; model-call counts therefore do not translate directly into wall-clock latency.
Figure 24: Generalization across model scales. Hierarchical grouping and permutation-based inference consistently mitigate large-option degradation across both small and large models.
Figure 25: Effect of hyperparameters. Larger group sizes improve hierarchical partitioning by preserving stronger candidates during local selection, while permutation-based inference benefits from a moderate number of shuffled views but exhibits diminishing returns at larger scales.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers