Correct but Unselectable: The Hidden Interface Tax in Multi-Candidate Reasoning

Qianli Ma; Zhiqing Tang; Fanshuai Meng; Zhi Yao; Hanshuai Cui; Weijia Jia

cs.LGOct 23, 2026

Correct but Unselectable: The Hidden Interface Tax in Multi-Candidate Reasoning

Authors: Qianli Ma, Zhiqing Tang, Fanshuai Meng, Zhi Yao, Hanshuai Cui, Weijia Jia

Organizations: Faculty of Arts and Science, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, Guangdong, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China · Guangdong Key Lab of AI and Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai 519087, China

Abstract

Small language models deployed under cost, latency, or privacy constraints cannot simply switch to larger models when reasoning fails. Multi-candidate generate-then-rerank offers a natural test-time scaling route. However, more candidates do not always yield higher accuracy. We identify a previously overlooked bottleneck between generation and selection: a correct answer may appear in the candidate pool yet fail to be selected because it is incomplete, unparseable, or misaligned with the answer mode the fixed selector expects. We term this loss the hidden interface tax and formalize a coverage-conversion decomposition that distinguishes raw oracle coverage from selectable coverage, the subset of problems where a correct candidate can actually be consumed by the selector. We propose Compatibility-Aware Candidate Construction (CACC), an interface layer between the candidate source and the fixed selector that repairs fragments, aligns answer modes, and removes selector-confusing noise without retraining either component. On numeric reasoning (GSM8K, competition_math), CACC raises both oracle coverage and final accuracy. On GPQA Diamond, CACC alone improves final accuracy from 5.6% to 19.2% (+13.6 pp); combined with proposer-side strengthening it reaches 25.8% (+20.2 pp). On MMLU-Pro, CACC exposes a coverage-conversion gap: answer-mode-compatible construction raises oracle coverage, but final accuracy decreases by 0.85 pp under the original selector; adding proposer-side strengthening improves final accuracy.

Figures & tables

Figure 1: The missing candidate-interface layer in small-model multi-candidate reasoning. Existing methods optimize the candidate source or the selector, while whether the pool contains parseable, answer-mode-compatible, and selector-consumable candidates remains an independent bottleneck.

Figure 2: The offline–online flow of CACC . Lower half: offline calibration produces a frozen rulebook and routing policy. Upper half: online inference applies gold-free diagnostics and construction to the raw candidate pool before the fixed selector.

Figure 3: Main results across reasoning regimes. Panel A: Base/CACC oracle, V|O, and final. Panel B: latent coverage upper bound and selected final accuracy with CACC -vs-Base delta. Panel C: $\Delta$ oracle, $\Delta$ V|O, and $\Delta$ final separating coverage, conversion, and final. Panel D: four-step MMLU-Pro mechanism chain. Exact numbers are in Table 1 and Appendix Tables 2 , 3 .

Regime	Dataset	Method	Oracle	V\|O	Final	$\Delta$ Final	Ratio
Numeric	GSM8K	Base	0.4882	0.5885	0.2873	ref.	1.00 $\times$
Numeric	GSM8K	SP †	0.6471	0.5714	0.3698	+8.25pp	1.29 $\times$
Numeric	GSM8K	CACC	0.6391	0.5682	0.3632	+7.59pp	1.26 $\times$
Numeric	GSM8K	CACC + SP †	0.6962	0.6017	0.4189	+13.16pp	1.46 $\times$
Numeric	CompMath	Base	0.3210	0.4732	0.1519	ref.	1.00 $\times$
Numeric	CompMath	SP †	0.4616	0.6014	0.2776	+12.57pp	1.83 $\times$

Table 1: Candidate-interface results.

\Delta

Final and Ratio are computed against the Base row within the same dataset. † uses a Qwen3-8B proposer (different compute regime from the Qwen3-1.7B generator). Paired tests for CACC vs Base are in Appendix Table 10 .

Unit	Comparison	$O_{b}$	$O_{a}$	$V_{b}$	$V_{a}$	$F_{b}$	$F_{a}$	$\Delta O$	$\Delta V$	$\Delta F$
MMLU-Pro	no- AMC numeric $\rightarrow$ CACC	0.0369	0.4767	0.2590	0.4777	0.0096	0.2277	+43.98	+21.87	+21.81
MMLU-Pro	CCR+ AMC $\rightarrow$ CACC	0.4775	0.4767	0.4754	0.4777	0.2270	0.2277	-0.08	+0.23	+0.07
MMLU-512	diverse $\rightarrow$ CACC	0.4609	0.5547	0.4492	0.3592	0.2070	0.1992	+9.38	-9.00	-0.78
MMLU-512	CACC $\rightarrow$ +hygiene	0.5547	0.5547	0.3592	0.4049	0.1992	0.2246	+0.00	+4.58	+2.54
GSM8K full	SP $\rightarrow$ PS x CACC	0.6111	0.7195	0.5943	0.4700	0.3632	0.3381	+10.84	-12.43	-2.50

Table 2: Coverage–selection decomposition on audit rows (outside Table 1 ).

O,V,F

denote oracle, V|O, and final;

b/a

denote before/after;

\Delta

in percentage points.

Type	Dataset	Method / transition	Oracle	V\|O	Final	$\Delta F$
Readout	MMLU-Pro	numeric scoring $\rightarrow$ MCQ reparse	0.0369 $\rightarrow$ 0.4072	0.2590 $\rightarrow$ 0.5800	0.0096 $\rightarrow$ 0.2362	+22.66pp
AMC	MMLU-Pro	MCQ reparse $\rightarrow$ CACC	0.4072 $\rightarrow$ 0.4767	0.5800 $\rightarrow$ 0.4777	0.2362 $\rightarrow$ 0.2277	-0.85pp
SP	MMLU-Pro	CACC $\rightarrow$ CACC + SP	0.4767 $\rightarrow$ 0.5527	0.4777 $\rightarrow$ 0.5312	0.2277 $\rightarrow$ 0.2936	+6.59pp
SP	GPQA	CACC $\rightarrow$ CACC + SP	0.4545 $\rightarrow$ 0.4899	0.4222 $\rightarrow$ 0.5258	0.1919 $\rightarrow$ 0.2576	+6.57pp
Recovery	MMLU-Pro	CACC $\rightarrow$ +hygiene	0.5547 $\rightarrow$ 0.5547	0.3592 $\rightarrow$ 0.4049	0.1992 $\rightarrow$ 0.2246	+2.54pp
Recovery	GPQA	CACC $\rightarrow$ +hygiene	0.5127 $\rightarrow$ 0.4931	0.3961 $\rightarrow$ 0.5386	0.2031 $\rightarrow$ 0.2656	+6.25pp

Table 3: Ablation, recovery, and interaction evidence. Reports oracle, V|O, and final before

\rightarrow

after with the final delta; sample sizes, wins/losses,

p

-values, scope, and artifact statuses are in Appendix Tables 10 , 11 , and 12 .

Control	Evaluation	$N$	Base	CACC	$\Delta F$
Target verifier, canonical	GPQA	198	16.16	24.92	+8.75
Target verifier, canonical	MMLU held-out	11,520	26.40	27.84	+1.44
32B candidate judge	GPQA	198	6.57	23.23	+16.67
Mistral generator	MMLU confirm	512	14.22	15.94	+1.72
Prospective ARC reuse	ARC-Challenge	256	69.92	70.70	+0.78
Fresh generation	Evaluation	$N$	CACC	CD	CD+ CACC

Table 4: Additional controls. Final accuracies are percentages; gains are percentage points. Frozen-pool rows reuse the original candidates. Target-verifier rows are three-seed means; Mistral is a five-seed mean. CD uses fresh paired generation and a common canonical selector, with its own baseline.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Evidence unit	$N$	Mode	Oracle	Final	V\|O	Use in paper
GSM8K Base	1319	numeric	644/1319	379/1319	0.5885	Numeric baseline
GSM8K CACC	1319	numeric	843/1319	479/1319	0.5682	Numeric main result
CompMath Base	3199	numeric	1027/3199	486/3199	0.4732	Numeric baseline
CompMath CACC	3199	numeric	1419/3199	825/3199	0.5814	Numeric main result
MMLU no- AMC numeric	12032	numeric	444/12032	115/12032	0.2590	Archived collapse provenance
MMLU MCQ reparse	12032	choice_letter	4900/12032	2842/12032	0.5800	Readout recovery

Appendix

Table 5: Evidence units, sizes, and core metrics. The table lists sample size, answer mode, oracle, final, and V|O for the main evidence units in the main text.

Component	Work and accounting boundary
Upstream generation	Audited contract: 16 sequences, at most 128 new tokens each; pool construction precedes retention.
Completion construction	One additional batch of 4 sequences, at most 160 new tokens each, per example.
Routing / normalization	Deterministic diagnostics, parsing, sanitization, and deduplication.
Fixed selector	Two yes/no continuation scores per retained candidate; batching determines forward-pass count.
Stronger proposer	Qwen3-8B replaces or augments the 1.7B source in the composition setting; accounted separately.

Appendix

Table 6: Inference operations in the audited configuration.

Diagnostic	Setting	Oracle	V\|O	Final	$\Delta F$	Note
Sample sweep	cap 8, samples 4	0.6422	0.5844	0.3753	ref.	Fewer samples but high conversion
Sample sweep	cap 8, samples 32	0.6429	0.4517	0.2904	-8.49pp	Similar oracle, lower conversion
Retained-cap sweep	cap 8	0.6391	0.5682	0.3632	ref.	CACC canonical retained cap
Retained-cap sweep	cap 16	0.8249	0.3998	0.3298	-3.34pp	Larger retained pool raises oracle but hurts conversion

Appendix

Table 7: GSM8K candidate-budget diagnostics: oracle, V|O, and final do not move synchronously when sample count or retained cap changes.

Setting	Final	$\Delta F$	Note
Canonical verifier	0.3632	ref.	CACC canonical selector
verifier128	0.3465	-1.67pp	Same-pool selector sensitivity check
verifier512	0.3594	-0.38pp	close to Canonical verifier final

Appendix

Table 8: GSM8K same-pool verifier sensitivity (final-only).

View	Parser / scoring path	$N$	Final	Oracle	V\|O	Use
legacy/no- AMC	archived numeric exact match	12032	115/12032	444/12032	0.2590	Numeric scoring near-collapse
legacy/no- AMC	benchmark-locked MCQ reparse	12032	2842/12032	4900/12032	0.5800	Readout recovery
CCR+ AMC	benchmark-locked MCQ reparse	12032	2731/12032	5745/12032	0.4754	Micro-ablation reference
CACC	benchmark-locked MCQ reparse	12032	2740/12032	5736/12032	0.4777	Gap exposure
CACC + SP	benchmark-locked MCQ reparse	12032	3532/12032	6650/12032	0.5311	Gap narrowing

Appendix

Table 9: MMLU-Pro parser and scoring provenance: this comparison is a scoring/readout-path ablation, not an ablation between two independent candidate pools.

Figure 4: Mechanistic analysis of the coverage–conversion gap, compatibility, policy tradeoffs, and recovery closure. Panel A: oracle, V|O, and final decomposition. Panel B: unconverted gap between oracle and final. Panel C: interaction delta matrices for $\Delta$ oracle, $\Delta$ V|O, and $\Delta$ final. Panel D: before $\rightarrow$ after slopes. Paired statistics are in Tables 3 and 10 .

Figure 5: Bootstrap confidence intervals for second-family paired recovery. Both closure intervals lie above zero, supporting recoverable compatibility.

Evidence unit	Split / scope	Before	After	$\Delta F$	w/l	Statistical detail	Artifact
Second-family recovery	MMLU confirm-512	102/512 (0.1992)	115/512 (0.2246)	+2.54pp	17/4	McNemar $p=0.007197$ ; CI [0.0078, 0.0430]	Paired JSONL
Second-family recovery	GPQA-128	26/128 (0.2031)	34/128 (0.2656)	+6.25pp	9/1	McNemar $p=0.02148$ ; CI [0.0156, 0.1172]	Paired JSONL
Selection rescue	MMLU-Pro	0.2936	0.3320	+3.84pp	613/150	McNemar $p=3.43{\times}10^{-67}$	Paired pred.
Selection rescue	GPQA	0.2576	0.2929	+3.53pp	10/3	McNemar $p=0.0923$	Paired pred.
Coverage pressure	GSM8K full	0.3632	0.3381	-2.50pp	summary	Oracle +10.84pp; V\|O -12.43pp	Summary only
Second-family smoke	GSM8K-128	27/128 (0.2109)	29/128 (0.2266)	+1.56pp	7/1	Smoke row	Paired rows available

Appendix

Table 10: Paired statistics and diagnostic details for the mechanism rows in Table 3 .

Evidence unit	Candidate pool	Prediction records	Paired statistics	Manuscript use
Main Base / CACC rows	available	available	row-level deltas	Frozen result rows
MMLU no- AMC numeric view	archived	available	provenance audit	Scoring/readout-path provenance
MMLU MCQ reparse	archived scoreboard	available	provenance audit	MCQ Readout recovery
Second-family closure	available	available	McNemar + bootstrap	Frozen paired recovery
Conversion-aware rescue	available	available	McNemar where available	Gap narrowing evidence
Power Sampling x CACC	summary + pool	summary metrics	aggregate diagnostic	Diagnostic summary-only

Appendix

Table 11: Artifact availability matrix: artifact completeness and paper use of each evidence unit.

Method	Task	Correct / $N$	Final	Extra cost	Artifact status
Self-Refine	GSM8K full	643/1319	0.4875	5.0 calls/example	Full row available
Self-Refine	CompMath numeric	789/3199	0.2466	17.0 calls/example	Full row available
Self-Refine	MMLU-Pro	2838/12032	0.2359	17.0 calls/example	Full row available
Self-Refine	GPQA Diamond	24/198	0.1212	17.0 calls/example	Full row available
Self-Refine	GSM8K-128	51/128	0.3984	eval-128	Small reference row
ToT-lite	GSM8K-128	37/128	0.2891	eval-128	Small search baseline row

Appendix

Table 12: External baseline provenance and artifact status.

Component	Diagnostic trigger	Online action	Frozen output
CCR	partial derivation, missing final answer, fragmented numeric form	repair final-answer span and recover parseable candidate text	candidate text plus parser status
AMC	Answer-mode mismatch, missing choice letter, Wrapper drift	map candidate to benchmark-locked mode and protect choice-letter wrapper	mode-aligned candidate
SCH	duplicated scaffold, template residue, selector-attractive noise	remove residue, deduplicate, and keep retained cap stable	hygienic retained pool
Routing policy	diagnostic vector $\phi$ and retained cap $K$	select rule sequence without gold access	frozen $\pi(\phi,K)$

Appendix

Table 13: Offline rulebook components used by CACC .

Error class	Diagnostic signal	Expected effect
Oracle miss	No candidate parses to gold	Low oracle
Fragment / partial	Reasoning stops before final answer	Low oracle / Low V\|O
Answer-mode mismatch	Numeric answer in MCQ or missing choice letter	Low MCQ final
Invalid final	Malformed final channel	Parser failure
Scaffold residue	Instruction or template leakage	Selector confusion
Wrapper drift	Second-family answer wrapper differs	Transfer loss

Appendix

Table 14: Candidate-interface error taxonomy used in the qualitative analysis.

Example id	Dataset	Gold	Audited failure / recovery pattern
gsm8k-test-00595	GSM8K	13	Oracle miss; CACC verifier selects 15 although gold is 13
competition-math-numeric-test-02878	CompMath	13	raw oracle hit exists, but verifier overrules first candidate and selects 11
mmlu-pro-test-08795	MMLU-Pro	A	no- AMC numeric view misses a choice-form solution; MCQ-aware CACC + SP later recovers final answer A
gsm8k-test-00462	GSM8K	71	CACC repairs Fragment / partial candidate into a boxed final answer
gpqa-diamond-train-recNFJjE5PPTqVJGv	GPQA	B	selected wrong candidate contains scaffold / instruction residue, illustrating hygiene pressure
mmlu-pro-test-08026	MMLU-Pro	D	stronger proposer adds/selects D after CCR/no- AMC misses the answer

Appendix

Table 15: Representative audit examples. These examples are used for post-hoc qualitative analysis only, not for offline rule calibration.

Interface item	Constraint	Reproducibility role
CCR prompt template	no gold access; repair only observable fragments	reproduces Fragment / partial candidate repair
AMC answer wrapper	numeric tasks keep exact numeric form; MCQ tasks hard-lock to choice_letter	fixes answer-mode alignment and readout path
Numeric parser normalization	normalize whitespace, delimiters, final numeric span, and wrapper variants	keeps open-ended scoring stable across candidate sources
MCQ parser configuration	preserve benchmark option set and final choice-letter endpoint	supports MMLU-Pro / GPQA reparse provenance
SCH sanitizer rules	remove Scaffold residue, duplicated wrappers, and selector-attractive debris	stabilizes verifier input distribution
Manifest binding	every prompt/parser/sanitizer version is tied to run id, seed, split, and scorer	enables rerunning table and figure generation scripts

Appendix

Table 16: Prompt, parser, and sanitizer interfaces bound to run manifests.

Claim	Evidence unit	Paper use
CACC improves numeric open-ended reasoning	GSM8K + competition_math	Direct final-gain evidence
MCQ benefits from AMC / scoring-path alignment	MMLU-Pro readout chain	Answer-mode alignment evidence
Second-family compatibility is recoverable	MMLU confirm-512 / GPQA-128	Paired recovery evidence
CACC + SP narrows MCQ gap	MMLU-Pro / GPQA	Proposer-side composition evidence
Conversion-aware rescue improves final selection	MMLU-Pro / GPQA	Selection-side composition evidence
Power Sampling x CACC diagnoses Coverage pressure	aggregate summary	Coverage–conversion diagnostic

Appendix

Table 17: Evidence scope and claim mapping.

Symbol	Meaning
$x$ , $y_{x}$ , $\mathcal{D}$	input instance, ground-truth answer, evaluation dataset
$m\in\{\text{numeric},\texttt{choice\_letter}{}\}$	benchmark-locked answer mode, fixed by registry $\mathcal{M}$
$C_{0}(x)=\{c_{1},\ldots,c_{n}\}$	raw candidate pool of size $n$ produced by the generator
$T_{m}$ , $R_{K}$ , $C_{K}(x,m)$	mode-aware candidate construction, retained-cap controller, retained pool of size $\leq K$
$\phi(x,c,m)$	gold-free diagnostic feature vector (parseability, Answer-mode mismatch, residue, …)
$\mathcal{R}$ , $\pi$	frozen rulebook ( $\textsc{CCR}{}/\textsc{AMC}{}/\textsc{SCH}{}$ ) and routing policy $\pi(\phi,K)\!\mapsto a$

Appendix

Table 18: Notation summary. Symbols are introduced in Sections 3 and 4 and used throughout the paper.

Benchmark	Split / $N$	Mode	Retained cap	Parser / scoring endpoint
GSM8K	clean test, $N{=}1319$	numeric	8	numeric exact match on final boxed span
competition_math:numeric	numeric subset, $N{=}3199$	numeric	8	numeric exact match on normalized final span
MMLU-Pro	test, $N{=}12032$	MCQ	8	benchmark-locked choice-letter reparse
GPQA Diamond	full, $N{=}198$	MCQ	8	benchmark-locked choice-letter reparse
MMLU-Pro confirm-512	confirm split, $N{=}512$	MCQ	8	second-family paired closure run
GPQA-128	subset, $N{=}128$	MCQ	8	second-family paired closure run

Appendix

Table 19: Per-benchmark setup details. The retained cap is shared across compared methods on each benchmark.

Evaluation	$N$	Seed	Base	CACC	$\Delta F$	95% CI	$p$
GPQA	198	14251	15.66	24.24	+8.59	[2.53,15.15]	.0137
GPQA	198	20260724	16.67	26.26	+9.60	[3.54,15.66]	.00432
GPQA	198	314159	16.16	24.24	+8.08	[1.52,14.65]	.0259
MMLU held-out	11,520	14251	26.35	28.07	+1.73	[1.04,2.43]	$6.48\times 10^{-7}$
MMLU held-out	11,520	20260724	26.52	28.15	+1.63	[0.94,2.34]	$4.23\times 10^{-6}$
MMLU held-out	11,520	314159	26.35	27.31	+0.96	[0.26,1.68]	.00799

Appendix

Table 20: Target-verifier canonical results on original pools. CIs are paired per-seed 95% bootstrap intervals for

\Delta F

(pp);

p

is the per-contrast exact McNemar value, without a cross-row multiplicity adjustment.

Evaluation	Cell	$O$	$V\|O$	$F$
GPQA ( $N=198$ )	Base	20.71	82.93	17.17
	CACC	51.01	45.54	23.23
	CD	75.25	39.60	29.80
	CD+ CACC	81.82	35.19	28.79
MMLU confirm ( $N=512$ )	Base	31.45	72.05	22.66
	CACC	48.83	60.00	29.30

Appendix

Table 21: Fresh-generation four-cell outcomes.

O

V|O

, and

F

are percentages.

Evaluation	$N$	Selector	Base	CACC	$\Delta F$	$p$
GPQA	198	Majority	16.67	19.70	+3.03	.2101
GPQA	198	8B option-logprob	17.68	28.28	+10.61	.00460
GPQA	198	1.7B candidate judge	5.56	18.69	+13.13	$4.23\times 10^{-5}$
GPQA	198	32B candidate judge	6.57	23.23	+16.67	$1.07\times 10^{-6}$
MMLU full	12,032	Majority	21.17	18.48	-2.69	$3.43\times 10^{-18}$
MMLU confirm	512	8B option-logprob	32.62	32.62	0.00	1.000

Appendix

Table 22: Selector-family results on original pools. Accuracies are percentages;

p

is the exact paired McNemar value for each row, without a table-wide adjustment.

Condition	$O$	$V$	$F$
Base	85.16	82.11	69.92
CACC	91.02	77.68	70.70

Appendix

Table 23: Prospective ARC-Challenge results (

N=256

). Oracle (

O

), conditional conversion (

V

), and final accuracy (

F

) are percentages.

Efficient Language Model Reasoning

Efficient language model reasoning studies how to improve problem-solving accuracy while reducing inference, memory, latency, and training costs.

See all 371 papers

Oct 1, 2026

ReSolve: Reusing Candidate Reasoning through Selective Generative Moderation

Sampling multiple solutions spends computation on intermediate deductions and unfinished arguments as well as final answers. We introduce ReSolve, a training-free inference procedure that reuses this candidate reasoning through selective generative moderation. An answer-distribution controller invokes a model to examine existing derivations when candidates disagree or lack a parseable answer, then incorporates the generated solution into a bounded loop. Under Hybrid scoring on 130 competition-mathematics problems evaluated with two independently sampled candidate pools, ReSolve obtains 100 and 99 correct answers, compared with 91 and 92 for voting over the same four candidates, with no correct-to-incorrect changes relative to that vote in either pool. Eight-sample self-consistency obtains 94 and 96 correct answers while consuming substantially more tokens; ReSolve uses 46.3% and 47.2% fewer tokens in the two evaluations. A controlled ablation removes visible derivations while retaining answer keys, vote counts, and the per-state output-cap rule, reducing accuracy from 100 to 93 correct despite increasing computation. Selective and always-on Uniform moderation both solve 97 problems, while selectivity reduces moderation tokens by approximately 54% and total pipeline tokens by 6.2%. These results support candidate reasoning as reusable inference computation. They do not establish an accuracy advantage over additional sampling or a distinct benefit from specialized route instructions.

Bangji Yang, Jiajun Fan, Hongbo Ma +6

University of Illinois at Urbana-Champaign · Rutgers University · University of Illinois Chicago +2

Efficient Language Model Reasoning Inference-Time Compute Allocation Efficient Language Model Inference LLM Mathematical Reasoning

Ask PDF

Jul 27, 2026

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.

Zongyou Yang, Yinghan Hou

Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom

LLM Evaluation LLM Answer Verification Language Model Generation Evaluation LLM Reliability

Ask PDF

Sep 26, 2026

Overwhelmed by Choice: Studying LLM Decision Making at Scale

Multiple-choice and candidate-selection evaluations are widely used to assess LLM reasoning and decision-making, yet most benchmarks contain relatively small candidate sets. It remains unclear whether conclusions drawn from these settings remain valid as the candidate space scales. We systematically evaluate LLMs as the number of competing candidates increases and find substantial accuracy degradation across tasks, prompting strategies, and model scales. Controlled analyses show that standard long-context retrieval explanations cannot fully account for this degradation. Instead, we identify two systematic failure patterns. First, gold-margin collapse: the score gap between the correct answer and the strongest distractor progressively shrinks, driven primarily by weakening confidence in the correct answer. Second, earlier candidate preferences become increasingly difficult to overturn, with later candidates exerting progressively weaker influence on the final prediction. Motivated by these findings, we evaluate hierarchical partitioning and permutation-based inference, which improve accuracy by roughly 20 percentage points at

N=160

on both HotpotQA and MIMIC. Overall, our results identify candidate-set scale as an important evaluation-protocol variable and show that strong small-option performance does not necessarily imply robust large-scale candidate comparison.

Yu-Chi Lin, Aryan Seth, Anshul Aravind +4

University of California, Los Angeles · University of Cincinnati

LLM Evaluation LLM Decision-Making Multiple-Choice Question Answering

Ask PDF

Abstract

Efficient Language Model Reasoning

Explore similar work

ReSolve: Reusing Candidate Reasoning through Selective Generative Moderation

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

Overwhelmed by Choice: Studying LLM Decision Making at Scale