Combining the answers a large language model (LLM) samples for a question into one decision is a test-time information fusion problem, usually solved by majority voting. Voting is unreliable on difficult questions, where the sampled answers share correlated errors, so the wrong answer can win and drawing more samples makes the decision worse. Selecting a candidate by reading a correctness signal from the model's hidden states is a promising alternative, but its accuracy varies across models and tasks, and no measure indicates when it can be trusted. In this paper, we propose CASE (Correctness-Axis SElection), a dynamic selection combiner that trains a linear gate on the answer-token hidden state and selects the highest-scoring candidate. Its main contribution is decodability, a leakage-free measure of how well the gate ranks a question's correct candidates above its incorrect ones, which predicts whether hidden-state selection will outperform voting. A conventional probe appears accurate only because of question-identity leakage, which vanishes under question-grouped evaluation. On held-out data, decodability predicts the accuracy gain of selection over voting with a Pearson correlation r=0.75 and a decision threshold near AUC=0.60. Across general and medical LLMs, CASE improves over voting by up to 19 points on medium-difficulty questions and 16.8 points on hard questions. Decodability depends on the aligned knowledge a model must recall, not on its scale, and its prediction transfers to an unseen scientific domain within 3.8 points. It thus provides a practical criterion, measurable in advance for a given model and task, for choosing between learned selection and majority voting.
Figures & tables
Figure 1: Two ways to fuse N candidate answers. Majority voting counts surface answers and fails when the correct answer is a minority; hidden-state selection (CASE) reads an internal correctness score from each candidate’s answer-token representation and returns the highest-scoring candidate.
Figure 2: The naive probe’s apparent success is question-identity leakage. (a) On LogiQA (Qwen2.5-7B, 651 questions), the naive random-split cross-validation looks informative ( 0.610 ), but grouping folds by question drops it to 0.542 and the selection-relevant within-question ranking AUC is at chance ( 0.502 ). (b) The leakage is not an artifact of that single cell: at each setting’s answer-token operating layer, random-split candidate accuracy (leaked) exceeds the leakage-free question-grouped accuracy in all 15 medical settings (Supplementary Table S17). Exact LogiQA values are in the text.
Model
Type
Bench
Diff
nq
AUC
vote%
CASE%
CASE − vote
95% CI
p
OpenBioLLM-8B
strong
MedQA
med
48
0.702
51.3
63.2
+11.9
[+0.6,+23.5]
.039
OpenBioLLM-8B
strong
MedQA
hard
288
0.606
3.5
11.1
+7.5
[+5.3,+9.8]
< .001
OpenBioLLM-8B
strong
MedMCQA
med
73
0.794
50.6
69.7
+19.1
[+11.1,+27.0]
< .001
OpenBioLLM-8B
strong
MedMCQA
hard
296
0.706
3.9
13.2
+9.3
[+7.1,+11.6]
< .001
OpenBioLLM-8B
strong
PubMedQA-cb
med
71
0.685
48.3
65.3
+17.0
[+7.3,+26.3]
.001
OpenBioLLM-8B
strong
PubMedQA-cb
hard
286
0.694
4.5
17.9
+13.4
[+10.8,+16.4]
< .001
Table 1: CASE vs. majority vote across models, benchmarks and difficulty (leakage-free; 95% bootstrap CI; two-sided p ). AUC is the within-question correctness ROC-AUC at the operating layer. cb = closed-book; ob = open-book. CASE − vote is computed from unrounded accuracies and can differ from the displayed vote% and CASE% by ±0.1 pp.
Figure 3: CASE minus majority vote (percentage points, medium difficulty) across models and benchmarks. Only knowledge-aligned medical models exceed voting; general and weakly-aligned medical models fall below it. Each row’s within-question decodability (AUC) is shown in parentheses: the gain crosses zero exactly as the AUC crosses the ≈0.60 threshold, so the decodability law can be read row by row.
Question (abridged)
Options — votes ⋅ gate g
Vote
CASE
Man with an itchy annular abdominal plaque; KOH prep confirms hyphae. Next best step in management?
A. Itraconazole — 6 ×⋅ g 0.90 (gold) B. Griseofulvin — 10 ×⋅ g 0.31 C. Topical clindamycin — 0 × D. Doxycycline — 0 ×
B
A
Boy with seizure disorder, apical murmur, calcified retinal lesions. Mutation in which gene?
A. NF1 (chr 17) — 8 ×⋅ g 0.71 B. NF2 (chr 22) — 0 × C. TSC1 (chr 9) — 8 ×⋅ g 1.00 (gold) D. VHL (chr 3) — 0 ×
A
C
Drug inhibits phosphate release by the myosin head; which cross-bridge step is blocked?
A. Myosin head cocking — 12 ×⋅ g 0.00 B. Exposure of binding sites — 0 × C. Myosin binding to actin — 0 × D. Power stroke — 4 ×⋅ g 0.29 (gold)
A
D
Table 2: Worked examples (OpenBioLLM-8B, MedQA; N=16 sampled candidates; leakage-free question-grouped gate). For each item: the option votes and mean CASE gate g , the majority-vote pick (wrong, red ), and the CASE pick (correct gold, teal ).
Figure 4: Single-variable controls. Each row changes one variable and holds the rest fixed; in every case the condition that requires the model to recall aligned knowledge it holds (teal) both raises the within-question decodability (AUC) and moves the fusion gain from below voting to above it, while the matched control (grey)—knowledge absent, shallow, or readable off a supplied passage—does neither. The recall row’s gains are the same-question, same-layer closed- vs. open-book comparison (Supplementary Table S16); each point’s AUC is that condition’s own operating-layer decodability (closed-book L24, open-book L20); the others hold the architecture, the task, or the domain fixed.
Model / benchmark
within-Q AUC
vote %
CASE %
medium (pp)
uncond. (pp)
OpenBioLLM-8B MedMCQA
0.794
50.6
69.7
+19.1
+6.8
OpenBioLLM-8B MedQA
0.702
51.3
63.2
+11.9
+4.0
OpenBioLLM-70B MedQA
0.691
49.5
59.8
+10.3
+0.7
OpenBioLLM-8B PubMedQA-cb
0.685
48.3
65.3
+17.0
+6.0
OpenBioLLM-8B PubMedQA-ob
0.650
53.2
61.9
+8.7
+1.7
BioMistral-7B MedQA
0.645
47.1
55.2
+8.1
+2.7
Table 3: The 15 model–benchmark points underlying the decodability → gain law, all at the pooled-best late operating layer (held-out r=0.75 , Supplementary Fig. S1; in-sample Pearson r=0.956 as a consistency check, p=3×10−8 , Spearman ρ=0.92 ; separating threshold AUC≈0.60 ). The medium column is the medium-difficulty gain that the law predicts (the mechanistic result); uncond. is the difficulty-unconditional gain over the whole candidate pool—the number a deployer who cannot bin questions by difficulty would experience. The unconditional gain is roughly a third of the medium-bin gain (easy questions dominate the pool and CASE does not help there), but near the boundary it scatters within ±0.4 pp of zero (e.g. Med42-8B MedMCQA −0.1 just above threshold, meditron-7B +0.2 just below) and is clearly positive only well above threshold, so the diagnostic predicts the sign of the deployable gain away from the boundary.
Figure 5: Within-question decodability predicts the fusion gain. (a) The empirical law: across 15 medical settings (filled, colored by decodability region) the within-question AUC predicts the CASE-minus-voting gain (Pearson r=0.96 ; fitted line), with a clean threshold at AUC≈0.60 ; open diamonds are out-of-sample GPQA settings (Section 6.7 ), not used in the fit. Error bars are 95% confidence intervals (horizontal: question-bootstrap of the within-question AUC; vertical: bootstrap of the gain); the AUC is markedly tighter than the gain, consistent with it being the more stable indicator. (b) The stylized model of Section 4 : predicted selection accuracy rises with decodability from the pool accuracy A toward 1 (Proposition 1) and overtakes the voting level V at a threshold above 21 (Proposition 2); the three curves are three correct-fractions A . (c) Out-of-sample validation: decodability measured on a calibration half predicts the deployment-half gain on held-out questions ( r=0.75 ; 13 settings, 10 splits each, mean ± s.d.).
N
single%
majority%
CASE%
CASE − maj
1
20.4
20.4
20.4
+0.0
2
20.7
20.7
25.2
+4.5
4
20.4
14.9
33.8
+18.9
8
20.7
10.0
42.1
+32.1
16
20.8
6.0
47.3
+41.3
Table 4: Byzantine N -scaling (OpenBioLLM × MedQA, minority-correct subset, 131 questions). Real draws without replacement from the candidate pool. CASE − maj is computed from unrounded accuracies and can differ from the displayed columns by ±0.1 pp.
Figure 6: Byzantine N -scaling. (a) On the minority-correct subset, as the candidate count N grows CASE climbs from 20.4 to 47.3% while majority voting collapses from 20.4 to 6.0% . (b) Resolved by difficulty, the CASE-minus-vote gain diverges on the minority-correct and hard bins ( +41.3 and +16.8 pp at N=16 ) but reverses on the medium bin ( −2.0 ), where voting recovers—the advantage is specific to the Byzantine hard regime. (c) The stylized model (Proposition 3, a=0.70 ) reproduces the divergence: predicted CASE accuracy rises with N while voting falls.
Figure 7: Answer-token hidden states (OpenBioLLM-8B, MedQA, layer 28) projected onto the correctness axis (mass-mean direction). (a) Marginal density of correct (green) vs. incorrect (red) answers; (b) the same states in 2-D, where a ∼ 19% cluster of incorrect answers separates cleanly on the far incorrect side—candidates the gate confidently and correctly scores as wrong, a recognizable error mode—while the harder remaining errors overlap the correct mass on the right. The pooled single-axis projection separates the two classes at AUC =0.73 ; this is the pooled projection AUC at the fixed layer 28, distinct from the within-question operating-layer decodability AUC (0.702) reported in Tables 1 and 3 .
Figure 8: Anatomy of the internal signal. (a) Within-question decodability emerges with scale for general models and is saturated by medical specialization. (b) The selection gain concentrates in the Byzantine (medium/hard) regime. (c) Adding the correctness direction during generation does not improve accuracy—the readout is not a causal lever.
Method
accuracy
vs. vote
extra cost / question
single agent
46.5%
—
—
majority vote
51.8%
—
0
generative verifier
63.0%
+11.1
∼∣options∣ fwd passes (cached; 4 on MedQA)
CASE
62.0%
+10.2
≈0 (reuse activations + 1 matvec)
Table 5: Generative verifier vs. CASE (OpenBioLLM × MedQA, medium difficulty). All four estimates come from the same diagnostic candidate pool and the same draws (best-of- N , N=4 , 200 draws), so the majority-vote baseline (51.8%) is shared. Gains over voting are computed from unrounded accuracies and can differ from the displayed percentages by ±0.1 pp (the verifier’s +11.1 vs. 63.0−51.8 ). The operating point differs from the medium cell of Table 1 (full-pool selection), so the CASE accuracy there, 63.2%, differs slightly within sampling uncertainty. To quantify the cost gap: the verifier’s ∣options∣ extra forward passes cost on the order of ∣options∣⋅2NparamsT≈2×1013 FLOP per question (8B model, T≈300 prompt tokens), whereas CASE adds one d=4096 matrix–vector product per candidate on activations already produced during generation, ≈105 FLOP per question—about eight orders of magnitude less marginal compute, though both share the upstream candidate-generation cost. This is an estimate from parameter and token counts, not a wall-clock benchmark.
Model
Benchmark
within-Q AUC
medium CASE − vote ( p )
hard CASE − vote ( p )
Qwen2.5-Math-7B
MATH-500
0.789
+8.1 (.043)
+2.5 ( < .001)
Qwen2.5-7B
MATH-500
0.763
+6.6 (n.s.)
+2.6 ( < .001)
Qwen2.5-Math-7B
GSM8K
0.820†
—
+11.9 (.001)
Qwen2.5-7B
GSM8K
0.595†
—
+0.3 (n.s.)
Table 6: Mathematics domains, at the operating layer as for the medical settings; p from the same question-level bootstrap. For MATH-500 the AUC is the medium-bin within-question AUC (the law’s axis, Eq. 11 ), with medium/hard n=17/233 (Qwen2.5-7B) and 27/236 (Qwen2.5-Math-7B). GSM8K is too ceiling-limited for the medium-bin axis (only 7 and 4 medium questions), so it is reported off-axis: pooled within-question AUC ( † ) and hard-bin gain over n=15 and 18 . n.s. = not significant.
medium
hard
Model
Subfield
AUC
vote%
CASE%
Δ vote
vote%
CASE%
Δ vote ( p )
Qwen2.5-14B
Phys
0.681
50.3
61.1
+10.8 (n.s.)
3.8
11.6
+7.9 ( < .001)
Qwen2.5-7B
Phys
0.661
54.0
58.2
+4.3 (n.s.)
4.6
12.0
+7.4 ( < .001)
Qwen2.5-14B
Chem
0.576
49.9
49.6
−0.3 (n.s.)
6.3
12.6
+6.2 ( < .001)
Qwen2.5-7B
Chem
0.550
54.9
47.4
−7.6 (n.s.)
6.5
14.2
+7.7 ( < .001)
Table 7: A non-medical knowledge domain (GPQA, graduate science; general models, late-layer operating point). Decodability spans the threshold (physics above, chemistry just below), and hidden-state selection beats voting on the hard ( Byzantine ) subset in every setting. The hard subset is well-powered (hard-bin n=133 – 136 ; Qwen-14B physics on its complete 187-question set), whereas the medium bins are small ( n=19 – 30 ), so the medium gains are mostly not significant (n.s. = not significant).
Model / benchmark
within-Q AUC
self-cert
seq-logp
pred-ent
CASE advantage
OpenBioLLM-8B MedMCQA
0.770
+4.9 n.s.
+13.3
+13.3
+3.7
OpenBioLLM-8B MedQA
0.759
+0.9 n.s.
+14.6
+13.4
+9.7
OpenBioLLM-8B PubMedQA-cb
0.707
−22.1
−4.8 n.s.
−0.1 n.s.
+23.2
Qwen2.5-7B MedQA
0.574
−3.3 n.s.
−3.8 n.s.
−3.2 n.s.
+4.6
meditron-7B MedQA
0.531
−14.9
−14.3
−14.4
+7.3
Table 8: CASE vs. near-free output-space selectors at medium difficulty, on a shared candidate pool. The self-cert / seq-logp / pred-ent columns give each selector’s gain over majority voting (pp); CASE advantage is CASE minus the best of the three—a paired comparison, positive whenever CASE beats every near-free selector. within-Q AUC is the pooled within-question ROC-AUC of the CASE gate (over all difficulties on this single seed-7 pool, so it differs slightly from the operating-layer AUC in Tables 1 and 3 ). n.s. = not significant at α=0.05 . Selector gains are over majority voting on a shared seed-7 candidate pool (operating layers L22–L27).
Figure 9: Deployment nomogram. A practitioner measures the within-question decodability on a small calibration set, locates the AUC on the horizontal axis, and reads the expected CASE − vote gain and its ±6 pp band off the law line. The vertical bands are the decision zones: below AUC0.55 retain voting; in the indeterminate band ( 0.55 – 0.65 ) the measurement cannot decide; above 0.65 deploy CASE. Open circles are the medium-difficulty gains the law is fit on (the mechanistic quantity); filled points are the difficulty-unconditional gains a deployer actually experiences (smaller, but positive exactly in the deploy zone).
A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decodability and selection, we distinguish read from write at the first answer token. Read asks whether the gold token can be decoded from intermediate residual states under same-relation decoy controls. Write asks whether the final readout ranks that token first among content tokens. Under three different readers, with a randomized-label control, a substantial fraction of failures remain readable while another content token is selected. We explain this through the selection margin at the final readout, the difference between the answer logit and the logit of its strongest alternative, which is answer support minus alternative support, and can also be split into a context-averaged baseline linked to token frequency and an item-specific term. Setting the answer support to the level typical of successful generations is sufficient to recover first-token selection for the majority of failures in most of the models we study; the original alternative remains ahead in most remaining failures under this edit, and this outcome follows directly from the readout geometry. Removing the frequency direction alone shifts selection but rarely recovers the answer. Prompt variants of the same fact that succeed supply support that transfers to failing variants through the residual stream and through late MLP outputs, with less consistent effects through late attention. First-token recovery leaves most full answers wrong, which limits the recovery achieved by these edits and separates three things that are easily conflated, decodability, recoverability, and generation.
Multiple-choice and candidate-selection evaluations are widely used to assess LLM reasoning and decision-making, yet most benchmarks contain relatively small candidate sets. It remains unclear whether conclusions drawn from these settings remain valid as the candidate space scales. We systematically evaluate LLMs as the number of competing candidates increases and find substantial accuracy degradation across tasks, prompting strategies, and model scales. Controlled analyses show that standard long-context retrieval explanations cannot fully account for this degradation. Instead, we identify two systematic failure patterns. First, gold-margin collapse: the score gap between the correct answer and the strongest distractor progressively shrinks, driven primarily by weakening confidence in the correct answer. Second, earlier candidate preferences become increasingly difficult to overturn, with later candidates exerting progressively weaker influence on the final prediction. Motivated by these findings, we evaluate hierarchical partitioning and permutation-based inference, which improve accuracy by roughly 20 percentage points at N=160 on both HotpotQA and MIMIC. Overall, our results identify candidate-set scale as an important evaluation-protocol variable and show that strong small-option performance does not necessarily imply robust large-scale candidate comparison.
Yu-Chi Lin, Aryan Seth, Anshul Aravind +4
University of California, Los Angeles · University of Cincinnati
Voting over multiple LLM responses is a common primitive in test-time scaling and ensemble inference. Collecting more responses can expand the candidate pool and increase the chance that a correct answer is discovered. Under a fixed call budget, a discovered answer still needs to accumulate enough support within the remaining calls to become the final plurality winner, creating a discovery-to-decision gap. In this work, we characterize this gap through the realized vote state and remaining call budget. We derive a sharp recoverability threshold and show that, as sampling proceeds, the observed candidate set can only expand while the set of reachable endpoint winners can only contract, inducing a candidate-level conversion window. Under a specified iid response law, the same state yields exact finite-horizon endpoint probabilities. We further show that merging wrong-answer identities preserves single-call correctness and cannot improve plurality accuracy, and that the effect of redistributing wrong-answer probability depends on the realized vote state. Singleton reachability yields a gold-free exact locking certificate. For a known answer universe, its first trigger is the earliest prefix at which all admissible continuations yield the same fixed-budget output. Empirically, most discovered-but-unselected correct answers lose reachability only after discovery. In a controlled Word16 study, input permutation improves raw-plurality accuracy by 21.1 points with essentially unchanged single-call correctness. Exact locking saves 28-30% of calls at a 16-call budget while preserving every fixed-budget output.