A decodability criterion predicts when hidden-state selection beats majority voting in large language models

Zhixiang wang; Ziliang Hong; Ulas Bagci

cs.AIAug 17, 2026

A decodability criterion predicts when hidden-state selection beats majority voting in large language models

Authors: Zhixiang wang, Ziliang Hong, Ulas Bagci

Organizations: Northwestern University, Chicago, United States

Abstract

Combining the answers a large language model (LLM) samples for a question into one decision is a test-time information fusion problem, usually solved by majority voting. Voting is unreliable on difficult questions, where the sampled answers share correlated errors, so the wrong answer can win and drawing more samples makes the decision worse. Selecting a candidate by reading a correctness signal from the model's hidden states is a promising alternative, but its accuracy varies across models and tasks, and no measure indicates when it can be trusted. In this paper, we propose CASE (Correctness-Axis SElection), a dynamic selection combiner that trains a linear gate on the answer-token hidden state and selects the highest-scoring candidate. Its main contribution is decodability, a leakage-free measure of how well the gate ranks a question's correct candidates above its incorrect ones, which predicts whether hidden-state selection will outperform voting. A conventional probe appears accurate only because of question-identity leakage, which vanishes under question-grouped evaluation. On held-out data, decodability predicts the accuracy gain of selection over voting with a Pearson correlation r=0.75 and a decision threshold near AUC=0.60. Across general and medical LLMs, CASE improves over voting by up to 19 points on medium-difficulty questions and 16.8 points on hard questions. Decodability depends on the aligned knowledge a model must recall, not on its scale, and its prediction transfers to an unseen scientific domain within 3.8 points. It thus provides a practical criterion, measurable in advance for a given model and task, for choosing between learned selection and majority voting.

Figures & tables

Figure 1: Two ways to fuse $N$ candidate answers. Majority voting counts surface answers and fails when the correct answer is a minority; hidden-state selection (CASE) reads an internal correctness score from each candidate’s answer-token representation and returns the highest-scoring candidate.

Figure 2: The naive probe’s apparent success is question-identity leakage. (a) On LogiQA (Qwen2.5-7B, 651 questions), the naive random-split cross-validation looks informative (

0.610

), but grouping folds by question drops it to

0.542

and the selection-relevant within-question ranking AUC is at chance (

0.502

). (b) The leakage is not an artifact of that single cell: at each setting’s answer-token operating layer, random-split candidate accuracy (leaked) exceeds the leakage-free question-grouped accuracy in all 15 medical settings (Supplementary Table S17). Exact LogiQA values are in the text.

Model	Type	Bench	Diff	$n_{q}$	AUC	vote%	CASE%	CASE $-$ vote	95% CI	$p$
OpenBioLLM-8B	strong	MedQA	med	48	0.702	51.3	63.2	$+11.9$	$[+0.6,+23.5]$	.039
OpenBioLLM-8B	strong	MedQA	hard	288	0.606	3.5	11.1	$+7.5$	$[+5.3,+9.8]$	$<$ .001
OpenBioLLM-8B	strong	MedMCQA	med	73	0.794	50.6	69.7	$+19.1$	$[+11.1,+27.0]$	$<$ .001
OpenBioLLM-8B	strong	MedMCQA	hard	296	0.706	3.9	13.2	$+9.3$	$[+7.1,+11.6]$	$<$ .001
OpenBioLLM-8B	strong	PubMedQA-cb	med	71	0.685	48.3	65.3	$+17.0$	$[+7.3,+26.3]$	.001
OpenBioLLM-8B	strong	PubMedQA-cb	hard	286	0.694	4.5	17.9	$+13.4$	$[+10.8,+16.4]$	$<$ .001

Table 1: CASE vs. majority vote across models, benchmarks and difficulty (leakage-free; 95% bootstrap CI; two-sided

p

). AUC is the within-question correctness ROC-AUC at the operating layer. cb

=

closed-book; ob

=

open-book. CASE

-

vote is computed from unrounded accuracies and can differ from the displayed vote% and CASE% by

\pm 0.1

pp.

Figure 3: CASE minus majority vote (percentage points, medium difficulty) across models and benchmarks. Only knowledge-aligned medical models exceed voting; general and weakly-aligned medical models fall below it. Each row’s within-question decodability (AUC) is shown in parentheses: the gain crosses zero exactly as the AUC crosses the

\approx 0.60

threshold, so the decodability law can be read row by row.

Question (abridged)	Options — votes $\cdot$ gate $g$	Vote	CASE
Man with an itchy annular abdominal plaque; KOH prep confirms hyphae. Next best step in management?	A. Itraconazole — 6 $\times\cdot$ g 0.90 (gold) B. Griseofulvin — 10 $\times\cdot$ g 0.31 C. Topical clindamycin — 0 $\times$ D. Doxycycline — 0 $\times$	B	A
Boy with seizure disorder, apical murmur, calcified retinal lesions. Mutation in which gene?	A. NF1 (chr 17) — 8 $\times\cdot$ g 0.71 B. NF2 (chr 22) — 0 $\times$ C. TSC1 (chr 9) — 8 $\times\cdot$ g 1.00 (gold) D. VHL (chr 3) — 0 $\times$	A	C
Drug inhibits phosphate release by the myosin head; which cross-bridge step is blocked?	A. Myosin head cocking — 12 $\times\cdot$ g 0.00 B. Exposure of binding sites — 0 $\times$ C. Myosin binding to actin — 0 $\times$ D. Power stroke — 4 $\times\cdot$ g 0.29 (gold)	A	D

Table 2: Worked examples (OpenBioLLM-8B, MedQA;

N=16

sampled candidates; leakage-free question-grouped gate). For each item: the option votes and mean CASE gate

g

, the majority-vote pick (wrong, red ), and the CASE pick (correct gold, teal ).

Figure 4: Single-variable controls. Each row changes one variable and holds the rest fixed; in every case the condition that requires the model to recall aligned knowledge it holds (teal) both raises the within-question decodability (AUC) and moves the fusion gain from below voting to above it, while the matched control (grey)—knowledge absent, shallow, or readable off a supplied passage—does neither. The recall row’s gains are the same-question, same-layer closed- vs. open-book comparison (Supplementary Table S16); each point’s AUC is that condition’s own operating-layer decodability (closed-book L24, open-book L20); the others hold the architecture, the task, or the domain fixed.

Model / benchmark	within-Q AUC	vote %	CASE %	medium (pp)	uncond. (pp)
OpenBioLLM-8B MedMCQA	0.794	50.6	69.7	$+19.1$	$+6.8$
OpenBioLLM-8B MedQA	0.702	51.3	63.2	$+11.9$	$+4.0$
OpenBioLLM-70B MedQA	0.691	49.5	59.8	$+10.3$	$+0.7$
OpenBioLLM-8B PubMedQA-cb	0.685	48.3	65.3	$+17.0$	$+6.0$
OpenBioLLM-8B PubMedQA-ob	0.650	53.2	61.9	$+8.7$	$+1.7$
BioMistral-7B MedQA	0.645	47.1	55.2	$+8.1$	$+2.7$

Table 3: The 15 model–benchmark points underlying the decodability

\rightarrow

gain law, all at the pooled-best late operating layer (held-out

r=0.75

, Supplementary Fig. S1; in-sample Pearson

r=0.956

as a consistency check,

p=3\times 10^{-8}

, Spearman

\rho=0.92

; separating threshold

\mathrm{AUC}\approx 0.60

). The medium column is the medium-difficulty gain that the law predicts (the mechanistic result); uncond. is the difficulty-unconditional gain over the whole candidate pool—the number a deployer who cannot bin questions by difficulty would experience. The unconditional gain is roughly a third of the medium-bin gain (easy questions dominate the pool and CASE does not help there), but near the boundary it scatters within

\pm 0.4

pp of zero (e.g. Med42-8B MedMCQA

-0.1

just above threshold, meditron-7B

+0.2

just below) and is clearly positive only well above threshold, so the diagnostic predicts the sign of the deployable gain away from the boundary.

Figure 5: Within-question decodability predicts the fusion gain. (a) The empirical law: across 15 medical settings (filled, colored by decodability region) the within-question AUC predicts the CASE-minus-voting gain (Pearson

r=0.96

; fitted line), with a clean threshold at

\mathrm{AUC}\approx 0.60

; open diamonds are out-of-sample GPQA settings (Section 6.7 ), not used in the fit. Error bars are 95% confidence intervals (horizontal: question-bootstrap of the within-question AUC; vertical: bootstrap of the gain); the AUC is markedly tighter than the gain, consistent with it being the more stable indicator. (b) The stylized model of Section 4 : predicted selection accuracy rises with decodability from the pool accuracy

A

toward

1

(Proposition 1) and overtakes the voting level

V

at a threshold above

\tfrac{1}{2}

(Proposition 2); the three curves are three correct-fractions

A

. (c) Out-of-sample validation: decodability measured on a calibration half predicts the deployment-half gain on held-out questions (

r=0.75

; 13 settings, 10 splits each, mean

\pm

s.d.).

$N$	single%	majority%	CASE%	CASE $-$ maj
1	20.4	20.4	20.4	$+0.0$
2	20.7	20.7	25.2	$+4.5$
4	20.4	14.9	33.8	$+18.9$
8	20.7	10.0	42.1	$+32.1$
16	20.8	6.0	47.3	$+41.3$

Table 4: Byzantine

N

-scaling (OpenBioLLM

\times

MedQA, minority-correct subset, 131 questions). Real draws without replacement from the candidate pool. CASE

-

maj is computed from unrounded accuracies and can differ from the displayed columns by

\pm 0.1

pp.

Figure 6: Byzantine

N

-scaling. (a) On the minority-correct subset, as the candidate count

N

grows CASE climbs from

20.4

47.3\%

while majority voting collapses from

20.4

6.0\%

. (b) Resolved by difficulty, the CASE-minus-vote gain diverges on the minority-correct and hard bins (

+41.3

and

+16.8

pp at

N=16

) but reverses on the medium bin (

-2.0

), where voting recovers—the advantage is specific to the Byzantine hard regime. (c) The stylized model (Proposition 3,

a=0.70

) reproduces the divergence: predicted CASE accuracy rises with

N

while voting falls.

Figure 7: Answer-token hidden states (OpenBioLLM-8B, MedQA, layer 28) projected onto the correctness axis (mass-mean direction). (a) Marginal density of correct (green) vs. incorrect (red) answers; (b) the same states in 2-D, where a

\sim

19% cluster of incorrect answers separates cleanly on the far incorrect side—candidates the gate confidently and correctly scores as wrong, a recognizable error mode—while the harder remaining errors overlap the correct mass on the right. The pooled single-axis projection separates the two classes at AUC

=0.73

; this is the pooled projection AUC at the fixed layer 28, distinct from the within-question operating-layer decodability AUC (0.702) reported in Tables 1 and 3 .

Figure 8: Anatomy of the internal signal. (a) Within-question decodability emerges with scale for general models and is saturated by medical specialization. (b) The selection gain concentrates in the Byzantine (medium/hard) regime. (c) Adding the correctness direction during generation does not improve accuracy—the readout is not a causal lever.

Method	accuracy	vs. vote	extra cost / question
single agent	46.5%	—	—
majority vote	51.8%	—	0
generative verifier	63.0%	$+11.1$	$\sim\|\text{options}\|$ fwd passes (cached; 4 on MedQA)
CASE	62.0%	$+10.2$	$\approx 0$ (reuse activations $+$ 1 matvec)

Table 5: Generative verifier vs. CASE (OpenBioLLM

\times

MedQA, medium difficulty). All four estimates come from the same diagnostic candidate pool and the same draws (best-of-

N

N=4

, 200 draws), so the majority-vote baseline (51.8%) is shared. Gains over voting are computed from unrounded accuracies and can differ from the displayed percentages by

\pm 0.1

pp (the verifier’s

+11.1

vs.

63.0-51.8

). The operating point differs from the medium cell of Table 1 (full-pool selection), so the CASE accuracy there, 63.2%, differs slightly within sampling uncertainty. To quantify the cost gap: the verifier’s

|\text{options}|

extra forward passes cost on the order of

|\text{options}|\cdot 2N_{\text{params}}T\approx 2\times 10^{13}

FLOP per question (8B model,

T\approx 300

prompt tokens), whereas CASE adds one

d{=}4096

matrix–vector product per candidate on activations already produced during generation,

\approx 10^{5}

FLOP per question—about eight orders of magnitude less marginal compute, though both share the upstream candidate-generation cost. This is an estimate from parameter and token counts, not a wall-clock benchmark.

Model	Benchmark	within-Q AUC	medium CASE $-$ vote ( $p$ )	hard CASE $-$ vote ( $p$ )
Qwen2.5-Math-7B	MATH-500	0.789	$+8.1$ (.043)	$+2.5$ ( $<$ .001)
Qwen2.5-7B	MATH-500	0.763	$+6.6$ (n.s.)	$+2.6$ ( $<$ .001)
Qwen2.5-Math-7B	GSM8K	$0.820^{\dagger}$	—	$+11.9$ (.001)
Qwen2.5-7B	GSM8K	$0.595^{\dagger}$	—	$+0.3$ (n.s.)

Table 6: Mathematics domains, at the operating layer as for the medical settings;

p

from the same question-level bootstrap. For MATH-500 the AUC is the medium-bin within-question AUC (the law’s axis, Eq. 11 ), with medium/hard

n=17/233

(Qwen2.5-7B) and

27/236

(Qwen2.5-Math-7B). GSM8K is too ceiling-limited for the medium-bin axis (only 7 and 4 medium questions), so it is reported off-axis: pooled within-question AUC (

\dagger

) and hard-bin gain over

n=15

and

18

. n.s.

=

not significant.

			medium			hard
Model	Subfield	AUC	vote%	CASE%	$\Delta$ vote	vote%	CASE%	$\Delta$ vote ( $p$ )
Qwen2.5-14B	Phys	0.681	50.3	61.1	$+10.8$ (n.s.)	3.8	11.6	$+7.9$ ( $<$ .001)
Qwen2.5-7B	Phys	0.661	54.0	58.2	$+4.3$ (n.s.)	4.6	12.0	$+7.4$ ( $<$ .001)
Qwen2.5-14B	Chem	0.576	49.9	49.6	$-0.3$ (n.s.)	6.3	12.6	$+6.2$ ( $<$ .001)
Qwen2.5-7B	Chem	0.550	54.9	47.4	$-7.6$ (n.s.)	6.5	14.2	$+7.7$ ( $<$ .001)

Table 7: A non-medical knowledge domain (GPQA, graduate science; general models, late-layer operating point). Decodability spans the threshold (physics above, chemistry just below), and hidden-state selection beats voting on the hard ( Byzantine ) subset in every setting. The hard subset is well-powered (hard-bin

n=133

–

136

; Qwen-14B physics on its complete 187-question set), whereas the medium bins are small (

n=19

–

30

), so the medium gains are mostly not significant (n.s.

=

not significant).

Model / benchmark	within-Q AUC	self-cert	seq-logp	pred-ent	CASE advantage
OpenBioLLM-8B MedMCQA	0.770	$+4.9$ n.s.	$+13.3$	$+13.3$	$+3.7$
OpenBioLLM-8B MedQA	0.759	$+0.9$ n.s.	$+14.6$	$+13.4$	$+9.7$
OpenBioLLM-8B PubMedQA-cb	0.707	$-22.1$	$-4.8$ n.s.	$-0.1$ n.s.	$+23.2$
Qwen2.5-7B MedQA	0.574	$-3.3$ n.s.	$-3.8$ n.s.	$-3.2$ n.s.	$+4.6$
meditron-7B MedQA	0.531	$-14.9$	$-14.3$	$-14.4$	$+7.3$

Table 8: CASE vs. near-free output-space selectors at medium difficulty, on a shared candidate pool. The self-cert / seq-logp / pred-ent columns give each selector’s gain over majority voting (pp); CASE advantage is CASE minus the best of the three—a paired comparison, positive whenever CASE beats every near-free selector. within-Q AUC is the pooled within-question ROC-AUC of the CASE gate (over all difficulties on this single seed-7 pool, so it differs slightly from the operating-layer AUC in Tables 1 and 3 ). n.s.

=

not significant at

\alpha=0.05

. Selector gains are over majority voting on a shared seed-7 candidate pool (operating layers L22–L27).

Figure 9: Deployment nomogram. A practitioner measures the within-question decodability on a small calibration set, locates the AUC on the horizontal axis, and reads the expected CASE

-

vote gain and its

\pm 6

pp band off the law line. The vertical bands are the decision zones: below

\mathrm{AUC}\,0.55

retain voting; in the indeterminate band (

0.55

–

0.65

) the measurement cannot decide; above

0.65

deploy CASE. Open circles are the medium-difficulty gains the law is fit on (the mechanistic quantity); filled points are the difficulty-unconditional gains a deployer actually experiences (smaller, but positive exactly in the deploy zone).

Language Model Decoding

Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.

See all 168 papers

On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode
Mar 14, 2026Valeria Ruscio, Keiran ThompsonHallucination in Language Models LLM Interpretability
Intuition Machines Inc.
Overwhelmed by Choice: Studying LLM Decision Making at Scale
Sep 26, 2026Yu-Chi Lin, Aryan Seth, Anshul Aravind +4LLM Evaluation LLM Decision-Making
University of California, Los Angeles · University of Cincinnati
From Discovery to Decision: Finite-Budget Recoverability in LLM Voting
Oct 1, 2026Shaoang Li, Jian LiLLM Inference Efficiency LLM Inference
Stony Brook University

Mar 14, 2026J/K move · Enter open · S save

On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode

Valeria Ruscio, Keiran Thompson

Intuition Machines Inc.

Hallucination in Language Models LLM Interpretability Factual Knowledge in Language Models Causal Interventions in Language Models

Open paper AskPDF arXiv

A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decodability and selection, we distinguish read from write at the first answer token. Read asks whether the gold token can be decoded from intermediate residual states under same-relation decoy controls. Write asks whether the final readout ranks that token first among content tokens. Under three different readers, with a randomized-label control, a substantial fraction of failures remain readable while another content token is selected. We explain this through the selection margin at the final readout, the difference between the answer logit and the logit of its strongest alternative, which is answer support minus alternative support, and can also be split into a context-averaged baseline linked to token frequency and an item-specific term. Setting the answer support to the level typical of successful generations is sufficient to recover first-token selection for the majority of failures in most of the models we study; the original alternative remains ahead in most remaining failures under this edit, and this outcome follows directly from the readout geometry. Removing the frequency direction alone shifts selection but rarely recovers the answer. Prompt variants of the same fact that succeed supply support that transfers to failing variants through the residual stream and through late MLP outputs, with less consistent effects through late attention. First-token recovery leaves most full answers wrong, which limits the recovery achieved by these edits and separates three things that are easily conflated, decodability, recoverability, and generation.

Abstract

Language Model Decoding

Explore similar work

On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode

Overwhelmed by Choice: Studying LLM Decision Making at Scale

From Discovery to Decision: Finite-Budget Recoverability in LLM Voting