From Discovery to Decision: Finite-Budget Recoverability in LLM Voting

Shaoang Li; Jian Li

cs.AIOct 1, 2026

From Discovery to Decision: Finite-Budget Recoverability in LLM Voting

Authors: Shaoang Li, Jian Li

Organizations: Stony Brook University

Abstract

Voting over multiple LLM responses is a common primitive in test-time scaling and ensemble inference. Collecting more responses can expand the candidate pool and increase the chance that a correct answer is discovered. Under a fixed call budget, a discovered answer still needs to accumulate enough support within the remaining calls to become the final plurality winner, creating a discovery-to-decision gap. In this work, we characterize this gap through the realized vote state and remaining call budget. We derive a sharp recoverability threshold and show that, as sampling proceeds, the observed candidate set can only expand while the set of reachable endpoint winners can only contract, inducing a candidate-level conversion window. Under a specified iid response law, the same state yields exact finite-horizon endpoint probabilities. We further show that merging wrong-answer identities preserves single-call correctness and cannot improve plurality accuracy, and that the effect of redistributing wrong-answer probability depends on the realized vote state. Singleton reachability yields a gold-free exact locking certificate. For a known answer universe, its first trigger is the earliest prefix at which all admissible continuations yield the same fixed-budget output. Empirically, most discovered-but-unselected correct answers lose reachability only after discovery. In a controlled Word16 study, input permutation improves raw-plurality accuracy by 21.1 points with essentially unchanged single-call correctness. Exact locking saves 28-30% of calls at a 16-call budget while preserving every fixed-budget output.

Figures & tables

Figure 1: Three 16-call trajectories (Qwen3.5-2B on forced-choice MMLU-Pro, chosen retrospectively with gold labels). Circles mark discovery, thin gray lines show other wrong answers, and dashed lines mark decision locking. In (b), the correct answer is unreachable from call 10, but H can still overtake B, so the output locks only at call 12.

Figure 2: Where covered correct answers are lost. (a) Covered failures (the correct answer is sampled but not selected at the endpoint) by class, as a percentage of each cohort’s covered failures. The labels at right give covered failures out of all trajectories (counts in Table 5 ). (b) The 553 forced-choice MMLU-Pro covered failures by the correct answer’s state after each call.

Figure 3: Forecasting from the realized state. (a) GSM endpoint calibration in the covered-but-unselected call-8 stratum (

n=490

, laws from 16 calls): observed-minus-forecast error by pre-specified forecast range, with family-bootstrap 95% intervals. The pooled analysis includes an adaptively added second wave ( Table 9 ). (b) Prospective Permuted-minus-Base effect forecasts against observed effects on 128 MMLU-Pro inputs (laws from 256 calls per arm), by pre-specified bin, with item-bootstrap 95% intervals ( Section 4 ).

Figure 4: Generator–selector interaction for Qwen3.5-4B on Word16, 16 responses per policy: (a) 256 pre-specified test items, (b) 384 fresh inputs. Numbers give the Permuted-minus-Base gain (points). Bars are 95% item-bootstrap intervals. Paired intervals and cells: Section G.2 for (a), Section G.4 for (b).

Figure 5: Lock-time law. (a) GSM-Symbolic,

B=16

: first certified prefix predicted from an independently estimated law and observed (4,000 trajectories). (b) Public GSM8K, Llama-3-8B/70B-Instruct (127 problems): calls saved, observed (lines) and predicted (crosses), with the averages of

\hat{m}/(1+\hat{m})

(dotted). (c) Per problem at

B=1024

: observed

\tau_{B}/B

against

1/(1+\hat{m})

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Exact response-law diagnostics. (a) Splitting equal-mass wrong identities preserves single-call correctness and coverage at each budget and improves plurality. Here

p_{\star}=0.3

and

p_{\bot}=0.1

. The dashed line shows coverage at 16 calls. (b) Arbitrary laws need not obey a scalar collision order. Laws A and B below both have

p_{\star}=0.3

and no abstention, yet B has lower collision and lower accuracy. Gold loses every count tie in both panels.

Study	Purpose	Data	Scale
Forced-choice MMLU-Pro	Illustrate recovery trajectories and conversion-window closure	200 items, five models, two sequences	2,000 trajectories
GSM-Symbolic	Prospective endpoint forecasting from a realized call-8 state	250 items in 100 template families, five models	5,000 forecast trajectories
Native generation	Test recovery dynamics under direct answer generation	1,200 MMLU-Pro and 200 Word16 items, Qwen3.5-2B/4B	2,800 trajectories
Diversification	Test representation diversity and selector interaction	256 held-out Word16 and 256 held-out MMLU-Pro items per task	16 calls per policy
Physical stopping	Measure request-time consequences of exact locking	32 MMLU-Pro items per checkpoint	batch sizes 1 and 4
Judge selector	Select after plurality’s window has closed	Call-12 states of 200 MMLU-Pro items, five pools	2,000 states, 268 vote-infeasible

Appendix

Table 1: Empirical studies and their role in the paper. The cohorts are analyzed separately.

	Forced-choice MMLU-Pro a	GSM-Symbolic	Native b	Thinking mode	Public GSM8K, MATH
Models	Qwen3.5-0.8B/2B/4B/27B, Gemma-4-E2B, Ministral-3-3B	the five 0.8B–4B checkpoints	Qwen3.5-2B/4B	Qwen3-14B/8B	Llama-3-8B/70B-Instruct
Prompting	chat, nonthinking, $\leq$ 40-word rationale	native chat, nonthinking	native chat, nonthinking	native chat, thinking	few-shot completion (5 GSM8K, 4 MATH)
Temperature	0.7	0.7	0.7 (High temp. 1.0)	0.6	0.6
top- $p$ / top- $k$	0.9 / model default	0.9 / 20	0.9 / 20	0.95 / 20	none / none
Repetition penalty	1.05	1.05	1.05	none	none
Max new tokens	96	1,024	4,096 c	16,384	512

Appendix

Table 2: Generation settings by protocol. In-house generation uses BF16 weights with Hugging Face Transformers, except the thinking-mode study, which uses vLLM 0.13.0. Every invalid call consumes budget and adds no vote.

Conversion-window outcome	Count	Share of discovered (%)
Infeasible at discovery	74	5.3
Closes later, before final call	433	31.3
Reachable through call 15, endpoint failure	46	3.3
Open through endpoint, gold selected	831	60.0

Appendix

Table 3: Timing of the gold candidate’s conversion window in forced-choice MMLU-Pro. The denominator is the 1,384 trajectories in which gold is discovered.

Figure 7: Forced-choice MMLU-Pro coverage and current selection across all 2,000 trajectories, with pointwise 95% item-bootstrap intervals. The per-call states of the 553 covered failures appear in Figure 2 b.

Model	Task	Converts from call 8	Blocked / eligible at 12	Late feasible	Late converts
2B	MMLU-Pro	$16.9\%$ (34/201)	$105/239$	$50/82$	$13/82$
4B	MMLU-Pro	$11.9\%$ (13/109)	$54/131$	$16/33$	$3/33$
2B	Word16	$59.7\%$ (37/62)	$0/59$	$20/21$	$8/21$
4B	Word16	$22.6\%$ (14/62)	$15/89$	$17/21$	$3/21$

Appendix

Table 4: State-conditioned recovery in the native-generation cohort. Eligible prefixes are covered but unselected. Blocked means

L(v_{12})>4

, and late means first discovery at calls 9–16.

Cohort	Trajectories	Coverage	Accuracy	I	F	L
Forced-choice MMLU-Pro	2,000	69.20	41.55	74	287	192
GSM-Symbolic	6,250	89.92	78.82	68	359	267
Native MMLU-Pro, 2B	1,200	83.08	61.75	32	131	93
Native MMLU-Pro, 4B	1,200	88.42	76.83	17	72	50
Native Word16, 2B	200	92.50	70.00	1	27	17
Native Word16, 4B	200	92.00	50.00	4	38	42

Appendix

Table 5: Endpoint coverage, accuracy, and unified failure counts. I: infeasible at discovery. F: feasible at discovery but never selected. L: selected and later lost. GSM-Symbolic includes the law-estimation sequences, and only 5,000 of its trajectories enter forecast evaluation. All rows use

B=16

. Public rows use trajectories from samples 2,001–10,000 of 127 GSM8K and 128 MATH problems. Invalid calls, as a share of all calls, are 0.0% for the forced-choice protocol (five checkpoints and 27B), 11.2% for GSM-Symbolic, 16.6% and 7.5% for native MMLU-Pro (2B, 4B), 9.2% and 0.1% for native Word16, 9.1% for fresh MMLU-Pro, 0.1% and 0.0% for public GSM8K and 6.9% and 6.8% for public MATH (8B, 70B).

Figure 8: Coverage and plurality accuracy against the budget on public samples from Llama-3-8B- and 70B-Instruct, for 127 GSM8K and 128 MATH problems. Trajectories are disjoint blocks of samples 2,001–10,000, from 63,500 or 64,000 trajectories at

B=16

to 889 or 896 at

B=1{,}024

. The shaded band is the share of trajectories that sample the correct answer without selecting it, and the numbers give the gap at

B=1{,}024

where it is at least 5 points (it is 3.0 points for 70B on GSM8K).

Task, model	$B$	Traj.	Coverage	Accuracy	Gap [95% CI]	Failures	Recoverable	Closed	Late converts
GSM8K, 8B	16	63,500	98.0	86.0	11.9 $[7.5,16.7]$	12.2	92.4	84.9	66/1,284
	64	15,875	99.3	87.0	12.3 $[7.3,17.8]$	12.4	98.3	95.6	0/75
	256	3,937	99.8	87.6	12.2 $[7.0,18.0]$	12.2	99.2	98.8	0/5
	1024	889	100.0	88.3	11.7 $[6.4,17.4]$	11.7	100.0	100.0	0/0
GSM8K, 70B	16	63,500	99.2	96.8	2.4 $[0.2,5.1]$	2.4	93.9	93.3	1/108
	1024	889	99.9	96.9	3.0 $[0.7,6.2]$	3.0	85.2	100.0	0/4

Appendix

Table 6: The discovery-to-decision gap at larger budgets on public samples (127 GSM8K and 128 MATH problems). Gap is coverage minus plurality accuracy, in points with problem-cluster bootstrap 95% intervals. Failures are covered failures as a share of covered trajectories. Recoverable is the share of covered failures still recoverable when gold first appeared. Closed is the share of covered failures whose correct answer became unreachable before the final call. Late converts counts late discoveries (

t>B/2

) selected at the endpoint, over all late discoveries.

Judge	States	Judge	Direct	Difference [95% CI]
Own pool, pooled	268	34.23	14.83	$+19.40$ $[+12.39,+26.84]$
Qwen3.5-0.8B	70	27.68	10.27	$+17.41$ $[+6.36,+30.05]$
Qwen3.5-2B	55	39.13	12.23	$+26.90$ $[+13.33,+41.41]$
Qwen3.5-4B	40	27.42	21.37	$+6.05$ $[-10.23,+23.57]$
Gemma-4-E2B	42	40.62	12.50	$+28.12$ $[+12.50,+43.75]$
Ministral-3-3B	61	35.56	14.44	$+21.11$ $[+7.77,+34.03]$

Appendix

Table 7: Judge selection in the 268 vote-infeasible call-12 states. Rates (%) and differences (points) use equal item weights with joint item-bootstrap 95% intervals. Each smaller model judging its own pool is primary. Per-model rows are unadjusted, and 27B judging the same states is secondary.

Stratum: comparator	States	Own-pool judges	Qwen3.5-27B
Covered, feasible: plurality at 16	248	$+22.31$ $[+13.60,+30.88]$	$+67.26$ $[+58.60,+74.83]$
Currently correct: plurality at 13	819	$-26.18$ $[-31.43,-21.18]$	$-13.93$ $[-19.42,-8.90]$
Gold absent: own direct solving	665	$+0.85$ $[-1.13,+3.18]$	$-0.62$ $[-4.02,+2.73]$
All states: plurality at 16	2,000	$+0.05$ $[-2.35,+2.50]$	$+32.50$ $[+28.40,+37.05]$

Appendix

Table 8: Judge selection in the other strata (descriptive). Differences are judge minus the stated comparator, in points, with equal item weights and joint item-bootstrap 95% intervals. The call-13 and call-16 comparators use one and four further plurality calls. Judging adds rationale context and a scoring pass, so these comparisons are not cost-matched.

Population	$n$	Observed	Forecast	Difference
Wave 1	235	0.289	0.311	$-2.1$
Wave 2	255	0.271	0.259	$+1.2$
Pooled	490	0.280	0.284	$-0.4$
Forecast $[0,0.05)$	189	0.063	0.008	$+5.5$
Forecast $[0.05,0.2)$	83	0.169	0.121	$+4.8$
Forecast $[0.2,0.5)$	93	0.333	0.350	$-1.7$

Appendix

Table 9: GSM-Symbolic calibration in the covered-but-unselected call-8 stratum. Differences are observed minus forecast, in percentage points. The pooled row combines the two evaluation waves, and the range rows show pooled calibration by forecast magnitude.

Comparator	Population	Difference	95% interval
Discovery-bin constant	covered, unselected	$-0.050$	$[-0.078,-0.015]$
Law-only	all forecasts	$-0.023$	$[-0.028,-0.019]$
Persistence	all forecasts	$-0.011$	$[-0.017,-0.005]$
Law-only	covered, unselected	$-0.032$	$[-0.051,-0.014]$
Persistence	covered, unselected	$-0.135$	$[-0.193,-0.083]$
MMLU-Pro-fitted logistic	all forecasts	$+0.001$	$[-0.001,+0.004]$

Appendix

Table 10: Wave-1 Brier-score contrasts for the state-conditioned plug-in forecast. Differences are forecast minus comparator, so negative values favor the state-conditioned forecast.

Data	Law	States	Observed	Forecast	Difference	$[0,0.05)$	$[0.5,1]$ (n)	AUC	Absent
GSM-Symbolic	16 calls (prospective)	490	0.280	0.284	$-0.4$ $[-3.6,+3.1]$	$+5.5$	$-11.9$ (125)	0.825	55
	64 calls	490	0.280	0.293	$-1.3$ $[-3.9,+1.4]$	$+3.0$	$-3.4$ (128)	0.871	11
GSM8K, 8B	16	6,841	0.215	0.245	$-3.0$ $[-7.8,+1.6]$	$+0.7$	$-13.5$ (1,492)	0.865	141
	2,000	6,841	0.215	0.219	$-0.3$ $[-1.2,+0.6]$	$-0.3$	$+0.6$ (1,145)	0.907	0
GSM8K, 70B	16	1,543	0.180	0.177	$+0.3$ $[-3.6,+6.4]$	$+1.4$	$+5.8$ (280)	0.948	4
	2,000	1,543	0.180	0.179	$0.0$ $[-1.6,+1.3]$	$+0.1$	$+1.0$ (293)	0.970	0

Appendix

Table 11: Forecast calibration in the covered-but-unselected call-8 stratum by law sample size. Differences are observed minus forecast, in points, with cluster-bootstrap 95% intervals over template families or problems. The last range is

[0.5,1]

, shown with its number of states. Absent counts the states whose correct answer never occurs in the law sample.

Figure 9: Observed minus forecast endpoint rate by forecast range in the covered-but-unselected call-8 stratum of public

B=16

trajectories, with laws from 16 (gray) or 2,000 (blue) disjoint samples of the same problem. Bars are problem-cluster bootstrap 95% intervals. The 70B GSM8K range

[0.2,0.5)

contains 19 states.

Forecast	Squared-error gain	Endpoint Brier	Effect squared error
Full state-based	—	0.0146 $[0.0079,0.0222]$	0.00092 $[0.00032,0.00170]$
Uniform wrong mass	0.0058 $[0.0010,0.0134]$	0.0212 $[0.0109,0.0334]$	0.00670 $[0.00198,0.01330]$
Wrong-identity shuffle	0.0030 $[0.0004,0.0068]$	0.0189 $[0.0100,0.0293]$	0.00390 $[0.00134,0.00730]$

Appendix

Table 12: Effect forecasts on 128 inputs and 256 states. The squared-error gain is control minus full loss (positive favors the full forecast) with the pre-specified 97.5% intervals. Brier scores over individual continuation outcomes and the squared error to the noisy observed effect are secondary, with 95% intervals.

Forecast bin	Inputs / states	Mean forecast	Mean observed effect
$[-1,-0.1)$	4 / 6	$-0.349$ $[-0.545,-0.165]$	$-0.391$ $[-0.585,-0.219]$
$[-0.1,-0.02)$	9 / 11	$-0.053$ $[-0.066,-0.040]$	$-0.060$ $[-0.096,-0.028]$
$[-0.02,0.02)$	117 / 224	$0.000$ $[-0.001,0.000]$	$0.000$ $[-0.001,0.001]$
$[0.02,0.1)$	3 / 3	$0.037$ $[0.029,0.051]$	$0.021$ $[0.000,0.031]$
$[0.1,1]$	9 / 12	$0.370$ $[0.221,0.495]$	$0.362$ $[0.250,0.482]$

Appendix

Table 13: Effect-forecast bins, fixed before validation, with states weighted equally and item-bootstrap 95% intervals. An input can contribute to two bins.

Direction	$\alpha$	Predicted	Observed
$a\to b$	0.25	$+0.47$	$+0.39$ $[+0.16,+0.67]$
$a\to b$	0.5	$+1.03$	$+1.06$ $[+0.46,+1.82]$
$b\to a$	0.25	$-0.06$	$-0.05$ $[-0.11,0.00]$
$b\to a$	0.5	$-0.12$	$-0.07$ $[-0.16,-0.01]$

Appendix

Table 14: Oracle recoding of two wrong identities on the independent Base continuations: pre-computed finite-difference predictions and observed endpoint changes, in points, with item-bootstrap 95% intervals.

Comparison	Effect	95% CI	Discordant $+/-$	98.33% CI	Holm-adjusted $p$	Decision
4B Base minus 2B Base	$-8.20$	$[-16.02,-0.39]$	$43/64$	$[-17.58,1.17]$	$0.105$	inconclusive
4B Permuted minus Base	$+21.09$	$[14.06,28.12]$	$74/20$	$[12.50,29.69]$	$5.54\times 10^{-8}$	supported
4B Probe minus High temp.	$-3.52$	$[-9.77,2.73]$	$31/40$	$[-11.33,4.30]$	$0.342$	inconclusive

Appendix

Table 15: Pre-specified Word16 primary comparisons on 256 paired test items. Formal decisions use Holm-adjusted exact paired tests over the three-comparison family.

Figure 10: All-item wrong collision on the 256 Word16 test items in each generation arm. Permutation substantially reduces repeated wrong identities while leaving single-call correctness nearly unchanged in the 4B comparison.

Generation	Raw plurality	Multiset-filtered	Borda fusion
Base	57.42	89.45	82.81
Permuted	78.52	95.31	85.55
High temp.	74.22	94.53	90.23

Appendix

Table 16: Selector dependence on the same 256 4B Word16 test items (accuracy, %).

Task	Generation	Raw	Filtered	Borda	Single call	Coverage	Invalid
Word16	Base	59.64	89.06	82.55	33.45	95.31	0.07
	Permuted	74.22	92.19	85.42	32.28	98.70	0.10
	High temp.	76.56	92.45	85.94	36.49	97.92	0.10
MMLU-Pro	Base	73.70	73.70	—	68.72	84.90	7.24
	Permuted	76.30	76.30	—	69.51	89.06	6.43
	High temp.	73.96	73.96	—	68.46	86.46	7.47

Appendix

Table 17: Fresh native cohort, 384 test inputs per task and sixteen calls per arm (percentages). Coverage is unchanged by filtering. Invalid calls include truncation and parse failures, and every call consumes budget. Borda fusion applies to Word16 only.

Figure 11: Mean recovery threshold of the correct answer on the 384 fresh Word16 inputs after 4, 8, 12 and 16 calls (cap 16), under raw plurality (a) and multiset filtering (b), for each generation arm. The thresholds use reference answers offline.

Arm	Valid	Truncated	Generation failure	Invalid letter	No final box
Base, $T=0.7$	90.87	8.06	1.04	0.02	0.01
Permuted, $T=0.7$	90.97	7.97	1.04	0.01	0.01
Base, $T=1.0$	90.35	8.58	1.04	0.00	0.03
Permuted, $T=1.0$	90.37	8.55	1.04	0.00	0.03

Appendix

Table 18: Status of the 12,288 calls per arm (percent). Truncated calls reached the 4,096-token cap and abstain.

Contrast	Base	Permuted	Effect (points)	95% interval	90% interval	Classification
	Accuracy (%)
At $T=0.7$	74.74	75.00	$+0.26$	$[-1.56,2.08]$	$[-1.30,1.69]$	practically equivalent
At $T=1.0$	74.48	73.96	$-0.52$	$[-2.47,1.43]$	$[-2.08,1.04]$	inconclusive
Interaction			$-0.78$	$[-3.09,1.52]$	$[-2.72,1.16]$	inconclusive

Appendix

Table 19: Fresh MMLU-Pro hold-out (768 inputs, Qwen3.5-4B, sixteen calls per cell). Accuracy is 16-call plurality correctness. Intervals are category-stratified item-bootstrap intervals, studentized for the interaction. Practical equivalence is a descriptive, unadjusted classification, and all Holm-adjusted

p

-values equal

1

Quantity	Base	Permuted	Base	Permuted
	$T=0.7$		$T=1.0$
All-call wrong collision	0.122	0.109	0.113	0.106
Collision among errors	0.693	0.634	0.676	0.599
Distinct wrong answers per input	0.79	0.95	0.84	1.00
Coverage at 16 calls (%)	86.9	88.0	85.9	87.2
Single-call correctness (%)	68.7	67.8	67.6	66.9

Appendix

Table 20: Secondary quantities on the fresh MMLU-Pro hold-out (descriptive). Collision among errors uses the inputs with at least two wrong votes, which are Base and Permuted subsets of 276 and 296 inputs at

T=0.7

and of 290 and 309 inputs at

T=1.0

. The certificate saving is the share of calls omitted in replay at

B=16

Figure 12: Distribution of the first certified prefix under the exact locking rule, split by whether the fixed-budget endpoint is correct. Most trajectories are certified before the full 16-call budget. The bar at 16 counts trajectories not certified before the budget ends.

Metric	MMLU-Pro ( $2{,}000$ )	GSM ( $5{,}000$ )
Locked by call 9	$33.85\%$	$47.10\%$
Locked by call 12	$66.45\%$	$70.06\%$
Not locked before call 16	$8.85\%$	$12.14\%$
First lock, quartiles	$9/11/14$	$9/10/13$
Mean calls consumed	$11.46$	$11.16$
Calls saved	$28.35\%$	$30.26\%$

Appendix

Table 21: Exact stopping replay with total budget 16. Output preservation follows from the certificate and is also verified on the stored trajectories.

Cohort	$B$	Trajectory $\leftarrow$ law	Checkpoints	Traj.	Obs.	Pred.	Difference [95% CI]	KS (bound)	MAD
GSM-Symbolic	16	$1$ – $4\leftarrow 0$	five a	4,000	32.64	32.48	$+0.16$ $[-0.17,+0.47]$	0.012 (0.022)	0.067
GSM-Symbolic	32	$1\\|2\leftarrow 0,3,4$	three	750	39.06	39.01	$+0.04$ $[-0.30,+0.39]$	0.010 (0.028)	0.040
GSM-Symbolic	32	$3\\|4\leftarrow 0,1,2$	three	750	39.06	39.04	$+0.02$ $[-0.32,+0.33]$	0.011 (0.029)	0.039
GSM-Symbolic	48	$1\\|2\\|3\leftarrow 0,4$	three	750	40.02	39.83	$+0.19$ $[-0.11,+0.49]$	0.013 (0.031)	0.033
GSM-Symbolic	64	$1\\|2\\|3\\|4\leftarrow 0$	three	750	40.32	40.20	$+0.12$ $[-0.28,+0.51]$	0.018 (0.044)	0.034
MMLU-Pro	16	seq. 0 $\leftarrow$ seq. 1	five	1,000	28.15	28.81	$-0.66$ $[-1.25,-0.03]$	0.039 (0.057)	0.092

Appendix

Table 22: Finite-budget validation (raw plug-in law from calls disjoint from the trajectory). Savings are percentages of the budget. Differences are observed minus predicted, in points, with cluster-bootstrap 95% intervals. KS is the Kolmogorov distance between predicted and observed lock-time distributions with its 97.5% bound. MAD is the mean absolute deviation of per-item

\tau_{B}/B

from

1/(1+\hat{m})

. The three checkpoints are Qwen3.5-4B, Gemma-4-E2B and Ministral-3-3B.

Figure 13: Cumulative distribution of the first certified prefix, predicted from an independently estimated law (gray) and observed (blue), for the seven cells of Table 22 , in table order. Dotted lines mark

B/2

, below which no prefix can be certified.

Model	$B$	Traj.	Obs.	Pred.	Difference [95% CI]	Asymptote	MAD [95% CI]	KS (bound)	Var. obs./CLT
Llama-3-8B	16	63,500	34.90	34.85	$+0.05$ $[-0.03,+0.14]$	39.53	0.0565 $[0.0533,0.0597]$	0.004 (0.008)	1.5 / 2.4
	32	31,750	37.34	37.29	$+0.05$ $[-0.03,+0.13]$		0.0297 $[0.0275,0.0319]$	0.007 (0.012)	3.4 / 4.7
	64	15,875	38.55	38.48	$+0.07$ $[-0.02,+0.16]$		0.0155 $[0.0138,0.0174]$	0.006 (0.013)	7.5 / 9.5
	128	7,874	39.10	39.03	$+0.07$ $[-0.02,+0.17]$		0.0087 $[0.0074,0.0102]$	0.006 (0.014)	16.5 / 18.9
	256	3,937	39.38	39.29	$+0.09$ $[-0.01,+0.19]$		0.0056 $[0.0045,0.0069]$	0.012 (0.024)	36.1 / 37.8
	512	1,905	39.48	39.42	$+0.05$ $[-0.04,+0.15]$		0.0042 $[0.0033,0.0053]$	0.010 (0.024)	68.0 / 75.7

Appendix

Table 23: Large-budget validation on public GSM8K (127 problems). Trajectories are disjoint blocks of samples 2,001–10,000, and the law comes from samples 1–2,000. Savings are percentages of the budget, and the asymptote is the population average of

\hat{m}/(1+\hat{m})

. MAD is the mean absolute deviation of per-problem

\tau_{B}/B

from

1/(1+\hat{m})

. KS is the Kolmogorov distance between predicted and observed lock-time distributions with its 97.5% bootstrap bound. The last column compares the mean observed within-problem variance of

\tau_{B}

with

B\hat{\sigma}^{2}/(1+\hat{m})^{3}

on problems with a confidently separated runner-up (69 for 8B, 27 for 70B).

Figure 14: Per problem on public GSM8K: observed

\tau_{B}/B

(mean over the problem’s trajectories) against

1/(1+\hat{m})

B=16

128

and

1024

, for Llama-3-8B-Instruct (top) and Llama-3-70B-Instruct (bottom). Table 23 reports all seven budgets. Deviations from the asymptote shrink as the budget grows.

Model	$B$	Traj.	Obs.	Pred.	Difference [95% CI]	Asymptote	MAD [95% CI]	KS (bound)	Var. obs./CLT
Llama-3-8B	16	64,000	13.28	13.19	$+0.09$ $[-0.02,+0.21]$	15.75	0.0419 $[0.0375,0.0464]$	0.006 (0.010)	1.4 / 3.3
	64	16,000	15.17	15.05	$+0.13$ $[-0.03,+0.28]$		0.0178 $[0.0159,0.0198]$	0.009 (0.018)	9.3 / 13.1
	256	3,968	15.70	15.57	$+0.12$ $[-0.07,+0.30]$		0.0100 $[0.0086,0.0116]$	0.012 (0.029)	46.5 / 52.3
	1024	896	15.82	15.72	$+0.10$ $[-0.09,+0.30]$		0.0088 $[0.0074,0.0101]$	0.017 (0.043)	203.1 / 209.3
Llama-3-70B	16	64,000	22.97	22.85	$+0.12$ $[+0.04,+0.21]$	26.25	0.0462 $[0.0417,0.0506]$	0.004 (0.008)	1.5 / 3.1
	64	16,000	25.60	25.51	$+0.09$ $[-0.02,+0.20]$		0.0154 $[0.0135,0.0176]$	0.005 (0.013)	8.7 / 12.3

Appendix

Table 24: Large-budget validation on public MATH for 128 problems, with laws from samples 1–2,000, disjoint trajectories from samples 2,001–10,000 and the conventions of Table 23 . Mean

\hat{m}

is 0.24 for 8B and 0.45 for 70B. The variance column uses problems with a confidently separated runner-up (98 and 86).

Model	$B$	Tokens omitted	Calls omitted	Token $-$ call	Obs. $-$ pred.	KS (bound)
Qwen3-14B	32	41.9 $[37.2,45.6]$	45.1 $[43.6,46.3]$	$-3.2$ $[-6.5,-0.7]$	$+0.16$ $[-0.03,+0.39]$	0.009 (0.021)
	24	41.3 $[37.0,44.8]$	44.1 $[42.7,45.3]$	$-2.8$ $[-6.0,-0.4]$	$+0.16$ $[-0.08,+0.43]$	0.011 (0.024)
	16	38.9 $[34.4,42.6]$	41.9 $[40.3,43.1]$	$-3.0$ $[-6.2,-0.4]$	$-0.08$ $[-0.50,+0.22]$	0.008 (0.026)
Qwen3-8B	32	40.6 $[35.8,44.6]$	44.1 $[42.3,45.6]$	$-3.5$ $[-6.6,-0.9]$	$-0.09$ $[-0.23,+0.03]$	0.011 (0.024)
	24	40.0 $[35.4,43.8]$	43.3 $[41.4,44.8]$	$-3.3$ $[-6.3,-0.8]$	$+0.05$ $[-0.21,+0.30]$	0.011 (0.029)
	16	38.4 $[34.0,42.1]$	41.3 $[39.6,42.7]$	$-2.9$ $[-5.8,-0.5]$	$+0.05$ $[-0.27,+0.37]$	0.009 (0.023)

Appendix

Table 25: Thinking-mode token accounting on 100 MATH-500 problems, with two 32-call sequences per problem and model (6,400 calls per model). Shares are percentages of the generated tokens or calls that the certificate omits at budget

B

. Token minus call and observed minus predicted saving are in points. Intervals are 95% problem-bootstrap intervals within difficulty levels. KS is the Kolmogorov distance between the predicted and observed lock-time distributions with its one-sided 95% bootstrap bound, the pre-specified level for this study. Predictions use the raw law of the other sequence, and answers are normalized boxed strings.

Model	Rule	Calls omitted	Tokens omitted	Guarantee
Qwen3-14B	Certificate	45.1	41.9	yes
	ASC 0.95	84.5	78.4	no
	ESC, window 8	70.0	62.0	no
Qwen3-8B	Certificate	44.1	40.6	yes
	ASC 0.95	83.0	77.1	no
	ESC, window 8	67.0	58.0	no

Appendix

Table 26: Exact locking and statistical stopping on the same thinking-mode sequences at

B=32

(200 sequences per model). Shares are percentages of calls and generated tokens omitted. ASC uses the posterior tail of Aggarwal et al. (2023) , computed post hoc. ESC uses nonoverlapping windows of eight calls, and a window with an abstention does not trigger. A change is an endpoint that differs from fixed-budget plurality. Only the certificate excludes changes by construction.

Figure 15: Physical request-time savings on 32 paired MMLU-Pro items per model and batch size, relative to the same-batch fixed-16 baseline. Bars are paired item-bootstrap 95% intervals, and each item has one timing. The larger output audit in Table 27 uses a separate cohort.

Method	Certificate	ASC	ESC	Bayesian
Changed outputs	0	1	6	22
Accuracy difference	$0$ (guaranteed)	$-0.083$	$-0.083$	$-0.750$
95% interval	$[0,0]$	$[-0.25,0]$	$[-0.417,0.25]$	$[-1.42,-0.17]$

Appendix

Table 27: Separate output audit: 1,200 native 4B MMLU-Pro items, sequential budget 16. Changes are relative to fixed-16 plurality, and accuracy differences are percentage points with pointwise 95% intervals.

Model	Batch	Certificate	ASC .95	ESC4	Bayesian
2B	1	$26.23\ [20.07,32.73]$	$38.35\ [24.61,51.66]$	$25.14\ [13.67,38.03]$	$46.29\ [31.21,60.31]$
2B	4	$14.38\ [8.71,20.11]$	$34.69\ [21.35,47.99]$	$25.72\ [12.86,40.64]$	$40.23\ [26.40,52.77]$
4B	1	$36.61\ [29.49,43.45]$	$55.94\ [40.51,69.37]$	$43.33\ [27.55,60.63]$	$79.93\ [67.79,87.77]$
4B	4	$19.74\ [14.38,24.80]$	$40.20\ [23.85,57.53]$	$35.57\ [19.35,54.50]$	$49.88\ [33.71,65.11]$

Appendix

Table 28: Paired request-time savings relative to the same-shape fixed-16 baseline. Each model/shape has 32 items, and intervals are 95% item-bootstrap intervals.

	Word16			MMLU-Pro			Shortest path
Method	Acc.	Calls	Time	Acc.	Calls	Time	Acc.	Calls	Time
Fixed-16	95.83	16.00	100.43	85.42	16.00	113.09	93.75	16.00	342.62
Certificate	95.83	14.50	90.82	85.42	11.75	88.67	93.75	13.50	289.50
ASC 0.95	95.83	13.50	86.19	85.42	6.33	58.00	93.75	10.92	235.03
ASC 0.99	95.83	15.17	94.94	85.42	10.33	82.78	93.75	13.00	279.66
ESC window 4	95.83	15.42	97.23	85.42	6.92	65.20	93.75	13.58	292.62

Appendix

Table 29: Fresh physical execution, 48 inputs per task: accuracy (%), mean calls and mean request time (s) per input. Methods other than parallel fixed-16 run at microbatch four.

Figure 16: Calls saved against changed outputs (a) and against accuracy change (b), relative to fixed-16 plurality, on 2,000 forced-choice MMLU-Pro trajectories: the plug-in curtailment family over

\epsilon

(

\alpha=1/2

), re-implemented ASC and ESC, a Dirichlet

(1/2)

posterior-mode rule and a majority lock. The two ESC curves stop at the same call and return either the answer of the unanimous window or the plurality of all calls so far. The exact certificate is the

\epsilon=0

member of the family. Point estimates, with intervals in Table 30 for the settings it lists.

Rule	Parameter	Mean stop	Saved %	Changed %	Accuracy change (points)
Exact certificate	$\epsilon=0$	11.46	28.35 $[27.28,29.33]$	0.00 $[0.00,1.83]$	0.00
Majority lock	$>B/2$	12.44	22.25 $[20.86,23.57]$	0.00 $[0.00,1.83]$	0.00
Plug-in curtailment, $\alpha=1/2$	$\epsilon=0.005$	7.07	55.80 $[53.84,57.69]$	2.10 $[1.45,2.80]$	$-0.05$ $[-0.55,+0.45]$
	$\epsilon=0.02$	6.12	61.76 $[59.99,63.47]$	3.50 $[2.70,4.35]$	$-0.50$ $[-1.10,+0.10]$
	$\epsilon=0.05$	4.40	72.51 $[71.03,73.96]$	7.85 $[6.55,9.15]$	$-1.45$ $[-2.40,-0.60]$
	$\epsilon=0.2$	2.65	83.42 $[82.55,84.28]$	17.60 $[15.80,19.60]$	$-3.85$ $[-5.35,-2.45]$

Appendix

Table 30: Stopping rules on 2,000 forced-choice MMLU-Pro trajectories,

B=16

: calls saved, changed outputs and accuracy change relative to fixed-16 plurality, with joint item-bootstrap 95% intervals (at zero observed changes, the upper bound is the conservative cluster-level bound

1.83\%

Checkpoint	Revision
Qwen/Qwen3.5-0.8B	2fc06364 715b967f 1860aea9 cf387788 75588b17
Qwen/Qwen3.5-2B	15852e8c 16360a2f ea060d61 5a32b452 70f8a8fc
Qwen/Qwen3.5-4B	851bf6e8 06efd8d0 a36b00dd f55e13cc b7b8cd0a
google/gemma-4-E2B-it	3e22461f 65e89153 144f8adb 70e3b8c2 cc9845a7
mistralai/Ministral-3-3B-Instruct-2512-BF16	b6d637be f2393152 b3da2b2f de72eecd ee30557e
Qwen/Qwen3.5-27B (judge and forced-choice replay, Sections E.5 and E.4 )	fc05daec 18b0a78c 049392ed 2e771dde 82bdf654

Appendix

Table 31: Checkpoint identities and pinned revisions.

Explore similar work

May 5, 2026cs.LG

Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference

Repeated sampling is a standard way to spend test-time compute, but its benefit is controlled by the latent distribution of correctness across examples, not by one-call accuracy alone. We study the binary correctness layer of repeated LLM inference under conditional-i.i.d. calls. One labeled call identifies the mean latent success probability; two labeled calls identify its second moment and hence the same-example correctness correlation that separates stable errors from recoverable call-level randomness. From these two moments, every fixed majority-vote budget has a sharp distribution-free two-call interval. The key technical reduction is that the infinite-dimensional moment problem has three-atom extremizers and quadratic dual certificates for every finite budget, so the bounds are exact rather than discretized or parametric. The first useful budget, three votes, has a closed form, width at most

1/8

, and a certified-improvement criterion. The infinite-vote endpoint is the limit of majority voting as the number of calls tends to infinity; it is also sharply bounded, but remains threshold-sensitive because it depends on latent mass around

q=1/2

. We add maximum-entropy and Latent-difficulty Gaussian-probit point completions, and experiments on LLM calls over QNLI and QQP show that empirical three- and five-vote accuracies are contained in the projected two-call regions while temperature changes and randomized model mixtures can create voting gains not ordered by one-call accuracy.

Yi Liu

York University

Majority Voting LLM Inference Optimization Call Random

Ask PDF

Sep 4, 2026cs.IR

Measuring Brand and Source Discovery under Repeated LLM Queries: A Finite-Sample Audit

Repeated-query audits must distinguish recovery of a collected set from completeness of possible outputs. We apply sample-based rarefaction to 4,500 responses from 50 buying questions, six configurations and 15 calls per cell. Historical-dictionary median ten-call recovery of the observed 15-call set ranges from 92.6% to 95.2%; re-adjudicating all 45,683 candidate strings changes this range to 89.5%-94.7%. Two blinded Gemini 3.1 Pro annotation roles assessed 600 complete answers, yielding micro F1 of 0.908 for canonical-name agreement and 0.975 for span-overlap agreement. This is AI-based evidence, without a human reference study. A separate matched roster analysis of 3,750 records per wave gives median single-call recovery of the observed five-call set of 80.0%-92.5% in February and 90.0%-100.0% in September, with question-subset dependence. Source accumulation also changes when API-returned hosts are restricted to those referenced by answer citation markers. These findings show that recovery percentages depend on extraction, question selection and the finite reference collection. They support explicit measurement definitions and sensitivity analyses, without establishing exhaustive repertoires, causal retrieval effects or a universal stopping rule.

Dmitrij Żatuchin

Department of Information Technologies, Estonian Entrepreneurship University of Applied Sciences (EUAS), Tallinn, Estonia · Rankfor.AI, Tallinn, Estonia

Model Auditing Large Language Model Responses Queries Discovery Brand-As-Memory Annotations

Ask PDF

Jan 29, 2026cs.LG

More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)

The performance of large language models (LLMs) on verifiable tasks is usually measured by pass@k, the probability of answering a question correctly at least once in k trials. At a fixed budget, a more suitable metric is coverage@cost, the average number of unique questions answered as a function of the total number of attempts. We connect the two metrics and show that the empirically-observed power-law behavior in pass@k leads to a sublinear growth of the coverage@cost (diminishing returns). To solve this problem, we propose Reset-and-Discard (ReD), a query method of LLMs that increases coverage@cost for a given budget, regardless of the pass@k form. Moreover, given a pass@k, we can quantitatively predict the savings in the total number of attempts using ReD. If pass@k is not available for the model, ReD can infer its power-law exponent. Experiments on three LLMs across coding (HumanEval), math (GSM8K), and reasoning (MMLU-Pro) benchmarks demonstrate that ReD substantially reduces the required attempts, tokens, and USD cost to reach a desired coverage, while also offering an efficient way to measure inference power-laws. ReD's advantage is maintained for imperfect verifiers and outperforms the tested allocation baselines.

Sagi Meir, Tommer D. Keidar, Noam Levi +2

School of Chemistry, Tel Aviv University, Tel Aviv 6997801, Israel · The Center for Physics and Chemistry of Living Systems, Tel Aviv University, Tel Aviv 6997801, Israel · School of Physics and Astronomy, Tel Aviv University, Tel Aviv 6997801, Israel +1

LLM Inference Optimization Token Budget Allocation

Ask PDF