Fewer Bits, More Tries: Recovering Capability Under Memory Constraints

Tomer Galanti; Josh Susskind; Eran Malach

cs.AIOct 11, 2026

Fewer Bits, More Tries: Recovering Capability Under Memory Constraints

Authors: Tomer Galanti, Josh Susskind, Eran Malach

Organizations: Texas A&M · Apple

Abstract

Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests. Single-user local inference can leave hardware underutilized, allowing parallel candidate generation to trade additional test-time computation for accuracy at modest latency cost. Across four code-generation benchmarks, verified retry recovers substantial accuracy lost to model size or compression. A 7B Qwen model with eight parallel attempts essentially matches a 32B model on MBPP+ and exceeds it on the other three benchmarks, while using about $4\times$ less model memory, running up to $6\times$ faster, and using less energy, as measured on Nvidia GPU. We observe similar tradeoff across model sizes, compression levels and hardware choices, although exact benefit depends on the hardware profile and problem setting. These results show that model capacity can be traded for verified test-time computation when correct solutions remain reachable within a modest retry budget.

Figures & tables

Figure 1: Retry trades a small amount of additional memory for higher accuracy. Each curve traces accuracy from

K=1

K=50

against model memory, defined as checkpoint weights plus KV cache. The dashed line marks the one-shot accuracy of native bf16 Coder-32B.

Checkpoint	Acc.	Wall speedup	Energy saving	Acc.	Wall speedup	Energy saving	Acc.	Wall speedup	Energy saving	Acc.	Wall speedup	Energy saving
	MBPP+			HumanEval+			LiveCodeBench			BigCodeBench
Coder-3B ( $K{=}8$ )	74.2%	$2.8\times$	$3.5\times$	87.5%	$2.2\times$	$2.7\times$	19.8%	$4.0\times$	$5.8\times$	58.2%	$1.5\times$	$1.7\times$
Coder-7B ( $K{=}8$ )	77.8%	$2.0\times$	$1.9\times$	89.4%	$2.2\times$	$2.4\times$	27.5%	$5.9\times$	$6.7\times$	62.7%	$1.3\times$	$1.3\times$
Coder-14B ( $K{=}8$ )	79.4%	$1.1\times$	$1.0\times$	89.5%	$1.0\times$	$0.9\times$	33.9%	$1.5\times$	$1.5\times$	66.3%	$0.6\times$	$0.6\times$
Coder-32B ( $K{=}1$ )	77.9%	$1.0\times$	$1.0\times$	85.1%	$1.0\times$	$1.0\times$	27.1%	$1.0\times$	$1.0\times$	56.6%	$1.0\times$	$1.0\times$

Table 1: Eight attempts from a smaller model can match or exceed one attempt from Coder-32B at lower serving cost. The 3B, 7B, and 14B models use

K=8

, while Coder-32B uses

K=1

. Accuracy comes from the main candidate pools; wall time and energy are measured on matched B200 workloads. Speedup and energy saving are Coder-32B/retried model, so values above

1\times

favor the smaller model; the Coder-32B row is the reference. See Appendix Table 4 for more details.

Figure 2: Parallel retry can make a smaller model both faster and more accurate. The curves show accuracy versus measured B200 wall-clock time as the number of attempts increases. The

\times

marks Coder-32B with one attempt, and the circles mark eight attempts.

Benchmark	One-shot 1-bit / bf16	At recovery 1-bit / bf16	Selected after 50 tries 1-bit / bf16	Any correct in 50 1-bit / bf16	Tries to recover	Wall-time ratio	Energy ratio
	Accuracy				Cost of recovering bf16 one-shot accuracy
HumanEval+	85.5% / 89.3%	91.5% / 89.3%	93.9% / 93.9%	98.2% / 98.2%	$2$	$0.72\times$	$1.30\times$
MBPP+	68.9% / 77.2%	79.8% / 77.2%	82.5% / 83.6%	87.3% / 88.6%	$4$	$0.42\times$	$0.73\times$
LiveCodeBench	14.6% / 16.8%	20.9% / 16.8%	34.9% / 34.3%	35.4% / 34.3%	$2$	$1.04\times$	$1.61\times$
BigCodeBench	19.2% / 26.7%	29.9% / 26.7%	63.4% / 67.0%	72.3% / 70.8%	$2$	$0.71\times$	$0.92\times$

Table 2: A few retries recover the one-shot accuracy lost to one-bit quantization. Each row compares the Bonsai Q1_0 one-bit checkpoint with the bf16 version of the same 27B model. “Tries to recover” is the smallest tested number of attempts for which the one-bit model matches or exceeds bf16 one-shot accuracy. Wall-time and energy ratios are bf16@1 / one-bit@recovery, so values above one favor the one-bit deployment and values below one favor bf16.

Benchmark	7B one-shot	7B, 8 tries generated tests	vs. 32B one-shot	7B, 50 tries generated tests	vs. 32B one-shot	7B, 50 tries public tests	32B one-shot	Weights (GB) 7B / 32B	Memory saving	Process peak (GB) 7B / 32B
	Accuracy							Memory
MBPP+	71.6%	76.8%	$-1.1$	78.6%	$+0.7$	79.9%	77.9%	5.57 / 65.53	$10.3\times$	11.73 / 73.02
HumanEval+	80.0%	82.9%	$-2.2$	83.5%	$-1.6$	87.8%	85.1%	5.57 / 65.53	$10.4\times$	11.73 / 73.02
LiveCodeBench	16.1%	25.0%	$-2.1$	28.0%	$+0.9$	31.4%	27.1%	5.57 / 65.53	$10.3\times$	11.73 / 73.02
BigCodeBench	46.3%	52.7%	$-3.9$	53.5%	$-3.2$	67.9%	56.6%	5.57 / 65.53	$10.3\times$	11.73 / 73.02

Table 3: Model-generated tests let a compact 7B model recover much of the 32B model’s one-shot accuracy. We report Coder-7B AWQ4 with model-generated verification at

K=8

and

K=50

, alongside public-test selection and the Coder-32B bf16 one-shot reference. “vs. 32B” is the accuracy difference from that reference.

Figure 3: Retry closes much of the gap between the 7B and 32B models when the smaller model still generates correct solutions. The curves show the accuracy difference between Coder-32B bf16 and Coder-7B AWQ4 as the number of attempts increases, using public tests, model-generated tests (CodeT), and the fraction of tasks with at least one correct candidate. The gap largely disappears on HumanEval+, MBPP+, and BigCodeBench, but remains on LiveCodeBench because the 7B model fails to generate correct solutions for some tasks.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

	MBPP+ ( $77.9\%$ ref.)					HumanEval+ ( $85.1\%$ ref.)					LiveCodeBench ( $27.1\%$ ref.)					BigCodeBench ( $56.6\%$ ref.)
Checkpoint	Acc.	Wall	Wall speedup	J/task	Energy saving	Acc.	Wall	Wall speedup	J/task	Energy saving	Acc.	Wall	Wall speedup	J/task	Energy saving	Acc.	Wall	Wall speedup	J/task	Energy saving
Coder-3B ( $K{=}8$ )	74.2%	3.12	$2.8\times$	32	$3.5\times$	87.5%	5.17	$2.2\times$	60	$2.7\times$	19.8%	7.46	$4.0\times$	120	$5.8\times$	58.2%	4.59	$1.5\times$	54	$1.7\times$
Coder-7B ( $K{=}8$ )	77.8%	4.31	$2.0\times$	59	$1.9\times$	89.4%	5.25	$2.2\times$	68	$2.4\times$	27.5%	5.06	$5.9\times$	104	$6.7\times$	62.7%	5.07	$1.3\times$	72	$1.3\times$
Coder-14B ( $K{=}8$ )	79.4%	7.75	$1.1\times$	110	$1.0\times$	89.5%	11.66	$1.0\times$	175	$0.9\times$	33.9%	20.08	$1.5\times$	450	$1.5\times$	66.3%	10.51	$0.6\times$	157	$0.6\times$
Coder-32B ( $K{=}1$ )	77.9%	8.57	$1.0\times$	112	$1.0\times$	85.1%	11.41	$1.0\times$	162	$1.0\times$	27.1%	29.82	$1.0\times$	692	$1.0\times$	56.6%	6.69	$1.0\times$	92	$1.0\times$

Appendix

Table 4: Eight attempts from a smaller model can match or exceed one attempt from Coder-32B at lower serving cost. The 3B, 7B, and 14B models use

K=8

, while Coder-32B uses

K=1

. Accuracy comes from the main candidate pools; wall time and energy are measured on matched B200 workloads. Speedup and energy saving are Coder-32B/retried model, so values above

1\times

favor the smaller model.

Benchmark	Checkpoint	pass@1	exec@8	exec@50	oracle@50	CodeT@8	Usable tests
HumanEval+	Bonsai Q1_0 (1-bit)	85.5%	93.3%	93.9%	98.2%	91.2%	0.707
	Bonsai AWQ4 (4-bit)	89.9%	94.1%	94.5%	98.2%	93.1%	0.866
	Qwen3.6-27B (bf16)	89.3%	93.6%	93.9%	98.2%	93.3%	0.927
MBPP+	Bonsai Q1_0 (1-bit)	68.9%	81.0%	82.5%	87.3%	78.2%	0.810
	Bonsai AWQ4 (4-bit)	76.6%	82.3%	83.3%	87.8%	80.8%	0.918
	Qwen3.6-27B (bf16)	77.2%	83.5%	83.6%	88.6%	82.1%	0.899

Appendix

Table 5: Accuracy and model-generated verification for the Qwen3.6-27B precision variants. All accuracy columns use the full task population for each experiment. “Usable tests” reports the fraction of tasks for which a usable generated test suite was produced.

$K$	Weights (GB) Q1 / bf16	Weight saving	Wall (s) Q1 / bf16	Wall-clock speedup	tok/s Q1 / bf16	Throughput ratio	J/task Q1 / bf16	Energy saving
1	3.80 / 54.66	$14.38\times$	271.7 / 359.7	$1.32\times$	222.8 / 156.5	$1.42\times$	1,656.2 / 3,946.8	$2.38\times$
8	3.80 / 54.66	$14.38\times$	1,549.1 / 2,719.3	$1.76\times$	307.4 / 166.0	$1.85\times$	10,367 / 30,382	$2.93\times$
50	3.80 / 54.66	$14.38\times$	8,603.8 / 16,477.5	$1.92\times$	346.8 / 171.1	$2.03\times$	59,151 / 185,108	$3.13\times$

Appendix

Table 6: Q1_0 is faster and more energy-efficient than bf16 at the same retry budget. Results use the HumanEval+ serving workload. Entries compare Q1_0 with bf16 at the same

K

. Speedup and energy saving are bf16/Q1_0, while throughput ratio is Q1_0/bf16. Values are medians over three repetitions.

				M3 Max, 64 GiB				M5 Pro, 64 GiB
Benchmark	Comparison	Accuracy dep./ref.	Memory saving ( $\times$ )	Speedup ( $\times$ )	Energy saving ( $\times$ )	Power ratio ( $\times$ )	Swap	Speedup ( $\times$ )	Energy saving ( $\times$ )	Power ratio ( $\times$ )	Swap
One-bit quantization
HumanEval+	Q1@4 / 27B@1	93.1/89.3	13.41	0.85	0.88	1.04	yes/no	1.67	1.06	0.63	yes/no
MBPP+	Q1@4 / 27B@1	79.8/77.2	13.45	0.86	0.83	0.97	yes/no	1.59	1.05	0.66	yes/no
LiveCodeBench	Q1@4 / 27B@1	25.6/16.8	13.31	0.87	0.87	1.00	yes/no	1.66	1.08	0.65	yes/no
BigCodeBench	Q1@4 / 27B@1	41.4/26.7	11.43	0.83	0.82	0.99	yes/no	1.58	1.03	0.65	yes/no

Appendix

Table 7: Physical-device recovery depends on retry depth. Accuracy is deployment/reference. Memory saving, speedup, energy saving, and power ratio are reference/deployment. In the model-scale block, semicolon-separated entries follow the order of the two comparisons in the second column. Model memory is checkpoint weights plus KV cache from realized sequence lengths.

Model size	Compact precision	Weights (GB) bf16 $\to$ compact	Weight saving	$\Delta$ pass@1	$\Delta$ exec@50	$\Delta$ pass@1	$\Delta$ exec@50	$\Delta$ pass@1	$\Delta$ exec@50	$\Delta$ pass@1	$\Delta$ exec@50	Largest $\|\Delta\|$ at $K=50$
		Memory		MBPP+		HumanEval+		LiveCodeBench		BigCodeBench
Coder-3B	GPTQ4	6.17 $\to$ 2.07	$2.98\times$	+0.7	+0.0	-1.0	+1.7	+0.0	+2.3	-1.7	-0.3	2.3
Coder-7B	AWQ4	15.23 $\to$ 5.57	$2.73\times$	+2.0	+1.3	-2.6	-3.1	-0.2	-2.3	-1.6	+0.1	3.1
Coder-32B	AWQ4	65.53 $\to$ 19.33	$3.39\times$	-0.7	-3.0	-0.6	+0.9	+0.6	+2.3	+0.3	+0.1	3.0
Qwen3-Coder MoE	AWQ4	61.07 $\to$ 16.81	$3.63\times$	+0.4	+0.3	+0.2	-0.7	-1.8	-2.3	-3.4	+0.8	2.3

Appendix

Table 8: Conventional quantization substantially reduces weight memory with relatively small changes in accuracy. Each row compares bf16 with the lowest-memory measured precision at the same model size. Accuracy differences are compact minus bf16, in percentage points. The MoE result uses a third-party AWQ4 checkpoint. Full results for every precision are given in Tables 9 and 11 .

Model	Precision	Weights (GB)	MBPP+			HumanEval+			LiveCodeBench			BigCodeBench
			pass@1	exec@50	CodeT@50	pass@1	exec@50	CodeT@50	pass@1	exec@50	CodeT@50	pass@1	exec@50	CodeT@50
Coder-3B	bf16	6.17	61.6%	77.5%	75.4%	77.4%	87.9%	84.2%	9.5%	26.3%	18.3%	41.3%	64.7%	49.7%
	GPTQ 8-bit	3.47	63.3%	78.3%	71.4%	77.7%	88.4%	82.9%	10.1%	28.6%	19.4%	40.7%	64.1%	48.6%
	AWQ 4-bit	2.69	62.6%	78.0%	71.4%	75.9%	90.9%	84.1%	8.7%	26.3%	17.7%	39.5%	64.0%	47.7%
	GPTQ 4-bit	2.07	62.3%	77.5%	71.2%	76.4%	89.6%	83.5%	9.5%	28.6%	17.7%	39.5%	64.4%	47.6%
Coder-7B	bf16	15.23	69.6%	78.6%	77.5%	82.6%	90.9%	84.1%	16.3%	33.7%	28.0%	47.9%	67.8%	56.2%

Appendix

Table 9: Qwen2.5-Coder accuracy across all measured precisions. For each model size, we report one-shot accuracy, accuracy after 50 attempts selected with public tests, and accuracy after 50 attempts selected with model-generated tests. The memory column reports checkpoint weight size. LiveCodeBench CodeT uses independently generated candidate pools.

Figure 4: Retry recovers lost one-shot accuracy only when correct solutions remain reachable. Panel (a) compares one-bit Bonsai with its bf16 Qwen3.6-27B parent, and panel (b) compares Coder-7B with Coder-32B. Bars show the change relative to the higher-precision or larger reference for accuracy after eight verified attempts and for the fraction of tasks solved at least once within 50 attempts. Quantization largely preserves candidate coverage, whereas reducing model size can lower it, most clearly on LiveCodeBench.

Benchmark	pass@1 3B $\to$ 32B	$\Delta$ pass@1	exec@50 3B $\to$ 32B	CodeT@50 3B $\to$ 32B	oracle@50 3B / 7B / 14B / 32B	oracle@50 range
MBPP+	$61.6\%\to 77.9\%$	+16.3	$77.5\%\to 82.9\%$	$75.4\%\to 81.1\%$	$86.8/87.3/86.0/87.3\%$	1.3
HumanEval+	$77.4\%\to 85.1\%$	+7.7	$87.9\%\to 89.3\%$	$84.2\%\to 85.7\%$	$93.9/92.7/94.5/92.1\%$	2.4
BigCodeBench	$41.3\%\to 56.6\%$	+15.4	$64.7\%\to 69.7\%$	$49.7\%\to 62.7\%$	$75.6/76.9/77.0/75.5\%$	1.5
LiveCodeBench	$9.5\%\to 27.1\%$	+17.6	$26.3\%\to 38.3\%$	$18.3\%\to 32.0\%$	$30.3/37.1/43.4/42.9\%$	13.1

Appendix

Table 10: Model scale mostly improves single-attempt reliability on MBPP+, HumanEval+, and BigCodeBench, but also expands the set of solvable tasks on LiveCodeBench. The endpoint columns compare 3B with 32B. The oracle@50 column shows all four native model sizes (3B / 7B / 14B / 32B), and the final column reports their range. Accuracy differences are in percentage points. LiveCodeBench CodeT uses independently generated candidate pools.

Precision	Weights (GB)	MBPP+		HumanEval+		LiveCodeBench		BigCodeBench
		pass@1	exec@50	pass@1	exec@50	pass@1	exec@50	pass@1	exec@50
bf16	61.07	77.2%	79.3%	88.5%	90.9%	32.7%	43.4%	57.0%	63.3%
FP8	31.18	77.1%	79.4%	88.2%	91.5%	31.6%	45.1%	57.8%	63.5%
GPTQ 8-bit	32.00	77.4%	79.1%	88.2%	90.9%	32.4%	45.1%	54.7%	62.6%
AWQ 4-bit	16.81	77.6%	79.6%	88.7%	90.2%	30.9%	41.1%	53.6%	64.1%
GPTQ 4-bit	16.93	76.8%	78.8%	87.8%	90.2%	31.5%	44.0%	52.6%	62.1%

Appendix

Table 11: Qwen3-Coder-30B-A3B remains relatively stable across precision levels. The model has 30.5B total parameters and 3.3B active parameters per token. The final two rows report the range across precisions for one-shot accuracy and accuracy after 50 verified attempts; brackets give paired 95% bootstrap intervals for the latter.

Model	Precision	exec@50	oracle@50	Public-test gap	CodeT@50	CodeT gap	Tasks with usable tests
Coder-3B	bf16	64.7%	75.6%	10.9	49.7%	15.0	99.9%
	GPTQ 8-bit	64.1%	74.5%	10.3	48.6%	15.5	100.0%
	AWQ 4-bit	64.0%	75.1%	11.1	47.7%	16.3	99.8%
	GPTQ 4-bit	64.4%	75.5%	11.1	47.6%	16.8	100.0%
Coder-7B	bf16	67.8%	76.9%	9.1	56.2%	11.6	99.9%
	GPTQ 8-bit	67.6%	76.7%	9.1	55.5%	12.1	100.0%

Appendix

Table 12: Verification, rather than candidate generation, remains a major bottleneck on BigCodeBench.

Figure 5: Verified retry has different effects across compact checkpoints. Points show retried accuracy minus one-shot reference accuracy, with 95% paired task-bootstrap intervals. Each comparison uses its common task set, so the plotted differences need not exactly match subtraction of the rounded full-population values in Table 13 . Blue intervals are above zero, red intervals are below zero, and gray intervals cross zero. Granite-Code uses Granite-Code-34B as its reference rather than Qwen2.5-Coder-32B, so its effect sizes are not directly comparable with the other rows.

Strategy	Retried checkpoint	One-shot reference	Selection	Retried weights (GB)	Weight saving	MBPP+		HumanEval+		LiveCodeBench
						Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$
Prune + distill	Minitron-8B	Coder-32B bf16	exec@50	16.83	$3.9\times$	71.7/77.9%	$\mathbf{-6.2\;[-10.5,-2.0]}$	75.0/85.1%	$\mathbf{-10.0\;[-15.9,-4.4]}$	26.3/27.1%	$-1.1\;[-5.1,+3.0]$
Native bf16	Coder-7B	Coder-32B bf16	exec@50	15.23	$4.3\times$	78.6/77.9%	$+0.7\;[-2.8,+4.2]$	90.9/85.1%	$\mathbf{+5.8\;[+1.8,+10.0]}$	33.7/27.1%	$\mathbf{+6.3\;[+1.8,+11.1]}$
Quantization	Coder-7B AWQ4	Coder-32B bf16	exec@50	5.57	$11.8\times$	79.9/77.9%	$+2.0\;[-1.7,+5.7]$	87.8/85.1%	$+2.8\;[-1.8,+7.3]$	31.4/27.1%	$+4.0\;[-0.6,+8.8]$
Reasoning distill	R1-Distill-1.5B	Coder-32B bf16	exec@50	3.55	$18.5\times$	71.7/77.9%	$\mathbf{-6.2\;[-10.7,-1.7]}$	79.9/85.1%	$-5.2\;[-11.4,+0.9]$	30.9/27.1%	$+3.5\;[-1.1,+8.0]$
Quantization	Coder-3B GPTQ8	Coder-32B bf16	exec@50	3.47	$18.9\times$	78.3/77.9%	$+0.4\;[-3.2,+3.9]$	88.4/85.1%	$+3.4\;[-0.9,+7.7]$	28.6/27.1%	$+1.2\;[-3.6,+5.9]$

Appendix

Table 13: Verified retry across several compact deployment strategies. Each row compares a compact checkpoint after 50 verified attempts with the one-shot reference shown in the third column. Accuracy is retried/reference, and

\Delta

is the paired accuracy difference where confidence intervals are available. Granite-Code uses Granite-Code-34B as its within-family reference.

$K$	Gain (%)	Wall-time multiplier	Gain (%)	Wall-time multiplier	Gain (%)	Wall-time multiplier
	Coder-3B		Coder-7B		Coder-32B
1	0	1.00	0	1.00	0	1.00
2	40	1.45	56	1.14	33	1.12
4	63	1.44	83	1.35	55	1.29
8	80	1.95	93	1.73	80	1.72
16	91	4.78	99	3.13	90	2.68

Appendix

Table 14: Most of the benefit of retry arrives at moderate budgets. For each model, “Gain” is the fraction of its improvement from

K=1

K=50

that has already been obtained at the displayed

K

. Wall time is reported relative to the same model at

K=1

	Coder-7B across retry budget			Model size at $K=50$
Benchmark	$K=1$	$K=8$	$K=50$	3B	7B	14B	32B
MBPP+	18.2%	6.0%	4.2%	5.0%	4.2%	4.0%	4.2%
HumanEval+	12.4%	5.8%	4.3%	3.7%	4.3%	3.7%	6.1%
LiveCodeBench	77.8%	55.4%	38.9%	48.6%	38.9%	30.9%	34.3%
BigCodeBench	40.1%	20.8%	13.9%	14.1%	13.9%	14.5%	15.1%

Appendix

Table 15: Retry sharply reduces how often the selector must fall back. The left block varies the retry budget for Coder-7B; values below

K=50

are averaged over 120 candidate subsets and

K=50

is exact. The right block compares native model sizes at

K=50

		MBPP+			HumanEval+			LiveCodeBench
Mode	$K$	Acc.	Wall (s)	J/task	Acc.	Wall (s)	J/task	Acc.	Wall (s)	J/task
Instruct	1	77.9%	16.5	243	87.9%	52.8	700	43.8%	114.5	1,707
	8	79.3%	53.3	866	89.9%	137.2	2,301	53.0%	199.4	4,317
	50	79.6%	130.2	2,454	90.2%	257.5	5,099	52.6%	467.6	11,156
Thinking	1	79.5%	124.2	2,008	91.2%	83.8	1,389	51.7%	186.0	3,899
	8	80.9%	218.3	4,899	92.9%	193.0	3,972	62.2%	578.2	13,670

Appendix

Table 16: Longer reasoning and parallel retry have different advantages across benchmarks. The Instruct and Thinking variants use the same Qwen3-30B-A3B architecture and weight footprint. Accuracy uses public-test selection on the full benchmark sets; wall time and energy come from matched 40-prompt B200 workloads and should be compared only within a benchmark. Bold rows highlight eight non-reasoning attempts versus one reasoning attempt.

Model	Type	pass@1	CodeT@50	exec@50	oracle@50
Coder-1.5B	native Coder	57.3%	68.3%	75.4%	84.7%
Coder-14B	native Coder	73.2%	79.4%	80.4%	86.0%
Qwen2.5-72B	general model	78.2%	78.7%	80.8%	84.4%

Appendix

Table 17: Additional model sizes on MBPP+ at

K=50

. All four accuracy measures in each row are computed from the same candidate pool. The 1.5B and 14B checkpoints belong to the native Qwen2.5-Coder scale ladder. Qwen2.5-72B is a separate general model and is included only as an additional accuracy comparison.

Figure 6: Parallel decoding reduces the energy cost of additional attempts. Solid curves show measured B200 energy for the native Qwen2.5-Coder models. Dotted lines show the energy that would be used if every attempt cost as much as a single attempt. Absolute energy should not be compared across panels because the benchmarks use different workloads and generation limits.

Figure 7: Quantization changes serving time as well as memory use. Each quantized checkpoint is compared with the bf16 model of the same size.

Benchmark	$K=1$	$K=8$	$K=50$
MBPP+	$1.00\times$	$0.52\times$	$0.51\times$
HumanEval+	$1.15\times$	$0.66\times$	$0.60\times$
LiveCodeBench	$1.04\times$	$0.79\times$	$0.62\times$
BigCodeBench	$0.92\times$	$0.54\times$	$0.52\times$

Appendix

Table 18: AWQ4 does not consistently speed up wide-batch decoding. Entries are Coder-7B wall-clock speedups

T_{\rm bf16}/T_{\rm AWQ4}

on B200, so values above

1\times

favor AWQ4.

Model	$\Delta$ accuracy (pp)	95% CI (pp)
Coder-1.5B	+6.08	[ 2.65, 9.79]
Coder-3B	+3.70	[ 1.06, 6.61]
Coder-7B	+3.44	[ 1.06, 5.82]
Coder-14B	+1.32	[-0.79, 3.44]
Coder-32B	+0.79	[-1.32, 2.91]

Appendix

Table 19: CodeT aggregation helps most for smaller models on MBPP+. Values are the accuracy difference between CodeT and simple test-pass counting at

K=50

, with paired 95% bootstrap intervals.

Figure 8: Deep retry separates failures of selection, generated tests, and candidate generation. Curves use Qwen2.5-Coder-7B candidate pools through

K=500

. Solid curves show public-test selection, CodeT, and simple test-pass counting. The remaining curves show the best accuracy possible from the generated-test outcome patterns and the fraction of tasks with at least one correct candidate. BigCodeBench has the largest gap between these two upper bounds, showing that its generated tests often fail to distinguish correct from incorrect programs.

Native Qwen2.5-Coder
	MBPP+		HumanEval+		LiveCodeBench
Checkpoint	Usable	Tests	Usable	Tests	Usable	Tests
Coder-1.5B	0.99	10.8	0.99	11.2	0.95	8.4
Coder-3B	1.00	11.5	1.00	11.7	0.98	9.8
Coder-7B	1.00	11.8	1.00	11.8	0.95	9.5
Coder-14B	1.00	12.0	1.00	11.9	0.89	8.9

Appendix

Table 20: The ability to produce usable generated tests varies substantially across model families. “Usable” is the fraction of tasks in the generated-test pool for which the tests can be parsed and executed. “Tests” is the mean number of generated tests.

			KV cache $V$ (GB)				Weights + KV $A$ (GB)				Accuracy (%)
Benchmark	Checkpoint	$W$ (GB)	$K{=}1$	$K{=}4$	$K{=}8$	$K{=}50$	$K{=}1$	$K{=}4$	$K{=}8$	$K{=}50$	$K{=}1$	$K{=}4$	$K{=}8$	$K{=}50$
MBPP+	Coder-3B AWQ4	2.69	0.01	0.04	0.07	0.45	2.70	2.73	2.76	3.14	62.5%	72.5%	75.0%	78.0%
	Coder-7B AWQ4	5.57	0.02	0.07	0.13	0.78	5.59	5.64	5.70	6.35	71.7%	77.6%	78.6%	79.9%
	Coder-7B bf16	15.23	0.02	0.07	0.13	0.78	15.25	15.30	15.36	16.01	69.6%	77.0%	78.0%	78.6%
	Coder-32B bf16	65.53	0.09	0.29	0.56	3.41	65.62	65.82	66.09	68.94	77.9%	80.8%	81.8%	82.9%
HumanEval+	Coder-3B AWQ4	2.69	0.02	0.07	0.13	0.80	2.71	2.76	2.82	3.49	75.8%	86.1%	88.5%	90.9%

Appendix

Table 21: Checkpoint weights, KV cache, and accuracy across retry budgets.

W

is checkpoint weight memory,

V

is the KV cache from Equation 3 , and

A=W+V

is their sum. The table contains the configurations used for the main accuracy–memory curves in Figure 1 . Memory values exclude runtime overhead; measured process-memory results are reported separately in Table 22 .

Configuration	Lowest successful utilization	Peak (GB)	Process-memory saving
Coder-7B AWQ4, $K=8$	0.045	11.55	$6.32\times$
Coder-7B AWQ4, $K=50$	0.046	11.73	$6.23\times$
Coder-7B bf16, $K=8$	0.098	21.58	$3.38\times$
Coder-32B bf16, $K=1$	0.370	73.02	$1.00\times$

Appendix

Table 22: Measured process memory for representative B200/vLLM configurations. For each configuration, we report the lowest tested gpu_memory_utilization setting that completed successfully and the corresponding peak process memory. The utilization setting controls vLLM’s cache allocation and is not itself a device-memory limit.

Explore similar work

Aug 22, 2026

More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning

Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy while fewer quantify local runtime and energy, characterize failure modes, or apply paired statistical comparisons under controlled conditions. This paper presents a controlled, documented procedure for evaluating locally hosted LLMs on mathematical reasoning. It combines fixed inference settings, hierarchical answer extraction and verification, explicit failure-mode classification, and per-question resource measurement, and reports accuracy with paired significance tests and effect sizes. We demonstrate it in a preliminary study of three compact open-weight models under five billion parameters, Gemma3:4b (Google), Phi3:3.8b (Microsoft), and Qwen3:4b (Alibaba), across datasets spanning Grade 8 Math, Calculus I, and Advanced Probability and Statistics. All models ran through the same local inference server on one workstation, using a shared prompt template, controlled settings, and a matched question set per dataset. No single model dominates. Qwen3:4b is most accurate on two datasets and Gemma3:4b on Calculus I, yet Gemma3:4b returns roughly three times more correct answers per watt-hour than Qwen3:4b on every dataset while generating far fewer output tokens; Qwen3:4b requires substantially more generation time, energy, and output per question. Phi3:3.8b is substantially less accurate on all three datasets; its low extraction-failure rate indicates incorrect answers rather than unparsed output, though we caveat possible prompt-format effects. These preliminary findings indicate that accuracy alone is an insufficient basis for selecting a local model.

Orion Powers, Daniella Seum, Khaled Slhoub

College of Engineering and Science Florida Institute of Technology Melbourne, Florida, USA

LLM Evaluation On-Device Language Model Inference LLM Reasoning Efficient Language Model Inference

Ask PDF

Jan 29, 2026

More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)

The performance of large language models (LLMs) on verifiable tasks is usually measured by pass@k, the probability of answering a question correctly at least once in k trials. At a fixed budget, a more suitable metric is coverage@cost, the average number of unique questions answered as a function of the total number of attempts. We connect the two metrics and show that the empirically-observed power-law behavior in pass@k leads to a sublinear growth of the coverage@cost (diminishing returns). To solve this problem, we propose Reset-and-Discard (ReD), a query method of LLMs that increases coverage@cost for a given budget, regardless of the pass@k form. Moreover, given a pass@k, we can quantitatively predict the savings in the total number of attempts using ReD. If pass@k is not available for the model, ReD can infer its power-law exponent. Experiments on three LLMs across coding (HumanEval), math (GSM8K), and reasoning (MMLU-Pro) benchmarks demonstrate that ReD substantially reduces the required attempts, tokens, and USD cost to reach a desired coverage, while also offering an efficient way to measure inference power-laws. ReD's advantage is maintained for imperfect verifiers and outperforms the tested allocation baselines.

Sagi Meir, Tommer D. Keidar, Noam Levi +2

School of Chemistry, Tel Aviv University, Tel Aviv 6997801, Israel · The Center for Physics and Chemistry of Living Systems, Tel Aviv University, Tel Aviv 6997801, Israel · School of Physics and Astronomy, Tel Aviv University, Tel Aviv 6997801, Israel +1

LLM Inference Efficiency LLM Evaluation Cost-Aware Inference Inference-Time Compute Allocation

Ask PDF

Oct 1, 2026

How Much Can Language Models Gain from Test-Time Computation?

How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.

Bangji Yang, Jingyuan Li, Jiajun Fan +8

University of Illinois at Urbana-Champaign · University of Washington · Tsinghua University

LLM Evaluation Test-Time Scaling

Ask PDF