Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It

Jonathan Williams; Esin Tureci Karthik R. Narasimhan

cs.LGOct 1, 2026

Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It

Authors: Jonathan Williams, Esin Tureci Karthik R. Narasimhan

Organizations: Princeton University

Abstract

Majority voting over sampled completions is the workhorse of test-time scaling, and reinforcement learning with verifiable rewards (RLVR) is the workhorse for making each completion better. The standard pipeline composes the two: train one policy with RLVR, then sample it many times and vote. We show that this composition is lossy. A vote can only overturn mistakes that its voters do not share, and RLVR sharpens a policy so that its samples increasingly make the same mistakes. With every method drawing exactly $160$ completions per problem, training a single LoRA adapter on the full RLVR budget raises single-sample accuracy on every model we test ( $1.5$ B- $8$ B). Yet on three of four models it leaves the majority vote below that of the untrained base model, by up to $4.8$ points. The damage builds during training: voter errors grow steadily more correlated, and the majority vote accuracy peaks early before falling by up to $7.0$ points. The cause is concentration, not RLVR itself. We split the same data and training budget across $K$ LoRA adapters, each trained on its own random disjoint shard, and call the result an adapter thicket. Thickets out-vote the fully trained adapter in all $16$ (model, $K$ ) settings, and for $K{\geq}4$ they stay within $0.8$ points of the base model or above it. A single adapter stopped early, at a thicket member's step count, is a strong control that matches thickets for small $K$ . For $K{\geq}8$ , thickets keep more of RLVR's single-sample gain and out-vote this control in six of eight settings. The cost of concentration also grows with the number of votes: from $16$ to $160$ votes, the thicket's lead over the fully trained adapter widens from $1.3$ to $3.3$ points. When the plan is to sample and vote, an RLVR budget is better spent broad than deep.

Figures & tables

Figure 1: One RLVR budget, one inference budget, two allocations. Left: GRPO-single-adapter trains one LoRA on the full

2{,}000

-problem pool with all

16

k rollouts. An adapter thicket trains

K

LoRAs on disjoint

{2000}/{K}

-example shards with

{16\mathrm{k}}/{K}

rollouts each. At inference every configuration pools exactly

160

completions per problem (

K

voters

\times{160}/{K}

samples). Right: full-pool majority vote at

K{=}16

on the math suite (macro mean over five benchmarks, three seeds), including the step-matched single-adapter control. The fully trained GRPO-single-adapter (red squares) falls below the base on three of four models. Deltas: thicket

-

GRPO-single-adapter.

Model	Method	$K{=}2$	$K{=}4$	$K{=}8$	$K{=}16$
Qwen2.5-1.5B	Base	51.3 $\pm$ 0.4	51.3 $\pm$ 0.4	51.3 $\pm$ 0.4	51.3 $\pm$ 0.4
	GRPO-single (matched)	54.6 $\pm$ 0.7	55.3 $\pm$ 2.0	53.7 $\pm$ 0.2	52.2 $\pm$ 0.9
	GRPO-single-adapter	51.6 $\pm$ 1.6	51.6 $\pm$ 1.6	51.6 $\pm$ 1.6	51.6 $\pm$ 1.6
	Thicket	55.2 $\pm$ 2.0	53.6 $\pm$ 0.7	54.6 $\pm$ 0.5	54.2 $\pm$ 0.2
Qwen2.5-3B	Base	63.8 $\pm$ 0.3	63.8 $\pm$ 0.3	63.8 $\pm$ 0.3	63.8 $\pm$ 0.3
	GRPO-single (matched)	61.9 $\pm$ 1.2	63.5 $\pm$ 1.2	63.3 $\pm$ 0.1	63.5 $\pm$ 0.8

Table 1: Majority-vote accuracy on the math suite under the fixed

160

-vote budget (canonical-class votes; macro means (%) over five benchmarks; mean

\pm

std over three seeds). Base and GRPO-single-adapter each vote over one

160

-completion pool, so their rows are constant across

K

by construction. Bold: best method per model and

K

by displayed mean; underlined: second best.

Figure 2: Single-adapter training raises voter accuracy and voter correlation together – and the vote pays for it. Checkpoints of the GRPO-single-adapter run at steps

25/50/125/250

and the final checkpoint (step

499

for Qwen,

500

for Llama; plotted at

500

), evaluated on the math suite with the same fixed

160

-completion budget per problem (grouped as

16

voters

\times

10

samples; mean over three seeds,

\pm 1

std band). Top: mean voter accuracy. Middle: mean pairwise error correlation

\bar{\rho}

. Bottom: majority vote over the

160

-vote pool. Dashed lines: the untrained base model. On every model

\bar{\rho}

rises monotonically with training, and the vote peaks by step

125

before declining.

		Coverage, math				Coverage, code
Model	Method ( $K{=}$ )	2	4	8	16	2	4	8	16
Qwen2.5-1.5B	Base	89.6 $\pm$ 0.3	89.6 $\pm$ 0.3	89.6 $\pm$ 0.3	89.6 $\pm$ 0.3	72.2 $\pm$ 0.4	72.2 $\pm$ 0.4	72.2 $\pm$ 0.4	72.2 $\pm$ 0.4
	GRPO-single (matched)	89.6 $\pm$ 0.4	89.3 $\pm$ 0.5	90.0 $\pm$ 0.3	89.5 $\pm$ 0.5	69.0 $\pm$ 1.8	70.2 $\pm$ 0.9	72.0 $\pm$ 0.5	71.9 $\pm$ 0.3
	GRPO-single-adapter	87.7 $\pm$ 0.7	87.7 $\pm$ 0.7	87.7 $\pm$ 0.7	87.7 $\pm$ 0.7	65.8 $\pm$ 1.4	65.8 $\pm$ 1.4	65.8 $\pm$ 1.4	65.8 $\pm$ 1.4
	Thicket	88.8 $\pm$ 1.0	89.5 $\pm$ 0.4	89.6 $\pm$ 0.5	89.8 $\pm$ 0.6	69.2 $\pm$ 1.4	70.6 $\pm$ 0.6	70.6 $\pm$ 0.5	70.3 $\pm$ 0.3
Qwen2.5-3B	Base	91.7 $\pm$ 0.1	91.7 $\pm$ 0.1	91.7 $\pm$ 0.1	91.7 $\pm$ 0.1	75.6 $\pm$ 1.1	75.6 $\pm$ 1.1	75.6 $\pm$ 1.1	75.6 $\pm$ 1.1

Table 2: Coverage (pass@pool: at least one correct completion among the

160

) on the math and code suites; macro means (%); mean

\pm

std over three seeds. Bold: best method per model, suite, and

K

; underlined: second best.

Figure 3: Single-sample accuracy vs. majority vote on the math suite (pass@1 = mean voter accuracy; majority vote over the fixed

160

-completion pool; macro means, mean over three seeds). The pink path follows the single adapter from the untrained base (gray circle) through steps

25/50/125/250

to its final checkpoint, GRPO-single-adapter (red square). These checkpoints are the step-matched pools for

K=16/8/4/2

. Blue triangles are thickets, labeled by

K

. Dashed line: the base model’s vote. Full training moves right but, on three of four models, down below the base. Thickets with

K{\geq}4

keep most of the rightward move where there is one, while staying within

0.8

p.p. of the base’s vote or above it. At

K\leq 4

the matched checkpoints sit at about the thickets’ pass@1, and the vote goes either way. At

K\geq 8

the thickets lie to the right of and above the matched checkpoints on three of four models.

Figure 4: The cost of concentration grows with the vote budget. Mean majority-vote margin on the math suite (p.p.) over the

16

(model,

K

) settings, as every method votes over its first

B

completions in generation order. Each setting’s margin is the difference of displayed three-seed means. Line colour marks the method in front, and markers mark the method subtracted. Red line with gray circles: GRPO-single-adapter

-

base. Blue line with red squares: thicket

-

GRPO-single-adapter. Dashed blue with gray circles: thicket

-

base. Dotted blue with pink diamonds: thicket

-

GRPO-single (matched).

B{=}160

is Table 1 . Table 19 in Appendix F gives the values.

Model	Shards	Majority vote	Filtered vote	Coverage	Voter acc.	$\bar{\rho}$
Qwen2.5-1.5B	Subject	54.2	55.2	89.6	34.4	0.37
	Random	54.8	56.0	89.6	34.4	0.37
Qwen2.5-3B	Subject	62.9	63.4	91.5	44.8	0.44
	Random	63.1	62.7	91.0	44.9	0.44
Qwen2.5-7B	Subject	69.1	69.3	92.7	55.2	0.51
	Random	69.3	69.5	92.1	55.0	0.51

Table 3: Subject-specialist vs. random disjoint shards at

K{=}7

(

184

training problems per adapter in both arms;

7\times 23=161

completions per problem; single seed, seed

42

). Macro means (%) over the five math benchmarks, except

\bar{\rho}

. Bold: better arm per model and column (ties bold both;

\bar{\rho}

is not bolded).

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

LoRA rank / $\alpha$ / dropout	64 / 128 / 0.0	prompts per step	4
LoRA targets	q,k,v,o,gate,up,down proj.	rollouts per prompt	8
learning rate	$2\times 10^{-5}$ (constant, 5% warmup)	rollout temperature	1.0
weight decay / grad. clip	0.01 / 0.1	max prompt / completion	1,024 / 2,048 tok.
KL coefficient	0.0	reward	binary verified
advantage	group-normalized (GRPO)	epochs	1

Appendix

Table 4: Shared GRPO/LoRA hyperparameters. The two training arms differ only in the training file.

Figure 5: The headline comparison at every voter count. Per-model majority-vote accuracy on the math suite (macro mean over five benchmarks, mean over three seeds) for all four methods, at

K\in\{2,4,8,16\}

. The

K{=}16

panel reproduces the right panel of Figure 1 . Number at the end of each red–blue connector: thicket

-

GRPO-single-adapter accuracy.

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	2	4	2	4	2	4	2	4
Qwen2.5-1.5B	MATH500	67.7 $\pm$ 0.3	67.7 $\pm$ 0.3	68.5 $\pm$ 1.3	69.9 $\pm$ 1.0	67.5 $\pm$ 1.7	67.5 $\pm$ 1.7	68.6 $\pm$ 1.0	69.1 $\pm$ 0.6
	GSM8K	86.4 $\pm$ 0.4	86.4 $\pm$ 0.4	84.1 $\pm$ 0.3	85.1 $\pm$ 0.3	82.4 $\pm$ 1.1	82.4 $\pm$ 1.1	84.6 $\pm$ 0.1	85.1 $\pm$ 0.4
	OlympiadBench	32.5 $\pm$ 0.4	32.5 $\pm$ 0.4	33.6 $\pm$ 0.5	33.7 $\pm$ 1.3	31.9 $\pm$ 0.3	31.9 $\pm$ 0.3	34.1 $\pm$ 0.4	34.3 $\pm$ 0.8
	Countdown	39.8 $\pm$ 1.9	39.8 $\pm$ 1.9	51.8 $\pm$ 2.2	54.7 $\pm$ 8.0	42.4 $\pm$ 10.5	42.4 $\pm$ 10.5	53.8 $\pm$ 9.8	47.8 $\pm$ 3.2
	Knights & Knaves	30.1 $\pm$ 0.5	30.1 $\pm$ 0.5	35.0 $\pm$ 4.1	33.3 $\pm$ 1.0	33.7 $\pm$ 1.7	33.7 $\pm$ 1.7	35.1 $\pm$ 1.6	31.7 $\pm$ 0.5

Appendix

Table 5: Per-benchmark majority-vote accuracy (%) at

K\in\{2,4\}

, fixed

160

-vote pools; mean

\pm

std over three seeds. Mean rows are macro means. Bold: best method per row and

K

; underlined: second best.

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	8	16	8	16	8	16	8	16
Qwen2.5-1.5B	MATH500	67.7 $\pm$ 0.3	67.7 $\pm$ 0.3	68.7 $\pm$ 0.2	67.8 $\pm$ 0.2	67.5 $\pm$ 1.7	67.5 $\pm$ 1.7	69.0 $\pm$ 0.9	68.8 $\pm$ 1.0
	GSM8K	86.4 $\pm$ 0.4	86.4 $\pm$ 0.4	86.0 $\pm$ 0.2	86.2 $\pm$ 0.3	82.4 $\pm$ 1.1	82.4 $\pm$ 1.1	85.6 $\pm$ 0.2	85.9 $\pm$ 0.1
	OlympiadBench	32.5 $\pm$ 0.4	32.5 $\pm$ 0.4	32.9 $\pm$ 0.9	32.2 $\pm$ 0.3	31.9 $\pm$ 0.3	31.9 $\pm$ 0.3	33.2 $\pm$ 0.4	33.9 $\pm$ 0.7
	Countdown	39.8 $\pm$ 1.9	39.8 $\pm$ 1.9	48.5 $\pm$ 1.0	45.0 $\pm$ 3.5	42.4 $\pm$ 10.5	42.4 $\pm$ 10.5	53.6 $\pm$ 0.3	49.6 $\pm$ 0.6
	Knights & Knaves	30.1 $\pm$ 0.5	30.1 $\pm$ 0.5	32.7 $\pm$ 0.3	29.7 $\pm$ 1.3	33.7 $\pm$ 1.7	33.7 $\pm$ 1.7	31.7 $\pm$ 2.1	32.6 $\pm$ 1.7

Appendix

Table 6: Per-benchmark majority-vote accuracy (%) at

K\in\{8,16\}

, fixed

160

-vote pools (format of Table 5 ).

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	2	4	2	4	2	4	2	4
Qwen2.5-1.5B	MATH500	69.7 $\pm$ 0.5	69.5 $\pm$ 0.6	69.6 $\pm$ 1.0	71.4 $\pm$ 1.0	68.1 $\pm$ 1.3	67.1 $\pm$ 0.7	70.9 $\pm$ 0.4	71.9 $\pm$ 0.8
	GSM8K	86.4 $\pm$ 0.3	86.4 $\pm$ 0.2	84.1 $\pm$ 0.3	85.0 $\pm$ 0.3	82.4 $\pm$ 1.1	82.4 $\pm$ 1.2	85.0 $\pm$ 0.4	85.0 $\pm$ 0.2
	OlympiadBench	35.9 $\pm$ 0.3	35.4 $\pm$ 0.5	35.4 $\pm$ 1.3	36.2 $\pm$ 1.7	31.9 $\pm$ 1.3	32.5 $\pm$ 0.9	35.4 $\pm$ 1.1	36.6 $\pm$ 0.6
	Countdown	38.0 $\pm$ 2.8	38.2 $\pm$ 2.7	51.3 $\pm$ 3.2	54.7 $\pm$ 8.0	41.7 $\pm$ 10.9	42.4 $\pm$ 10.5	53.8 $\pm$ 9.8	47.8 $\pm$ 3.2
	Knights & Knaves	28.5 $\pm$ 0.6	29.4 $\pm$ 0.6	34.9 $\pm$ 2.3	33.0 $\pm$ 1.3	34.0 $\pm$ 3.0	33.2 $\pm$ 1.5	36.2 $\pm$ 1.5	32.2 $\pm$ 0.4

Appendix

Table 7: Per-benchmark accuracy (%) under confidence-filtered voting, all four methods,

K\in\{2,4\}

; fixed

160

-vote pools; mean

\pm

std over three seeds. Bold: best method per row and

K

; underlined: second best.

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	8	16	8	16	8	16	8	16
Qwen2.5-1.5B	MATH500	69.9 $\pm$ 0.6	69.5 $\pm$ 0.4	70.9 $\pm$ 0.2	70.8 $\pm$ 0.7	67.3 $\pm$ 1.0	67.2 $\pm$ 0.7	71.0 $\pm$ 0.4	70.0 $\pm$ 0.7
	GSM8K	86.1 $\pm$ 0.2	86.3 $\pm$ 0.4	85.8 $\pm$ 0.2	86.3 $\pm$ 0.2	82.4 $\pm$ 1.1	82.4 $\pm$ 1.1	85.3 $\pm$ 0.5	86.3 $\pm$ 0.6
	OlympiadBench	35.8 $\pm$ 0.4	34.7 $\pm$ 0.8	35.2 $\pm$ 0.6	34.5 $\pm$ 0.7	32.6 $\pm$ 1.0	32.8 $\pm$ 1.4	36.6 $\pm$ 1.0	35.3 $\pm$ 0.2
	Countdown	38.8 $\pm$ 2.1	38.1 $\pm$ 3.0	47.6 $\pm$ 2.4	45.0 $\pm$ 3.5	42.4 $\pm$ 10.5	42.4 $\pm$ 10.5	53.6 $\pm$ 0.3	49.6 $\pm$ 0.6
	Knights & Knaves	29.4 $\pm$ 1.1	29.4 $\pm$ 0.8	31.6 $\pm$ 0.6	28.6 $\pm$ 2.6	32.8 $\pm$ 2.6	34.4 $\pm$ 2.1	31.1 $\pm$ 1.6	32.6 $\pm$ 1.7

Appendix

Table 8: Per-benchmark accuracy (%) under confidence-filtered voting,

K\in\{8,16\}

(format of Table 7 ).

Figure 6: Majority-vote accuracy vs. voter count

K

(fixed

160

-vote budget at every

K

): suite macro (top row) and each benchmark, all models (columns). Lines: seed means; bands:

\pm 1

std. Base and GRPO-single-adapter are flat by construction (one

160

-completion pool regrouped at every

K

Model	$K$	MATH500	GSM8K	OlympiadBench	Countdown	Knights & Knaves	Mean
Qwen2.5-1.5B	2	70.9 $\pm$ 0.4	85.0 $\pm$ 0.4	35.4 $\pm$ 1.1	53.8 $\pm$ 9.8	36.2 $\pm$ 1.5	56.3 $\pm$ 1.7
	4	71.9 $\pm$ 0.8	85.0 $\pm$ 0.2	36.6 $\pm$ 0.6	47.8 $\pm$ 3.2	32.2 $\pm$ 0.4	54.7 $\pm$ 0.9
	8	71.0 $\pm$ 0.4	85.3 $\pm$ 0.5	36.6 $\pm$ 1.0	53.6 $\pm$ 0.3	31.1 $\pm$ 1.6	55.5 $\pm$ 0.5
	16	70.0 $\pm$ 0.7	86.3 $\pm$ 0.6	35.3 $\pm$ 0.2	49.6 $\pm$ 0.6	32.6 $\pm$ 1.7	54.8 $\pm$ 0.2
Qwen2.5-3B	2	77.9 $\pm$ 0.3	91.1 $\pm$ 0.6	45.3 $\pm$ 1.4	47.6 $\pm$ 6.1	53.7 $\pm$ 1.3	63.1 $\pm$ 1.7
	4	78.7 $\pm$ 1.1	91.5 $\pm$ 0.2	45.3 $\pm$ 0.6	52.6 $\pm$ 2.6	55.3 $\pm$ 3.7	64.7 $\pm$ 0.4

Appendix

Table 9: Thicket accuracy under confidence-filtered voting (each voter keeps its top-

q

completions by mean log-likelihood,

q

chosen by per-benchmark five-fold CV; fixed

160

-vote pools; mean

\pm

std over three seeds). Mean column: macro over the five benchmarks. Bold: the displayed value improves on the same setting’s plain majority vote.

Model	$K$	MATH500	GSM8K	OlympiadBench	Countdown	Knights & Knaves	Mean
Qwen2.5-1.5B	2	69.7 $\pm$ 0.5	86.4 $\pm$ 0.3	35.9 $\pm$ 0.3	38.0 $\pm$ 2.8	28.5 $\pm$ 0.6	51.7 $\pm$ 0.6
	4	69.5 $\pm$ 0.6	86.4 $\pm$ 0.2	35.4 $\pm$ 0.5	38.2 $\pm$ 2.7	29.4 $\pm$ 0.6	51.8 $\pm$ 0.4
	8	69.9 $\pm$ 0.6	86.1 $\pm$ 0.2	35.8 $\pm$ 0.4	38.8 $\pm$ 2.1	29.4 $\pm$ 1.1	52.0 $\pm$ 0.4
	16	69.5 $\pm$ 0.4	86.3 $\pm$ 0.4	34.7 $\pm$ 0.8	38.1 $\pm$ 3.0	29.4 $\pm$ 0.8	51.6 $\pm$ 0.3
Qwen2.5-3B	2	78.9 $\pm$ 0.4	91.6 $\pm$ 0.5	45.7 $\pm$ 1.3	51.7 $\pm$ 1.5	54.2 $\pm$ 3.4	64.4 $\pm$ 0.4
	4	78.8 $\pm$ 0.4	91.6 $\pm$ 0.2	45.6 $\pm$ 0.9	51.0 $\pm$ 1.5	54.4 $\pm$ 3.0	64.3 $\pm$ 0.1

Appendix

Table 10: Base-model accuracy under confidence-filtered voting (format of Table 9 ; three seeds).

Model	$K$	MATH500	GSM8K	OlympiadBench	Countdown	Knights & Knaves	Mean
Qwen2.5-1.5B	2	68.1 $\pm$ 1.3	82.4 $\pm$ 1.1	31.9 $\pm$ 1.3	41.7 $\pm$ 10.9	34.0 $\pm$ 3.0	51.6 $\pm$ 1.7
	4	67.1 $\pm$ 0.7	82.4 $\pm$ 1.2	32.5 $\pm$ 0.9	42.4 $\pm$ 10.5	33.2 $\pm$ 1.5	51.5 $\pm$ 1.8
	8	67.3 $\pm$ 1.0	82.4 $\pm$ 1.1	32.6 $\pm$ 1.0	42.4 $\pm$ 10.5	32.8 $\pm$ 2.6	51.5 $\pm$ 1.7
	16	67.2 $\pm$ 0.7	82.4 $\pm$ 1.1	32.8 $\pm$ 1.4	42.4 $\pm$ 10.5	34.4 $\pm$ 2.1	51.8 $\pm$ 1.6
Qwen2.5-3B	2	74.4 $\pm$ 1.5	90.0 $\pm$ 0.3	42.3 $\pm$ 1.9	35.7 $\pm$ 18.1	53.2 $\pm$ 5.0	59.1 $\pm$ 5.3
	4	74.3 $\pm$ 2.0	89.8 $\pm$ 0.5	42.3 $\pm$ 2.4	35.7 $\pm$ 18.1	52.5 $\pm$ 5.2	58.9 $\pm$ 5.5

Appendix

Table 11: GRPO-single-adapter accuracy under confidence-filtered voting (format of Table 9 ; three seeds).

Model	$K$	MATH500	GSM8K	OlympiadBench	Countdown	Knights & Knaves	Mean
Qwen2.5-1.5B	2	69.6 $\pm$ 1.0	84.1 $\pm$ 0.3	35.4 $\pm$ 1.3	51.3 $\pm$ 3.2	34.9 $\pm$ 2.3	55.0 $\pm$ 0.0
	4	71.4 $\pm$ 1.0	85.0 $\pm$ 0.3	36.2 $\pm$ 1.7	54.7 $\pm$ 8.0	33.0 $\pm$ 1.3	56.1 $\pm$ 2.0
	8	70.9 $\pm$ 0.2	85.8 $\pm$ 0.2	35.2 $\pm$ 0.6	47.6 $\pm$ 2.4	31.6 $\pm$ 0.6	54.2 $\pm$ 0.4
	16	70.8 $\pm$ 0.7	86.3 $\pm$ 0.2	34.5 $\pm$ 0.7	45.0 $\pm$ 3.5	28.6 $\pm$ 2.6	53.1 $\pm$ 0.7
Qwen2.5-3B	2	77.2 $\pm$ 0.3	90.9 $\pm$ 0.3	44.3 $\pm$ 0.7	46.2 $\pm$ 4.5	54.6 $\pm$ 2.8	62.6 $\pm$ 1.3
	4	76.6 $\pm$ 0.7	91.1 $\pm$ 0.6	44.4 $\pm$ 1.4	50.8 $\pm$ 3.5	56.0 $\pm$ 2.2	63.8 $\pm$ 1.6

Appendix

Table 12: GRPO-single (matched) accuracy under confidence-filtered voting (format of Table 9 ; three seeds).

Figure 7: Most frequently selected CV filter strength per (model, $K$ ) setting and method (mode over $75$ selections: three seeds $\times$ five math benchmarks $\times$ five folds; ties toward the weaker filter, consistent with the selection rule; darker $=$ more aggressive filtering). The most frequent choice is no filtering in $13$ of $16$ settings for GRPO-single-adapter and for GRPO-single (matched), in $9$ for the base model, and in $8$ for the thicket. Because real-filter selections split between ${1}/{2}$ and ${1}/{4}$ , the mode understates how often each pool filters. Across all $1{,}200$ selections per method, the CV picks a real filter in $67\%$ (base), $64\%$ (thicket), $59\%$ (GRPO-single (matched)), and $53\%$ (GRPO-single-adapter) of cases.

Figure 8: Plain majority vote (dashed) vs. confidence-filtered voting (solid) on the math-suite macro at

160

votes, per method (rows) and model (columns); mean of three seeds. By displayed means, the solid curve sits above the dashed one in all

16

settings for the base model and GRPO-single (matched) and in

15

16

for the thicket. For GRPO-single-adapter it is above in

6

settings, level in

4

, and below in

6

. No bands are drawn, because bands of the two curves would hide a gain of about half a point. Tables 9 – 12 give the stds.

Figure 9: Why confidence is worth acting on. Distributions of the per-response mean log-likelihood

\overline{\ell}

for correct (green) vs. incorrect (gray) completions at

K{=}16

, over all

160

completions per problem, pooled over the three seeds and the five math benchmarks (non-null responses; histogram ranges clipped to the

0.5

–

99.5

th percentiles; dashed lines: medians, dark green for correct and gray for incorrect). x-axes differ across panels. Correct completions have the higher median in every method–model pool – the signal confidence-filtered voting exploits. The fully trained GRPO-single-adapter (third row) concentrates its completions much closer to

\overline{\ell}{=}0

than the other pools.

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	2	4	2	4	2	4	2	4
Qwen2.5-1.5B	MATH500	93.5 $\pm$ 0.8	93.5 $\pm$ 0.8	94.0 $\pm$ 0.7	93.9 $\pm$ 0.3	92.7 $\pm$ 1.1	92.7 $\pm$ 1.1	92.8 $\pm$ 0.3	93.7 $\pm$ 0.2
	GSM8K	99.1 $\pm$ 0.1	99.1 $\pm$ 0.1	98.6 $\pm$ 0.2	99.0 $\pm$ 0.2	98.1 $\pm$ 0.3	98.1 $\pm$ 0.3	98.9 $\pm$ 0.3	98.9 $\pm$ 0.1
	OlympiadBench	67.9 $\pm$ 0.4	67.9 $\pm$ 0.4	69.1 $\pm$ 0.3	68.4 $\pm$ 0.1	66.3 $\pm$ 0.9	66.3 $\pm$ 0.9	68.2 $\pm$ 1.0	69.3 $\pm$ 0.9
	Countdown	87.8 $\pm$ 1.0	87.8 $\pm$ 1.0	86.2 $\pm$ 2.7	85.1 $\pm$ 2.1	81.6 $\pm$ 4.5	81.6 $\pm$ 4.5	84.2 $\pm$ 4.3	85.3 $\pm$ 1.8
	Knights & Knaves	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	99.9 $\pm$ 0.2	100.0 $\pm$ 0.0

Appendix

Table 13: Per-benchmark coverage (pass@pool %) on the math suite,

K\in\{2,4\}

; mean

\pm

std over three seeds. Bold: best method per row and

K

; underlined: second best.

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	8	16	8	16	8	16	8	16
Qwen2.5-1.5B	MATH500	93.5 $\pm$ 0.8	93.5 $\pm$ 0.8	94.1 $\pm$ 0.4	92.5 $\pm$ 0.4	92.7 $\pm$ 1.1	92.7 $\pm$ 1.1	93.9 $\pm$ 0.5	93.4 $\pm$ 0.2
	GSM8K	99.1 $\pm$ 0.1	99.1 $\pm$ 0.1	99.1 $\pm$ 0.2	99.1 $\pm$ 0.1	98.1 $\pm$ 0.3	98.1 $\pm$ 0.3	98.8 $\pm$ 0.2	98.8 $\pm$ 0.2
	OlympiadBench	67.9 $\pm$ 0.4	67.9 $\pm$ 0.4	68.9 $\pm$ 0.1	68.6 $\pm$ 0.6	66.3 $\pm$ 0.9	66.3 $\pm$ 0.9	68.7 $\pm$ 0.1	69.7 $\pm$ 0.8
	Countdown	87.8 $\pm$ 1.0	87.8 $\pm$ 1.0	88.0 $\pm$ 1.3	87.6 $\pm$ 2.7	81.6 $\pm$ 4.5	81.6 $\pm$ 4.5	86.4 $\pm$ 2.0	87.1 $\pm$ 1.9
	Knights & Knaves	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	99.9 $\pm$ 0.2	100.0 $\pm$ 0.0	100.0 $\pm$ 0.0	99.9 $\pm$ 0.2	100.0 $\pm$ 0.0

Appendix

Table 14: Per-benchmark coverage (pass@pool %) on the math suite,

K\in\{8,16\}

(format of Table 13 ).

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	2	4	2	4	2	4	2	4
Qwen2.5-1.5B	MBPP	93.4 $\pm$ 0.0	93.4 $\pm$ 0.0	90.6 $\pm$ 2.4	92.4 $\pm$ 1.0	88.1 $\pm$ 1.6	88.1 $\pm$ 1.6	91.4 $\pm$ 0.9	92.7 $\pm$ 0.3
	HumanEval	96.7 $\pm$ 0.9	96.7 $\pm$ 0.9	92.5 $\pm$ 3.0	93.9 $\pm$ 0.6	88.8 $\pm$ 1.4	88.8 $\pm$ 1.4	92.9 $\pm$ 1.4	93.9 $\pm$ 0.6
	LeetCode	26.5 $\pm$ 0.3	26.5 $\pm$ 0.3	24.0 $\pm$ 3.7	24.4 $\pm$ 2.1	20.5 $\pm$ 2.6	20.5 $\pm$ 2.6	23.4 $\pm$ 2.8	25.1 $\pm$ 1.1
	Mean	72.2 $\pm$ 0.4	72.2 $\pm$ 0.4	69.0 $\pm$ 1.8	70.2 $\pm$ 0.9	65.8 $\pm$ 1.4	65.8 $\pm$ 1.4	69.2 $\pm$ 1.4	70.6 $\pm$ 0.6
Qwen2.5-3B	MBPP	95.2 $\pm$ 0.5	95.2 $\pm$ 0.5	93.0 $\pm$ 2.8	92.9 $\pm$ 1.8	90.8 $\pm$ 1.1	90.8 $\pm$ 1.1	92.6 $\pm$ 1.3	93.1 $\pm$ 1.8

Appendix

Table 15: Per-benchmark coverage (pass@pool %) on the code suite,

K\in\{2,4\}

; mean

\pm

std over three seeds. Bold: best method per row and

K

; underlined: second best.

		Base		GRPO-single (matched)		GRPO-single-adapter		Thicket
Model	Benchmark ( $K{=}$ )	8	16	8	16	8	16	8	16
Qwen2.5-1.5B	MBPP	93.4 $\pm$ 0.0	93.4 $\pm$ 0.0	93.8 $\pm$ 0.1	93.7 $\pm$ 0.8	88.1 $\pm$ 1.6	88.1 $\pm$ 1.6	93.3 $\pm$ 0.1	93.0 $\pm$ 0.5
	HumanEval	96.7 $\pm$ 0.9	96.7 $\pm$ 0.9	96.7 $\pm$ 0.7	96.1 $\pm$ 1.5	88.8 $\pm$ 1.4	88.8 $\pm$ 1.4	94.1 $\pm$ 1.4	93.5 $\pm$ 0.9
	LeetCode	26.5 $\pm$ 0.3	26.5 $\pm$ 0.3	25.6 $\pm$ 0.9	25.9 $\pm$ 0.4	20.5 $\pm$ 2.6	20.5 $\pm$ 2.6	24.3 $\pm$ 0.3	24.6 $\pm$ 1.5
	Mean	72.2 $\pm$ 0.4	72.2 $\pm$ 0.4	72.0 $\pm$ 0.5	71.9 $\pm$ 0.3	65.8 $\pm$ 1.4	65.8 $\pm$ 1.4	70.6 $\pm$ 0.5	70.3 $\pm$ 0.3
Qwen2.5-3B	MBPP	95.2 $\pm$ 0.5	95.2 $\pm$ 0.5	94.8 $\pm$ 0.5	95.6 $\pm$ 0.2	90.8 $\pm$ 1.1	90.8 $\pm$ 1.1	94.2 $\pm$ 0.9	95.4 $\pm$ 0.7

Appendix

Table 16: Per-benchmark coverage (pass@pool %) on the code suite,

K\in\{8,16\}

(format of Table 15 ).

Figure 10: Coverage vs. voter count

K

on the math suite (fixed

160

-completion pools). Lines: seed means; bands:

\pm 1

std. Base and GRPO-single-adapter are flat by construction. In the Knights & Knaves row, coverage saturates at

100\%

and coinciding lines hide one another.

Figure 11: Coverage vs. voter count

K

on the code suite (fixed

160

-completion pools). Lines: seed means; bands:

\pm 1

std. Base and GRPO-single-adapter are flat by construction.

	What full RLVR does		Thicket margin (MV, p.p.)
Model	$H$ (pass@1)	$\Delta\bar{\rho}$	thicket $-$ base	base $-$ GRPO-single-adapter	$G$
Qwen2.5-1.5B	+7.3	+0.13	+2.9	$-$ 0.3	+2.6
Qwen2.5-3B	+1.0	+0.07	+0.3	+4.8	+5.1
Qwen2.5-7B	+0.7	+0.05	$-$ 0.5	+1.7	+1.2
Llama-3.1-8B	+9.7	+0.19	+1.4	+4.1	+5.5

Appendix

Table 17: Decomposing the thicket’s vote margin over GRPO-single-adapter at

K{=}16

and

160

votes (math macro, mean over three seeds).

G=(\text{thicket}-\text{base})+(\text{base}-\text{GRPO-single-adapter{}})

: the RLVR gain the thicket retains, plus the damage that concentrated training inflicts and the thicket avoids.

H

and

\Delta\bar{\rho}

are the pass@1 and error-correlation changes that full single-adapter training causes relative to the base.

Figure 12: Voter accuracy vs. error correlation at

K{=}16

(per-voter macro means over each suite’s benchmarks, voters of all three seeds pooled; large markers are method means). The fully trained GRPO-single-adapter moves up and far to the right of the base model. The step-matched control stays close to the base model in both correlation and accuracy. Thicket members add less correlation than GRPO-single-adapter in every panel, and gain accuracy over the base model in all but Qwen2.5-7B math (

-0.1

		Voter accuracy (%)				Error correlation $\bar{\rho}$
Suite	Model	Base	Matched	Single	Thicket	Base	Matched	Single	Thicket
Math	Qwen2.5-1.5B	28.9 $\pm$ 0.1	31.1 $\pm$ 0.1	36.2 $\pm$ 0.5	34.0 $\pm$ 0.1	0.30 $\pm$ 0.00	0.32 $\pm$ 0.00	0.43 $\pm$ 0.01	0.37 $\pm$ 0.00
	Qwen2.5-3B	44.1 $\pm$ 0.1	44.1 $\pm$ 0.2	45.1 $\pm$ 0.8	44.8 $\pm$ 0.1	0.42 $\pm$ 0.00	0.42 $\pm$ 0.00	0.49 $\pm$ 0.01	0.43 $\pm$ 0.00
	Qwen2.5-7B	55.2 $\pm$ 0.1	55.2 $\pm$ 0.0	55.9 $\pm$ 0.4	55.1 $\pm$ 0.0	0.51 $\pm$ 0.00	0.51 $\pm$ 0.00	0.56 $\pm$ 0.01	0.51 $\pm$ 0.00
	Llama-3.1-8B	29.2 $\pm$ 0.0	30.0 $\pm$ 0.2	38.9 $\pm$ 0.2	35.3 $\pm$ 0.0	0.27 $\pm$ 0.00	0.28 $\pm$ 0.00	0.46 $\pm$ 0.01	0.34 $\pm$ 0.00
Code	Qwen2.5-1.5B	34.7 $\pm$ 0.1	34.9 $\pm$ 0.2	41.0 $\pm$ 1.5	37.5 $\pm$ 0.2	0.45 $\pm$ 0.00	0.46 $\pm$ 0.00	0.65 $\pm$ 0.03	0.50 $\pm$ 0.01

Appendix

Table 18: Mean voter accuracy (single completion, %) and mean pairwise error correlation

\bar{\rho}

K{=}16

; macro means over each suite’s benchmarks, mean

\pm

std over three seeds. Matched: GRPO-single (matched); Single: GRPO-single-adapter.

Figure 13: Complete plot set, math suite: per-voter accuracy vs.

\bar{\rho}

for

K\in\{2,4,8,16\}

(rows) and all models (columns); voters of all three seeds pooled; large markers are method means.

Figure 14: Complete plot set, code suite (protocol as in Figure 13 ).

Mean margin (p.p.) over the 16 (model, $K$ ) settings (settings with a positive margin)
	Votes per problem $B$ (first $B$ responses of each method)
	16	32	48	64	96	128	160
GRPO-single-adapter $-$ Base	+0.15 (8)	$-$ 0.40 (8)	$-$ 1.08 (4)	$-$ 1.43 (4)	$-$ 1.93 (0)	$-$ 2.28 (4)	$-$ 2.58 (4)
Thicket $-$ GRPO-single-adapter	+1.33 (15)	+1.68 (16)	+2.01 (16)	+2.25 (16)	+2.86 (16)	+3.26 (16)	+3.28 (16)
Thicket $-$ Base	+1.48 (13)	+1.28 (13)	+0.93 (10)	+0.83 (8)	+0.94 (11)	+0.98 (10)	+0.71 (9)
Thicket $-$ GRPO-single (matched)	+0.56 (11)	+0.47 (11)	+0.28 (10)	+0.28 (9)	+0.23 (9)	+0.34 (11)	+0.27 (10)

Appendix

Table 19: The vote budget (math majority vote over each method’s first

B

completions; three-seed means). Upper rows: mean margin (p.p.) over the

16

(model,

K

) settings, each the difference of displayed means, with the number of settings where it is positive in parentheses. Lower rows: each method’s majority vote, averaged over the

16

settings,

\pm

the std over seeds of that average.

B{=}160

is Table 1 , and

B{=}16

is Table 20 . Figure 4 plots all ten budgets. Base and GRPO-single-adapter do not depend on

K

, so the count for GRPO-single-adapter

-

Base moves in steps of four (one model at a time).

Model	Method	$K{=}2$	$K{=}4$	$K{=}8$	$K{=}16$
Qwen2.5-1.5B	Base	44.6 $\pm$ 0.3	44.6 $\pm$ 0.3	44.6 $\pm$ 0.3	44.6 $\pm$ 0.3
	GRPO-single (matched)	47.9 $\pm$ 1.4	46.6 $\pm$ 1.7	46.1 $\pm$ 0.4	45.0 $\pm$ 0.1
	GRPO-single-adapter	46.0 $\pm$ 1.7	46.0 $\pm$ 1.7	46.0 $\pm$ 1.7	46.0 $\pm$ 1.7
	Thicket	48.1 $\pm$ 1.0	47.5 $\pm$ 0.9	46.9 $\pm$ 0.6	46.4 $\pm$ 1.0
Qwen2.5-3B	Base	56.9 $\pm$ 0.9	56.9 $\pm$ 0.9	56.9 $\pm$ 0.9	56.9 $\pm$ 0.9
	GRPO-single (matched)	56.9 $\pm$ 1.0	57.7 $\pm$ 0.8	57.8 $\pm$ 0.7	56.6 $\pm$ 0.9

Appendix

Table 20: Majority-vote accuracy on the math suite at

16

votes (canonical-class votes; macro means (%) over five benchmarks; mean

\pm

std over three seeds). A thicket member contributes its first

{16}/{K}

completions. Every other method contributes the first

16

completions of its pool, so the base and GRPO-single-adapter rows are constant across

K

by construction. Bold: best method per model and

K

by displayed mean (ties bold both); underlined: second best.

		Coverage, math				Coverage, code
Model	Method ( $K{=}$ )	2	4	8	16	2	4	8	16
Qwen2.5-1.5B	Base	73.1 $\pm$ 0.4	73.1 $\pm$ 0.4	73.1 $\pm$ 0.4	73.1 $\pm$ 0.4	61.3 $\pm$ 1.1	61.3 $\pm$ 1.1	61.3 $\pm$ 1.1	61.3 $\pm$ 1.1
	GRPO-single (matched)	74.7 $\pm$ 0.8	74.4 $\pm$ 1.5	73.7 $\pm$ 0.7	73.3 $\pm$ 0.5	61.4 $\pm$ 2.2	61.4 $\pm$ 1.1	62.5 $\pm$ 0.8	61.9 $\pm$ 0.9
	GRPO-single-adapter	72.8 $\pm$ 1.5	72.8 $\pm$ 1.5	72.8 $\pm$ 1.5	72.8 $\pm$ 1.5	58.2 $\pm$ 0.8	58.2 $\pm$ 0.8	58.2 $\pm$ 0.8	58.2 $\pm$ 0.8
	Thicket	74.5 $\pm$ 0.6	74.2 $\pm$ 0.8	74.3 $\pm$ 0.6	73.9 $\pm$ 0.7	61.4 $\pm$ 1.9	62.1 $\pm$ 1.1	62.4 $\pm$ 0.7	63.5 $\pm$ 0.3
Qwen2.5-3B	Base	80.6 $\pm$ 0.0	80.6 $\pm$ 0.0	80.6 $\pm$ 0.0	80.6 $\pm$ 0.0	68.1 $\pm$ 0.5	68.1 $\pm$ 0.5	68.1 $\pm$ 0.5	68.1 $\pm$ 0.5

Appendix

Table 21: Coverage at

16

votes (at least one correct completion among the

16

pooled, with the first-

16

convention of Table 20 ) on the math and code suites; macro means (%); mean

\pm

std over three seeds. Bold: best method per model, suite, and

K

; underlined: second best.

Figure 15: Majority-vote accuracy (math macro, mean over three seeds) as each method votes over its first

B

completions,

B=16,32,\dots,160

, per voter count

K

(line shade) and method (rows). The base-model and GRPO-single-adapter curves are identical at every

K

, so only the

K{=}16

curve is visible in their rows. No bands are drawn here. Figure 16 draws

\pm 1

std bands.

Figure 16: The same curves as Figure 15 , with the four methods overlaid in each panel, per voter count

K

(rows) and model (columns). Bands:

\pm 1

std over seeds.

		Majority vote		Filtered vote
Model	Benchmark	Subject	Random	Subject	Random
Qwen2.5-1.5B	MATH500	69.4	69.4	71.3	71.5
	GSM8K	86.0	85.8	85.5	85.8
	OlympiadBench	32.6	33.6	35.4	37.8
	Countdown	49.3	49.8	49.3	49.8
	Knights & Knaves	33.8	35.2	34.4	35.2

Appendix

Table 22: Per-benchmark majority vote and confidence-filtered vote (%) for subject-specialist vs. random disjoint shards at

K{=}7

; single seed (

42

). Bold: better arm per row and vote type (ties bold both).

Explore similar work

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
Jul 28, 2026Pixel Nomand, Elena Voss, Marcus Hale +1Reinforcement Learning With Verifiable Reward Token Budget Allocation
University of Wisconsin–Madison · University of Washington
Diversifying RLVR Rollouts via First-Token Exploration
May 27, 2026Soeun Kim, Albert NoReinforcement Learning With Verifiable Reward Verifiable Rewards
Department of Artificial Intelligence, Yonsei University
When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR
May 19, 2026Yuchun Miao, Sen Zhang, Yuqi Zhang +4Reinforcement Learning With Verifiable Reward Verifiable Rewards
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University · Meituan Longcat Team · The University of Sydney +1

Jul 28, 2026, cs.LGJ/K move · Enter open · S save

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes

University of Wisconsin–Madison · University of Washington

Reinforcement Learning With Verifiable Reward Token Budget Allocation Policy Gradient Verdict

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).

Abstract

Explore similar work

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Diversifying RLVR Rollouts via First-Token Exploration

When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR