Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL

Dylan P. Wilson; Chengsong Huang; Jiaxin Huang

cs.LGOct 14, 2026

Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL

Authors: Dylan P. Wilson, Chengsong Huang, Jiaxin Huang

Organizations: Dartmouth College · Washington University in St. Louis

Abstract

Reinforcement learning with self-generated labels, such as majority vote over a model's own samples, often recovers most of the gain of training with verifiable rewards. This is puzzling from the usual prompt-level view, in which majority vote only returns a model's modal answer and cannot add information about correctness. We argue that the relevant unit is the batch-level update. Because parameters are shared, a prompt's own gradient accounts for less than 1% of the change in that prompt's output at practical batch sizes, so a wrong label matters only to the extent that label errors share a direction across prompts. We decompose the majority-vote update into shrinkage and misdirection relative to the gold update, and show that misdirection vanishes with batch size whenever errors are incoherent across prompts, however often the labels are wrong. Measuring per-response gradients on five models, we find that majority-vote errors are as incoherent as a random-reward null, that the majority-vote update agrees with the gold update as closely as two independent gold updates agree with each other, and that its noise-corrected cosine with the gold update is 1 within error. Randomly flipping up to 30% of gold labels leaves the update direction intact, while the same fraction flipped by a shared rule destroys it, and rewards that favor a kind of response on every prompt, such as a learned reward model, fall measurably short. The equivalence holds at every checkpoint we examine but is local: gold-reward training still ends ahead on held-out majority accuracy. Error coherence, not error rate, determines when self-generated labels can stand in for verification.

Figures & tables

		Split-half cosine		Expected update
Model	$N$	Gold vs. gold (ceiling)	MV vs. gold	$c_{\infty}$ (95% CI)
Qwen3-4B-Base †	331	0.647	0.647	1.006 (.979, 1.023)
Qwen3-4B-Base	164	0.606	0.600	0.999 (.965, 1.043)
Qwen3-1.7B-Base	636	0.842	0.834	0.998 (.988, 1.006)
Qwen2.5-7B	151	0.799	0.818	1.010 (.968, 1.021)
Llama-3.2-3B-Instruct	247	0.371	0.360	0.997 (.921, 1.072)

Table 1: MV is as close to gold as gold is to itself. We split the prompt groups into two disjoint halves and compare summed gradients across the halves. Gold vs. gold is the ceiling: how well two independent gold updates agree. MV vs. gold pairs the MV update on one half with the gold update on the other; MV is indistinguishable from gold when the two columns match.

c_{\infty}

removes the dependence on the number of prompt groups

N

by estimating the cosine between the expected MV and gold updates;

c_{\infty}=1

means they point in the same direction. Table 12 gives the full results.

	Right majority			Wrong majority
Checkpoint	Gold	MV error $E$	Null	Gold	MV error $E$	Null
Base	22.41	0.39	0.30	0 6.55	0.23	0.69
Step 10	0 6.52	0.01	0.36	0 0.82	0.15	0.50
Step 20	27.99	0.14	$-$ 0.15	13.44	0.55	0.66

Table 2: MV errors are as incoherent as a random reward. Mean pairwise cosine between prompt gradients (

\times 10^{-3}

); a positive value means the gradients share a direction. Gold gradients share one, while the MV error

E

stays at the level of a Gaussian random reward (null). Qwen3-4B-Base along the MV run, split by whether the base model’s majority answer is right or wrong. Because this majority is computed on separate samples, it can differ from the majority within a 16-response training group, so

E

is not exactly zero on right-majority prompts. Checkpoints use different prompt draws; Table 13 adds the MV gradients.

Independent noise: gold labels flipped at random with probability $\varepsilon$
Reward	Wrong (%)	$\rho_{e}$ ( $\times 10^{-3}$ )	$c_{\infty}$ (95% CI)
$\varepsilon=0.05$	0 5.5	0 1.4	1.000 (.979, 1.019)
$\varepsilon=0.10$	0 9.8	0 0.8	1.003 (.977, 1.026)
$\varepsilon=0.20$	20.4	0 1.2	1.011 (.969, 1.059)
$\varepsilon=0.30$	30.8	0 0.2	1.036 ‡
Shared-rule noise: the same fraction flipped by response length

Table 3: Error structure, not error rate, sets the update direction. Disattenuated cosine

c_{\infty}

between the expected update of each reward and the expected gold update, all on the same Qwen3-1.7B-Base responses at initialization (

N=636

prompt groups). A value of 1 means the reward moves the model in the gold direction; an interval below 1 means its errors add a systematic deviation. Wrong is the percentage of responses whose reward differs from gold;

\rho_{e}

is the mean pairwise alignment of the per-prompt errors (

\times 10^{-3}

). Random vote rewards a randomly chosen response’s answer; self-certainty vote the answer with the highest aggregated self-certainty.

Checkpoint	Cosine	Shrinkage $\beta$	Misdirection $R_{\perp}$	Error/signal	Ceiling
Base	0.927	$-$ 0.055	0.428	0.432	0.647 †
Step 10	0.810	$-$ 0.143	0.621	0.637	0.265
Step 20	0.972	$-$ 0.020	0.239	0.240	0.658

Table 4: The MV update stays close to the gold update on the same batch. Qwen3-4B-Base along the MV run. Writing

G_{\mathrm{mv}}=(1-\beta)G_{\mathrm{gold}}+E_{\perp}

(Section 4 ), shrinkage

\beta

is the fraction of the gold step that MV does not deliver (negative: MV moves further along the gold direction than gold does), and misdirection

R_{\perp}=\|E_{\perp}\|/\|G_{\mathrm{gold}}\|

is the size of the part pointing elsewhere. Cosine is

\cos(G_{\mathrm{mv}},G_{\mathrm{gold}})

on the same batch, error/signal is

\|\sum E\|/\|\sum G_{\mathrm{gold}}\|

, and ceiling is the split-half gold vs. gold cosine, which is anomalously low at step 10.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model and data
Setting	Value
Base model	Qwen3-4B-Base
Training data	DeepScaleR (29k problems)
Framework	verl 0.8.0 ( Sheng et al., 2025 )
Training steps	26
Batch

Appendix

Table 5: Training configuration shared by the gold and MV runs.

	Base	Step 20
Mixed-outcome prompts (of 256)	125	165
wrong- / right-majority	41 / 84	81 / 84
$\cos(G_{\mathrm{mv}},G_{\mathrm{gold}})$	0.898	0.972
$\\|G_{\mathrm{mv}}\\|/\\|G_{\mathrm{gold}}\\|$	1.092	1.009
Shrinkage $\beta$	$+0.019$	$+0.020$
Misdirection $R_{\perp}$	0.480	0.239

Appendix

Table 6: Same-prompt replication (Section 5 ). Base and step 20 evaluated on the identical 256 step-21 prompts, each scored on its own on-policy rollouts. Shrinkage is unchanged while misdirection halves, and the two reproduce the measured batch cosine. Steps 5 and 15 are absent because the run does not checkpoint there.

Model	$N$	Mean-centered	Std-normalized
Qwen3-1.7B-Base	636	0.997 (.989, 1.006)	0.994 (.982, 1.008)
Qwen3-4B-Base (on-policy)	164	0.999 (.965, 1.043)	1.000 (.968, 1.034)
Qwen3-4B-Base step 20	165	1.002 (.981, 1.015)	1.002 (.985, 1.013)
Qwen2.5-7B	151	0.998 (.968, 1.027)	0.996 (.962, 1.030)
Llama-3.2-3B-Instruct	247	0.996 (.929, 1.078)	0.998 (.911, 1.096)
Llama-3.1-8B-Instruct	208	1.007 (.944, 1.079)	1.017 (.929, 1.119)

Appendix

Table 7: Std-normalized advantages give the same answer.

c_{\infty}

between the MV and gold updates with advantages mean-centered (

s_{i}=1

) and with

s_{i}

set to the group standard deviation, the weighting the training loss uses, computed from the same stored sketches. Step 10 is omitted: its own split-half agreement is

Z\approx 0.25

, below the threshold at which

c_{\infty}

is reported.

Run	maj@1	maj@16	maj@64	pass@64
MV, seed A	0.325	0.525	0.545	0.858
MV, seed B	0.314	0.505	0.537	0.866
Gold, seed A	0.348	0.550	0.585	0.843
Gold, seed B	0.339	0.557	0.595	0.866
Within-reward spread (MV)	0.011	0.019	0.007	0.007
Within-reward spread (gold)	0.008	0.007	0.010	0.022

Appendix

Table 8: Run-to-run variation is comparable to the gold–MV gap (Section 5 ). Two MV runs and two gold runs, trained from different seeds, all evaluated at step 10 on the same 134 level-5 problems with 64 samples each. maj@

k

for

k<64

averages over random

k

-subsets drawn without replacement; at

k=64

it uses the whole pool. Answers are clustered and graded exactly as the evaluation harness does, including its treatment of ties, so these numbers are directly comparable with those quoted in Section 5 .

Step	1	8	10	15	20	26
Parse rate	0.668	0.843	0.840	0.811	0.748	0.682
Majority share	0.382	0.408	0.420	0.465	0.511	0.548
Maj. correct	0.344	0.348	0.398	0.352	0.316	0.367

Appendix

Table 9: Answer parsing along the gold run of Qwen3-4B-Base, over the 4,096 responses generated at each step. Parse rate is the fraction of responses with an extractable answer, majority share the fraction of parsed responses holding the group’s plurality answer, and maj. correct the fraction of groups whose plurality answer matches the reference.

Method	Source	Settings	Range	Mean recovery
Online MV
TTRL †	Zuo et al. (2026)	2	109–111%	110%
Plurality-vote GRPO	Zhao et al. (2026b)	2	0 69–98%	0 83%
SRT	Shafayat et al. (2025)	4	0 55–100%	0 81%
Modified label-free rewards
RESTRAIN	Yu et al. (2026)	2	0 81–97%	0 89%

Appendix

Table 10: Label-free methods recover most of the gain of ground-truth RL. Recovery is

(\text{method}-\text{base})/(\text{GT-RL}-\text{base})

, computed from benchmark-averaged scores reported within the same paper; values are means over settings, with per-setting results in Table 16 . For MV, only online variants (labels from the current policy) are included.

		Growth exponent $\alpha(B)$
Checkpoint	$B$ range	MV error $E$	Gold	Gap
Base	00 8 $\,\to\,$ 16	0.524	0.581	0.057
	32 $\,\to\,$ 64	0.500	0.720	0.220
	64 $\,\to\,$ 128	0.524	0.813	0.289
	128 $\,\to\,$ 164	0.509	0.874	0.365
Step 20	16 $\,\to\,$ 32	0.506	0.684	0.179

Appendix

Table 11: Errors grow like a random walk while the gold signal grows coherently. Local growth exponents

\alpha(B)

of summed gradients between successive batch sizes, Qwen3-4B-Base. By Proposition 1 ,

\alpha=1/2

indicates incoherent vectors and

\alpha\to 1

a dominant shared direction.

Model	$N$	Gold vs. gold	MV vs. gold	MV vs. MV	$\Delta$ (95% CI)
Qwen3-4B-Base †	331	0.647	0.647	0.646	( $-$ .014, $+$ .014)
Qwen3-1.7B-Base	636	0.842	0.834	0.831	( $-$ .016, $-$ .003)
Qwen2.5-7B	151	0.799	0.818	0.843	( $+$ .010, $+$ .044)
Llama-3.2-3B-Instruct	247	0.371	0.360	0.352	( $-$ .022, $+$ .010)
Llama-3.1-8B-Instruct	208	0.438	0.399	0.362	( $-$ .050, $-$ .016)

Appendix

Table 12: Full split-half results behind Table 1 .

\Delta

is the 95% interval of (MV vs. gold) minus (gold vs. gold); an interval containing zero means MV cannot be distinguished from gold. MV vs. MV is the agreement of MV with itself across halves.

	Right majority				Wrong majority
Checkpoint	Gold	MV	MV error $E$	Null	Gold	MV	MV error $E$	Null
Base	22.41	20.06	0.39	0.30	0 6.55	0 9.26	0.23	0.69
Step 10	0 6.52	0 6.52	0.01	0.36	0 0.82	0 2.79	0.15	0.50
Step 20	27.99	26.94	0.14	$-$ 0.15	13.44	14.71	0.55	0.66

Appendix

Table 13: Full version of Table 2 , adding the MV gradients themselves. Mean pairwise cosine between prompt gradients (

\times 10^{-3}

), Qwen3-4B-Base along the MV run.

(a) Majority-vote accuracy
	Base	Step 20	Change (95% CI)
maj@1	0.185	0.374	$+$ 0.188 ( $+$ .158, $+$ .218) ∗
maj@4	0.292	0.466	$+$ 0.174 ( $+$ .145, $+$ .206) ∗
maj@16	0.443	0.543	$+$ 0.100 ( $+$ .066, $+$ .136) ∗
maj@64	0.503	0.585	$+$ 0.082 ( $+$ .030, $+$ .138) ∗
(b) Majority-answer flips, base $\rightarrow$ step 20

Appendix

Table 14: MV training repairs wrong majorities on held-out problems. 134 level-5 MATH-500 problems, base model against step 20 of the MV run (66 wrong-majority and 68 right-majority problems at base); majorities use all 64 samples. (a) maj@

k

and its change, with a 95% bootstrap interval over problems. (b) Problems whose majority answer changes; the resampling null is the flip rate between two disjoint halves of 32 samples at step 20. ∗ The interval excludes zero in (a) and the resampling null in (b).

Model	Raw	Length-controlled (95% CI)
Qwen3-4B-Base †	0.990	0.973 (.934, 1.029)
Qwen3-4B-Base (on-policy)	0.950	0.941 (.871, .993)
Qwen3-1.7B-Base	0.902	0.969 (.955, .971)
Qwen2.5-7B	0.536	0.959 (.926, 1.017)
Llama-3.2-3B-Instruct	0.855	0.786 (.727, .862)
Llama-3.1-8B-Instruct	0.883	0.756 (.738, .825)

Appendix

Table 15: INTUITOR’s alignment with gold depends on the model. Disattenuated cosine

c_{\infty}

between the INTUITOR and gold updates at initialization, raw and after controlling for response length.

Zuo et al. (2026) — GT-RL = Oat-Zero (same backbone), greedy decoding
Method	Model	Train set	Benchmarks averaged	Base	GT-RL	Method	Recovery
TTRL	Qwen2.5-Math-1.5B	test set	AIME24, AMC, MATH-500	28.5	49.1	51.0	109%
TTRL	Qwen2.5-Math-7B	test set	AIME24, AMC, MATH-500	35.3	62.0	65.0	111%
Zhao et al. (2026b) — GT-RL = GRPO with gold answers, greedy decoding
Plurality-vote GRPO	Qwen2.5-3B	MATH	GSM8K, MATH-500	60.9	73.1	72.8	98%
Plurality-vote GRPO	Llama-3.2-3B-Inst	MATH	GSM8K, MATH-500	56.2	60.4	59.1	69%

Appendix

Table 16: Per-setting numbers behind Table 10 . Scores are the average over the listed benchmarks (pass@1, %), including GSM8K ( Cobbe et al., 2021 ) . Recovery

=(\text{Method}-\text{Base})/(\text{GT-RL}-\text{Base})

. SRT values are read off Shafayat et al. (2025) , Fig. 3 (final step), and are approximate.

	Coherence $\cos(v_{t},v_{s})$	Growth exponent
MV error $E_{t}$	0.030	0.594
Gold gradient	0.400	0.762
Random reward (floor)	0.001	–
Sign-flip reference ( $\sqrt{T}$ null)	–	0.540 (.522, .576)

Appendix

Table 17: The per-step MV errors do not share a direction (Section 4.4 ). Qwen3-4B-Base along the MV run, each checkpoint sketched on its own on-policy rollouts. Coherence is the mean cosine over the ten step pairs

t\neq s

; the random reward is evaluated through the same pipeline and its expected update is the zero vector, so it gives the floor. Exponent is the log–log slope of

\|\sum_{t\leq T}v_{t}\|

T

. The sign-flip reference randomises the sign of each

E_{t}

, preserving all norms while destroying any consistent direction, and so is the

\sqrt{T}

null for these data.

Step size	$d_{\text{own}}$	$d_{\text{other}}$	Own/other
$10^{-7}$	$7.18\times 10^{-6}$	$3.70\times 10^{-6}$	1.94
$10^{-8}$	$7.23\times 10^{-8}$	$4.19\times 10^{-8}$	1.72

Appendix

Table 18: A prompt’s own gradient barely outweighs another prompt’s. Change in the output distribution of prompt

q

after a small step along

q

’s own gradient (

d_{\text{own}}

) or along another prompt’s gradient (

d_{\text{other}}

), measured on identical tokens (Qwen3-1.7B-Base, 8 prompts, fp64). A fully tabular model would give own/other

\to\infty

; a fully shared one would give 1.

Model	$B{=}8$	$B{=}16$	$B{=}64$	$B{=}128$	KL-share slope	Shared energy	$B^{\ast}$
	Own share (%)				Scaling
Qwen3-1.7B-Base	0 9.81	0 3.37	0.38	0.13	$-$ 1.37	2.27%	43
Qwen3-4B-Base	0 3.36	0 0.86	0.18	0.09	$-$ 1.31	2.63%	37
Llama-3.2-3B-Instruct	29.74	25.69	0.83	–	$-$ 1.15	1.49%	66

Appendix

Table 19: A single prompt’s contribution shrinks with batch size. Own share :

d_{\text{own}}/d_{\text{batch}}

, the fraction of the change in a prompt’s output caused by its own gradient within a batch of size

B

(step size

10^{-7}

). Scaling , computed over every prompt group from the gradient sketches: the KL-share slope is the log–log slope of the KL change from swapping one prompt against

B

(i.i.d. theory:

-2

); writing

g_{q}=\mu+\xi_{q}

, the shared energy is

\|\mu\|^{2}

as a fraction of the total, and

B^{\ast}

is the batch size at which the shared component overtakes the prompt-specific one.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR
Apr 28, 2026Xinjie Chen, Biao Fu, Jing Wu +4RL for Language Model Reasoning Mathematical Reasoning
Tongyi Lab, Alibaba Group, Hangzhou, China
CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning
Aug 10, 2026Ambuj Mehrish, Sebastiano VasconReward Modeling RL for Language Models
CVML Lab Ca’ Foscari University of Venice
Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards
May 27, 2026Saurabh Dash, Pierre Clavier, John Dang +4Reinforcement Learning Instruction Following
Cohere Labs · Cohere · Adaption Labs +1

Apr 28, 2026J/K move · Enter open · S save

JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR

Xinjie Chen, Biao Fu, Jing Wu, Guoxin Chen+3

Tongyi Lab, Alibaba Group, Hangzhou, China

RL for Language Model Reasoning Mathematical Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) enhances the reasoning of large language models (LLMs), but standard RLVR often depends on human-annotated answers or carefully curated reward specifications. In machine-checkable domains, label-free alternatives such as majority voting or LLM-as-a-judge remove annotation cost but can introduce false positives that destabilize training. We introduce JURY-RL, a label-free RLVR framework that decouples answer proposal from reward disposal: votes from model rollouts propose a candidate answer, and a formal verifier determines whether that candidate can receive positive reward. Concretely, only rollouts matching the plurality-voted answer are rewarded when that answer is successfully verified in Lean. When verification is inconclusive, we invoke ResZero (Residual-Zero), a fallback reward that discards the unverified plurality proposal and redistributes a zero-mean, variance-preserving signal over the residual answers. This design maintains a stable optimization gradient without reinforcing unverifiable consensus. Across three backbone models trained on mathematical data, JURY-RL consistently outperforms other label-free baselines on mathematical reasoning benchmarks and transfers competitively to code generation and general benchmarks. It attains pass@1 performance comparable to supervised ground-truth training, with superior generalization demonstrated by higher pass@k and response diversity.

Abstract

RL for Language Model Reasoning

Explore similar work

JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR

CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards