Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
Figures & tables
Figure 1: Training dynamics of TTRL and CoVerRL on Qwen3-1.7B-Base. CoVerRL maintains high reward accuracy while steadily improving label accuracy, whereas TTRL suffers from reward accuracy degradation as output diversity collapses.
Figure 2: Overview of the CoVerRL framework. The pipeline illustrates the Pseudo-Label Generation process and the Online Dual-Role Co-Evolution strategy, facilitating a mutual bootstrapping process where generation and verification capabilities jointly evolve.
Model
Method
MATH500
AMC
AIME24
GPQA
Average
Qwen3-1.7B-Base
Base Model
53.5 / 53.3
24.6 / 24.5
3.8 / 3.3
27.5 / 27.3
27.4 / 27.1
TTRL
65.1 / 65.0
31.1 / 30.9
5.2 / 5.2
30.9 / 30.7
33.1 / 33.0
CoVerRL (Ours)
69.0 / 71.9
36.0 / 38.6
9.8 / 10.6
32.9 / 33.6
36.9 / 38.7
Δ
+3.9 / +6.9
+4.9 / +7.7
+4.6 / +5.4
+2.0 / +2.9
+3.8 / +5.7
Llama-3.2-3B-Instruct
Base Model
42.7 / 41.0
17.0 / 15.7
4.6 / 5.0
26.9 / 26.1
22.8 / 22.0
TTRL
52.6 / 52.2
23.8 / 23.3
13.8 / 14.0
29.8 / 28.2
30.0 / 29.4
Table 1: Main results on mathematical reasoning and out-of-domain benchmarks. Each cell displays Acc.@first / Acc.@final. Δ indicates the improvement of CoVerRL over TTRL.
Model
MATH500
AMC
AIME24
GPQA
Qwen3-1.7B-Base
55.8
28.5
11.7
38.7
+ CoVerRL
81.2
64.4
48.9
57.9
Llama-3.2-3B-Instruct
57.0
50.9
55.8
56.9
+ CoVerRL
77.2
76.2
75.6
55.2
Qwen2.5-7B
54.0
30.9
12.9
35.7
+ CoVerRL
86.5
70.9
58.8
58.4
Table 2: Self-verification accuracy on four benchmarks.
Figure 3: Co-evolution dynamics across three model backbones. As training progresses, verification accuracy improves through contrastive learning, which in turn enables better filtering of self-consistent errors and leads to higher label accuracy than TTRL.
Figure 6
Figure 6: Comparison of CoVerRL against naive consistency filtering on Qwen3-1.7B-Base. Naive filtering ( τ∈{0.1,0.2} ) improves label accuracy initially but converges to TTRL baseline, while failing to improve reward accuracy. CoVerRL achieves sustained gains on both metrics.
Model
Score
Qwen3-1.7B-Base
46.8
Llama-3.2-3B-Instruct
58.6
Qwen2.5-7B
49.7
Meta-Llama-3-70B-Instruct
66.2 ∗
gpt-4-turbo-2024-04-09
67.3 ∗
gpt-4o-2024-05-13
72.5 ∗
Table 3: Performance on RewardBench. Scores marked with ∗ are from the RewardBench leaderboard.
Figure 7: Comparison of training response length and validation accuracy on MATH500 between the Thinking mode and Non-thinking mode on Qwen3-1.7B-Base. Non-thinking mode suffers from length collapse (left), leading to inferior verification accuracy (right).
Figure 8: Comparison of TTRL + PAG and CoVerRL on Llama-3.2-3B-Instruct. TTRL + PAG exhibits verifier collapse with Correct Recall dropping to near zero (left), causing accuracy degradation (right). CoVerRL maintains stable training through balanced sampling.
Method
MATH500
AMC
AIME24
GPQA
CoVerRL
69.0 / 71.9
36.0 / 38.6
9.8/10.6
32.9 / 33.6
w/o Ver. Filtering
67.1/70.3
34.6/38.1
7.1/9.2
29.6/29.4
w/o Ver. Update
64.6/64.6
34.3/34.4
5.6/5.6
29.1/29.3
w/o Self-Correction
67.1/68.4
35.3/35.2
6.7/7.8
31.0/25.4
w/o AA-GRPO
67.7/71.1
35.1/38.0
8.5/10.2
31.2/31.9
Table 4: Ablation study on Qwen3-1.7B-Base. Results are reported as Acc.@first / Acc.@final.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Table 12
Method
MATH500
AMC
AIME24
GPQA
CoVerRL
69.0 /71.9
36.0/38.6
9.8/10.6
32.9/33.6
w/o least ratio
68.6/ 72.0
36.6/39.8
7.5/9.6
31.0/32.1
w/o clip-higher
66.8/68.9
33.1/31.6
6.5/7.9
31.2/30.2
Appendix
Table 7: Ablation Study of CoVerRL on Qwen3-1.7B-Base. Results are reported as Acc.@first / Acc.@final.
Figure 14
Method
MATH500
AMC
AIME24
GPQA
TTRL
65.1/65.0
31.1/30.9
5.2/5.2
30.9/30.7
TTRL w/ filter-0.1
63.3/63.1
33.1/33.1
3.8/3.8
31.1/30.7
TTRL w/ filter-0.2
66.2/66.0
34.8/34.9
8.1/7.9
31.7/31.3
CoVerRL
69.0 /71.9
36.0/38.6
9.8 /10.6
32.9 / 33.6
CoVerRL w/ filter-0.2
68.7/ 72.1
37.0 / 39.2
9.2/ 12.1
30.9/31.6
Appendix
Table 8: Impact of the constant filtering threshold. Results are reported as Acc.@first / Acc.@final.
Model
First Generation
Verification
Correction
Total
Qwen3-1.7B-Base
798.6
535.5
1132.7
2466.9
Llama-3.2-3B-Instruct
794.6
434.0
1159.7
2388.3
Qwen2.5-7B
924.1
399.0
849.3
2172.4
Appendix
Table 9: Average number of training samples per step of CoVerRL.
Model
First Generation
Verification
Correction
Total
Qwen3-1.7B-Base
0.9145
0.8981
0.9000
0.9042
Llama-3.2-3B-Instruct
0.9482
0.9502
0.9399
0.9445
Qwen2.5-7B
0.9032
0.8688
0.8518
0.8768
Appendix
Table 10: Weighted average reward accuracy per step of CoVerRL.