Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.
Figures & tables
Figure 1: Comparison of Co-RL with prior label-free RL methods. Both TTRL and Co-rewarding derive rewards from self-generated agreement, and CoMAS scores multi-turn interactions with one of its own agents acting as judge. Co-RL instead derives rewards directly from peer votes. Beyond two agents, the votes pass along a directed ring ( N=3 shown).
Figure 2: (a) Agreement between the two models, (b) pseudo-label accuracy, and (c) evaluation performance; (d) Error overlap before RL for two pairs each from a different family, the same family, and the same model under a different seed.
Figure 3: Overview of Co-RL with two agents. Each agent samples K responses to the same unlabeled question and generates a pseudo label with majority vote. Each rollout is then rewarded by agreement with the cohort’s pseudo label, and updates its own policy, with no sharing parameters and no gradient exchange except the cross-reward process.
Method
GSM8K
MATH500
AMC
HEval
GPQA
MBPP
LCB
Avg
Qwen2.5-3B
Base
73.4
56.6
28.9
39.0
21.2
52.2
13.7
40.7
GT-Reward
76.2
64.6
36.1
65.2
20.7
54.4
14.5
47.4
TTRL
80.4
66.4
31.3
63.4
22.2
51.8
15.9
47.3
RENT
75.6
62.8
31.3
59.2
18.2
52.4
14.5
44.9
Intuitor
74.9
64.2
26.5
59.8
27.3
50.4
16.4
45.6
Table 1: Full performance across seven benchmarks for 3B models (%). For each benchmark, the best label-free result is shown in bold and the second best is underlined , with ties sharing the marking. Base and GT-Reward serve as references and are excluded from the ranking. Co-RL (Same family) trains two agents initialized from the same base model. Co-RL (Different family) pairs one agent from each of the two families. Co-RL (Different family+) further decouples the training data. Appendix D.1 extends the comparison to 7B and 8B models.
Method
GSM8K
MATH-500
HumanEval
MBPP
MMLU
GPQA
SciBench
Avg
Base
85.40
55.00
73.78
55.80
63.20
28.79
36.47
56.92
MAPoRL
85.80
55.40
75.61
57.00
63.20
31.47
39.08
58.22
TTRL
88.20
56.80
73.78
59.00
63.80
27.23
38.48
58.18
CoMAS
87.20
55.80
77.44
59.20
65.60
29.69
37.68
58.94
Co-RL (Different family)
89.5
68.6
82.32
68.00
65.80
29.69
36.87
62.97
Table 2: Comparison under the CoMAS multi-agent RL setting (%). All methods train Qwen2.5-3B-Instruct on the same prompt mixture and are evaluated following the CoMAS protocol. Results for prior methods are reported from Xue et al. (2026) .
Model
Method
GSM8K
MATH500
AMC
HEval
GPQA
MBPP
LCB
Avg
Qwen2.5-3B
Base
73.4
56.6
28.9
39.0
21.2
52.2
13.7
40.7
GT-Reward
76.2
64.6
36.1
65.2
20.7
54.4
14.5
47.4
TTRL
80.4
66.4
31.3
63.4
22.2
51.8
15.9
47.3
Co-RL (Different family)
79.8
66.3
33.6
64.6
23.2
56.0
15.8
48.5
Llama-3.2-3B-Instruct
Base
73.6
43.8
18.1
51.2
21.2
50.8
12.0
38.7
GT-Reward
78.8
53.8
25.3
60.4
20.7
50.2
12.1
43.0
Table 3: Three-agent Co-RL with heterogeneous model families (%). Qwen2.5-3B, Llama-3.2-3B-Instruct, and Qwen3-1.7B are jointly trained in a single Co-RL run. For each model, we compare against the base model, training with ground-truth rewards (GT-Reward), and self-rewarding with majority-vote pseudo-labels (TTRL).
Backbone
Data
Method
MathVision
MathVerse
MathVista
We-Math
Avg
InternVL-3.5-2B
open-r1
GT-Reward
26.55
35.33
59.60
59.31
45.20
Base
24.77
34.21
55.60
57.87
43.11
TTRL
25.86
34.24
57.60
62.47
45.04
Co-RL (Different family)
26.25
34.92
58.90
61.55
45.40
MMR1
GT-Reward
25.99
34.37
59.00
59.25
44.65
Base
24.77
34.21
55.60
57.87
43.11
Table 4: Vision-language results for the small pair, Qwen2.5-VL-3B with InternVL3.5-2B, trained separately on open-r1 and MMR1 (%). Base is graded once with the corrected multiple-choice grader and is therefore identical across the two training sets. Base and GT-Reward serve as references and are excluded from the ranking.
Figure 4: Training dynamics at four scales, one column per backbone (Qwen2.5-3B, Llama-3.2-3B, Qwen2.5-7B, Llama-3.1-8B). (a) MATH-500 validation accuracy, (b) standard deviation of the reward within a rollout group, normalized to its value at the first step, and (c) mean completion length. Runs marked diverged leave the plotted range.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
B correct
B wrong
A correct
both correct
only A correct
A wrong
only B correct
both wrong
Appendix
Table 5: The four outcomes for a pair of models A and B. Every problem falls into exactly one cell, and all four diversity measures are counts over these cells.
Decoupling
Pair
κ↓
c↑ (%)
w↓ (%)
u↑ (%)
3B tier
different family
Llama-3.2-3B × Phi-3.5-mini
0.31
32.8
3.0
53.0
different family
Qwen2.5-3B × Llama-3.2-3B
0.38
31.2
2.4
63.0
different family
Qwen2.5-3B × Phi-3.5-mini
0.38
31.2
4.0
55.4
different family
Qwen2.5-3B × MiniCPM3-4B
0.41
29.4
4.4
60.4
same family
Qwen2.5-3B × Qwen3-1.7B-Base
0.52
24.2
4.2
63.2
Appendix
Table 6: Error decoupling before RL, by what the two models differ in, sorted by κ within each block.
Partner
Decoupling
κ↓
c↑ (%)
w↓ (%)
Anchor: Qwen2.5-3B
itself, new seed
seed only
0.56
22.0
4.4
Qwen3-1.7B-Base
same family
0.52
24.2
4.2
MiniCPM3-4B
different family
0.41
29.4
4.4
Phi-3.5-mini
different family
0.38
31.2
4.0
Llama-3.2-3B
different family
0.38
31.2
2.4
Appendix
Table 7: One model held fixed, partner varied. Capability is identical to the seed-only row along each ladder, so the source of the partner is the only variable.
Method
GSM8K
MATH500
AMC
HEval
GPQA
MBPP
LCB
Avg
Qwen2.5-7B
Base
82.9
70.0
39.8
47.6
18.7
62.8
21.1
49.0
GT-Reward
84.8
77.6
49.4
56.1
23.7
64.4
25.5
54.5
TTRL
80.6
74.8
39.8
51.8
25.8
65.4
23.9
51.7
RENT
78.8
75.4
47.0
50.6
29.8
61.6
26.2
52.8
Intuitor
82.9
75.4
41.0
51.8
28.3
64.0
24.8
52.6
Appendix
Table 8: Results at 7B and 8B on the seven-benchmark suite (%). Base and GT-Reward serve as references and are excluded from the ranking.
Backbone
Method
MathVision
MathVerse
MathVista
We-Math
Avg
Qwen2.5-VL-7B
GT-Reward
26.74
41.07
71.90
67.01
51.68
Base
23.36
33.32
56.60
62.47
43.94
TTRL
23.62
37.26
69.40
65.23
48.88
Co-RL (Different family)
26.87
38.43
71.00
68.22
51.13
InternVL-3.5-8B
GT-Reward
37.24
43.35
69.30
73.51
55.85
Base
29.21
36.65
65.70
60.69
48.06
Appendix
Table 9: Vision-language results at 7B to 12B on open-r1, with InternVL3.5-8B as the shared partner.
Figure 5: Training dynamics for Qwen2.5-VL-7B trained with InternVL3.5-8B on open-r1. (a) Evaluation accuracy, (b) mean completion length, and (c) the accuracy of the exchanged pseudo-labels together with the agreement between the two agents. Co-RL keeps improving and holds its completion length, while TTRL peaks and then degrades in both.
Setting
GSM8K
MATH-500
AMC
Avg
TTRL (Qwen2.5-3B)
88.2
68.8
39.8
65.6
TTRL (Llama-3.2-3B)
65.7
56.0
27.7
49.8
TTRL (ensemble)
88.2
68.0
38.6
64.9
Co-RL (Qwen2.5-3B)
87.4
72.8
37.4
65.9
Co-RL (Llama-3.2-3B)
87.3
58.8
33.7
59.9
Co-RL (ensemble)
90.1
70.8
39.8
66.9
Appendix
Table 10: Matched-budget comparison between TTRL and Co-RL on text reasoning benchmarks. Both settings train the same two base models, Qwen2.5-3B and Llama-3.2-3B-Instruct. The ensemble rows pool four rollouts from each of the two models for majority voting (maj@8, T=0.6 ). Avg is the macro-average over the three benchmarks. For each benchmark, the best result is in bold and the second best is underlined , with ties sharing the marking.
Setting
MathVision
MathVerse
MathVista
We-Math
Avg
open-r1
TTRL (Qwen2.5-VL)
22.96
31.45
61.10
63.39
44.73
TTRL (InternVL3.5)
29.67
38.91
62.30
67.24
49.53
TTRL (ensemble)
27.24
35.13
65.40
67.41
48.80
Co-RL (Qwen2.5-VL)
25.43
35.66
64.80
65.80
47.92
Co-RL (InternVL3.5)
30.46
38.60
63.30
67.87
50.06
Appendix
Table 11: Matched-budget comparison between TTRL and Co-RL on multimodal reasoning benchmarks. Both settings train the same two base models, Qwen2.5-VL-3B and InternVL3.5-2B. The ensemble rows pool four rollouts from each of the two models for majority voting (maj@8, T=0.6 , top- p 0.95). Rows are grouped by training set. MMR1 runs use the corrected multiple-choice grader and open-r1 runs the legacy grader, so the two blocks are not compared against each other.
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of LLMs, but often depends on external supervision from human annotations or gold-standard solutions. Reinforcement learning from internal feedback (RLIF) has recently emerged as a scalable unsupervised alternative, using signals extracted from the model itself. However, existing RLIF methods typically rely on a single internal reward, which can lead to reward hacking, entropy collapse, and degraded reasoning structure. We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components: an answer-level reward based on cluster voting and a completion-level reward based on token-wise self-certainty. To combine these signals robustly, we apply GDPO-based normalization to reduce reward-scale imbalance. We further introduce KL-Cov regularization, which targets low-entropy token distributions responsible for disproportionate entropy reduction, preserving exploration and preventing late-stage collapse. Across mathematical reasoning and code-generation benchmarks, our method improves stability and robustness over prior unsupervised RL approaches, while achieving performance close to supervised RLVR methods. These results show that complementary internal rewards, combined with targeted regularization, can support stable long-horizon reasoning without relying on external ground-truth supervision. Code will be released soon.
Current label-free RLVR approaches for large language models (LLMs), such as TTRL and Self-reward, have demonstrated effectiveness in improving the performance of LLMs on complex reasoning tasks. However, these methods rely heavily on accurate pseudo-label estimation and converge on spurious yet popular answers, thereby trapping in a dominant mode and limiting further improvements. Building on this, we propose Dual Consensus Reinforcement Learning (DCRL), a novel self-supervised training method which is capable of generating more reliable learning signals through a two-stage consensus mechanism. The model initially acts as an anchor, producing dominant responses; then it serves as an explorer, generating diverse auxiliary signals via a temporary unlearning process. The final training target is derived from the harmonic mean of these two signal sets. Notably, the process operates entirely without external models or supervision. Across eight benchmarks and diverse domains, DCRL consistently improves Pass@1 over majority vote while yielding more stable training dynamics. These results demonstrate that DCRL establishes a scalable path toward stronger reasoning without labels.
Kaixuan Du, Meng Cao, Hang Zhang +3
School of Automation Science and Electrical Engineering, Beihang University
Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.