Current label-free RLVR approaches for large language models (LLMs), such as TTRL and Self-reward, have demonstrated effectiveness in improving the performance of LLMs on complex reasoning tasks. However, these methods rely heavily on accurate pseudo-label estimation and converge on spurious yet popular answers, thereby trapping in a dominant mode and limiting further improvements. Building on this, we propose Dual Consensus Reinforcement Learning (DCRL), a novel self-supervised training method which is capable of generating more reliable learning signals through a two-stage consensus mechanism. The model initially acts as an anchor, producing dominant responses; then it serves as an explorer, generating diverse auxiliary signals via a temporary unlearning process. The final training target is derived from the harmonic mean of these two signal sets. Notably, the process operates entirely without external models or supervision. Across eight benchmarks and diverse domains, DCRL consistently improves Pass@1 over majority vote while yielding more stable training dynamics. These results demonstrate that DCRL establishes a scalable path toward stronger reasoning without labels.
Figures & tables
Figure 1: An overview of Dual Consensus Reinforcement Learning (DCRL). Specifically, the policy model assumes two roles: (1) an anchor that generates dominant and reliable responses; (2) an explorer that produces diverse auxiliary signals through a temporary unlearning process.
Figure 2: Output distributions of the Anchor and Explorer models. The Explorer model, after the unlearning process, generates a more diverse distribution.
Figure 3: Comparison between Majority vote and Dual Consensus : Majority vote tends to fall into spurious consensus by over-relying on dominant but potentially incorrect response modes, while Dual Consensus mitigates this issue by converting the anchor model (which captures dominant reasoning patterns) into an explorer model via temporary unlearning. This transformation enables the framework to explore diverse alternative response modes, thereby balancing the reliability of current dominant patterns and the diversity of potential valid alternatives, and ultimately achieving more accurate answer selection.
Methods
Math
Multi-Task
Average
Math
GSM8K
AIME24
Minerva.
AMC
Olympiad.
MMLU.
GPQA.
Llama3.2-3B-Instruct
Vanilla
42.4
76.1
4.5
11.7
20.6
14.6
26.4
22.3
27.3
GRPO
49.2
79.7
13.5
14.9
23.2
15.3
32.8
23.8
31.5
RENT
45.4
78.5
9.4
11.2
20.9
15.2
30.3
22.7
29.2
TTRL
44.6
71.8
10.2
12.4
21.5
14.6
25.9
21.4
27.8
Table 1: Main Results (%) of DCRL Trained on DAPO-Math-14k: DCRL Outperforms All Label-Free Baselines. The best results are highlighted in bold . DCRL exceeds all unsupervised methods and nearly matches GRPO with gold labels.
Figure 4: Training Dynamics of Dual Consensus on Qwen3-8B-Base. DCRL-Anchor in Fig. 4(b) refers to the majority vote of the anchor model in DCRL.
Figure 5: Comparison of reward signal between Majority Vote and Dual Consensus.
Methods
Datasets
Average
MATH500
AIME24
AMC
GPQA
Qwen2.5-Math-1.5B
Vanilla
30.6
5.6
23.4
15.4
18.7
TTRL
72.9
17.0
45.3
22.2
39.3
DCRL(ours)
74.5
17.7
46.6
22.8
40.4
Δ (-TTRL)
↑ 2.2%
↑ 4.1%
↑ 2.8%
↑ 2.7%
↑ 2.7%
Table 2: Results (%) of Test-Time Adaptation Trained on Different Datasets: DCRL Consistently Outperforms the TTRL Baseline. All results are evaluated under the same experimental settings and reported as the average pass@1 over 16 independent seeds.
Figure 6: Label accuracy curves (smoothed) of test-time adaptation for DCRL and TTRL across different tasks.
Method
MATH
GSM8K
AIME24
MMLU.
GPQA.
Average
DCRL (Full)
79.2
93.3
14.7
56.7
37.9
56.3
- w/o Harmonic Election
78.7
93.4
14.3
56.2
37.8
56.0
- w/o Conservative Reward
78.3
93.3
12.2
57.2
36.2
55.4
- w/o Dynamic Sampling
76.4
90.3
14.3
52.0
34.4
53.4
Table 3: Ablation Studies to Analyze the Contribution of DCRL Core Modules with the Qwen3-8B-Base Model.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Batch size
128
Mini batch size
128
Micro batch size
8
Max prompt length
4096
Max response length
3072
Learning rate
1×10−6
Appendix
Table 4: Hyperparameter Settings for DCRL Framework.
Figure 7: Detailed Results of Ablation Studies with Qwen3-8B-Base, including pass@16 on training dataset and label accuracy
Unlearn LR ( ηu )
MATH
GSM8K
AIME24
MMLU.
GPQA.
Average
1×10−7
78.6
92.7
12.6
55.5
37.2
55.3
3×10−7
79.2
93.3
14.7
56.7
37.9
56.3
5×10−7
77.6
92.7
11.5
54.3
36.4
54.5
1×10−6 (Failed)
67.3
88.1
12.1
50.2
35.1
50.5
Appendix
Table 5: Sensitivity Analysis of Unlearning Learning Rate ( ηu ) on Qwen3-8B-Base.
Consensus Strategy
MATH
GSM8K
AIME24
MMLU.
GPQA.
Avg.
Majority Vote (Anchor Only)
78.7
93.4
14.3
56.2
37.8
56.0
Majority Vote (Anchor + Explorer)
64.7
90.3
14.3
52.0
34.4
51.5
Harmonic Mean (Ours)
79.2
93.3
14.7
56.7
37.9
56.3
Appendix
Table 6: Comparison of Different Consensus Strategies for Pseudo-Label Selection on Qwen3-8B-Base.
Figure 8: Curves of Different Consensus Strategies for Pseudo-Label Selection on Qwen3-8B-Base.
Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55% to over 85%, confirming that both capabilities genuinely co-evolve.
Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.
Dong Yan, Jian Liang, Yanbo Wang +3
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.
Yunhao Yang, Yuexin Bian, Yunjie Tian +6
Johns Hopkins University · UC San Diego · Independent Researcher +1