What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
Authors: Dong Yan, Jian Liang, Yanbo Wang, Shuo Lu, Ran He, Tieniu Tan
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.
Figures & tables
Figure 1: Comparison of pseudo-labeling strategies under weak consensus. (a) Majority voting assigns the positive label despite dispersed answer distribution. (b) SCRL abstains from positive labeling when consensus is insufficient and identifies negative labels.
Figure 2: Overview of the SCRL framework. SCRL addresses test-time label noise through three components: selective positive pseudo-labeling enforces strict consensus thresholds to prevent reinforcing unreliable majorities; entropy-gated negative pseudo-labeling identifies negative labels by isolating answers that are both rare and exhibit high uncertainty, pruning the search space without eliminating valid candidates; dynamic reward shaping constructs distribution-aware rewards that scale with consensus strength and penalize uncertainty trajectories.
Method
AIME25
AMC
MATH-500
Minerva
GPQA
Avg
32 candidate responses, 16 training samples
Qwen2.5-3B
1.9
23.2
50.0
21.4
21.6
23.6
+ TTRL ( Zuo et al., 2025 )
2.6
39.4
66.9
31.6
25.0
33.1
+ SCRL (Ours)
8.4
41.5
68.2
29.5
25.7
34.7
Δ
+5.8
+2.1
+1.3
-2.1
+0.7
+1.6
Qwen2.5-Math-7B
4.6
34.0
46.5
10.1
23.0
23.6
Table 1: Main results on various reasoning benchmarks. We report the pass@1 accuracy (%) across five datasets under two rollout budgets. Results denoted by * represent peak performance before significant degradation during training. Δ shows the performance gain over TTRL ( Zuo et al., 2025 ) .
Method
AIME24
MATH / AMC ⋆
Average
Instruct Models
Llama-3.2-1B †
1.5
24.7
13.1
+ TTRL †
6.7
27.8
17.3
+ COMPASS †
3.5
28.7
16.1
+ SCRL
6.7
39.7
23.2
Δ
+3.2
+11.0
+7.1
Table 2: Pass@1 accuracy (%) on Llama-3-Instruct models. Column denoted by ⋆ refers to MATH-500 for Llama-3.2-1B-Instruct and AMC for Llama-3.1-8B-Instruct. Results denoted by † and ‡ are reported from COMPASS ( Tang et al., 2025 ) and RESTRAIN ( Yu et al., 2025b ) , respectively. Δ shows the performance gain over the corresponding baseline.
Figure 3: Statistics of positive and negative pseudo-label estimation on the AMC dataset using Qwen2.5-3B.
Method
AIME25
AMC
SCRL
8.4
41.5
w/o Selective Labeling
3.9 ↓4.5
41.8 ↑0.3
w/o Negative Labeling
1.2 ↓7.2
37.7 ↓3.8
w/o Entropy Gate
3.6 ↓4.8
39.3 ↓2.2
w/o Dynamic Reward
5.1 ↓3.3
39.8 ↓1.7
Table 3: Ablation study of SCRL components on Qwen2.5-3B. The table reports pass@1 accuracy (%).
Method
AMC
MATH-500
Minerva
Pass@1
Pass@16
Pass@1
Pass@16
Pass@1
Pass@16
Qwen2.5-3B
23.2
67.5
50.0
86.4
21.4
50.0
+ TTRL ( Zuo et al., 2025 )
27.3
54.2
58.5
84.8
23.4
48.9
+ SCRL (Ours)
27.5
60.2
65.1
87.0
26.3
49.3
Table 4: Generalization performance of Qwen2.5-3B trained on AIME25. Bold shows the best result.
τpos
τneg
AIME25
AMC
MATH
0.25
0.0625
19.2
64.5
85.5
0.25
0.125
19.0
66.6
85.4
0.375
0.0625
22.1
66.3
85.0
0.375
0.125
22.8
67.5
86.2
Table 5: Parameter analysis of τpos and τneg on Qwen2.5-Math-7B. The table reports pass@1 accuracy (%).
Type
AIME24
AMC
MATH-500
Minerva
Correct Trajectories
0.208
0.225
0.245
0.258
Incorrect Trajectories
0.288
0.279
0.272
0.316
Overall Average
0.286
0.266
0.259
0.303
Table 6: Average trajectory-level entropy for correct and incorrect answers across mathematical reasoning benchmarks. Results are computed using Qwen2.5-3B during the initial rollout phase.
Method
Max Generation Length
10,240
15,360
Qwen3-4B
38.9
51.9
+ TTRL
50.9
59.3
+ SCRL
53.6
60.2
Δ
+2.7
+0.9
Table 7: Pass@1 accuracy (%) on large reasoning models using Qwen3-4B under maximum generation length of 10k and 15k tokens. Δ shows the performance gain over TTRL ( Zuo et al., 2025 ) .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Qwen2.5-3B
Qwen2.5-Math-7B
Max Prompt Length
512
512
Max Response Length
3072
3072
Train Batch Size
8
8
PPO Mini Batch Size
1
1
PPO Micro Batch Size per GPU
2
2
N GPUs per Node
4
8
Appendix
Table 8: Training hyperparameter configuration for Qwen2.5-3B and Qwen2.5-Math-7B based on the verl ( Sheng et al., 2025 ) framework.
Figure 4: Training dynamics of SCRL and TTRL on Qwen2.5-3B across three mathematical benchmarks.
Figure 5: Training dynamics of SCRL and TTRL on Qwen2.5-Math-7B across three mathematical benchmarks.
τmarg
λH
AIME25
AMC
0.125
0
19.0
64.4
0.125
0.2
19.0
63.7
0.125
0.5
18.5
66.5
0
0.1
18.8
66.5
0.0625
0.1
18.8
68.8
0.25
0.1
22.6
65.6
Appendix
Table 9: Parameter analysis of τmarg and λH on Qwen2.5-Math-7B. The table reports pass@1 accuracy (%). Bold shows the configuration in main experiments.