Leveraging the model's internal information as the self-reward signal in Reinforcement Learning (RL) has received extensive attention due to its label-free nature. While prior works have made significant progress in applying the Test-Time Scaling (TTS) strategies to RL, the discrepancy in internal information between test and training remains inadequately addressed. Moreover, Test-Time Training based on voting-based TTS strategies often suffers from reward hacking problems. To address these issues, we propose DistriTTRL, which leverages the distribution prior of the model's confidence during RL to progressively optimize the reward signal, rather than relying solely on single-query rollouts. Additionally, we mitigate the phenomenon of consistent reward hacking caused by the voting-based TTS strategies through diversity-targeted penalties. Benefiting from this training mechanism where model capability and self-reward signals complement each other, and the mitigation of reward hacking, DistriTTRL has achieved significant performance improvements across multiple models and benchmarks.
Figures & tables
Figure 1: Overview of DistriTTRL . During the pseudo-label estimation process at each step, we first calibrate the Global Confidence Distribution from previous steps using the Local Confidence Distribution of all queries in the current step, then guide the distribution of specific queries in the current step with the calibrated distribution prior. Additionally, we computes diversity from each query’s rollouts and adjusts the advantage accordingly.
Figure 2: Confidence distribution shift across training steps. Trained on AMC using Qwen3-8B with 32 samples per question. Each distribution aggregates 15 consecutive steps.
Figure 3: Trend of majority ratio (left) and accuracy (right, average of 16) during the TTT process of training Qwen2.5-7B on AIME2024. GT denotes direct supervision with ground truth, while Voting denotes using Majority Voting for pseudo-label estimation.
Model
Method
AIME2024
AMC
MATH-500
Avg.
Qwen2.5-7B-Base
Base Model
7.50
33.06
59.11
33.22
TTRL
22.08
52.26
80.38
51.57
TTRL-GT
62.71
82.61
90.75
78.69
TTRL-WSC
23.07 + 0.99
54.07 + 1.81
80.30 − 0.08
52.48 + 0.91
DistriTTRL-WSC
22.08 + 0.00
57.23 + 4.97
80.67 + 0.29
53.33 + 1.76
DistriTTRL-GMM
23.54 + 1.46
56.48 + 4.22
81.46 + 1.08
53.83 + 2.26
Table 1: Main Results of DistriTTRL across benchmarks (16 repeats). -GT denotes using ground truth directly as labels to measure the upper bound of the method. -WSC denotes using Weighted Self-Consistency (Weighted Majority Voting) as the voting method, while -GMM denotes using DistriVoting as the voting method. Bold and underline respectively indicate the optimal and suboptimal results excluding GT.
Figure 4: Effect of diversity-targeted penalty in DistriTTRL on mitigating consistency reward hacking, training Qwen2.5-7B on AIME2024.
Figure 5: Impact of voting budget on the accuracy of different TTS strategies, evaluated on AIME2024 using DeepSeek-R1-8B (64 repeats). Complete results provided in Appendix §B .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Impact of sampling budget on the accuracy of different test-time scaling strategies across four additional benchmarks (DeepSeek-R1-8B, averaged over 64 runs).
Budget
Method
HMMT2025
GPQA-D
AIME2024
AIME2025
BRUMO2025
8
MoB
69.48
67.22
88.39
80.83
89.27
BoN
69.69
66.26
87.60
78.75
88.33
SC
71.82
66.88
89.32
80.57
89.95
WSC
72.03
67.04
88.96
80.36
90.31
DeepConf
71.09
67.37
89.06
80.52
90.31
DistriVoting
72.71
66.94
89.27
81.20
90.05
Appendix
Table 2: Complete voting accuracy comparison results of different test-time scaling strategies under varying sampling budgets across 5 benchmarks using DeepSeek-R1-8B, averaged over 64 repetitions.
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.
Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.
Kunbin Xu, Xingzuo Li, Xuefeng Bai +1
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
Reasoning models have advanced rapidly, but the dominant reinforcement learning from verifiable rewards (RLVR) recipe remains surprisingly narrow: sample many responses and reward each with a single bit indicating whether the final answer is correct. Yet many settings provide rich feedback, including execution traces, tool outputs, expert corrections, and model self-evaluations. We study how to use such feedback through a distributional variant of the classic imitation learning algorithm DAgger, where the learner has local access to an expert distribution on states visited by the current policy. This yields a simple forward cross-entropy objective that admits a blackbox expert and whose sequence-level gradient {conduct rich credit assignment by propagating} future expert-student disagreement back to earlier decisions. We show that prior RL with self-distillation objectives based on reverse KL or Jensen-Shannon fail to guarantee monotonic policy improvement: even when the expert has higher reward, their updates may increase probability on worse actions. In contrast, we show that forward cross-entropy admits monotonic policy improvement and enjoys guarantees on regret. We further show that our objective optimizes a lower bound on teacher-weighted likelihood of success, leading to improved Pass@N. Empirically, our approach, DistIL, improves over RLVR and RL with self-distillation baselines across a variety of domains: scientific reasoning, coding, and solving hard mathematical problems.
Rishabh Agrawal, Jacob Fein-Ashley, Paria Rashidinejad
Large Reasoning Models have demonstrated remarkable performance with the advancement of test-time scaling techniques, which enhances prediction accuracy by generating multiple candidate responses and selecting the most reliable answer. While prior work has analyzed that internal model signals like confidence scores can partly indicate response correctness and exhibit a distributional correlation with accuracy, such distributional information has not been fully utilized to guide answer selection. Motivated by this, we propose DistriVoting, which incorporates distributional priors as another signal alongside confidence during voting. Specifically, our method (1) first decomposes the mixed confidence distribution into positive and negative components using Gaussian Mixture Models, (2) then applies a reject filter based on positive/negative samples from them to mitigate overlap between the two distributions. Besides, to further alleviate the overlap from the perspective of distribution itself, we propose SelfStepConf, which uses step-level confidence to dynamically adjust inference process, increasing the separation between the two distributions to improve the reliability of confidences in voting. Experiments across 16 models and 5 benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches.