Test-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation. Because population-majority bias is generally unidentifiable without external supervision, we focus on mitigating the downstream effects of finite-rollout variation. We propose Prompt-Conditional Distributionally Robust DPO (PC-Dr.DPO), which conditions within-prompt robust reweighting on observed vote reliability to make updates from sampling-sensitive prompts more conservative. We further introduce a cache-based mechanism to reduce self-supervised forgetting during online adaptation. Across six mathematical and knowledge-intensive reasoning benchmarks and five model backbones, PC-Dr.DPO matches or outperforms the strongest TTRL baseline in 24 of 30 settings.
Figures & tables
Figure 1: Distribution of prompt-level modal-answer agreement for Qwen2.5-Math-1.5B-Instruct on the 83-problem AMC benchmark. For each prompt x , we sample m=32 rollouts and compute the fraction producing the most frequent final answer. The broad distribution indicates substantial prompt-level heterogeneity: values near one correspond to nearly unanimous rollout groups, whereas smaller values indicate fragmented answer distributions that are more sensitive to resampling.
Method
Math Reasoning
Knowledge Reasoning
A24
A25
AMC
M500
GPQA-D
MMLU-M
Qwen2.5-1.5B-Instruct
Base
2.9
0.8
22.0
50.7
24.1
39.5
RENT
0.0
∗
19.2
55.7
∗
5.2
TTRL
3.3
∗
33.6
62.3
23.5
10.9
Co-Reward
3.3
∗
31.4
59.6
23.3
10.8
Table 1: Test-time adaptation accuracy (%) under Avg@32 evaluation. The best result among adaptation methods is bolded . ∗ denotes training collapse.
Figure 2: Adaptation dynamics of Qwen2.5-Math-1.5B-Instruct on the AMC benchmark. Left: fraction of problems in the initial majority-vote-solvable set that are recovered by the adapted policy. Middle: difference between the number of problems solved by the current policy and the initial majority-vote reference; the dashed line at zero denotes full recovery. Right: decomposition of the currently solved problems into those recovered from the initial majority-vote-solvable set and newly discovered solutions. The policy progressively recovers most initially available solutions while occasionally solving problems outside the initial set.
Method
AMC
MATH500
Base model
34.3
63.5
PC-Dr.DPO w/ constant β
42.0
72.3
PC-Dr.DPO w/ frequency MV
44.6
71.4
PC-Dr.DPO
45.8
73.3
Table 2: Ablation of the reliability-conditioned temperature and score-based majority selection on Qwen2.5-3B-Instruct. We report Avg@32 accuracy (%); the best adapted result is bolded .
Figure 3: Effect of the rollout count m during test-time adaptation of Qwen2.5-Math-1.5B-Instruct on AMC. Left: pseudo-label accuracy across adaptation steps. The faint trajectories show the per-step measurements, while the darker curves show their smoothed trends. Right: validation Avg@32 accuracy throughout adaptation. Increasing m reduces the variability of sampled pseudo-labels and produces more stable downstream improvement.
Figure 4: Self-reinforcing collapse of Qwen2.5-1.5B-Instruct during test-time adaptation on AIME25. Left: accuracy of the sampled pseudo-labels relative to the ground-truth answers. Middle: validation Avg@32 accuracy, which eventually decreases to zero. Right: average modal-answer fraction within each rollout group. Although the modal-answer fraction steadily approaches one, the pseudo-label accuracy remains close to zero. Thus, the policy becomes increasingly confident in predominantly incorrect pseudo-labels, reinforcing systematic population-majority bias and ultimately producing adaptation collapse.
Figure 5: Effect of cache replay during adaptation of Llama-3.2-3B-Instruct on MATH500. Left: cache-replay dynamics, including the cumulative number of prompts whose solutions have been cached and the number of previously mastered prompts recalled at each adaptation step. Middle: pseudo-label accuracy with and without cache replay; faint curves show per-step values and darker curves show smoothed trajectories. Right: validation Avg@32 accuracy throughout adaptation. Cache replay periodically revisits earlier self-consistent solutions, reducing degradation in pseudo-label quality and mitigating self-supervised forgetting. Removing cache replay lowers final Avg@32 accuracy from 58.4% to 56.3% .
Figure 6: Additional adaptation dynamics on MATH500 for Llama-3.2-3B-Instruct and Qwen2.5-Math-1.5B-Instruct. In each row, the problems solved by the initial rollout majority define a fixed reference set. Left: fraction of this initial set recovered by the current policy. Middle: difference between the number of problems solved by the current policy and the initial majority-vote reference; the dashed line at zero denotes full numerical recovery of the initial reference level. Right: decomposition of the currently solved problems into solutions recovered from the initial reference set and newly discovered solutions. Both backbones recover most initially available solutions and occasionally solve problems outside the initial reference set, while incomplete recovery indicates degradation induced by systematic pseudo-label errors.
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.
Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.
Dong Yan, Jian Liang, Yanbo Wang +3
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.
Kunbin Xu, Xingzuo Li, Xuefeng Bai +1
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China