Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for Recursive Self-Improving AI (RSI) by adapting Large Reasoning Models (LRMs) on unlabeled test inputs, using self-consensus rewards derived from majority voting over sampled rollouts. However, majority voting can mistake popularity for correctness: a spurious yet high-frequency unverified consensus may become a biased reward signal, causing test-time RL to reinforce frequent but wrong answers and collapse into an incorrect mode. We address this false-popular failure mode with T3RL (Tool-Verification for Test-Time Reinforcement Learning), a verification-aware test-time RL framework. T3RL grounds pseudo-label construction in external tool evidence. Concretely, a verifier utilizes external tool evidence (e.g., from code execution) to upweight verified rollouts during a verification-aware voting, producing more reliable pseudo-labels for training. Across various math difficulties (MATH-500, AMC, and AIME 2024) and diverse backbone families, T3RL significantly improves over TTRL, with better performance on harder problems. T3RL is positioned as a verified online data synthesizer, highlighting the role of tool verification in reliable online adaptation. T3RL is extensible to verifiable domains and provides training and inference efficiency.
Figures & tables
Figure 1: The concept of T 3 RL . Top: T 3 RL introduces test-time verification into self-evolvement via tool-executed evidence (e.g., code interpreter) to stabilize training with verified rollouts. Bottom: Majority-vote pseudo-labels can be spurious. T 3 RL introduces verification to suppress false-popular pseudo-labels. Right: T 3 RL achieves consistent gains with evidence-grounded test-time training.
Figure 2: T 3 RL: Tool Verification for Test-Time Reinforcement Learning. Verifier: an LLM verifier parses each sampled rollout yi into a candidate answer ai and, with tool feedback, produces a binary validity flag vi∈{0,1} . Tool verification: the verifier compiles the rollout’s claimed computations into lightweight Python and queries a code interpreter to obtain executable evidence. Verification-weighted majority voting: unverified rollouts receive unit vote mass, and verified rollouts receive ω vote mass. A verification-aware pseudo-label y~∗ is formed by weighted voting.
Figure 3: Unverified consensus versus tool-verified consensus. ( 3(a) ) Spurious reward in the reinforced cycle of TTRL : the false-popular answer is rewarded and its vote share grows over rounds. ( 3(b) ) Tool verification can adjust the false-popular answer.
Figure 4: False-majority share across benchmarks.
Table 1: Main performance of T 3 RL. (a) Improvement across benchmark and backbone models. (b) Difficulty breakdown performance on MATH-500 (L5 is the hardest). The relative improvement of T 3 RL over TTRL increases with difficulty level ( Table 1 ).
Figure 6: Ablation on verifier and verification tool . Left: Adding an LLM verifier improves TTRL even without tool execution. Right: Code execution significantly strengthens verification.
Policy
Verifier
Acc.
Prec.
Rec.
Δ AIME
Math-1.5B
Coder-1.5B
73.2
25.0
51.5
+5.0
Llama-3.2-3B
Coder-1.5B
73.3
39.4
59.1
+3.8
Qwen-3-4B
Coder-1.5B
75.0
38.2
40.7
+3.6
Table 3: Verifier quality on policy rollouts.
Figure 8: Ablations on tool position and training robustness. Left: allowing the policy to call tools during rollouts ( TTRL-Agent ) can hurt performance, while T 3 RL improves by restricting tool use to the verifier for reward shaping. Right: verifier-shaped rewards reduce run-to-run variability, indicating more stable optimization under unlabeled test-time RL.
Table 9
Table 6: Non-math generalization on Knights-and-Knaves puzzles with an SMT-LIB2/Z3 example.
Adaptation setting AIME 2024 AMC MATH-500 TTRL adapted on AIME 15.8±2.142.9±2.465.5±2.4 T 3 RL adapted on AIME 20.8±1.645.4±2.568.0±2.0 TTRL adapted on AMC 11.5±1.648.9±1.666.6±1.4 T 3 RL adapted on AMC 15.1±2.050.9±2.770.4±1.7 TTRL adapted on MATH-500 10.8±3.443.2±2.773.0±3.8 T 3 RL adapted on MATH-500 14.7±1.945.7±2.074.6±3.2
Accuracy gain (pp) A C M A C M +5.0+2.5+2.5+3.6+2.0+3.8+3.9+2.5+1.6 T 3 RL better
Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.
Test-time reinforcement learning has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models in a completely label-free manner. Despite existing studies focusing on Pass@1 performance, optimizing Pass@k remains under-explored yet critical in label-free settings, which measures generation coverage for sustained exploration. Optimizing Pass@k in label-free setting is highly non-trivial, as directly applying the Pass@k advantage designs effective for RLVR yields unsatisfactory performance. Through in-depth empirical analysis, we discover the root causes hindering performance: pseudo-label estimations for low-confidence samples have a high probability of being incorrect, while candidate answers for high-confidence samples suffer from severe diversity collapse. To overcome these hurdles, we propose TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), a novel confidence-adaptive framework that expands Pass@k coverage and improves Pass@1 performance. Based on our key insight that verification capability generally leads generation capability, TTRL-CoCoV employs a confidence-conditioned mechanism: for high-confidence samples, it bootstraps verifier and applies an exploration-enhancing reward to prevent diversity collapse; for low-confidence samples, it delegates pseudo-label selection to the verifier to filter incorrect pseudo-labels; and for medium-confidence samples, it bypasses verification entirely. Extensive experiments demonstrate that TTRL-CoCoV outperforms the best competing methods across 6 widely-recognized benchmarks, achieves average absolute gains of +9.8% in Pass@1 and +18.7% in Pass@16 over TTRL, and even achieves absolute Pass@1 improvements of up to +5.0% across multiple reasoning benchmarks when compared against fully supervised RL methods. Our code repository: https://github.com/shanjf666/CoCoV.
Jiahui Li, Jianfeng Shan, Wenpei Chen +5
1Sun Yat-Sen University · University of Science and Technology of China · 3National University of Singapore
Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams by deriving pseudo-rewards from majority voting consensus. However, existing TTRL methods rely exclusively on positive pseudo-labeling strategies. Such reliance becomes vulnerable under challenging scenarios where answer distributions are highly dispersed, resulting in weak consensus that inadvertently reinforces incorrect trajectories as supervision signals. In this paper, we propose SCRL (Selective-Complementary Reinforcement Learning), a robust test-time reinforcement learning framework that effectively mitigates label noise amplification. SCRL develops Selective Positive Pseudo-Labeling, which enforces strict consensus criteria to filter unreliable majorities. Complementarily, SCRL introduces Entropy-Gated Negative Pseudo-Labeling, the first negative supervision mechanism in TTRL, to reliably prune incorrect trajectories based on generation uncertainty. Extensive experiments on multiple reasoning benchmarks demonstrate that SCRL achieves substantial improvements over baselines, while maintaining robust generalization and training stability under constrained rollout budgets. Our code is available at https://github.com/Jasper-Yan/SCRL.
Dong Yan, Jian Liang, Yanbo Wang +3
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
Test-time reinforcement learning (TTRL) always adapts models at inference time via pseudo-labeling, leaving it vulnerable to spurious optimization signals from label noise. Through an empirical study, we observe that responses with medium consistency form an ambiguity region and constitute the primary source of reward noise. Crucially, we find that such spurious signals can be even amplified through group-relative advantage estimation. Motivated by these findings, we propose a unified framework, Debiased and Denoised test-time Reinforcement Learning (DDRL), to mitigate spurious signals. Concretely, DDRL first applies a frequency-based sampling strategy to exclude ambiguous samples while maintaining a balanced set of positive and negative examples. It then adopts a debiased advantage estimation with fixed advantages, removing the bias introduced by group-relative policy optimization. Finally, DDRL incorporates a consensus-based off-policy refinement stage, which leverages the rejection-sampled dataset to enable efficient and stable model updates. Experiments on three large language models across multiple mathematical reasoning benchmarks demonstrate that DDRL consistently outperforms existing TTRL baselines. The code will soon be released at https://github.com/yuyongcan/DDRL.
Yongcan Yu, Lingxiao He, Jian Liang +5
1NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · *Work done during an internship at Meituan · School of Artificial Intelligence, University of Chinese Academy of Sciences +2