Tool Verification for Test-Time Reinforcement Learning

Ruotong Liao; Nikolai Röhrich; Xiaohan Wang; Yuhui Zhang; Yasaman Samadzadeh; Volker Tresp; Serena Yeung-Levy

cs.AIMar 2, 2026

Tool Verification for Test-Time Reinforcement Learning

Authors: Ruotong Liao, Nikolai Röhrich, Xiaohan Wang, Yuhui Zhang, Yasaman Samadzadeh, Volker Tresp, Serena Yeung-Levy

Organizations: Ludwig-Maximilians University of Munich · MCML · Stanford University

Abstract

Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for Recursive Self-Improving AI (RSI) by adapting Large Reasoning Models (LRMs) on unlabeled test inputs, using self-consensus rewards derived from majority voting over sampled rollouts. However, majority voting can mistake popularity for correctness: a spurious yet high-frequency unverified consensus may become a biased reward signal, causing test-time RL to reinforce frequent but wrong answers and collapse into an incorrect mode. We address this false-popular failure mode with T $^3$ RL (Tool-Verification for Test-Time Reinforcement Learning), a verification-aware test-time RL framework. T $^3$ RL grounds pseudo-label construction in external tool evidence. Concretely, a verifier utilizes external tool evidence (e.g., from code execution) to upweight verified rollouts during a verification-aware voting, producing more reliable pseudo-labels for training. Across various math difficulties (MATH-500, AMC, and AIME 2024) and diverse backbone families, T $^3$ RL significantly improves over TTRL, with better performance on harder problems. T $^3$ RL is positioned as a verified online data synthesizer, highlighting the role of tool verification in reliable online adaptation. T $^3$ RL is extensible to verifiable domains and provides training and inference efficiency.

Figures & tables

Figure 1: The concept of T 3 RL . Top: T 3 RL introduces test-time verification into self-evolvement via tool-executed evidence (e.g., code interpreter) to stabilize training with verified rollouts. Bottom: Majority-vote pseudo-labels can be spurious. T 3 RL introduces verification to suppress false-popular pseudo-labels. Right: T 3 RL achieves consistent gains with evidence-grounded test-time training.

Figure 2: T 3 RL: Tool Verification for Test-Time Reinforcement Learning. Verifier: an LLM verifier parses each sampled rollout $y_{i}$ into a candidate answer $a_{i}$ and, with tool feedback, produces a binary validity flag $v_{i}\in\{0,1\}$ . Tool verification: the verifier compiles the rollout’s claimed computations into lightweight Python and queries a code interpreter to obtain executable evidence. Verification-weighted majority voting: unverified rollouts receive unit vote mass, and verified rollouts receive $\omega$ vote mass. A verification-aware pseudo-label $\tilde{y}^{*}$ is formed by weighted voting.

Panel of Figure 3 — Figure 3: Unverified consensus versus tool-verified consensus. ( 3(a) ) Spurious reward in the reinforced cycle of TTRL : the false-popular answer is rewarded and its vote share grows over rounds. ( 3(b) ) Tool verification can adjust the false-popular answer.

Panel of Figure 4 — Figure 4: False-majority share across benchmarks.

Panel of Table 1 — Table 1: Main performance of T 3 RL. (a) Improvement across benchmark and backbone models. (b) Difficulty breakdown performance on MATH-500 (L5 is the hardest). The relative improvement of T 3 RL over TTRL increases with difficulty level ( Table 1 ).

Panel of Figure 6 — Figure 6: Ablation on verifier and verification tool . Left: Adding an LLM verifier improves TTRL even without tool execution. Right: Code execution significantly strengthens verification.

Policy	Verifier	Acc.	Prec.	Rec.	$\Delta$ AIME
Math-1.5B	Coder-1.5B	73.2	25.0	51.5	$+5.0$
Llama-3.2-3B	Coder-1.5B	73.3	39.4	59.1	$+3.8$
Qwen-3-4B	Coder-1.5B	75.0	38.2	40.7	$+3.6$

Table 3: Verifier quality on policy rollouts.

Panel of Figure 8 — Figure 8: Ablations on tool position and training robustness. Left: allowing the policy to call tools during rollouts ( TTRL-Agent ) can hurt performance, while T 3 RL improves by restricting tool use to the verifier for reward shaping. Right: verifier-shaped rewards reduce run-to-run variability, indicating more stable optimization under unlabeled test-time RL.

Table 9

Panel of Table 6 — Table 6: Non-math generalization on Knights-and-Knaves puzzles with an SMT-LIB2/Z3 example.

Adaptation setting AIME 2024 AMC MATH-500 TTRL adapted on AIME

15.8\pm 2.1

42.9\pm 2.4

65.5\pm 2.4

T 3 RL adapted on AIME

\mathbf{20.8\pm 1.6}

45.4\pm 2.5

68.0\pm 2.0

TTRL adapted on AMC

11.5\pm 1.6

48.9\pm 1.6

66.6\pm 1.4

T 3 RL adapted on AMC

\mathbf{15.1\pm 2.0}

\mathbf{50.9\pm 2.7}

\mathbf{70.4\pm 1.7}

TTRL adapted on MATH-500

10.8\pm 3.4

43.2\pm 2.7

73.0\pm 3.8

T 3 RL adapted on MATH-500

\mathbf{14.7\pm 1.9}

\mathbf{45.7\pm 2.0}

\mathbf{74.6\pm 3.2}

Accuracy gain (pp) A C M A C M

+5.0

+2.5

+2.5

+3.6

+2.0

+3.8

+3.9

+2.5

+1.6

T 3 RL better

Table 7: Cross-dataset adaptation (A/C/M: AIME/AMC/MATH-500). Inset: accuracy gain.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: System Prompt for the Verifier.

Panel of Figure 10 — Figure 10: Qualitative examples of the two primary failure modes encountered when deploying an undersized verifier.

Model / Method	AIME 2024	AMC	MATH-500	Avg
Qwen-2.5-0.5B Vanilla (Baseline)	0.0	4.8	7.9	4.2
w/ TTRL	0.4	12.0	34.6	15.7
w/ T 3 RL (0.5B verifier)	0.0	10.8	32.0	14.3
$\Delta$ ( T 3 RL $-$ TTRL)	-0.4	-1.2	-2.6	-1.4
Rel. (% over TTRL)	$\downarrow$ 100.0%	$\downarrow$ 10.0%	$\downarrow$ 7.5%	$\downarrow$ 8.9%

Appendix

Table 8: Performance comparison illustrating failure cases with a weak (0.5B) verifier.

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 48 papers

Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification
Jun 2, 2026Jiahui Li, Jianfeng Shan, Wenpei Chen +5Reinforcement Learning LLM Answer Verification
1Sun Yat-Sen University · University of Science and Technology of China · 3National University of Singapore
What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time
Mar 20, 2026Dong Yan, Jian Liang, Yanbo Wang +3RL for Language Model Reasoning Test-Time RL
School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Nanjing University
Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning
Apr 23, 2026Yongcan Yu, Lingxiao He, Jian Liang +5Reinforcement Learning RL for Language Model Reasoning
1NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · *Work done during an internship at Meituan · School of Artificial Intelligence, University of Chinese Academy of Sciences +2

Jun 2, 2026J/K move · Enter open · S save

Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification

Jiahui Li, Jianfeng Shan, Wenpei Chen, Shunyu Wu+4

1Sun Yat-Sen University · University of Science and Technology of China · 3National University of Singapore

Reinforcement Learning LLM Answer Verification RL for Language Model Reasoning Test-Time RL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Test-time reinforcement learning has emerged as a promising paradigm for enhancing the complex reasoning abilities of large language models in a completely label-free manner. Despite existing studies focusing on Pass@1 performance, optimizing Pass@k remains under-explored yet critical in label-free settings, which measures generation coverage for sustained exploration. Optimizing Pass@k in label-free setting is highly non-trivial, as directly applying the Pass@k advantage designs effective for RLVR yields unsatisfactory performance. Through in-depth empirical analysis, we discover the root causes hindering performance: pseudo-label estimations for low-confidence samples have a high probability of being incorrect, while candidate answers for high-confidence samples suffer from severe diversity collapse. To overcome these hurdles, we propose TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), a novel confidence-adaptive framework that expands Pass@k coverage and improves Pass@1 performance. Based on our key insight that verification capability generally leads generation capability, TTRL-CoCoV employs a confidence-conditioned mechanism: for high-confidence samples, it bootstraps verifier and applies an exploration-enhancing reward to prevent diversity collapse; for low-confidence samples, it delegates pseudo-label selection to the verifier to filter incorrect pseudo-labels; and for medium-confidence samples, it bypasses verification entirely. Extensive experiments demonstrate that TTRL-CoCoV outperforms the best competing methods across 6 widely-recognized benchmarks, achieves average absolute gains of +9.8% in Pass@1 and +18.7% in Pass@16 over TTRL, and even achieves absolute Pass@1 improvements of up to +5.0% across multiple reasoning benchmarks when compared against fully supervised RL methods. Our code repository: https://github.com/shanjf666/CoCoV.

Abstract

Test-Time RL

Explore similar work

Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification

What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time

Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning