Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning

Minghui Xu; Zi Wang

cs.AIAug 28, 2026

Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning

Authors: Minghui Xu, Zi Wang

Organizations: Department of Energy Science and Engineering Stanford University

Abstract

Current large language models (LLMs) increasingly benefit from external tool integration, especially for tasks requiring reliable computation and verification. Motivated by this, we study calculator tool calling for improving mathematical reasoning on the Countdown task. We first analyze reasoning failures and find that calculation errors account for a substantial portion of incorrect responses. We then construct supervised fine-tuning datasets to teach the model useful tool-use patterns and how to interpret returned outputs. Building on this tool-formatted policy, we apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO, using automatically verifiable final-answer rewards. To enable a more reliable evaluation, we construct a fresh 1,024-problem held-out Countdown benchmark with no exact overlap with the training data. Our results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k. Among the RL methods, Tool-DAPO achieves the strongest performance, improving pass@1 from 35.8% for Tool-SFT to 66.0%. Further analysis shows that RL encourages more effective tool use even when only final-answer rewards are provided. These findings suggest that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.

Figures & tables

Figure 1: Schematic of RL for tool-integrated LLM on math reasoning.

Figure 2: SFT response outcomes by difficulty. Bars show the percentage of responses for 3-number and 4-number Countdown problems, grouped by correct answers, calculation errors, target/planning errors, their overlap, and other number-use/format errors.

Figure 3: Pass@

k

comparison on Countdown reasoning. The left panel reports results on our 1,024-problem fresh held-out benchmark, while the right panel reports results on the original 50-problem public test split.

Figure 4: Comparison of reinforcement learning methods under tool integration. The curves show pass@

k

for Tool-SFT, Tool-RLOO, Tool-RLOO++, and Tool-DAPO on the 1,024-problem fresh test set and the 50-problem public test set.

Method	1024-sample test set					50-sample test set
Method	$k{=}1$	$k{=}2$	$k{=}4$	$k{=}8$	$k{=}16$	$k{=}1$	$k{=}2$	$k{=}4$	$k{=}8$	$k{=}16$
Tool-SFT	35.8	48.9	59.6	67.6	73.7	40.1	56.0	68.5	76.1	82.0
Tool-RLOO	56.6	62.8	67.0	70.6	74.0	61.1	66.5	69.5	72.0	74.0
Tool-RLOO++	56.8	61.8	65.2	68.2	70.8	63.2	68.4	71.1	72.8	74.0
Tool-GRPO	44.8	54.5	63.0	70.3	76.7	41.1	50.6	58.1	64.3	72.0
Tool-DAPO	66.0	70.1	72.7	74.7	76.6	68.5	73.1	76.3	79.1	82.0

Table 1: Pass@

k

(%) on the 1024-sample and 50-sample Countdown test sets. Best in bold .

Figure 5: Correctness distribution across RLOO training. Each curve shows the fraction of examples with a given number of correct responses among 16 sampled responses.

Figure 6: Correctness distribution across Tool-RLOO training. The tool-augmented model shows a similar shift toward high-correctness examples, with extended training further increasing the fraction of examples for which most sampled responses are correct.

Figure 7: Correctness distribution across Tool-DAPO training. Compared with the RLOO-based baselines, Tool-DAPO substantially increases the fraction of examples with 16 out of 16 correct sampled responses, while more effectively reducing the fraction of examples with 0 out of 16 correct responses.

Figure 8: Tool-use and accuracy dynamics during Tool-RLOO training.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Numbers	$[99,11,46,48]$
Target	$44$
Reference solution	$(99-11)/(48-46)$
RLOO pass@16	$0/16$
Tool-RLOO pass@16	$4/16$

Appendix

Table 2: Example where plain RLOO fails but Tool-RLOO succeeds.

Hyperparameter	Value
Initial policy	SFT checkpoint
Dataset	Countdown / Tool-Countdown
Tool execution	No / Yes
Max turns	– / 10
Training steps	100/200
Batch size	128

Appendix

Table 3: Hyperparameters for RLOO and Tool-RLOO. The two settings share the same RL hyperparameters; Tool-RLOO additionally enables tool execution during rollout.

Hyperparameter	Value
Initial policy	Tool-SFT checkpoint
Dataset	Tool-Countdown
Tool execution	Yes
Max turns	10
Training steps	100
Batch size	128

Appendix

Table 4: Hyperparameters for Tool-RLOO++. Tool-RLOO++ follows Tool-RLOO and modifies the advantage normalization.

Hyperparameter	Value
Initial policy	SFT checkpoint
KL reference	Frozen SFT checkpoint
Dataset	Tool-Countdown
Tool execution	Yes
Max tool calls	5
Training steps	250

Appendix

Table 5: Hyperparameters for Tool-GRPO. The trainer is a faithful single-update GRPO (DeepSeekMath,

\mu=1

): the importance ratio is identically 1 and PPO-style clipping is inert.

Hyperparameter	Value
Initial policy	Tool-SFT checkpoint
Dataset	Tool-Countdown
Tool execution	Yes
Max turns	10
Training steps	100
Batch size	128

Appendix

Table 6: Hyperparameters for Tool-DAPO.

LLM Mathematical Reasoning

Large language model mathematical reasoning studies how AI systems solve, explain, verify, and generalize across mathematical problems and proofs.

See all 270 papers

Mar 17, 2026

Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

Through encouraging self-exploration, reinforcement learning from verifiable rewards (RLVR) has significantly advanced the mathematical reasoning capabilities of large language models. As the starting point for RLVR, the capacity of supervised fine-tuning (SFT) to memorize new chain-of-thought trajectories provides a crucial initialization that shapes the subsequent exploration landscape. However, existing research primarily focuses on facilitating exploration during RLVR training, leaving exploration-aware SFT under-explored. To bridge this gap, we propose Offline eXploration-Aware (OXA) fine-tuning. Specifically, OXA optimizes two objectives: promoting low-confidence verified teacher-distillation data to internalize previously uncaptured reasoning patterns, and suppressing high-confidence incorrect self-distillation data to redistribute probability mass of incorrect patterns toward potentially correct candidates. Experimental results across 6 benchmarks show that OXA consistently improves mathematical reasoning performance, especially achieving an average gain of

+6

Pass@1 and

+5

Pass@

k

points compared to conventional SFT on the Qwen2.5-1.5B-Math. Crucially, OXA elevates initial policy entropy, and performance gains persist throughout extensive RLVR training, demonstrating the long-term value of OXA.

Yongyu Mu, Jiali Zeng, Fandong Meng +2

NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China

LLM Mathematical Reasoning Supervised Fine-Tuning RL Fine-Tuning

Ask PDF

Jun 2, 2026

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.

Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10

1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2

LLM Alignment RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Ask PDF

May 18, 2026

Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning

Large language models (LLMs) have increasingly leveraged tool invocation to enhance their reasoning capabilities. However, existing approaches typically tightly couple tool invocation with immediate execution. Such immediate tool interaction may disrupt the reasoning coherence of LLMs and constrain their expressivity, ultimately degrading reasoning performance. To this end, for the first time, we propose and formalize the problem of decoupling tool invocation from execution during reasoning, and introduce delayed execution with explicit control to enhance tool-integrated reasoning (TIR). Furthermore, we propose a hierarchical control framework and theoretically derive a surrogate loss that enables an implicitly hierarchical policy to learn behavior equivalent to that of an explicit hierarchical policy, leading to the proposed IH-GRPO algorithm. Extensive experiments on IH-GRPO achieve absolute improvements of 1.87%, 2.16%, and 2.53% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across six out-of-domain mathematical reasoning benchmarks over the strongest baseline method, while also yielding consistent performance gains in other domains. Our code is available at https://github.com/Lumina04/IH-GRPO-01.

Li Wang, Xiaohan Wang, Xiaodong Lu +5

Meituan · Tsinghua University

RL for Language Model Reasoning Hierarchical RL LLM Tool Use LLM Mathematical Reasoning RL for Tool Use

Ask PDF

Abstract

LLM Mathematical Reasoning

Explore similar work

Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning