Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning

Minghui Xu; Zi Wang

cs.AIAug 28, 2026

Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning

Authors: Minghui Xu, Zi Wang

Organizations: Department of Energy Science and Engineering Stanford University

Abstract

Current large language models (LLMs) increasingly benefit from external tool integration, especially for tasks requiring reliable computation and verification. Motivated by this, we study calculator tool calling for improving mathematical reasoning on the Countdown task. We first analyze reasoning failures and find that calculation errors account for a substantial portion of incorrect responses. We then construct supervised fine-tuning datasets to teach the model useful tool-use patterns and how to interpret returned outputs. Building on this tool-formatted policy, we apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO, using automatically verifiable final-answer rewards. To enable a more reliable evaluation, we construct a fresh 1,024-problem held-out Countdown benchmark with no exact overlap with the training data. Our results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k. Among the RL methods, Tool-DAPO achieves the strongest performance, improving pass@1 from 35.8% for Tool-SFT to 66.0%. Further analysis shows that RL encourages more effective tool use even when only final-answer rewards are provided. These findings suggest that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.

Figures & tables

Figure 1: Schematic of RL for tool-integrated LLM on math reasoning.

Figure 2: SFT response outcomes by difficulty. Bars show the percentage of responses for 3-number and 4-number Countdown problems, grouped by correct answers, calculation errors, target/planning errors, their overlap, and other number-use/format errors.

Figure 3: Pass@

k

comparison on Countdown reasoning. The left panel reports results on our 1,024-problem fresh held-out benchmark, while the right panel reports results on the original 50-problem public test split.

Figure 4: Comparison of reinforcement learning methods under tool integration. The curves show pass@

k

for Tool-SFT, Tool-RLOO, Tool-RLOO++, and Tool-DAPO on the 1,024-problem fresh test set and the 50-problem public test set.

Method	1024-sample test set					50-sample test set
Method	$k{=}1$	$k{=}2$	$k{=}4$	$k{=}8$	$k{=}16$	$k{=}1$	$k{=}2$	$k{=}4$	$k{=}8$	$k{=}16$
Tool-SFT	35.8	48.9	59.6	67.6	73.7	40.1	56.0	68.5	76.1	82.0
Tool-RLOO	56.6	62.8	67.0	70.6	74.0	61.1	66.5	69.5	72.0	74.0
Tool-RLOO++	56.8	61.8	65.2	68.2	70.8	63.2	68.4	71.1	72.8	74.0
Tool-GRPO	44.8	54.5	63.0	70.3	76.7	41.1	50.6	58.1	64.3	72.0
Tool-DAPO	66.0	70.1	72.7	74.7	76.6	68.5	73.1	76.3	79.1	82.0

Table 1: Pass@

k

(%) on the 1024-sample and 50-sample Countdown test sets. Best in bold .

Figure 5: Correctness distribution across RLOO training. Each curve shows the fraction of examples with a given number of correct responses among 16 sampled responses.

Figure 6: Correctness distribution across Tool-RLOO training. The tool-augmented model shows a similar shift toward high-correctness examples, with extended training further increasing the fraction of examples for which most sampled responses are correct.

Figure 7: Correctness distribution across Tool-DAPO training. Compared with the RLOO-based baselines, Tool-DAPO substantially increases the fraction of examples with 16 out of 16 correct sampled responses, while more effectively reducing the fraction of examples with 0 out of 16 correct responses.

Figure 8: Tool-use and accuracy dynamics during Tool-RLOO training.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Numbers	$[99,11,46,48]$
Target	$44$
Reference solution	$(99-11)/(48-46)$
RLOO pass@16	$0/16$
Tool-RLOO pass@16	$4/16$

Appendix

Table 2: Example where plain RLOO fails but Tool-RLOO succeeds.

Hyperparameter	Value
Initial policy	SFT checkpoint
Dataset	Countdown / Tool-Countdown
Tool execution	No / Yes
Max turns	– / 10
Training steps	100/200
Batch size	128

Appendix

Table 3: Hyperparameters for RLOO and Tool-RLOO. The two settings share the same RL hyperparameters; Tool-RLOO additionally enables tool execution during rollout.

Hyperparameter	Value
Initial policy	Tool-SFT checkpoint
Dataset	Tool-Countdown
Tool execution	Yes
Max turns	10
Training steps	100
Batch size	128

Appendix

Table 4: Hyperparameters for Tool-RLOO++. Tool-RLOO++ follows Tool-RLOO and modifies the advantage normalization.

Hyperparameter	Value
Initial policy	SFT checkpoint
KL reference	Frozen SFT checkpoint
Dataset	Tool-Countdown
Tool execution	Yes
Max tool calls	5
Training steps	250

Appendix

Table 5: Hyperparameters for Tool-GRPO. The trainer is a faithful single-update GRPO (DeepSeekMath,

\mu=1

): the importance ratio is identically 1 and PPO-style clipping is inert.

Hyperparameter	Value
Initial policy	Tool-SFT checkpoint
Dataset	Tool-Countdown
Tool execution	Yes
Max turns	10
Training steps	100
Batch size	128

Appendix

Table 6: Hyperparameters for Tool-DAPO.

LLM Mathematical Reasoning

Large language model mathematical reasoning studies how AI systems solve, explain, verify, and generalize across mathematical problems and proofs.

See all 270 papers

Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
Mar 17, 2026Yongyu Mu, Jiali Zeng, Fandong Meng +2LLM Mathematical Reasoning Supervised Fine-Tuning
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China
Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Jun 2, 2026Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10LLM Alignment RL for Language Model Reasoning
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2
Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning
May 18, 2026Li Wang, Xiaohan Wang, Xiaodong Lu +5RL for Language Model Reasoning Hierarchical RL
Meituan · Tsinghua University

Mar 17, 2026J/K move · Enter open · S save

Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

Yongyu Mu, Jiali Zeng, Fandong Meng, JingBo Zhu+1

NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Pattern Recognition Center, WeChat AI, Tencent Inc, China

LLM Mathematical Reasoning Supervised Fine-Tuning RL Fine-Tuning

Open paper AskPDF arXiv

Through encouraging self-exploration, reinforcement learning from verifiable rewards (RLVR) has significantly advanced the mathematical reasoning capabilities of large language models. As the starting point for RLVR, the capacity of supervised fine-tuning (SFT) to memorize new chain-of-thought trajectories provides a crucial initialization that shapes the subsequent exploration landscape. However, existing research primarily focuses on facilitating exploration during RLVR training, leaving exploration-aware SFT under-explored. To bridge this gap, we propose Offline eXploration-Aware (OXA) fine-tuning. Specifically, OXA optimizes two objectives: promoting low-confidence verified teacher-distillation data to internalize previously uncaptured reasoning patterns, and suppressing high-confidence incorrect self-distillation data to redistribute probability mass of incorrect patterns toward potentially correct candidates. Experimental results across 6 benchmarks show that OXA consistently improves mathematical reasoning performance, especially achieving an average gain of

+6

Pass@1 and

+5

Pass@

k

points compared to conventional SFT on the Qwen2.5-1.5B-Math. Crucially, OXA elevates initial policy entropy, and performance gains persist throughout extensive RLVR training, demonstrating the long-term value of OXA.

Abstract

LLM Mathematical Reasoning

Explore similar work

Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning