While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.
Figures & tables
Figure 1: ICRL training workflow. The model is trained through a multi-stage curriculum that gradually reduces the number of in-context examples in the rollout template. At each stage, the LLM generates tool-augmented rollouts, receives rewards, and updates its policy via reinforcement learning, enabling a transition from imitation to autonomous tool use.
Few-Shot Prompt Template
Solve the following problem step by step. You must conduct reasoning inside <think> … </think> every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <search> query </search> and it will return results between <information> … </information> . You can search as many times as you want. Finally, provide the answer inside <answer> … </answer> .
Here are some examples:
Example Problem: qdemo
Example Solution: <think> … </think> <search> … </search>
<information> … </information> <think> … </think> <answer> a </answer>
Table 1: Few-shot rollout template in ICRL .
Violation
Rationale
No <answer> tag
Must provide structured answer
Unbalanced <answer> tags
Proper XML structure required
No <think> tag
Should demonstrate reasoning
Unbalanced <think> tags
Proper XML structure required
No <search> usage
Should utilize available tool
Empty answer content
Answer must be substantive
Table 2: Format violation penalties for computing rewardformat .
Figure 4
Model
Method
Difficult Question Answering
Average
TriviaQA
HotpotQA
2Wiki
Musique
Bamboogle
Qwen2.5-3B
Direct
28.8
14.9
24.4
2.0
2.4
14.50
CoT
3.2
2.1
2.1
0.2
0.0
1.52
IRCoT
31.2
16.4
17.1
6.7
24.0
19.08
Search-o1
47.2
22.1
21.8
5.4
32.0
25.70
RAG
54.4
25.5
22.6
4.7
8.0
23.04
Table 3: Main Results of ICRL : Exact Match (EM) Accuracy (%) on various difficult QA datasets. The best performance is set bold . The second best performance is underlined .
Model
Method
SFT
Difficult Question Answering
Average
TriviaQA
HotpotQA
2Wiki
Musique
Bamboogle
Qwen2.5-3B
O 2 -Searcher
✓
59.7
38.8
37.4
16.0
34.4
37.26
ICRL
✗
72.6
35.4
39.2
20.0
33.6
40.16
Table 4: Results: Exact Match (EM) Accuracy (%) on various QA datasets. O 2 -Searcher applies cold-start SFT before RL, while our method (ICRL) applies RL without SFT.
Question
When did the president who set the precedent of a two term limit enter office?
Golden Answer
April 30, 1789
Model Response
<think> I need to know which president set this precedent. I also need to know when he entered office. Let me start by finding out which president it is. </think> <search> president who set two term limit precedent </search> <information> Doc 1(Title: “Twenty-second Amendment to the United States Constitution”) Until the amendment’s ratification, the president had not been subject to term limits, but both George Washington and Thomas Jefferson (the first and third … Doc 2(Title: “Amendment 22 – Term Limits for the Presidency”) The decision for George Washington – the First President of the United States – to voluntarily step down from office after two terms, established an unofficial … Doc 3(Title: “The first U.S. president set his own term limit”) Washington wrote … </information> <think> It seems George Washington set the precedent. Now I need to find out when he entered office. </think> <search> when did George Washington enter office as president </search> <information> Doc 1(Title: “Presidency of George Washington”) Washington was inaugurated as the first President of the United States on April 30, 1789… Doc 2(Title: “George Washington”) Washington was unanimously elected … Doc 3(Title: “President Washington’s First Term (1789-1792)”) George Washington was inaugurated … </information> <think> Based on the information, George Washington entered office on April 30, 1789. </think> <answer> April 30, 1789 </answer>
Table 5: Example of multi-turn search-augmented question answering. The model correctly identifies George Washington as the president who set the two-term precedent and retrieves his inauguration date through two search queries.
Figure 2: Comparison of Qwen-7B trained for three stages (320) vs four stages (321~0). (a) EM Accuracy across five QA datasets. (b) Cumulative finish percent vs number of search turns, aggregated across all datasets.
Model
Method
Difficult Question Answering
Average
TriviaQA
HotpotQA
2Wiki
Musique
Bamboogle
Qwen2.5-14B
Direct
52.0
22.6
28.2
6.0
15.2
24.80
CoT
56.4
24.6
25.8
9.0
40.0
31.16
ICRL
75.0
43.2
61.8
25.6
53.6
51.84
Table 6: Results: Qwen2.5-14B models Exact Match (EM) Accuracy (%) on various QA datasets. The best performance is set bold .
Figure 3: Training dynamics comparison across different few-shot settings (3-shot, 2-shot, 0-shot) for Qwen-7B model.
Model
Method
SFT
Math QA
AIME2024
AIME2025
Qwen3-8B
ReTool
✓
67.0
49.3
ICRL
✗
64.1
51.7
Table 7: Results: Accuracy (%) on Math QA datasets.
Tool-Integrated Reasoning (TIR) has emerged as a promising direction by extending Large Language Models' (LLMs) capabilities with external tools during reasoning. Existing TIR methods typically rely on external tool documentation during reasoning. However, this leads to tool mastery difficulty, tool size constraints, and inference inefficiency. To mitigate these issues, we explore Tool-Internalized Reasoning (TInR), aiming at facilitating reasoning with tool knowledge internalized into LLMs. Achieving this goal presents notable requirements, including tool internalization and tool-reasoning coordination. To address them, we propose TInR-U, a tool-internalized reasoning framework for unified reasoning and tool usage. TInR-U is trained through a three-phase pipeline: 1) tool internalization with a bidirectional knowledge alignment strategy; 2) supervised fine-tuning warm-up using high-quality reasoning annotations, and 3) reinforcement learning with TInR-specific rewards. We comprehensively evaluate our method across in-domain and out-of-domain settings. Experiment results show that TInR-U achieves superior performance in both settings, highlighting its effectiveness and efficiency.
Qiancheng Xu, Yongqi Li, Fan Liu +3
The Hong Kong Polytechnic University · Southeast University · University of Edinburgh +1
Current large language models (LLMs) increasingly benefit from external tool integration, especially for tasks requiring reliable computation and verification. Motivated by this, we study calculator tool calling for improving mathematical reasoning on the Countdown task. We first analyze reasoning failures and find that calculation errors account for a substantial portion of incorrect responses. We then construct supervised fine-tuning datasets to teach the model useful tool-use patterns and how to interpret returned outputs. Building on this tool-formatted policy, we apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO, using automatically verifiable final-answer rewards. To enable a more reliable evaluation, we construct a fresh 1,024-problem held-out Countdown benchmark with no exact overlap with the training data. Our results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k. Among the RL methods, Tool-DAPO achieves the strongest performance, improving pass@1 from 35.8% for Tool-SFT to 66.0%. Further analysis shows that RL encourages more effective tool use even when only final-answer rewards are provided. These findings suggest that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.
Minghui Xu, Zi Wang
Department of Energy Science and Engineering Stanford University
Tool calling extends large language models (LLMs) by enabling grounded interaction with external executable interfaces, thereby supporting environment-coupled problem solving. However, mainstream in-context learning (ICL) approaches typically incorporate detailed tool documentation and usage examples directly into the context. This results in substantial inference overhead and heightened risks of hallucination as the context length grows. Conversely, while tuning-based methods improve general tool-calling capabilities, they often fail to effectively internalize the specific details of previously seen tools, thereby retaining a dependency on in-context documentation. To address these limitations, we propose ParaTool, a framework that projects each tool into a dedicated, loadable set of parameters. By equipping a dynamic integration of these parameterized tools, the LLM can perform tool calling without relying on in-context documents or examples. Specifically, our approach consists of three stages: (1) parametric tool pre-training encapsulates the knowledge of different tools into independent parameter modules; (2) soft tool selection employs a gating network to dynamically weigh and aggregate relevant tool parameters; and (3) parametric tool fine-tuning jointly updates tool parameters to align the training and inference processes. Experiments on Stable ToolBench and BFCL demonstrate that ParaTool significantly outperforms strong ICL-based baselines, achieving superior performance while reducing computational complexity.
Zekai Yu, Qi Meng, Qizhi Chu +3
1Beijing University of Posts and Telecommunications, Beijing, China