In-Context Reinforcement Learning for Tool Use in Large Language Models

Yaoqi Ye; Yiran Zhao; Keyu Duan; Zeyu Zheng; Kenji Kawaguchi; Cihang Xie; Michael Qizhe Shieh

cs.AIMar 9, 2026

In-Context Reinforcement Learning for Tool Use in Large Language Models

Authors: Yaoqi Ye, Yiran Zhao, Keyu Duan, Zeyu Zheng, Kenji Kawaguchi, Cihang Xie, Michael Qizhe Shieh

Organizations: National University of Singapore · Salesforce AI Research · University of California Berkeley · University of California, Santa Cruz

Abstract

While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.

Figures & tables

Figure 1: ICRL training workflow. The model is trained through a multi-stage curriculum that gradually reduces the number of in-context examples in the rollout template. At each stage, the LLM generates tool-augmented rollouts, receives rewards, and updates its policy via reinforcement learning, enabling a transition from imitation to autonomous tool use.

Few-Shot Prompt Template
Solve the following problem step by step. You must conduct reasoning inside <think> … </think> every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <search> query </search> and it will return results between <information> … </information> . You can search as many times as you want. Finally, provide the answer inside <answer> … </answer> .

Here are some examples:
Example Problem: $q_{\text{demo}}$
Example Solution: <think> … </think> <search> … </search>
<information> … </information> <think> … </think> <answer> $a$ </answer>

Table 1: Few-shot rollout template in ICRL .

Violation	Rationale
No <answer> tag	Must provide structured answer
Unbalanced <answer> tags	Proper XML structure required
No <think> tag	Should demonstrate reasoning
Unbalanced <think> tags	Proper XML structure required
No <search> usage	Should utilize available tool
Empty answer content	Answer must be substantive

Table 2: Format violation penalties for computing

\mathrm{reward}_{\text{format}}

Figure 4

Model	Method	Difficult Question Answering					Average
Model	Method	TriviaQA	HotpotQA	2Wiki	Musique	Bamboogle	Average
Qwen2.5-3B	Direct	28.8	14.9	24.4	2.0	2.4	14.50
	CoT	3.2	2.1	2.1	0.2	0.0	1.52
	IRCoT	31.2	16.4	17.1	6.7	24.0	19.08
	Search-o1	47.2	22.1	21.8	5.4	32.0	25.70
	RAG	54.4	25.5	22.6	4.7	8.0	23.04

Table 3: Main Results of ICRL : Exact Match (EM) Accuracy (%) on various difficult QA datasets. The best performance is set bold . The second best performance is underlined .

Model	Method	SFT	Difficult Question Answering					Average
Model	Method	SFT	TriviaQA	HotpotQA	2Wiki	Musique	Bamboogle	Average
Qwen2.5-3B	O 2 -Searcher	✓	59.7	38.8	37.4	16.0	34.4	37.26
Qwen2.5-3B	ICRL	✗	72.6	35.4	39.2	20.0	33.6	40.16

Table 4: Results: Exact Match (EM) Accuracy (%) on various QA datasets. O 2 -Searcher applies cold-start SFT before RL, while our method (ICRL) applies RL without SFT.

Question

When did the president who set the precedent of a two term limit enter office?

Golden Answer

April 30, 1789

Model Response

<think> I need to know which president set this precedent. I also need to know when he entered office. Let me start by finding out which president it is. </think> <search> president who set two term limit precedent </search> <information> Doc 1(Title: “Twenty-second Amendment to the United States Constitution”) Until the amendment’s ratification, the president had not been subject to term limits, but both George Washington and Thomas Jefferson (the first and third … Doc 2(Title: “Amendment 22 – Term Limits for the Presidency”) The decision for George Washington – the First President of the United States – to voluntarily step down from office after two terms, established an unofficial … Doc 3(Title: “The first U.S. president set his own term limit”) Washington wrote … </information> <think> It seems George Washington set the precedent. Now I need to find out when he entered office. </think> <search> when did George Washington enter office as president </search> <information> Doc 1(Title: “Presidency of George Washington”) Washington was inaugurated as the first President of the United States on April 30, 1789… Doc 2(Title: “George Washington”) Washington was unanimously elected … Doc 3(Title: “President Washington’s First Term (1789-1792)”) George Washington was inaugurated … </information> <think> Based on the information, George Washington entered office on April 30, 1789. </think> <answer> April 30, 1789 </answer>

Table 5: Example of multi-turn search-augmented question answering. The model correctly identifies George Washington as the president who set the two-term precedent and retrieves his inauguration date through two search queries.

Panel of Figure 2 — Figure 2: Comparison of Qwen-7B trained for three stages (320) vs four stages (321~0). (a) EM Accuracy across five QA datasets. (b) Cumulative finish percent vs number of search turns, aggregated across all datasets.

Model	Method	Difficult Question Answering					Average
Model	Method	TriviaQA	HotpotQA	2Wiki	Musique	Bamboogle	Average
Qwen2.5-14B	Direct	52.0	22.6	28.2	6.0	15.2	24.80
	CoT	56.4	24.6	25.8	9.0	40.0	31.16
	ICRL	75.0	43.2	61.8	25.6	53.6	51.84

Table 6: Results: Qwen2.5-14B models Exact Match (EM) Accuracy (%) on various QA datasets. The best performance is set bold .

Panel of Figure 3 — Figure 3: Training dynamics comparison across different few-shot settings (3-shot, 2-shot, 0-shot) for Qwen-7B model.

Model	Method	SFT	Math QA
Model	Method	SFT	AIME2024	AIME2025
Qwen3-8B	ReTool	✓	67.0	49.3
Qwen3-8B	ICRL	✗	64.1	51.7

Table 7: Results: Accuracy (%) on Math QA datasets.

RL for Tool Use

Reinforcement learning for tool use trains agents to choose, execute, and learn from external tools through interaction and feedback.

See all 190 papers

TInR: Exploring Tool-Internalized Reasoning in Large Language Models
Apr 12, 2026Qiancheng Xu, Yongqi Li, Fan Liu +3LLM Tool Use Efficient Language Model Reasoning
The Hong Kong Polytechnic University · Southeast University · University of Edinburgh +1
Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
Aug 28, 2026Minghui Xu, Zi WangLLM Mathematical Reasoning RL for Tool Use
Department of Energy Science and Engineering Stanford University
ParaTool: Shifting Tool Representations from Context to Parameters
May 28, 2026Zekai Yu, Qi Meng, Qizhi Chu +3LLM Inference Efficiency LLM Tool Use
1Beijing University of Posts and Telecommunications, Beijing, China

Apr 12, 2026J/K move · Enter open · S save

TInR: Exploring Tool-Internalized Reasoning in Large Language Models

Qiancheng Xu, Yongqi Li, Fan Liu, Hongru Wang+2

The Hong Kong Polytechnic University · Southeast University · University of Edinburgh+1

LLM Tool Use Efficient Language Model Reasoning

Open paper AskPDF arXiv

Tool-Integrated Reasoning (TIR) has emerged as a promising direction by extending Large Language Models' (LLMs) capabilities with external tools during reasoning. Existing TIR methods typically rely on external tool documentation during reasoning. However, this leads to tool mastery difficulty, tool size constraints, and inference inefficiency. To mitigate these issues, we explore Tool-Internalized Reasoning (TInR), aiming at facilitating reasoning with tool knowledge internalized into LLMs. Achieving this goal presents notable requirements, including tool internalization and tool-reasoning coordination. To address them, we propose TInR-U, a tool-internalized reasoning framework for unified reasoning and tool usage. TInR-U is trained through a three-phase pipeline: 1) tool internalization with a bidirectional knowledge alignment strategy; 2) supervised fine-tuning warm-up using high-quality reasoning annotations, and 3) reinforcement learning with TInR-specific rewards. We comprehensively evaluate our method across in-domain and out-of-domain settings. Experiment results show that TInR-U achieves superior performance in both settings, highlighting its effectiveness and efficiency.

Abstract

RL for Tool Use

Explore similar work

TInR: Exploring Tool-Internalized Reasoning in Large Language Models

Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning

ParaTool: Shifting Tool Representations from Context to Parameters