E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning

Weiyang Guo; Zesheng Shi; Liye Zhao; Jiayuan Ma; Zeen Zhu; Junxian He; Min Zhang; Jing Li

cs.AIApr 10, 2026

E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning

Authors: Weiyang Guo, Zesheng Shi, Liye Zhao, Jiayuan Ma, Zeen Zhu, Junxian He, Min Zhang, Jing Li

Organizations: Harbin Institute of Technology, Shenzhen, China · Huawei Technologies Co., Ltd. · Hong Kong University of Science and Technology

Abstract

While Large Language Models (LLMs) have demonstrated significant potential in Tool-Integrated Reasoning (TIR), existing training paradigms face significant limitations: Zero-RL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3-TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert "anchors" and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model's knowledge boundaries, effectively balancing exploration diversity with training efficiency.Experimental results demonstrate that E3-TIR achieves a 6 performance improvement over traditional paradigms on tool-use tasks, while requiring less than 10 of the synthetic data. Furthermore, in terms of ROI, a comprehensive metric integrating performance, data cost, and training efficiency we achieve a 1.46x gain compared to baselines. Code is available at https://github.com/yuki-younai/E3-TIR.

Figures & tables

Figure 1: (a) Zero-RL: Inefficient exploration and slow convergence. (b) SFT+RL: Vulnerable to overfitting and performance collapse after a high start. (c)

\text{E}^{3}

-TIR: Enables rapid adaptation via expert guidance with stable, continuous growth.

Figure 2: Statistical analysis of the limitations of current training paradigms.

Figure 3: Illustration of our $\text{E}^{3}$ -TIR framework. (a) Branching exploration from expert anchors and dynamic experience filtering. (b) Hybrid advantage estimation and advantage-aware gradient detachment.

Figure 4: Distribution of Standard Deviation and Entropy under Different sampling strategies.

Method	Mathematical Reasoning					Knowledge-Intensive Reasoning					Avg.	RMR.
	AIME24	AIME25	AMC23	GSM8K	MATH	2Wiki.	Bamb.	HQA	MuSi.	SimQA	Avg.	RMR.
Qwen2.5-3B-Instruct	4.5	8.6	38.3	65.3	81.0	19.2	27.5	11.5	6.3	5.9	26.8	$-$
+ Only SFT	6.0	10.8	35.6	58.3	81.0	30.3	35.2	35.8	16.3	19.5	32.9	7.4
+ SFT-then-RL	11.2	16.5	51.8	69.1	85.1	54.2	47.3	50.5	25.3	30.6	44.2	6.1
+ Zero-RL	12.0	13.3	48.7	71.9	85.4	46.7	50.8	48.8	23.2	31.1	43.2	7.9
+ $\text{E}^{3}$ -TIR (Ours)	19.0	20.8	56.0	72.3	85.2	52.3	50.3	51.4	26.9	32.6	46.7	6.2

Table 1: Overall results on 10 challenging reasoning tasks. The top two results are highlighted in bold and underlined . The abbreviations of the datasets: Math (Math500), HQA (HotpotQA), 2Wiki. (2wikiMultiHopQA), MuSi. (MuSiQue), SimQA (SimpleQA) and Bamb (Bamboogle).

Panel of Figure 5 — Figure 5: Comparison of the training curves on Qwen2.5-3B-Instruct.

LLM-based Math Agent
Method	Computational Reasoning					Knowledge-Intensive Reasoning					Avg.
	AIME24	AIME25	AMC23	GSM8K	MATH	2Wiki.	Bamb.	HQA	MuSi.	SimQA	Avg.
Qwen2.5-3B-Instruct	4.5	8.6	38.3	65.3	81.0	19.2	27.5	11.5	6.3	5.9	26.8
ToRL	10.0	20.0	53.0	72.0	84.4	24.0	25.4	17.9	8.3	6.5	32.2
SimpleTIR	17.5	19.1	54.3	69.4	83.9	16.1	24.2	16.5	7.2	8.8	31.7
LLM-based Search Agent

Table 2: Overall results on 10 challenging reasoning tasks. The top two results are highlighted in bold and underlined . All baseline methods use Qwen2.5-3B as the backbone. For baselines lacking a 3B Instruct version, we replicate them using the exact parameter settings from their open-source code.

Figure 6: Ablation analysis of different hyper-parameters in Expert Branch Sample.

Figure 7: Ablation analysis of clip ratio and gradient norm on different mix policy optimization.

Method	Knowledge.		Computational.
Method	HotQA	Musique.	ACM23	MATH
$\text{E}^{3}$ -TIR (Warm up)	44.9	19.9	43.3	65.0
w/o $\hat{A}_{\text{global}}$	$35.4_{{\color[rgb]{1,0,0}(-9.5)}}$	$13.7_{{\color[rgb]{1,0,0}(-6.2)}}$	$36.8_{{\color[rgb]{1,0,0}(-6.5)}}$	$56.3_{{\color[rgb]{1,0,0}(-8.7)}}$
w/o $\hat{A}_{\text{tree}}$	$42.6_{{\color[rgb]{1,0,0}(-2.3)}}$	$17.8_{{\color[rgb]{1,0,0}(-2.1)}}$	$43.5_{{\color[rgb]{1,0,0}(+0.2)}}$	$63.1_{{\color[rgb]{1,0,0}(-1.9)}}$

Table 3: Ablation on Hybrid Advantages Estimation.

Method	Knowledge.		Computational.
Method	HotQA	Musique.	AMC23	MATH
$\text{E}^{3}$ -TIR	51.4	26.9	56.0	85.2
w/o Branch	$47.2_{{\color[rgb]{1,0,0}(-4.2)}}$	$23.1_{{\color[rgb]{1,0,0}(-3.8)}}$	$52.3_{{\color[rgb]{1,0,0}(-3.7)}}$	$83.6_{{\color[rgb]{1,0,0}(-1.6)}}$
w/o AAGD	$15.3_{{\color[rgb]{1,0,0}(-36.1)}}$	$8.2_{{\color[rgb]{1,0,0}(-18.7)}}$	$24.8_{{\color[rgb]{1,0,0}(-31.2)}}$	$48.6_{{\color[rgb]{1,0,0}(-36.6)}}$
w/o Hybrid Adv.	$49.5_{{\color[rgb]{1,0,0}(-1.9)}}$	$25.6_{{\color[rgb]{1,0,0}(-1.3)}}$	$55.2_{{\color[rgb]{1,0,0}(-0.8)}}$	$84.1_{{\color[rgb]{1,0,0}(-1.1)}}$
Mix SFT+RL	$22.4_{{\color[rgb]{1,0,0}(-29.0)}}$	$11.3_{{\color[rgb]{1,0,0}(-15.6)}}$	$32.5_{{\color[rgb]{1,0,0}(-23.5)}}$	$56.8_{{\color[rgb]{1,0,0}(-28.4)}}$

Table 4: Component ablation on Qwen2.5-3B-Instruct. w/o Branch: remove expert-guided branching; w/o AAGD: remove advantage-aware gradient detaching; w/o Hybrid Adv.: remove expert-tree advantage; Mix SFT+RL: weighted SFT+RL Lv et al. (2025) .

Figure 8: Comparison of Code Failed Rate and Solve-None curves across different training stages.

Method	Avg Score	Avg Calls	Fail Rate	Redundancy
Only SFT	32.9	2.24	4.5%	0.6%
SFT-then-RL	44.2	1.82	3.8%	0.8%
Zero-RL	43.2	2.52	7.4%	0.5%
$\text{E}^{3}$ -TIR	46.7	1.97	4.0%	0.6%

Table 5: Tool-calling behavior audit on Qwen2.5-3B-Instruct. Avg Calls: average tool invocations per query; Fail Rate: proportion of invalid tool calls; Redundancy: ratio of repeated calls.

Figure 9: (a) Cost-Benefit Heatmap during the Warm-up Stage. (b) Comparison with Other Mix Strategies.

Method	ALFWorld	SCIWorld	Avg.
Only SFT	62.5	43.0	52.8
SFT-then-RL	81.5	65.0	73.3
Zero-RL	86.5	63.5	75.0
$\text{E}^{3}$ -TIR	89.0	67.5	78.3

Table 6: Generalization on long-horizon agent tasks using Qwen2.5-3B-Instruct. Expert data: 2K trajectories from AgentGYM agentgym per task.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Warm Up	Post-RL	Zero-RL
$\gamma$	0.5	0	0
$\lambda$	0.1	0	0
n	8	16	16
m	8	0	0
k	3	0	0
optimizer	AdamW	AdamW	AdamW

Appendix

Table 7: Hyperparameters and configurations for different Training.

Table 17

Figure 10: Variation of the sample size and rewards in the preheating stage of the mixed strategy.

Example from Musique
Question:
What country has the border troops of the country of the literature of the country of Karl Maron citizenship?
Labeled Answer:
East Germany
Model Output by $\text{E}^{3}$ -TIR (Qwen2.5-3B)
<think> To solve this question, I need to find out the country whose border troops have the citizenship of the country where the literature of the country of Karl Maron is from.

Appendix

Table 10: An example from

\text{E}^{3}

-TIR on the Musique dataset, with special symbols used in think content, search queries, returned results and final answer highlighted with purple box , green box and red box , respectively.

Example from Dataset
Question:
Find the number of ordered pairs $(x,y)$ , where both $x$ and $y$ are integers between $-100$ and $100$ inclusive, such that $12x^{2}-xy-6y^{2}=0$ .
Labeled Answer: 117
Model Output by $\text{E}^{3}$ -TIR (Qwen2.5-3B)
<think> To solve the problem, we need to find the number of ordered pairs $(x,y)$ such that both $x$ and $y$ are integers between $-100$ and $100$ inclusive, and they satisfy the equation $12x^{2}-xy-6y^{2}=0$ .
First, let’s rearrange the equation to make it easier to solve: $12x^{2}-xy-6y^{2}=0.$ This is a quadratic equation in terms of $x$ . We can use the quadratic formula to solve for $x$ : $12x^{2}-xy-6y^{2}=0.$ The quadratic formula for $ax^{2}+bx+c=0$ is given by: $x=\frac{-b\pm\sqrt{b^{2}-4ac}}{2a}.$ In our equation, $a=12$ , $b=-y$ , and $c=-6y^{2}$ . So the solutions for $x$ are: $x=\frac{y\pm\sqrt{y^{2}+24\cdot 6y^{2}}}{2\cdot 12}=\frac{y\pm\sqrt{y^{2}+144y^{2}}}{24}=\frac{y\pm\sqrt{145y^{2}}}{24}=\frac{y\pm y\sqrt{145}}{24}.$ This gives us two solutions: $x=\frac{y(1+\sqrt{145})}{24}\quad\text{and}\quad x=\frac{y(1-\sqrt{145})}{24}.$ For $x$ to be an integer, $\frac{y(1+\sqrt{145})}{24}$ and $\frac{y(1-\sqrt{145})}{24}$ must be integers. Since $\sqrt{145}$ is not an integer, $y$ must be a multiple of 24 for $\frac{y(1+\sqrt{145})}{24}$ and $\frac{y(1-\sqrt{145})}{24}$ to be integers. Let’s denote $y=24k$ , where $k$ is an integer. Then the solutions for $x$ become: $x=\frac{24k(1+\sqrt{145})}{24}=1+\sqrt{145}k\quad\text{and}\quad x=\frac{24k(1-\sqrt{145})}{24}=1-\sqrt{145}k.$ Since $\sqrt{145}$ is not an integer, $k$ must be such that $1+\sqrt{145}k$ and $1-\sqrt{145}k$ are integers. This means $k$ must be an integer that makes $1+\sqrt{145}k$ and $1-\sqrt{145}k$ integers. The only way this can happen is if $k=0$ , but $k=0$ would make $x=1$ . However, we need to check if there are any other values of $k$ that might work.

Appendix

Table 11: An example from

\text{E}^{3}

-TIR on the AIME25 dataset, with special symbols used in think content.

Example from Dataset
Question: Let $p$ be the least prime number for which there exists a positive integer $n$ such that $n^{4}+1$ is divisible by $p^{2}$ . Find the least positive integer $m$ such that $m^{4}+1$ is divisible by $p^{2}$ . Labeled Answer: 110
Model Output by $\text{E}^{3}$ -TIR (Qwen2.5-3B)
<think> To solve this problem, we need to follow these steps: 1. Identify the smallest prime number $p$ for which there exists a positive integer $n$ such that $n^{4}+1$ is divisible by $p^{2}$ . 2. Once we find this prime $p$ , we need to find the smallest positive integer $m$ such that $m^{4}+1$ is divisible by $p^{2}$ . Let’s start by iterating over prime numbers and checking the condition for $n^{4}+1$ being divisible by $p^{2}$ . We will use Python to find the smallest prime $p$ and then the smallest $m$ .First, we will write a function to check if a number is prime and then iterate over prime numbers to find the smallest $p$ . Let’s write the code to achieve this. </think> <python> ⬇ 1 from sympy import isprime , primerange 2 # Function to find the smallest prime p 3 def find_smallest_prime_p (): 4 p = 2 5 while True : 6 found = False 7 # Check for the first 1000 values of n to find if there exists an n 8 # such that (n^4 + 1) is divisible by p^2 9 for n in range (1, 1000): 10 if ( n 4 + 1) % ( p 2) == 0: 11 found = True 12 break 13 if found : 14 return p 15 p = next_prime ( p ) 16 17 # Find the smallest prime p 18 smallest_prime_p = find_smallest_prime_p () 19 print (( smallest_prime_p , smallest_m )) </python> <result> Code execution failed: Traceback (most recent call last): File “<string>”, line 26, in <module> File “<string>”, line 15, in find_smallest_prime_p NameError: name ‘next_prime’ is not defined </result> <think> It seems that the previous approach had some issues. Let’s solve this problem step-by-step more methodically. 1. Identify the smallest prime $p$ such that there exists a positive integer $n$ for which $n^{4}+1$ is divisible by $p^{2}$ . 2. Once we find $p$ , determine the smallest integer $m$ such that $m^{4}+1$ is divisible by $p^{2}$ . We will start by iterating over prime numbers and check for each prime $p$ whether there exists an $n$ such that $n^{4}+1\equiv 0\pmod{p^{2}}$ . This means $n^{4}\equiv-1\pmod{p^{2}}$ . We will then check for the smallest $m$ that satisfies this condition for $p^{2}$ . Let’s write the corrected and more analytical code to solve this: 1. Generate a list of prime numbers. 2. For each prime $p$ , check if there exists an $n$ such that $n^{4}\equiv-1\pmod{p^{2}}$ . 3. Once we find the smallest such $p$ , find the smallest $m$ such that $m^{4}\equiv-1\pmod{p^{2}}$ . </think> <python> ⬇ 1 from sympy import isprime , primerange , next_prime 2 # Function to find the smallest m for a given prime p 3 def find_smallest_m_for_prime_p ( p ): 4 m = 1 5 while True : 6 if ( pow ( m , 4, p 2) + 1) % ( p 2) == 0: 7 return m 8 m += 1 9 # Iterate over primes 10 primes = list ( primerange (2, 10000)) 11 for p in primes : 12 found = False 13 # Check for an n such that n^4 + 1 is divisible by p^2 14 # The search range for n can be up to p^2 15 for n in range (1, 1000): 16 if ( pow ( n , 4, p 2) + 1) % ( p 2) == 0: 17 found = True 18 break 19 if found : 20 smallest_prime_p = p 21 break 22 smallest_m = find_smallest_m_for_prime_p ( smallest_prime_p ) 23 print (( smallest_prime_p , smallest_m )) </python> <result> (17, 110) </result> <think> The smallest prime number $p$ for which there exists a positive integer $n$ such that $n^{4}+1$ is divisible by $p^{2}$ is $17$ . The least positive integer $m$ such that $m^{4}+1$ is divisible by $17^{2}=289$ is $110$ .Thus, the final answer is $\boxed{110}$ . </think>
<answer> The final answer is 110 </answer>

Appendix

Table 12: An example from

\text{E}^{3}

-TIR on the AIME24 dataset, with special symbols used in think content.

Tool-Integrated Reasoning

See all 11 papers

TInR: Exploring Tool-Internalized Reasoning in Large Language Models
Apr 12, 2026Qiancheng Xu, Yongqi Li, Fan Liu +3LLM Tool Use Efficient Language Model Reasoning
The Hong Kong Polytechnic University · Southeast University · University of Edinburgh +1
Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning
Jun 2, 2026Hongye Cao, Nuo Yan, Haoyuan Deng +5Agentic RL RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210093, China · Nanyang Technological University, Singapore · China Mobile NineVerse Artificial Intelligence Technology (Beijing) Co., Ltd., and Institute of Artificial Intelligence, NineVerse, Beijing 100032, China
Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning
May 7, 2026Qianjia Cheng, Yuchen Zhang, Zhilin Wang +9Tool-Integrated Reasoning RL for Language Model Reasoning
Zhejiang University · Shanghai AI Laboratory · Peking University +4

Apr 12, 2026J/K move · Enter open · S save

TInR: Exploring Tool-Internalized Reasoning in Large Language Models

Qiancheng Xu, Yongqi Li, Fan Liu, Hongru Wang+2

The Hong Kong Polytechnic University · Southeast University · University of Edinburgh+1

LLM Tool Use Efficient Language Model Reasoning

Open paper AskPDF arXiv

Tool-Integrated Reasoning (TIR) has emerged as a promising direction by extending Large Language Models' (LLMs) capabilities with external tools during reasoning. Existing TIR methods typically rely on external tool documentation during reasoning. However, this leads to tool mastery difficulty, tool size constraints, and inference inefficiency. To mitigate these issues, we explore Tool-Internalized Reasoning (TInR), aiming at facilitating reasoning with tool knowledge internalized into LLMs. Achieving this goal presents notable requirements, including tool internalization and tool-reasoning coordination. To address them, we propose TInR-U, a tool-internalized reasoning framework for unified reasoning and tool usage. TInR-U is trained through a three-phase pipeline: 1) tool internalization with a bidirectional knowledge alignment strategy; 2) supervised fine-tuning warm-up using high-quality reasoning annotations, and 3) reinforcement learning with TInR-specific rewards. We comprehensively evaluate our method across in-domain and out-of-domain settings. Experiment results show that TInR-U achieves superior performance in both settings, highlighting its effectiveness and efficiency.

Abstract

Tool-Integrated Reasoning

Explore similar work

TInR: Exploring Tool-Internalized Reasoning in Large Language Models

Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning

Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning