When to Trust Tools? Adaptive Tool Trust Calibration For Tool-Integrated Math Reasoning
Authors: Ruotao Xu, Yixin Ji, Yu Luo, Jinpeng Li, Dong Li, Peifeng Li, Juntao Li, Min Zhang
Organizations: School of Computer Science and Technology, Soochow University · Department of Foundation Model, 2012 Labs, Huawei · Harbin Institute of Technology, Shenzhen (HITSZ)
Large reasoning models (LRMs) have achieved strong performance enhancement through scaling test time computation, but due to the inherent limitations of the underlying language models, they still have shortcomings in tasks that require precise computation and extensive knowledge reserves. Tool-Integrated Reasoning (TIR) has emerged as a promising paradigm that incorporates tool call and execution within the reasoning trajectory. Although recent works have released some powerful open-source TIR models, our analysis reveals that these models still suffer from critical deficiencies. We find that when the reasoning of the model conflicts with the tool results, the model tends to believe in its own reasoning. And there are cases where the tool results are correct but are ignored by the model, resulting in incorrect answers, which we define as "Tool Ignored''. This indicates that the model does not know when to trust or ignore the tool. To overcome these limitations, We introduce Adaptive Tool Trust Calibration (ATTC), a novel framework that guides the model to adaptively choose to trust or ignore the tool results based on the confidence score of generated code blocks. The experimental results from various open-source TIR models of different sizes and across multiple datasets demonstrate that ATTC effectively reduces the "Tool Ignored" issue, resulting in a performance increase of 4.1% to 7.5%.
Figures & tables
Figure 1: Case of the “Tool Ignored” phenomenon. The model arrive at an incorrect answer of 18 through reasoning, while the tool provide the correct result of 17. The model ignore the tool and give an incorrect answer.
Figure 2: The proportion of contradictions between model reasoning and tool results in true and false cases. The color proportions in the column indicate the proportion that the model chooses to believe in its own reasoning or tool results.
Figure 3: Proportion of “Tool Ignored” phenomenon of different TIR models on four datasets.
Figure 4: An overview of the Adaptive Tool Trust Calibration (ATTC) method.
Model
MATH 500
Minerva Math
Olympiad
AIME24
AMC23
Avg
Models based on Qwen2.5-7B
ToRL-7B
82.2
33.5
49.9
43.3
65.0
54.8
+ATTC
84.8
43.8
52.4
46.7
72.5
60.0 +5.2
Effective TIR-7B
82.8
30.5
51.9
42.3
70.0
55.5
+ATTC
85.8
42.3
53.5
46.7
77.5
61.2 +5.7
VerlTool-7B
82.0
31.6
49.8
40.0
67.5
54.2
Table 1: Pass@1 performance of the proposed ATTC method across various tool integrated reasoning models on various math benchmarks.
Model
MATH 500
Minerva
Olympiad
AIME24
AMC23
Avg
Tok ↓
Time ↓
Tok ↓
Time ↓
Tok ↓
Time ↓
Tok ↓
Time ↓
Tok ↓
Time ↓
Tok ↓
Time ↓
Models based on Qwen2.5-7B
EffectiveTIR-7B
1195
241
838
120
1203
318
1638
129
1503
78
1275
177
+ATTC
833
177
901
119
1283
316
1636
126
1398
71
1210 -5%
161 -9%
SimpleTIR-7B
1538
389
2683
373
2774
624
4444
278
2737
241
2835
381
+ATTC
1316
323
2534
328
2898
653
3723
206
2428
165
2579 -9%
335 -12%
Table 2: Efficiency performance of the proposed ATTC method across various tool integrated reasoning models on various math benchmarks. "Tok" denotes the average token count for a single question. "Time" denotes the average single reasoning time of each benchmark, and the unit is seconds. ↓ indicates that lower values are better.
Model
AIME25 Pass@1
AIME25 Pass@32
ToRL-7B
27.8
30.0
+Ours
32.2
36.7
Effective TIR-7B
30.0
66.7
+Ours
36.7
70.0
SimpleTIR-7B
28.9
53.3
+Ours
33.3
56.7
Table 3: Pass@1 and Pass@32 performances of the proposed ATTC method across various tool integrated reasoning models on AIME 2025.
Figure 5: Average performance of ATTC on five datasets under varying settings of the threshold λ .
Figure 6: The linear relationship between PRM score and confidence score.
Figure 7: Average number of tool calls per question for four models under the Vanilla and ATTC settings.
Figure 8: The proportion of “Tool Ignored” among false cases for four models across five datasets under the Vanilla and ATTC settings.
Figure 9: Case of the same problem optimized by ATTC. The model arrive at an incorrect answer of 18 through reasoning, while the tool provide the correct result of 17. The model trust the tool and give an correct answer.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Contradiction Ratio Prompt Template
Figure 11: “Tool Ignored” Prompt Template
Model
Temperature
Top_p
ToRL-7B
0
1
Effective TIR-7B
0.6
0.95
VerlTool-7B
0
1
SimpleTIR-7B&32B
1.0
0.7
ReTool-32B&4B
1.0
0.7
DemyAgent-4B
1.0
0.6
Appendix
Table 4: Temperature and top_p of different TIR models.
School of Computer Science and Technology, Beijing Institute of Technology, China · 3Independent, China · School of Computer Science and Technology, Huazhong University of Science and Technology, China