Agents' Overreliance on Unreliable Tools

Hoyeol Yang; Woojung Song; Taewon Kim; Jonghyun Song; Seoyeon Park; Yohan Jo

cs.AISep 4, 2026

Agents' Overreliance on Unreliable Tools

Authors: Hoyeol Yang, Woojung Song, Taewon Kim, Jonghyun Song, Seoyeon Park, Yohan Jo

Organizations: Seoul National University, Seoul, Republic of Korea

Abstract

LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the right tools, assuming that tool returns are reliable. However, tools can return plausible but incorrect outputs. We evaluate 14 models with three tools (web search, an LLM sub-agent, and a code executor), corrupting their returns to examine whether agents overrely on unreliable tools. Agents adopt corrupted returns at high rates, with mean adoption exceeding one third for every tool and reaching 68.0% for web search. Corrupted returns also frequently override correct answers that agents give without tools, and this reliance persists in more complex tasks and in tasks that combine multiple tools. Agents often make more tool calls under corrupted returns, and reasoning traces show them questioning a return and even stating the correct answer. Nevertheless, they still pass the corrupted content to users, rarely warning them of the conflict. We further test interventions spanning prompts, tool metadata, post-training, and activation steering. Verification prompts reduce adoption across all three tools while largely preserving accuracy with correct returns, and reliability labels provide partial mitigation in web search. The tested post-training methods offer limited gains, and the effect of activation steering depends on the model and intervention strength. Overall, our findings provide a basis for diagnosing tool overreliance and for developing agents that assess tool returns rather than assuming their reliability.

Figures & tables

		Search					Sub-Agent				Code Executor
		base acc. $\uparrow$	Adoption rate $\downarrow$			Override rate $\downarrow$	base acc. $\uparrow$	Adoption rate $\downarrow$		Override rate $\downarrow$	base acc. $\uparrow$	Adoption rate $\downarrow$		Override rate $\downarrow$
		base acc. $\uparrow$	P1	P2	P3	P1	base acc. $\uparrow$	P1	P2	P1	base acc. $\uparrow$	P1	P2	P1
Qwen	3-235B	53.8	75.7	19.0	47.7	66.4	30.6	38.6	23.8	21.4	68.8	37.9	20.4	34.0
	3-235B-Think	57.2	57.7	11.7	36.2	44.1	33.4	35.5	16.3	22.7	91.1	19.2	9.8	17.0
	3-30B	42.3	81.9	19.7	47.3	75.2	23.6	42.2	24.8	21.5	65.6	52.2	26.8	52.2
	3-30B-Think	44.1	63.1	11.9	30.9	38.5	27.5	32.5	14.4	24.0	87.6	39.4	30.9	37.0

Table 1: Base accuracy by tool and adoption and override rates by corruption condition (%).

Figure 2: P1 adoption and override rates by item type.

One tool
Tool	Adoption rate $\downarrow$	Override rate $\downarrow$
Search	58.9	47.3
Sub-Agent	94.7	91.9
Code Executor	21.5	21.4
Sub-Agent + Code Executor
Sub-Agent	49.1	48.3

Table 2: Mean P1 adoption and override rates across more realistic tasks and two-tool combinations (%).

	Tool		User		RAG
Tool	A	O	A	O	A	O
Search	75.4	66.7	77.4	71.1	82.7	78.9
Sub-Agent	39.0	26.0	36.7	25.6	37.4	27.1
Code Executor	40.2	39.4	0 1.8	0 0.2	0 3.8	0 1.5

Table 3: Mean P1 adoption rates (A) and override rates (O) across six models under each presentation (%).

Figure 3: Tool-call ratios relative to correct returns.

	Search		Sub-Agent		Code Executor
Policy	P0	P1	P0	P1	P0	P1
Plain	94.2	75.4	68.1	39.0	99.8	40.2
Compare	90.9	67.6	63.7	29.3	100.0	31.2
Verify	94.2	70.0	69.9	35.2	99.9	30.4
Disclose	93.3	72.0	66.4	38.0	99.9	38.6

Table 5: Mean prompt intervention results across six models (%). P0 reports accuracy with correct returns and P1 reports adoption rate with corrupted returns. Bold indicates the highest P0 and lowest P1 per tool.

Figure 4: P1 adoption rate (solid circles) and accuracy (dashed squares) with steering directions estimated after the first tool return.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Paper	Journal / conference	Publication date
AI	Deep Residual Learning for Image Recognition ( He et al., 2016 )	CVPR 2016 *	2015-12-10
	Attention Is All You Need ( Vaswani et al., 2017 )	NeurIPS 2017 *	2017-06-12
	Adam: A Method for Stochastic Optimization ( Kingma & Ba, 2015 )	ICLR 2015 *	2014-12-22
Medicine	Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries ( Sung et al., 2021 )	CA: A Cancer Journal for Clinicians	2021-02-04
	Hallmarks of Cancer: The Next Generation ( Hanahan & Weinberg, 2011 )	Cell	2011-03-04
	Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine ( Polack et al., 2020 )	New England Journal of Medicine	2020-12-10

Appendix

Table 9: Pre-cutoff papers used in the sub-agent evaluation. An asterisk marks papers whose publication date is based on the initial arXiv release; other dates use the journal’s online or issue publication date. Month-level dates are shown where no day-level date is available.

Domain	Paper	Journal / conference	Publication date
AI	Dataset Distillation via a Noise-Unconstrained Generative Model ( Zhang et al., 2026c )	IEEE TPAMI	2026-05-06
	Causality-Aware Spatiotemporal Adversarial Learning for Knowledge-Data Fault Diagnosis in Large-Scale Industrial Processes ( Zhang et al., 2026a )	IEEE TNNLS	2026-07-23
	A Quaternion Rotation-Enhanced Differential Privacy Framework for Image Privacy Protection ( Gao et al., 2026 )	IEEE TNNLS	2026-08-11
Medicine	Cuproptosis-immunity crosstalk informs strategy to overcome immunotherapy resistance ( Lei et al., 2026 )	Cell	2026-06-22
	Mechanism of actin thin filament pointed-end elongation by leiomodin ( Brotzman et al., 2026 )	Nature Communications	2026-07-20
	Zonated mechanosensing by PIEZO1 controls liver regeneration ( Zhang et al., 2026d )	Science	2026-07-02

Appendix

Table 10: Post-cutoff papers used in the sub-agent evaluation. Dates are first-public-release dates.

Domain	Work	Source	Publication date
Literature	Moby-Dick; or, The Whale	Project Gutenberg #2701	1851
	Pride and Prejudice	Project Gutenberg #1342	1813
	Jane Eyre: An Autobiography	Project Gutenberg #1260	1847
	Frankenstein; Or, The Modern Prometheus	Project Gutenberg #84	1818
	Dracula	Project Gutenberg #345	1897
	Alice’s Adventures in Wonderland	Project Gutenberg #11	1865

Appendix

Table 11: Novels used in the sub-agent evaluation.

Model	Adopting	Resolved	Warning YES	Rate (%)
Qwen3-235B	525	525	1	0.2
Qwen3-235B-Think	261	261	17	6.5
Qwen3-30B	711	711	1	0.1
Qwen3-30B-Think	633	633	0	0.0
Gemini-3.7-flash	81	81	0	0.0
Gemini-3.1-flash-lite	277	277	3	1.1

Appendix

Table 13: Code Executor final-answer warning rates for Section 4.5 , P1 and P2 combined. GPT-5.6-Luna judges the user-facing final answer with the same rubric for all models. The denominator includes adopting runs with available final-answer text and a resolved warning label.

Model	Adopting	Warning YES	Rate (%)
Qwen3-235B	1578	15	1.0
Qwen3-235B-Think	1223	0	0.0
Qwen3-30B	1694	13	0.8
Qwen3-30B-Think	1223	0	0.0
Gemini-3.7-flash	701	1	0.1
Gemini-3.1-flash-lite	846	11	1.3

Appendix

Table 14: Search final-answer warnings among adopting runs, P1–P3 combined, conditional on the target-bearing exposure criterion in Table 1 . Qwen3.5-27B supplies the final-answer warning flag.

Reference	Traces ( $n$ )	Agreement [95% CI]	$\kappa$	Precision	Recall	F1
R1	403	87.3 [83.6, 90.7]	0.702	0.901	0.917	0.909
R2	372	83.9 [79.4, 88.0]	0.657	0.786	0.986	0.874
R3	398	85.2 [81.2, 88.8]	0.682	0.810	0.974	0.885
Consensus	394	87.8 [84.0, 91.2]	0.735	0.844	0.979	0.906

Appendix

Table 15: GPT-5.6-Luna agreement with human trace annotations. Agreement and its 95% item-cluster bootstrap interval are percentages. Precision, recall, and F1 use human YES as the positive class. All rows evaluate the conflict label used in Table 4 ; R1–R3 denote the three annotators.

	Search			Sub-Agent		Code Executor
Model	P1	P2	P3	P1	P2	P1	P2
Qwen3-235B	66.4	12.0	43.9	21.4	13.7	34.0	18.8
Qwen3-235B-Think	44.1	0 6.8	30.2	22.7	10.9	17.0	8.1
Qwen3-30B	75.2	12.8	41.7	21.5	20.9	52.2	22.6
Qwen3-30B-Think	38.5	0 4.3	15.3	24.0	4.2	37.0	27.2
Gemini-3.7-flash	21.3	0 0.9	17.7	23.4	15.7	7.1	0.0

Appendix

Table 16: The override rate at each corruption level (%). P1 is repeated from Table 1 for comparison.

Model	Base acc.	P1 adoption rate	P1 override rate $O^{(1)}$
Gemma-4-31B	76.9	2.0	0.3
Gemma-4-12B	73.0	1.7	1.1
GPT-5.4-mini	69.6	17.5	17.6
GPT-5.4-nano	69.1	27.8	27.6
GPT-OSS-120B	73.8	33.4	35.0
GPT-OSS-20B	71.0	31.8	32.7

Appendix

Table 21: GSM-Hard Code Executor extension on 871 problems (%). The adoption rate is conditioned on at least one tool call;

O^{(1)}

additionally conditions on a correct single no-tool response.

Model	Training recipe	$\Delta$ P0 accuracy [95% CI]	$\Delta$ P1 adoption rate [95% CI]
Qwen3-30B	SFT	-1.2 [-4.1, +1.7]	-1.2 [-5.1, +2.5]
	SFT (recovery)	-2.2 [-5.0, +0.7]	-4.5 [-8.1, -0.8]
	DPO	-1.7 [-4.5, +1.2]	-4.8 [-8.8, -0.6]
	GRPO	-0.7 [-3.3, +1.7]	-2.5 [-6.0, +1.0]
GPT-OSS-20B	SFT	-11.0 [-14.8, -7.2]	+12.1 [+7.1, +17.3]
	SFT (recovery)	-8.9 [-12.4, -5.5]	+9.7 [+4.6, +14.9]

Appendix

Table 31: Post-training changes on Search relative to the untrained reference in Table 7 (percentage points). Intervals use 10,000 paired item-bootstrap resamples without multiplicity adjustment. The resampling pool is the same 418 questions, with each checkpoint retaining its own eligible P1 episodes. Appendix F.4 gives the evaluation settings.

Search
	Plain		Compare		Verify		Disclose
Item type	P0	P1	P0	P1	P0	P1	P0	P1
Never changing	97.7	72.6	96.0	60.0	97.6	66.5	96.7	69.9
Slow changing	92.4	72.9	87.5	65.7	92.5	65.8	91.5	68.3
Fast changing	92.4	80.3	88.8	76.0	92.1	77.6	91.4	77.3
Sub-Agent

Appendix

Table 34: Prompt intervention results by item type (%). P0 accuracy and the P1 adoption rate are averaged without weighting across the six models. Search P1 uses target-bearing exposed episodes within each item type.

		Search
Model	Arm	P0	P1
Gemma-4-12B	Base	93.1	79.1
	10% corrupt	71.8	80.6
	25% corrupt	73.0	82.2
	80% corrupt	58.6	81.3

Appendix

Table 36: Gemma-4-12B distillation with corrupted tool returns on held-out Search (%). P0 reports accuracy with an uncorrupted return; P1 reports adoption conditional on target-bearing exposure, as in Table 1 .

Model	Base	SFT	SFT (recovery)	DPO	GRPO
Qwen3-30B	367	387	384	365	366
GPT-OSS-20B	398	383	392	396	404
Gemma-4-31B	348	376	363	351	354
Model	Base	10% corrupt	25% corrupt	80% corrupt
Gemma-4-12B	359	175	180	91

Appendix

Table 38: Eligible Search P1 episodes for post-training, out of 418 evaluated questions per checkpoint. All P0 accuracy cells retain 418 judgments. The same exposure criterion applies to every checkpoint.

Model	Layer	P0 episodes	P1/P2 episodes	AUROC
Qwen3-30B	47	300	711	0.997
Gemma-4-31B	43	300	236	1.000
GPT-OSS-20B	7	248	736	0.993

Appendix

Table 39: Distinguishing correct from corrupted returns among episodes that accept the returned value. Counts refer to accepted correct returns (P0) and adopted corrupted returns (P1/P2). AUROC uses five-fold cross-validation grouped by problem. All activations are measured after the first tool return.

		P0	P1
Model	$\alpha$	Accuracy	Adoption rate	Accuracy	Failures
Qwen3-30B	$-1$	—	27.0	0.0	0.0
	$-0.5$	—	83.0	2.0	0.0
	$-0.25$	—	86.0	3.0	0.0
	$0$	100.0	58.0	23.0	0.0
	$+0.25$	100.0	45.0	29.0	0.0

Appendix

Table 40: Steering on the same 100 held-out Code Executor problems (%). Positive

\alpha

adds each model’s corrupted-minus-correct direction; negative

\alpha

reverses it. Directions are estimated after the first tool return. All rates use 100 problems; — denotes an unmeasured condition.

Explore similar work

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
Sep 14, 2026Yiwei Yang, Haoxiang Zhang, Bingbing Wen +6LLM Agent Reliability RL for Tool Use
University of Washington · University of California San Diego · Stanford University
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley
Don't Blindly Trust It: How Unreliable Feedback Breaks Tool-Using LLM Agents
Jun 19, 2026Chubin Zhang, Zhenglin Wan, Xingrui Yu +5AI Agent Reliability Tool-Use Evaluation
Nanyang Technological University, Singapore · National University of Singapore, Singapore · CFAR, Agency for Science, Technology and Research, Singapore +3

Sep 14, 2026J/K move · Enter open · S save

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

Yiwei Yang, Haoxiang Zhang, Bingbing Wen, Yao Lu+5

University of Washington · University of California San Diego · Stanford University

LLM Agent Reliability RL for Tool Use

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.

Abstract

Explore similar work

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Don't Blindly Trust It: How Unreliable Feedback Breaks Tool-Using LLM Agents