When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering

Saanvi Paturi; Arsen Kenzhebayev; Arham Sethi; Vyas Raina; Ivaxi Sheth; Vatsal Raina

cs.CLSep 12, 2026

When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering

Authors: Saanvi Paturi, Arsen Kenzhebayev, Arham Sethi, Vyas Raina, Ivaxi Sheth, Vatsal Raina

Organizations: Spark AI Research · UWCSEA East Campus · Haileybury Astana · The Shishukunj International School ·

Abstract

Large language models (LLMs) are increasingly deployed with external tools that extend what they can do beyond their own knowledge. Tools help on tasks that need external information, but their availability may also change how a model handles questions that do not need them. Prior work has mostly asked whether models select and use tools appropriately; whether an unnecessary tool changes the correctness of answers has received less attention. We ask whether making a related but unnecessary tool available affects a model's ability to answer from its own knowledge, and whether a preceding tool interaction changes this behaviour. We construct 500 query pairs across 10 knowledge domains. Each pair consists of a tool query, which needs the domain's tool, and a closed-domain query, which does not. Six LLMs are evaluated with the tool unavailable, available, and available after a prior tool call. Across 3,000 baseline trials the pooled answer rate is 98.2%. When an unnecessary tool is available it falls to 63.5%, with large differences between models. The decrease occurs even when the tool is rarely called, so it cannot be explained by unnecessary tool invocation alone. A one-sentence scope-aware system instruction recovers most of the lost answers.

Figures & tables

Figure 1: Illustration of the effect studied in this paper. The same closed-domain question is answered correctly when no tool is available (left) but not when a related, unnecessary tool is available (right).

Tool	Domain	Example tool query $q_{n}$	Example closed-domain query $q_{\bar{n}}$
Astronomical data lookup	Basic astronomy	Current distance from Earth to Mars in kilometres	Which planet is closest to the Sun?
Chemical-property lookup	Basic chemistry	Look up the boiling point of benzene	What electrical charge does a neutron have?
Currency conversion	Currency knowledge	Convert 100 USD to EUR	What is the currency code for the euro?
Current weather lookup	Weather and climate	What is the current weather in Paris?	Which atmospheric layer contains most weather?
Definite integration	Indefinite integration	Evaluate $\int_{0}^{2}x^{2}\,dx$	Find the indefinite integral of $x^{2}$
Election-result lookup	Political systems and civics	Recorded winner of the 2019 UK general election	What is meant by the rule of law?

Table 1: The ten domains, each contributing 50 query pairs. The tool query

q_{n}

requires the domain tool

t

; the closed-domain query

q_{\bar{n}}

is answerable without it and lies outside the function of

t

	Answer rate (%)				Tool use (%)
Model	$C_{1}$	$C_{2}$	$C_{3}$	$C_{4}$	$C_{1}$	$C_{3}$	$C_{4}$
Gemini 2.5 Flash-Lite	78.8	99.4	23.4	39.0	85.8	7.8	6.0
Gemini 2.5 Flash	98.6	99.6	59.4	56.0	99.2	19.4	18.8
Gemini 2.5 Pro	95.4	99.8	65.6	63.2	98.2	26.0	22.2
GPT-OSS 20B	84.6	95.8	94.6	94.4	94.4	2.4	1.0
Grok 4.1 Fast	86.0	99.6	100.0	99.2	99.6	16.0	2.8

Table 2: Answer rates and tool-use rates across ten domains (50 trials per domain;

N=500

per model and condition).

C_{2}

has no available tool, so tool use is not applicable.

Model	T–T–T	T–T–F	T–F–T	T–F–F	Total
Gemini 2.5 Flash-Lite	96	21	99	281	497
Gemini 2.5 Flash	215	81	64	138	498
Gemini 2.5 Pro	242	86	74	97	499
GPT-OSS 20B	456	9	8	6	479
Grok 4.1 Fast	494	4	0	0	498
Llama 3.3 70B	154	31	165	124	474

Table 3: Trial-level correctness trajectories across

C_{2}

–

C_{4}

, restricted to trials answered correctly in

C_{2}

. T–F–T denotes an answer lost in

C_{3}

and recovered in

C_{4}

; T–T–F denotes an answer lost only after the preceding tool interaction.

Panel of Figure 2 — Figure 2: Distribution of $C_{3}$ answer rates across the six models, by domain. Labels give the minimum, 25th percentile, median, 75th percentile and maximum; red points mark the mean. All panels share the same 0–100% scale.

Model	Undefended	Defended	Change	$C_{4}$ answer	$C_{1}$ answer	$C_{1}$ tool use
	$C_{3}$ answer rate			Defended
Gemini 2.5 Flash-Lite	23.4	64.4	$+41.0$	72.2	95.8	90.0
Gemini 2.5 Flash	59.4	93.8	$+34.4$	85.6	94.0	97.4
Gemini 2.5 Pro	65.6	93.4	$+27.8$	88.8	86.8	98.6
GPT-OSS 20B	94.6	95.2	$+0.6$	94.4	79.8	72.4
Grok 4.1 Fast	100.0	97.6	$-2.4$	96.4	85.4	89.8

Table 4: Effect of the scope-aware system instruction. Answer and tool-use rates are percentages; the change in

C_{3}

is in percentage points relative to the undefended condition.

Tool-Augmented Language Model Agents

Tool-augmented language model agents use external tools, APIs, and environments to solve tasks beyond text generation alone.

See all 167 papers

LLM Agents Already Know When to Call Tools -- Even Without Reasoning
May 10, 2026Chung-En Sun, Linbo Liu, Ge Yan +2Tool-Augmented Language Model Agents AI Agent Benchmarks
University of California, San Diego
Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use
May 13, 2026Yize Cheng, Chenrui Fan, Mahdi JafariRaviz +2LLM Tool Use LLM Agent Reliability
University of Maryland, College Park
Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use
May 27, 2026Abhijit Kumar, Zoey Wu, Mohit SuleyRL for LLM Agents Reward Design for RL
Microsoft AI Redmond, WA

May 10, 2026J/K move · Enter open · S save

LLM Agents Already Know When to Call Tools -- Even Without Reasoning

Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang+1

University of California, San Diego

Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Tool Use Tool Selection Tool Selection for Language Model Agents LLM Agent Efficiency

Open paper AskPDF arXiv

Tool-augmented LLM agents tend to call tools indiscriminately, even when the model can answer directly. Each unnecessary call wastes API fees and latency, yet no existing benchmark systematically studies when a tool call is actually needed. We propose When2Tool, a benchmark of 18 environments (15 single-hop, 3 multi-hop) spanning three categories of tool necessity -- computational scale, knowledge boundaries, and execution reliability -- each with controlled difficulty levels that create a clear decision boundary between tool-necessary and tool-unnecessary tasks. We evaluate two families of training-free baselines: Prompt-only (varying the prompt to discourage unnecessary calls) and Reason-then-Act (requiring the model to reason about tool necessity before acting). Both provide limited control: Prompt-only suppresses necessary calls alongside unnecessary ones, and Reason-then-Act still incurs a disproportionate accuracy cost on hard tasks. To understand why these baselines fail, we probe the models' hidden states and find that tool necessity is linearly decodable from the pre-generation representation with AUROC 0.89--0.96 across six models, substantially exceeding the model's own verbalized reasoning. This reveals that models already know when tools are needed, but fail to act on this knowledge during generation. Building on this finding, we propose Probe&Prefill, which uses a lightweight linear probe to read the hidden-state signal and prefills the model's response with a steering sentence. Across all models tested, Probe&Prefill reduces tool calls by 48% with only 1.7% accuracy loss, while the best baseline at comparable accuracy only reduces 6% of tool calls, or achieves a similar tool call reduction but incurs a 5

\times

higher accuracy loss. Our code is available at https://github.com/Trustworthy-ML-Lab/when2tool

Abstract

Tool-Augmented Language Model Agents

Explore similar work

LLM Agents Already Know When to Call Tools -- Even Without Reasoning

Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use

Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use