MLLMs Fail to Refuse when Using Tools Agentically

Rikiya Takehi; Ryo Hachiuma; Shaona Ghosh; Dan Zhao; Yu-Chiang Frank Wang; Yusuke Hirota

cs.AIOct 2, 2026

MLLMs Fail to Refuse when Using Tools Agentically

Authors: Rikiya Takehi, Ryo Hachiuma, Shaona Ghosh, Dan Zhao, Yu-Chiang Frank Wang, Yusuke Hirota

Organizations: MIT · NVIDIA

Abstract

Agentic multimodal large language models (MLLMs) have recently pushed the frontier of visual reasoning by calling tools such as zooming and tagging. Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests. Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%. Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.

Figures & tables

Figure 1: Multimodal Large Language Models (MLLMs) are more likely to fail refusing harmful requests when using tools. The example shows a user request on how to pickpocket, which the models should refuse to answer. Values in the bar chart are $\Delta\text{RFR}$ . Detailed results in Table 1 .

Model	MM-SafetyBench		VLSBench		HoliSafe		Average
Model	no-tool	tool-use	no-tool	tool-use	no-tool	tool-use	no-tool	tool-use	$\Delta$ RFR
Open-weights
Qwen3-VL	28.7	34.1	34.4	50.4	22.3	26.0	28.5	36.9	${\scriptstyle+}8.4_{\pm 1.2}$
Qwen3.5	41.1	41.5	50.7	56.2	30.1	33.8	40.7	43.8	${\scriptstyle+}3.2_{\pm 1.1}$
Kimi-K2.5	42.5	47.1	49.3	51.6	26.4	27.6	39.4	42.1	${\scriptstyle+}2.7_{\pm 1.1}$
Kimi-K2.6	34.1	39.1	40.5	57.3	17.3	26.6	30.6	41.0	${\scriptstyle+}10.4_{\pm 1.7}$

Table 1: Average Refusal Failure Rate (RFR) on each of the three benchmarks (lower is better). The colors ( ) indicate the level of increase in refusal failure compared to the no-tool setting. Average is the unweighted mean over the three benchmarks;

\Delta

RFR is reported with its 95% confidence interval.

Figure 2: Agentic MLLMs fail to refuse more often as the number of tool calls increases. Refusal failure rate is averaged over three benchmarks, and the shades represent the 95% confidence interval.

Figure 3: Agentic MLLMs successfully refuse when the original request is reinjected. The values are averaged over three benchmarks.

Figure 4: Qualitative example of context dilution on HoliSafe dataset. Tool-using Qwen3-VL performs 5 tool calls and explains how to disable a door lock, while no-tool Qwen3-VL successfully refuses the request due to safety reasons.

Figure 5: Comparison between no-tool and tool-using MLLMs on how final answers start. Only the samples from which models refused successfully are used to ensure fair comparison.

Figure 6: Qualitative example of safety focus displacement on HoliSafe dataset. Tool-using Claude Opus 4.6 starts from an observation summary of the image, while no-tool Claude Opus 4.6 directly refuses the harmful request.

Figure 7: Comparison of agentic MLLM failure rate with a standard zooming tool against that with a fake zooming tool that outputs the original image. We observe that tool outputs do not change the refusal rate.

	Tool availability
Prompt structure	No-tool RFR	Tool-use RFR	$\boldsymbol{\Delta_{\mathrm{tool}}}$
Minimal	22.0	25.8	${\scriptstyle+}3.8_{\pm 0.4}$
Structured	22.1	26.3	${\scriptstyle+}4.2_{\pm 0.5}$
$\boldsymbol{\Delta_{\mathrm{struct}}}$	${\scriptstyle+}0.1_{\pm 0.3}$	${\scriptstyle+}0.5_{\pm 0.4}$	–

Table 2: Factorial prompt ablation on 5 proprietary models. The right column reports the tool-use effect (Tool-use

-

No-tool), and the bottom row reports the structured-prompt effect (Structured

-

Minimal).

Dataset	Gemini-3-Flash	Agentic Vision	$\Delta$ RFR
HoliSafe	22.1	38.4	+16.4
MM-SafetyBench	35.6	51.5	+15.9
VLSBench	41.5	54.8	+13.3
Average	33.1	48.2	+15.2

Table 3: Native agentic-vision validation on Gemini-3-Flash Agentic Vision

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$N$	Split / subset	Coverage
MM-SafetyBench [ 35 ]	1,680	SD_TYPO	13 safety scenarios. We use all benchmark categories from the SD_TYPO image-text split.
VLSBench [ 21 ]	2,241	HF train	Visual-leakless multimodal safety benchmark covering 6 categories and 19 subcategories.
HoliSafe [ 29 ]	4,016	HF test	Holistic image-text safety benchmark covering 7 categories, 18 subcategories, and 5 image-text risk types. We exclude 15 failed samples following our loader.

Appendix

Table 4: Datasets used in our evaluation.

N

denotes the number of evaluated image–query pairs after applying our dataset loaders.

Model	$b/c$	$p$	$b/c$	$p$	$b/c$	$p$
	HoliSafe		MM-SafetyBench		VLSBench
Claude Opus 4.6	228/96	$\mathbf{1.6\mathrm{e}{-13}}$	146/58	$\mathbf{6.1\mathrm{e}{-10}}$	171/60	$\mathbf{1.5\mathrm{e}{-13}}$
Gemini-2.5-Pro	350/232	$\mathbf{1.0\mathrm{e}{-6}}$	183/95	$\mathbf{1.4\mathrm{e}{-7}}$	256/138	$\mathbf{2.8\mathrm{e}{-9}}$
Kimi-K2.5	297/214	$\mathbf{2.4\mathrm{e}{-4}}$	192/126	$\mathbf{2.6\mathrm{e}{-4}}$	236/192	$\mathbf{0.038}$
Qwen3-VL	387/265	$\mathbf{2.0\mathrm{e}{-6}}$	179/119	$\mathbf{6.3\mathrm{e}{-4}}$	324/224	$\mathbf{2.2\mathrm{e}{-5}}$
Qwen3.5	388/241	$\mathbf{4.6\mathrm{e}{-9}}$	191/121	$\mathbf{7.4\mathrm{e}{-5}}$	231/141	$\mathbf{3.1\mathrm{e}{-6}}$

Appendix

Table 5: Significance testing results.

Figure 8: Complete results of Figure 2 in the main paper.

Figure 9: Qualitative example of context dilution on MM-SafetyBench.

Figure 10: Qualitative example of context dilution on VLSBench.

Figure 11: Qualitative example of safety focus displacement on VLSBench

MLLM Safety

See all 28 papers

Tool Mediation Alters Refusal Mechanisms in Large Language Models
Sep 28, 2026Abel Rodríguez, Giuseppe Garofalo, Lieven Desmet +1Language Model Safety Evaluation LLM Agent Safety
DistriNet, KU Leuven
Before Retrieval Begins: Safety and Tool-Use Decisions in LLM Agents
Oct 16, 2026Aditya Nawal, Manit Baser, Mohan GurusamyLLM Agent Safety Tool-Using Language Agents
National University of Singapore
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Sep 27, 2026Tianzhuo Yang, Zirui Mi, Yantao Huang +4LLM Agent Evaluation LLM Agents
Peking University · Beijing Academy of Artificial Intelligence

Sep 28, 2026J/K move · Enter open · S save

Tool Mediation Alters Refusal Mechanisms in Large Language Models

Abel Rodríguez, Giuseppe Garofalo, Lieven Desmet, Vera Rimmer

DistriNet, KU Leuven

Language Model Safety Evaluation LLM Agent Safety Language Model Robustness Refusal Behavior in Language Models Refusal Mechanisms in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed with access to external tools, yet harmful tool-mediated interactions are less likely to be refused when compared to regular conversational ones. As this change in refusal behavior remains underexplored, we investigate its underlying mechanisms across a diverse set of open-weight language models. We find that information about the harmfulness of a request remains strongly encoded in the model's representations and transfers across conversational and tool-mediated inputs. Evidence from representation geometry and neuron-level analysis further indicates that the two interaction modes systematically distribute harm-related computation differently. Crucially, while conversational inputs can be refused at relatively low levels of perceived harmfulness, tool-mediated inputs remain permissive until harmfulness crosses a substantially higher effective refusal threshold. Moreover, tool-mediated refusal is also more brittle: progressively weakening the refusal computation disrupts tool-mediated refusal at lower intervention strengths than conversational refusal, even when benign capabilities remain intact. Together, our findings indicate that tool mediation does not simply reduce the internal perception of harm, but instead impacts its conversion into refusal. Overall, this suggests tool-mediated environments may intrinsically reduce robustness of models to harmful requests, and that conventional safety evaluations may not fully transfer to LLM agents.

Abstract

MLLM Safety

Explore similar work

Tool Mediation Alters Refusal Mechanisms in Large Language Models

Before Retrieval Begins: Safety and Tool-Use Decisions in LLM Agents

AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents