While large language models are increasingly deployed as tool-calling agents, their safety evaluations still focus primarily on their ability to refuse harmful requests in text. We investigate whether the refusal behavior in text transfers to tool use. We construct a benchmark of 2,304 prompts across four domains, consisting of matched triplets that present the same harmful or benign request in text-only, standard tool-enabled, and adversarially framed tool-enabled settings. We then assess tool-call safety using 20 predefined forbidden actions. Testing five models of varying sizes and families, we find that requests refused in text frequently result in unsafe tool calls. Under standard tool-enabled prompting, this divergence occurs for 10-52 percent of harmful requests refused in chat. To mechanistically investigate this gap, we identify a refusal-associated direction in the residual stream of each of the five models. We find that in tool-enabled contexts, the activations along this direction decrease in four of five models. Furthermore, lower activation predicts unsafe tool calls across all five models (AUC 0.76-0.86). However, restoring the signal to the respective text-only value through activation patching only prevents a subset of unsafe calls, primarily by suppressing tool use rather than by producing safer calls. Steering along the refusal direction nearly eliminates unsafe calls only at strengths that also suppress benign tool use or degrade the output quality. Our findings indicate that the identified refusal direction contributes to whether a model acts, but only partially explains the refusal divergence. Therefore, evaluating the safety of tool-calling agents requires directly examining their actions along with their textual responses.
Figures & tables
Text
Unsafe-tool
Divergence
Model
refusal
Normal
Stress
Normal
Stress
Qwen3-14B
0.753
0.544
0.677
0.519
0.657
Mistral-7B-Instruct-v0.3
0.435
0.354
0.448
0.287
0.347
Llama-3.1-70B-Instruct
0.760
0.289
0.289
0.229
0.229
c4ai-command-r7b
0.667
0.112
0.141
0.098
0.098
Gemma-3-27B-it
0.633
0.172
0.151
0.132
0.107
Table 1: Conditioned divergence between text refusal and tool-call safety (all bf16). Bootstrap 95% confidence intervals (10,000 cluster resamples of the 128 prompt cells) lie within ±0.09 of every divergence value and ±0.05 of every refusal rate.
Figure 1: Conditioned divergence: of the harmful requests each model strictly refuses in chat (R shown per model), the fraction whose matched tool-enabled run makes an unsafe call.
Model
Neutral
Safety-reinforced
Tool-encouraging
Qwen3-14B
0.63
0.39
0.58
Mistral-7B
0.25
0.23
0.49
Command-R-7B
0.08
0.04
0.18
Gemma-3-27B
0.25
0.03
0.20
Llama-3.1-70B
0.37
0.11
0.25
Table 2: System-prompt framing modulates the transfer gap but does not close it, shown as conditioned divergence under normal framing by system condition.
Model
Layer
Suppression ( Δ , t )
Ablation
Addition ( 1× )
AUC / Patching
Qwen3-14B
33/40
66.8,t=5.6
78%→94%
0%→33%
0.808 / 24% (9)
Mistral-7B
26/32
1.4,t=5.1
49%→33%
0%→0%
0.760 / 20% (6)
Command-R-7B
26/32
26.4,t=9.7
68%→33%
0%→41%‡
0.768 / 34% (6)
Gemma-3-27B
51/62
2902,t=4.0
66%→100%
0%→1%
0.790 / 90% (3)
Llama-3.1-70B
67/80
0.2,t=0.4
71%→59%
1%→24%
0.864 / 13% (3)
Table 3: Mechanistic summary. Layer and Suppression (paired drop in refusal activation from no-tool to tool-normal; mean Δ , t ; n=96 ) are from the original run. Ablation: judged refusal on harmful no-tool prompts before and after removing r^ at every layer ( n=240 , coherent outputs; Gemma uses the mean-out direction of Table 6 ). Addition: judged refusal on benign prompts after adding 1× the projection gap (Table 7 ). AUC: low refusal activation predicting an unsafe call. Patching: share of unsafe calls that revert when the projection is restored to its no-tool level, with the share reverting via a safe call in parentheses (Table 4 ). ‡ half the outputs degenerate
Figure 2: Refusal activation by mode (Qwen3-14B). Tool context suppresses the projection relative to no-tool chat.
Figure 3: Steering along r^ in tool mode. Per model, by coefficient: harmful unsafe-call and any-call rates, benign any-call rate (task completion) and judged over-refusal. Hollow markers in the shaded band mark coefficients where at least a quarter of outputs are degenerate and no rate is meaningful. Removing nearly all unsafe calls takes coefficients at which benign tool use collapses or outputs degenerate (Table 9 ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Unsafe base
Flip (95% CI)
via safe call
via no call
Unsafe ∣ call
Qwen3-14B
50%
24% (18, 31)
9%
15%
89%
Mistral-7B
32%
20% (12, 28)
6%
13%
93%
Command-R-7B
12%
34% (20, 49)
6%
29%
92%
Gemma-3-27B
13%
90% (79, 97)
3%
87%
80%
Llama-3.1-70B
61%
13% (8, 18)
3%
9%
96%
Appendix
Table 4: Activation patching on harmful tool-enabled prompts: the refusal projection at the operating layer is overwritten with its value on the matched no-tool prompt and unsafe calls that revert are counted ( n=300 baseline prompts per model; flips are over the unsafe subset, with bootstrap 95% CIs). “Via safe call” means the patched model still called a tool and the call scored safe; “via no call” means it emitted no tool call. Unsafe ∣ call is the unsafe rate among the calls the patched model still makes. Every generation was screened for degenerate output (0–8% per model).
Model
Layer
cos(r^text,r^behav)
cos(r^text,r^tool)
r^behav old vs. new
Qwen3-14B
33/40
0.859
0.741
0.975
Mistral-7B
26/32
0.711
0.773
0.978
Command-R-7B
26/32
0.746
0.394
0.978
Gemma-3-27B
51/62
0.822
0.797
0.958
Llama-3.1-70B
67/80
0.693
0.213
0.979
Appendix
Table 5: Alignment between the three directions, at each model’s operating layer. r^text is the request-type direction (harmful vs. benign, no-tool); r^tool is the same contrast extracted under tool prompts; r^behav is the behavior-defined direction (refused vs. complied harmful). The final column recomputes r^behav under the previous refusal labels: the two agree at 0.96 – 0.98 everywhere, so this check does not depend on the relabel.
KL of r^text
Refusal after ablating
Model
generic
dataset
Layers
none
r^text
mean-out
random
mean-dir
Qwen3-14B
1.59
2.31
4/21
0.78
0.94
0.95
0.78
– ‡
Mistral-7B
0.25
0.91
0/17
0.49
0.33
0.35
0.47
0.50
Command-R-7B
0.04
0.14
8/17
0.68
0.33
0.48
0.69
0.47
Gemma-3-27B
11.34
10.95
0/32
0.66
– ‡
1.00
0.68
– ‡
Llama-3.1-70B
0.28
0.40
11/41
0.69
0.62
0.64
0.69
0.34
Appendix
Table 6: Direction admissibility and ablation controls, harmful no-tool prompts ( n=240 ). KL is the mean next-token KL(clean∥ablated) on harmless prompts under all-layer projection removal; Arditi et al. [2024] require <0.1 . “Generic” is 48 ordinary requests (their protocol); “dataset” is this benchmark’s benign prompts. “Layers” counts candidate layers in the 35–85% depth band passing the threshold on generic prompts. Refusal is judged on coherent outputs; ‡ marks conditions with ≥25% degenerate output, whose rates are not meaningful. Mean-out projects the mean activation out of the direction before ablating; random is a unit Gaussian direction; mean-dir ablates the mean-activation direction itself. n=120 .
Model
0×
1×
1.5×
2×
3×
Qwen3-14B
0.00
0.33 (2%)
0.53 (18%)
– (91%)
– (100%)
Mistral-7B
0.00
0.00 (0%)
0.01 (0%)
0.06 (5%)
– (98%)
Command-R-7B
0.00
0.41 (50%)
– (100%)
– (100%)
– (100%)
Gemma-3-27B
0.00
0.01 (1%)
– (91%)
– (100%)
– (100%)
Llama-3.1-70B
0.01
0.24 (0%)
–
0.34 (8%)
–
Appendix
Table 7: Activation addition on benign no-tool prompts ( n=240 ) at multiples of the diff-in-means gap ( Arditi et al. [2024] use 1× ). Each cell is judged refusal among coherent outputs, with the degenerate share in parentheses; cells above 25% degenerate carry no meaningful rate. n=120 , 1× and 2× only.
Model
Unsafe (base → abl.)
Δ (95% CI)
Calls (base → abl.)
Unsafe ∣ calls
Qwen3-14B
0.460→0.235
−0.225(−0.29,−0.17)
0.590→0.370
0.78→0.64
Mistral-7B
0.325→0.350
+0.025(−0.03,+0.07)
0.425→0.470
0.77→0.74
Command-R-7B
0.120→0.180
+0.060(+0.02,+0.10)
0.135→0.230
0.89→0.78
Gemma-3-27B
0.130→0.000
−0.130(−0.17,−0.09)
0.175→0.000
n/a
Llama-3.1-70B
0.450→0.360
−0.090(−0.14,−0.04)
0.740→0.565
0.61→0.64
Appendix
Table 8: Ablation’s effect on behavior rather than wording, on harmful tool-enabled prompts ( n=200 , paired). Unsafe-tool-call scoring is a predicate over the parsed calls, so no refusal classifier enters this measurement. “Calls” is the share of prompts on which the model emits any tool call; “Unsafe ∣ calls” is the unsafe rate among those. Gemma’s ablated outputs here are the degenerate raw-direction outputs, so its row is a capability collapse, not a safety gain.
Harmful
Benign
Model
c
Any call
Unsafe
Unsafe ∣ call
Any call
Over-refusal
Deg.
Qwen3-14B
0
0.61
0.54
0.88
0.66
0.00
0%
200
0.48
0.39
0.80
0.63
0.01
0%
450
0.32
0.24
0.76
0.48
0.08
2%
700 ‡
0.07
0.03
0.43
0.22
0.44
51%
Mistral-7B
0
0.39
0.27
0.70
0.51
0.03
0%
Appendix
Table 9: Steering along r^ in tool mode, by coefficient c . Harmful side: share of prompts with any tool call, unsafe-call rate, and the unsafe rate among calls made. Benign side: share with any tool call (task completion) and judged over-refusal. Deg. is the share of degenerate outputs; rows above 25% are marked ‡ and their rates are artefacts.
Large language models increasingly act as agents that call external tools, yet most tool-safety research examines only what follows a tool call. The prior decision of whether to call the tool remains largely unexamined and can influence downstream safety. Across the seven open-weight models, the unweighted model-average harm rate after admitted harmful requests is 74.5%. To identify what governs this decision, we apply a factorial intervention to the input request. Within our generated construction, neutralizing the requested-operation component produces the largest admission contrast (19 pp). A linear direction estimated from this contrast influences held-out admission in four of seven models; effects on operation-neutralized requests in some models show that it is not an operation-specific mechanism. We then test whether written refusal is behaviorally and representationally coupled to the same decision. Written refusal accompanies 33.6% of admitted harmful calls. Plain-text refusal training reduces unsafe admission in models that retain benign tool use, showing that response-level safety training can transfer to the call decision. Together, these results establish a partial, model-dependent relation between written refusal and tool admission. They motivate direct evaluation and supervision of tool admission rather than treating text refusal as its proxy.
Large language models deployed as agents increasingly interact with external systems through tool calls--actions with real-world consequences that text outputs alone do not carry. Safety evaluations, however, overwhelmingly measure text-level refusal behavior, leaving a critical question unanswered: does alignment that suppresses harmful text also suppress harmful actions? We introduce the GAP benchmark, a systematic evaluation framework that measures divergence between text-level safety and tool-call-level safety in LLM agents. We test six frontier models across six regulated domains (pharmaceutical, financial, educational, employment, legal, and infrastructure), seven jailbreak scenarios per domain, three system prompt conditions (neutral, safety-reinforced, and tool-encouraging), and two prompt variants, producing 17,420 analysis-ready datapoints. Our central finding is that text safety does not transfer to tool-call safety. Across all six models, we observe instances where the model's text output refuses a harmful request while its tool calls simultaneously execute the forbidden action--a divergence we formalize as the GAP metric. Even under safety-reinforced system prompts, 219 such cases persist across all six models. System prompt wording exerts substantial influence on tool-call behavior: TC-safe rates span 21 percentage points for the most robust model and 57 for the most prompt-sensitive, with 16 of 18 pairwise ablation comparisons remaining significant after Bonferroni correction. Runtime governance contracts reduce information leakage in all six models but produce no detectable deterrent effect on forbidden tool-call attempts themselves. These results demonstrate that text-only safety evaluations are insufficient for assessing agent behavior and that tool-call safety requires dedicated measurement and mitigation.
Agentic multimodal large language models (MLLMs) have recently pushed the frontier of visual reasoning by calling tools such as zooming and tagging. Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests. Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%. Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.