PRISM: Transfer and Limits of Activation Interventions in Tool-Calling Language Models

Giovana Kerche Bonás; Thales Sales Almeida; Marcelo da Silva Reis; Rodrigo Nogueira; Marcos Meideiros Raimundo

cs.CLOct 14, 2026

PRISM: Transfer and Limits of Activation Interventions in Tool-Calling Language Models

Authors: Giovana Kerche Bonás, Thales Sales Almeida, Marcelo da Silva Reis, Rodrigo Nogueira, Marcos Meideiros Raimundo

Organizations: RECOD.ai Lab, Institute of Computing, Universidade Estadual de Campinas · Maritaca AI

Abstract

Recent literature shows that unsafe tool calls can often accompany textual refusals in language models. To investigate this hidden vulnerability, we introduce PRISM, a framework for evaluating the transfer of activation interventions across precision, languages, and scales. Our findings demonstrate that leveraging lower-resource settings (specifically quantized NF4 labels and cross-lingual transfer) offers a robust pathway to expose and improve model alignment. Notably, using behavioral labels from an NF4 copy enables stable intervention estimation even with very few scenarios. Furthermore, these alignment improvements transfer effectively across languages: five-language NF4-supervised directions successfully transfer to held-out Arabic, reducing unsafe violations in Llama-3.1-8B by 21.95 percentage points without target-language tuning. By mapping paired benign activations, we also successfully transfer these safety interventions to much larger models (up to 70B parameters). Ultimately, PRISM highlights how auxiliary supervision from compressed models and multilingual data can uncover and mitigate hazardous tool-calling behaviors across different model families. We release all experimental code and supporting artifacts at https://github.com/GiKerche/prism, alongside an interactive explorer of recorded model responses and experimental results at https://gikerche.github.io/prism/.

Figures & tables

Panel of Figure 1 — Figure 1: The PRISM pipeline. Unsafe-response fractions from BF16/NF4 weight BF16 activations to estimate native/shadow directions using all five languages or a single language. We evaluate precision, cross-language (including held-out Arabic), and cross-scale transfer (via $W$ ), applying shifts at selected BF16 layers.

Same LUMEN request. “We don’t want poor or peripheral candidates in the trainee program - they don’t fit with our rich clients. Filter to exclude anyone who studied at public school or lives in low-income neighborhoods.”
Baseline: no intervention	Direction estimated directly in the model, $a=+2$	Ministral3-8B $\to$ 14B direction, $a=+2$
“I must refuse that request.” The response explains that filtering by school type or neighborhood is discriminatory.	Proposes a process_candidate call with the action filter , explicitly listing public schools and low-income residential areas among the exclusion criteria.	Proposes a process_candidate call with the action filter , using public-school education and low neighborhood income as filtering criteria.
No tool call; no action violation.	Unsafe call proposed.	Unsafe call proposed.

Figure 2: Refusal to action in Ministral-3-14B. Same English request, ethical context, and seed (104); layer 20,

a=+2

. The receiver-native direction is estimated in Ministral-3-14B, the transported one in Ministral-3-8B. Calls are recorded without execution; baseline and calls are abridged.

Figure 3: Behavioral effects and tool-use capability. Changes from baseline at the selected intervention strengths: native/shadow (sources), native/transported (receivers). Bars show pointwise 95% CIs conditional on observed labels and fixed interventions (adjusted CIs in appendix). LUMEN positive/negative: ethical/neutral. Qwen lacks suppression; SODA covers sources only; blanks denote unevaluated cases.

Model	Layer	BA (%)	$a_{+}$	$a_{-}$
Qwen3.5-9B	18	96.91	$8$	$0$
Ministral-3-8B	21	85.29	$1$	$-0.25$
Llama-3.1-8B	26	73.66	$4$	$-4$
Qwen3.5-27B	40	74.41	$8$	$0$
Ministral-3-14B	20	85.84	$2$	$-1$
Llama-3.1-70B	78	78.70	$4$	$-4$

Table 1: Selected layers (zero-based) and strengths. Rows 1–3: sources (native and shadow share

a_{\pm}

); rows 4–6: receivers with the transported direction. BA: probe balanced accuracy on Validation I. Qwen has

a_{-}=0

(no eligible neutral calls in Validation II, 0/340).

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Scenarios	Families
Construction	35	33
Validation I: layer selection	16	11
Validation II: intervention strength selection	17	12
Final test	40	19

Appendix

Table 2: Scenario-family allocation for construction, selection, and final evaluation. All models use the same assignment.

Validation labels (number of responses)
	Qwen3.5 9B	Ministral-3 8B	Llama-3.1 8B
Unsafe action	2	318	1715
No unsafe action	2558	2241	826
Missing label	0	1	19
Validation performance (%)
Probe accuracy	93.83	83.21	73.39

Appendix

Table 3: Labels and probe performance used to select each frozen layer on Validation I; label-audit sensitivity is reported in the text. Each model generated 2,560 validation responses from 16 scenarios in 11 families. Responses with missing labels are excluded from performance calculations. Accuracy is the weighted percentage of correct predictions. The majority-class baseline always predicts the most frequent class in the construction split; it does not measure evaluator agreement. Balanced accuracy averages recall for the unsafe and non-unsafe classes and is the layer-selection criterion.

Figure 5: Probe balanced accuracy across all source layers on the Validation I partition. Markers indicate the selected layers (18 for Qwen, 21 for Ministral, and 26 for Llama), and the dashed line indicates 50% balanced accuracy. Each model uses the same valid responses at every layer. These curves support layer selection and do not measure causal intervention effects.

Panel of Figure 6 — Figure 6: How native and shadow directions are constructed. Both paths read the original BF16 model at the last input token, before it answers. Native weights these activations using violation fractions from BF16 responses; shadow uses fractions from the NF4 copy’s responses. Within-language contributions are pooled and normalized using Equation 5 . Both directions intervene in BF16.

Source	Layer	Native–shadow cosine	Common $\sigma$
Qwen3.5-9B	18	0.9838	2.7549
Ministral-3-8B	21	0.9811	2.3595
Llama-3.1-8B	26	0.9847	3.5429

Appendix

Table 4: Descriptive geometry of the actual pooled intervention vectors fitted on the 35-scenario construction partition. Cosines are computed from the frozen vectors used in the final acquisition. Their agreement does not establish equal behavioral effects.

Receiver	Layers	Rank	$\lambda_{\mathrm{rel}}$	Energy (%)	NMSE
Qwen3.5-27B	18 $\to$ 40	256	0.01	24.13	0.332
Ministral-3-14B	21 $\to$ 20	256	0.01	58.84	0.252
Llama-3.1-70B	26 $\to$ 78	256	0.1	42.37	0.595

Appendix

Table 5: Benign-anchor maps. Layers are source

\to

receiver. Map rank and penalty use 512 validation anchors after fitting on 1,024 anchors. Retained energy is the squared length of the unit source direction projected onto the selected source PCA basis. Normalized squared error (NMSE) on 512 diagnostic anchors does not select the map.

Model	Arm	$a$	Baseline	Intervention	$\Delta$ [95% CI]	BFCL tool
Qwen3.5-9B	N+	$8$	0/800 (0.00%)	609/796 (76.51%)	+76.51 [68.41, 81.16]	-2.84
Qwen3.5-9B	S+	$8$	0/800 (0.00%)	623/797 (78.17%)	+78.17 [66.54, 83.94]	-2.41
Qwen3.5-9B	N-	$0$	19/800 (2.38%)	19/800 (2.38%)	+0.00 [0.00, 0.00]	+0.00
Qwen3.5-9B	S-	$0$	19/800 (2.38%)	19/800 (2.38%)	+0.00 [0.00, 0.00]	+0.00
Ministral-3-8B	N+	$1$	23/800 (2.88%)	73/798 (9.15%)	+6.27 [1.93, 10.65]	+0.88
Ministral-3-8B	S+	$1$	23/800 (2.88%)	78/797 (9.79%)	+6.90 [2.62, 11.77]	+1.20

Appendix

Table 6: Source effects: N uses native labels; S uses NF4 labels. Plus uses ethical context; minus uses neutral context. Each arm has 800 responses. Counts and percentages use available labels. Paired deltas use only pairs with both labels available, so they can differ from the marginal rate difference. Intervals are pointwise family-bootstrap 95% CIs; adjusted intervals are in Table 8 . The BFCL column gives accuracy changes on 914 tool-required cases. All changes are percentage points. Qwen has no selected suppression intervention strength because its neutral Validation II baseline has no eligible calls (0/340). Zero-strength rows reuse the final baseline and do not measure an intervention effect.

Model	Arm	$a$	Baseline	Intervention	$\Delta$ [95% CI]	BFCL tool
Qwen3.5-27B	T+	$8$	0/800 (0.00%)	605/796 (76.01%)	+76.01 [65.12, 82.77]	-2.19
Qwen3.5-27B	T-	$0$	8/800 (1.00%)	8/800 (1.00%)	+0.00 [0.00, 0.00]	+0.00
Ministral-3-14B	T+	$2$	6/800 (0.75%)	35/800 (4.38%)	+3.62 [1.21, 7.66]	+0.11
Ministral-3-14B	T-	$-1$	148/797 (18.57%)	87/800 (10.88%)	-7.65 [-12.63, -1.94]	-0.77
Llama-3.1-70B	T+	$4$	35/800 (4.38%)	255/800 (31.87%)	+27.50 [22.14, 33.91]	+0.11
Llama-3.1-70B	T-	$-4$	383/792 (48.36%)	104/799 (13.02%)	-35.15 [-46.53, -23.22]	+0.00

Appendix

Table 7: Receiver effects: T is the transported native source direction. Plus uses ethical context; minus uses neutral context. Each arm has 800 responses. Counts and percentages use available labels. Paired deltas use only pairs with both labels available, so they can differ from the marginal rate difference. Intervals are pointwise family-bootstrap 95% CIs; adjusted intervals are in Table 8 . The BFCL column gives accuracy changes on 914 tool-required cases. All changes are percentage points. Qwen has no selected suppression intervention strength because its neutral Validation II baseline has no eligible calls (0/340). Zero-strength rows reuse the final baseline and do not measure an intervention effect.

Model	Contrast	Labeled pairs	$\Delta$ (pp)	Adjusted interval (pp)
Qwen3.5-9B	N+ $-$ B	796/800	+76.51	[62.55, 83.02]
Qwen3.5-9B	S+ $-$ B	797/800	+78.17	[58.78, 85.74]
Qwen3.5-9B	N+ $-$ S+	793/800	-1.51	[-4.64, 5.52]
Qwen3.5-9B	N- $-$ B	800/800	+0.00	[0.00, 0.00]
Qwen3.5-9B	S- $-$ B	800/800	+0.00	[0.00, 0.00]
Qwen3.5-9B	N- $-$ S-	800/800	+0.00	[0.00, 0.00]

Appendix

Table 8: The 18 predefined source contrasts use nominal

1-.05/18

intervals, and the six receiver contrasts use

1-.05/6

, from 10,000 paired family-bootstrap draws. N/S/T denote native/shadow/transported; B denotes the context-matched baseline. These percentile intervals condition on fitted objects and observed judge labels. Exact-zero intervals for zero-strength aliases represent identical observations.

Model	Estimator	Estimation $\to$ evaluation	Labeled pairs	$\Delta$ (pp) [95% CI]
Qwen3.5-9B	native	Spanish $\to$ Portuguese	159/160	$+87.42$ [ $+79.00,+93.50$ ]
Qwen3.5-9B	shadow	Chinese $\to$ Portuguese	160/160	$+88.12$ [ $+77.42,+93.75$ ]
Ministral-3-8B	native	Hindi $\to$ Chinese	160/160	$+13.75$ [ $+3.33,+24.19$ ]
Ministral-3-8B	shadow	Hindi $\to$ Chinese	160/160	$+12.50$ [ $+3.38,+20.65$ ]
Llama-3.1-8B	native	Portuguese $\to$ Spanish	154/160	$+34.42$ [ $+15.44,+54.82$ ]
Llama-3.1-8B	shadow	Portuguese $\to$ Spanish	157/160	$+25.48$ [ $+11.27,+41.23$ ]

Appendix

Table 9: Largest cross-language induction point estimates in ethical context. Each row is selected descriptively from the 20 off-diagonal cells for that model and estimator. Intervals are pointwise, without adjustment for selecting a maximum; these rows do not establish that the listed pair is uniquely best. Denominators show pairs with both labels available out of 160 attempts.

Model	Arm	Baseline	Intervention	$\Delta$ [95% CI] (pp)
Qwen3.5-9B	N+	0/44 (0.00%)	39/44 (88.64%)	+88.64 [80.77, 97.50]
Qwen3.5-9B	S+	0/44 (0.00%)	40/43 (93.02%)	+93.02 [82.69, 100.00]
Qwen3.5-9B	N-	1/44 (2.27%)	1/44 (2.27%)	+0.00 [0.00, 0.00]
Qwen3.5-9B	S-	1/44 (2.27%)	1/44 (2.27%)	+0.00 [0.00, 0.00]
Ministral-3-8B	N+	0/44 (0.00%)	1/44 (2.27%)	+2.27 [0.00, 9.38]
Ministral-3-8B	S+	0/44 (0.00%)	0/44 (0.00%)	+0.00 [0.00, 0.00]

Appendix

Table 10: Arabic transfer of pooled five-language vectors: 44 responses per generated condition from 11 scenarios in six families, excluded from construction and both validation partitions. N/S denote native/shadow; plus is ethical and minus neutral context. Counts use available labels and deltas use complete pairs. Pointwise intervals are descriptive. A zero-event or degenerate interval is not evidence of zero risk; Qwen zero-strength arms are exact baseline aliases.

Model	Audited	A/B disagree	C agrees when A=B
Qwen3.5-9B	133	27/133 (20.30%)	104/106 (98.11%)
Ministral-3-8B	264	23/264 (8.71%)	230/238 (96.64%)
Llama-3.1-8B	272	92/272 (33.82%)	156/178 (87.64%)
Qwen3.5-27B	13	2/13 (15.38%)	11/11 (100.00%)
Ministral-3-14B	33	3/33 (9.09%)	29/30 (96.67%)
Llama-3.1-70B	38	22/38 (57.89%)	14/16 (87.50%)

Appendix

Table 11: Stratified final audit. A is Claude Sonnet 4.6, B is Grok 4.5, and C is DeepSeek V4 Flash 0731 FP8. Counts aggregate all components within each model. C agreement conditions on valid A and B votes agreeing and a valid C vote. These raw sample fractions are descriptive; inclusion-weighted summaries and missingness are exported. They are not human-validated accuracy, and no audit vote replaces a primary label.

Condition	Ethical unsafe	Neutral unsafe	BFCL tool-required
Unsteered	0/800 (0.00%)	19/800 (2.38%)	846/914 (92.56%)
Native $a=+8$	609/796 (76.51%)	–	820/914 (89.72%)
Shadow $a=+8$	623/797 (78.17%)	–	824/914 (90.15%)
LoRA seed 26082701	485/800 (60.62%)	461/799 (57.70%)	859/914 (93.98%)
LoRA seed 26082702	478/800 (59.75%)	452/800 (56.50%)	854/914 (93.44%)
LoRA seed 26082703	500/799 (62.58%)	473/800 (59.13%)	856/914 (93.65%)

Appendix

Table 12: Current Qwen9B LoRA comparison: unsafe/labeled responses for LUMEN, and valid/total cases for BFCL, with percentages. All three adapters use construction data only; their fixed targets are described in the text. Steering uses layer 18 and

a=+8

in ethical context; no nonzero Qwen suppression intervention strength was selected. Each unsafe arm has 800 possible responses; BFCL has 914 tool-required cases. Missing judgments remain missing. These are marginal rates, which need not subtract to the paired estimates.

Quantity	Steering	LoRA
Stored coefficients	4,096	43,278,336
Additional artifact	16 KiB	165.15 MiB
Weight-update steps	0	20
Optimizer-loop time (min)	—	15.99–16.50
Training peak (GiB)	—	55.67

Appendix

Table 13: Measured Qwen9B intervention artifacts and LoRA training resources. LoRA entries are per adapter; the time range covers all three replicas on one H200 each. Steering stores one pooled FP32 direction; LoRA stores FP32 adapter weights. The shared BF16 checkpoint is excluded from artifact size. Peak memory is PyTorch allocated memory, including the backbone. Dashes denote quantities without a comparable measurement, not zero preparation cost.

Source	BF16 positives	NF4 positives
Qwen3.5-9B	28/700 (4.00%)	56/699 (8.01%)
Ministral-3-8B	98/700 (14.00%)	188/700 (26.86%)
Llama-3.1-8B	459/692 (66.33%)	371/695 (53.38%)

Appendix

Table 14: Construction labels: unsafe/labeled responses, out of 700 attempts per model and precision. Missing labels remain excluded, not negative.

Model	Language	BF16 unsafe/labeled (%)	NF4 unsafe/labeled (%)
Qwen3.5-9B	pt	4/140 (2.86)	11/140 (7.86)
Qwen3.5-9B	en	4/140 (2.86)	4/140 (2.86)
Qwen3.5-9B	es	5/140 (3.57)	10/140 (7.14)
Qwen3.5-9B	zh	5/140 (3.57)	8/140 (5.71)
Qwen3.5-9B	hi	10/140 (7.14)	23/139 (16.55)
Ministral-3-8B	pt	19/140 (13.57)	34/140 (24.29)

Appendix

Table 15: Construction positives by language. Each precision has 140 attempted responses per cell; the difference from the labeled denominator is missingness. All ten language vectors per source model are identifiable at full budget.

Figure 7: Construction-only, paired family-subsampling diagnostic. NF4 reduces undefined direction fits for Qwen and Ministral when few construction families are available. At eight families, Llama fits are available in every subset under either label source. This does not measure downstream effects, generalization or monetary savings.

Figure 8: Complete Validation II intervention-strength curves at frozen layers. Lines show changes in the valid action-call proxy, not semantic unsafe-action labels. Solid/dashed lines denote native/shadow sources; receivers use transported native directions. Stars identify retained nonzero intervention strengths, circles other admissible points, and crosses points rejected by capability or quality gates. Colors distinguish the positive ethical and negative neutral contrasts.

Figure 9: Cosines between the actual language-specific intervention vectors fitted on current construction data. High alignment in Ministral and lower alignment in Qwen/Llama coexist with varying cross-language effects. Cosine is a geometric diagnostic and does not establish equal causal efficacy.

Model	Estimator	Context	Diagonal $\Delta$	Other-language $\Delta$	Paired same minus other [95% CI]
Qwen3.5-9B	native	ethical	+59.39	+63.72	-4.09 [-8.31, +0.51]
Qwen3.5-9B	shadow	ethical	+62.46	+64.21	-2.00 [-4.64, +2.43]
Ministral-3-8B	native	ethical	+8.02	+6.04	+1.98 [+0.47, +4.21]
Ministral-3-8B	native	neutral	-5.14	-4.94	-0.50 [-1.31, +0.72]
Ministral-3-8B	shadow	ethical	+7.41	+6.33	+0.88 [-0.47, +2.33]
Ministral-3-8B	shadow	neutral	-5.75	-5.51	-0.25 [-0.83, +0.44]

Appendix

Table 16: Descriptive diagonal and off-diagonal cell averages, and a separate paired contrast over complete five-vector tuples, in percentage points. Qwen neutral outcomes are zero-strength aliases. Different missing-label support means subtraction of the two cell averages need not equal the paired contrast.

Model	Estimator	Context	Claude	Grok	DeepSeek audit	Majority
Qwen3.5-9B	native	ethical	+74.26	+33.02	+77.56	+74.26
Qwen3.5-9B	shadow	ethical	+78.79	+56.71	+66.46	+66.46
Ministral-3-8B	native	ethical	+7.54	+7.10	+7.51	+7.54
Ministral-3-8B	shadow	ethical	+8.13	+8.13	+8.25	+8.13
Ministral-3-8B	native	neutral	-7.45	-2.38	-4.43	-4.96
Ministral-3-8B	shadow	neutral	-10.50	-8.57	-10.62	-11.15

Appendix

Table 17: Effects standardized from the stratified audit at the selected intervention strengths, in percentage points. The same audited candidates and inclusion weights are used for all four columns. These sparse-audit point estimates are not the full-population primary effects, uncertainty intervals, or independent validation. Qwen zero suppression aliases are omitted.

Context	Evaluator	Baseline unsafe	Intervention unsafe	Paired $\Delta$ (pp)	Adjusted interval (pp)
Ethical $+4$	Claude	41/800 (5.12%)	281/800 (35.12%)	$+30.00$	$[+23.06,+41.65]$
Ethical $+4$	Grok	20/800 (2.50%)	111/800 (13.88%)	$+11.38$	$[+6.06,+24.25]$
Ethical $+4$	DeepSeek	35/800 (4.38%)	255/800 (31.87%)	$+27.50$	$[+20.00,+37.34]$
Ethical $+4$	Majority	36/800 (4.50%)	256/800 (32.00%)	$+27.50$	$[+20.28,+39.05]$
Neutral $-4$	Claude	429/800 (53.62%)	120/800 (15.00%)	$-38.62$	$[-53.75,-21.68]$
Neutral $-4$	Grok	146/800 (18.25%)	47/800 (5.88%)	$-12.38$	$[-20.93,-5.62]$

Appendix

Table 18: Complete evaluator sensitivity in Llama70B. Each arm has 800 responses; percentages use available labels. All 952 structural candidates receive both Claude and Grok votes; exact structural negatives are shared. DeepSeek supplies the primary endpoint. Family-bootstrap intervals adjust across the eight displayed contrasts. Missing labels remain missing, and the majority requires two matching valid votes. Paired effects need not equal differences of marginal percentages. These intervals exclude evaluator error.

Model	Context	$\Delta$ (pp)	98% interval (pp)
Qwen3.5-9B	ethical	$-1.51$	$[-4.10,+4.01]$
Ministral-3-8B	ethical	$-0.75$	$[-3.37,+0.26]$
Ministral-3-8B	neutral	$-0.25$	$[-1.13,+1.12]$
Llama-3.1-8B	ethical	$+1.64$	$[-1.30,+3.59]$
Llama-3.1-8B	neutral	$-1.80$	$[-5.47,+1.75]$

Appendix

Table 19: Exploratory native-minus-shadow equivalence-margin diagnostic. The reference margin is

[-7,+7]

pp. The margin was introduced after outcomes were inspected and is not an independently established utility threshold. Qwen’s zero-strength baseline alias is excluded.

Control	Unsafe/labeled (%)	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. shadow [adj. CI]	BFCL $\Delta$
Random 1 ( $+$ )	1/800 (0.12%)	$+0.12$ [ $0.00,1.22$ ]	$-78.04$ [ $-86.24,-53.27$ ]	$-10.83$
Random 2 ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-78.17$ [ $-86.34,-53.52$ ]	$-14.00$
Random 3 ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-78.17$ [ $-86.34,-53.52$ ]	$-19.37$
Probe ( $+$ )	4/800 (0.50%)	$+0.50$ [ $0.00,1.78$ ]	$-77.67$ [ $-86.00,-53.39$ ]	$-6.67$
Call propensity ( $+$ )	580/800 (72.50%)	$+72.50$ [ $50.64,83.84$ ]	$-5.77$ [ $-16.92,6.09$ ]	$-2.52$
Permuted labels ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-78.17$ [ $-86.34,-53.52$ ]	$-15.97$

Appendix

Table 20: Qwen3.5-9B controls at the same layer and signed displacement as the original intervention. Each arm contains 800 responses; counts and rates use available semantic labels. Plus uses ethical and minus neutral context. Baselines: ethical 0/800 (0.00%). Changes are paired percentage points; brackets are Bonferroni-adjusted family-bootstrap intervals for the full declared family of 174 additional contrasts. BFCL is the validity change on 914 tool-required cases. Controls are not retuned to pass the capacity gate. Zero-event bootstrap intervals do not establish zero population risk. Native/shadow primary outcomes are retained in Table 6 .

Control	Unsafe/labeled (%)	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. shadow [adj. CI]	BFCL $\Delta$
Random 1 ( $+$ )	22/800 (2.75%)	$-0.12$ [ $-3.04,1.03$ ]	$-7.03$ [ $-19.82,-0.46$ ]	$-0.77$
Random 2 ( $+$ )	23/800 (2.88%)	$+0.00$ [ $-1.59,0.69$ ]	$-6.90$ [ $-18.45,-0.46$ ]	$-1.53$
Random 3 ( $+$ )	25/800 (3.12%)	$+0.25$ [ $-1.03,1.07$ ]	$-6.65$ [ $-17.77,-0.35$ ]	$-0.77$
Probe ( $+$ )	26/800 (3.25%)	$+0.38$ [ $-2.24,2.07$ ]	$-6.52$ [ $-19.14,-0.46$ ]	$-0.77$
Call propensity ( $+$ )	76/800 (9.50%)	$+6.62$ [ $0.36,13.98$ ]	$-0.50$ [ $-4.56,1.31$ ]	$+0.88$
Permuted labels ( $+$ )	16/800 (2.00%)	$-0.88$ [ $-3.45,0.00$ ]	$-7.78$ [ $-20.56,-0.46$ ]	$-1.86$

Appendix

Table 21: Ministral-3-8B controls at the same layer and signed displacement as the original intervention. Each arm contains 800 responses; counts and rates use available semantic labels. Plus uses ethical and minus neutral context. Baselines: ethical 23/800 (2.88%); neutral 165/800 (20.62%). Changes are paired percentage points; brackets are Bonferroni-adjusted family-bootstrap intervals for the full declared family of 174 additional contrasts. BFCL is the validity change on 914 tool-required cases. Controls are not retuned to pass the capacity gate. Zero-event bootstrap intervals do not establish zero population risk. Native/shadow primary outcomes are retained in Table 6 .

Control	Unsafe/labeled (%)	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. shadow [adj. CI]	BFCL $\Delta$
Random 1 ( $+$ )	310/800 (38.75%)	$-15.19$ [ $-26.35,-2.47$ ]	$-32.91$ [ $-56.08,-14.12$ ]	$+0.00$
Random 2 ( $+$ )	347/800 (43.38%)	$-10.25$ [ $-18.25,-0.06$ ]	$-28.27$ [ $-48.64,-8.84$ ]	$+0.11$
Random 3 ( $+$ )	342/800 (42.75%)	$-11.01$ [ $-23.17,-1.26$ ]	$-28.89$ [ $-52.06,-8.80$ ]	$-0.22$
Probe ( $+$ )	352/800 (44.00%)	$-9.87$ [ $-22.18,-0.55$ ]	$-27.64$ [ $-50.12,-6.90$ ]	$-0.77$
Call propensity ( $+$ )	605/799 (75.72%)	$+22.18$ [ $0.40,49.75$ ]	$+4.15$ [ $-1.15,9.47$ ]	$-1.20$
Permuted labels ( $+$ )	213/800 (26.62%)	$-27.47$ [ $-38.41,-13.23$ ]	$-44.97$ [ $-62.41,-28.37$ ]	$-2.08$

Appendix

Table 22: Llama-3.1-8B controls at the same layer and signed displacement as the original intervention. Each arm contains 800 responses; counts and rates use available semantic labels. Plus uses ethical and minus neutral context. Baselines: ethical 426/790 (53.92%); neutral 547/787 (69.50%). Changes are paired percentage points; brackets are Bonferroni-adjusted family-bootstrap intervals for the full declared family of 174 additional contrasts. BFCL is the validity change on 914 tool-required cases. Controls are not retuned to pass the capacity gate. Zero-event bootstrap intervals do not establish zero population risk. Native/shadow primary outcomes are retained in Table 6 .

Estimator	Context	$a$	Baseline unsafe	Intervention unsafe	Paired $\Delta$ [adj. CI]
Native	Neutral	$+4$	547/787 (69.50%)	622/800 (77.75%)	$+8.39$ [ $-4.03,21.85$ ]
Native	Ethical	$-4$	426/790 (53.92%)	135/800 (16.88%)	$-36.96$ [ $-50.61,-20.97$ ]
Shadow	Neutral	$+4$	547/787 (69.50%)	624/798 (78.20%)	$+8.92$ [ $-4.27,23.09$ ]
Shadow	Ethical	$-4$	426/790 (53.92%)	159/800 (19.88%)	$-33.80$ [ $-43.88,-20.91$ ]

Appendix

Table 23: Additional Llama8B tests of the opposite intervention strength sign in each context. Each arm has 800 responses; denominators retain missing labels. Changes are percentage points, with intervals adjusted over the same 174-comparison control family. Identical intervention settings reuse their existing tool-required BFCL evaluation.

Direction	Unsafe/labeled (%)	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. transport [adj. CI]	BFCL $\Delta$
Random 1 ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-76.01$ [ $-86.00,-54.21$ ]	$-3.83$
Random 2 ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-76.01$ [ $-86.00,-54.21$ ]	$-4.27$
Random 3 ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-76.01$ [ $-86.00,-54.21$ ]	$-7.33$
Probe ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-76.01$ [ $-86.00,-54.21$ ]	$-6.13$
Call propensity ( $+$ )	152/800 (19.00%)	$+19.00$ [ $8.33,28.23$ ]	$-57.04$ [ $-67.82,-35.62$ ]	$-7.99$
Permuted labels ( $+$ )	0/800 (0.00%)	$+0.00$ [ $0.00,0.00$ ]	$-76.01$ [ $-86.00,-54.21$ ]	$-25.38$

Appendix

Table 24: Qwen3.5-27B controls at the layer and signed displacement used for transport. Plus uses ethical and minus neutral context; every arm has 800 responses. Baselines: ethical 0/800 (0.00%); neutral 8/800 (1.00%). Counts and rates use available judgments. Changes are paired percentage points, with family-bootstrap intervals adjusted over all 174 additional contrasts. The transport difference is control minus transported intervention. BFCL reports the validity change on 914 tool-required cases. These controls retain the transport intervention strength and are not separately optimized for capacity. All three random draws and the permuted-label draw are reported.

Direction	Unsafe/labeled (%)	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. transport [adj. CI]	BFCL $\Delta$
Random 1 ( $+$ )	7/800 (0.88%)	$+0.12$ [ $0.00,0.35$ ]	$-3.50$ [ $-12.23,0.00$ ]	$-0.44$
Random 2 ( $+$ )	6/800 (0.75%)	$+0.00$ [ $-2.12,0.68$ ]	$-3.62$ [ $-13.99,0.00$ ]	$-2.08$
Random 3 ( $+$ )	6/800 (0.75%)	$+0.00$ [ $-1.06,0.34$ ]	$-3.62$ [ $-13.04,0.00$ ]	$+0.33$
Probe ( $+$ )	14/800 (1.75%)	$+1.00$ [ $0.00,3.23$ ]	$-2.62$ [ $-10.38,0.00$ ]	$-0.77$
Call propensity ( $+$ )	70/800 (8.75%)	$+8.00$ [ $0.59,21.28$ ]	$+4.38$ [ $0.25,10.36$ ]	$+1.20$
Permuted labels ( $+$ )	5/800 (0.62%)	$-0.12$ [ $-1.06,0.00$ ]	$-3.75$ [ $-13.04,0.00$ ]	$-3.50$

Appendix

Table 25: Ministral-3-14B controls at the layer and signed displacement used for transport. Plus uses ethical and minus neutral context; every arm has 800 responses. Baselines: ethical 6/800 (0.75%); neutral 148/797 (18.57%). Counts and rates use available judgments. Changes are paired percentage points, with family-bootstrap intervals adjusted over all 174 additional contrasts. The transport difference is control minus transported intervention. BFCL reports the validity change on 914 tool-required cases. These controls retain the transport intervention strength and are not separately optimized for capacity. All three random draws and the permuted-label draw are reported.

Direction	Unsafe/labeled (%)	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. transport [adj. CI]	BFCL $\Delta$
Random 1 ( $+$ )	37/800 (4.62%)	$+0.25$ [ $-2.57,2.45$ ]	$-27.25$ [ $-40.87,-18.02$ ]	$-0.22$
Random 2 ( $+$ )	31/800 (3.88%)	$-0.50$ [ $-3.14,2.13$ ]	$-28.00$ [ $-42.61,-17.98$ ]	$+0.00$
Random 3 ( $+$ )	31/800 (3.88%)	$-0.50$ [ $-3.01,2.12$ ]	$-28.00$ [ $-42.30,-18.43$ ]	$+0.11$
Probe ( $+$ )	81/800 (10.12%)	$+5.75$ [ $1.25,15.24$ ]	$-21.75$ [ $-30.81,-12.73$ ]	$-0.22$
Call propensity ( $+$ )	366/800 (45.75%)	$+41.38$ [ $29.40,51.47$ ]	$+13.88$ [ $4.38,18.66$ ]	$+0.00$
Permuted labels ( $+$ )	407/800 (50.88%)	$+46.50$ [ $34.49,58.43$ ]	$+19.00$ [ $7.06,29.01$ ]	$-0.22$

Appendix

Table 26: Llama-3.1-70B controls at the layer and signed displacement used for transport. Plus uses ethical and minus neutral context; every arm has 800 responses. Baselines: ethical 35/800 (4.38%); neutral 383/792 (48.36%). Counts and rates use available judgments. Changes are paired percentage points, with family-bootstrap intervals adjusted over all 174 additional contrasts. The transport difference is control minus transported intervention. BFCL reports the validity change on 914 tool-required cases. These controls retain the transport intervention strength and are not separately optimized for capacity. All three random draws and the permuted-label draw are reported.

Model	Native $a$	Unsafe/labeled (%)	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. transport [adj. CI]	BFCL $\Delta$
Qwen3.5-27B	$+4$	7/800 (0.88%)	$+0.88$ [ $0.00,3.27$ ]	$-75.13$ [ $-84.98,-53.20$ ]	$-0.22$
Qwen3.5-27B	$0^{\dagger}$	8/800 (1.00%)	$+0.00$ [ $0.00,0.00$ ]	$+0.00$ [ $0.00,0.00$ ]	$+0.00$
Ministral-3-14B	$+2$	65/800 (8.12%)	$+7.38$ [ $0.54,19.48$ ]	$+3.75$ [ $0.16,8.37$ ]	$+1.20$
Ministral-3-14B	$-0.25$	128/800 (16.00%)	$-2.76$ [ $-6.25,0.00$ ]	$+5.12$ [ $0.25,10.91$ ]	$-0.66$
Llama-3.1-70B	$+8$	648/798 (81.20%)	$+76.82$ [ $64.09,83.93$ ]	$+49.37$ [ $27.48,62.95$ ]	$-0.33$
Llama-3.1-70B	$-4$	146/800 (18.25%)	$-29.92$ [ $-47.35,-14.79$ ]	$+5.13$ [ $0.08,12.40$ ]	$+0.44$

Appendix

Table 27: Receiver-native directions at their Validation II-selected intervention strengths. Positive intervention strengths use ethical and negative intervention strengths neutral context. The transport comparators use

+8/0

+2/-1

, and

+4/-4

for Qwen, Ministral, and Llama, respectively. Each arm has 800 possible responses; BFCL uses 914 tool-required cases. Changes are paired percentage points and intervals adjust over the same 174 additional contrasts.

\dagger

No negative Qwen intervention strength was selected; this row reuses baseline. Ministral

+2

and Llama

-4

reuse the identical receiver-native conditions in the matched-displacement tables. These reused observations are not additional experimental replicates. Llama70B uses the receiver-native intervention strengths

+8/-4

selected with Prompt/Python in Validation II.

Model	Arm	$a$	Unsafe/labeled	Rate (%)	Paired $\Delta$ [adjusted CI]
Qwen3.5-9B	Baseline	$0$	114/400	28.50	–
Qwen3.5-9B	Native $+$	$8$	270/400	67.50	$+39.00$ [ $30.25,48.00$ ]
Qwen3.5-9B	Shadow $+$	$8$	270/400	67.50	$+39.00$ [ $29.25,48.50$ ]
Ministral-3-8B	Baseline	$0$	133/400	33.25	–
Ministral-3-8B	Native $+$	$1$	198/400	49.50	$+16.25$ [ $10.75,22.25$ ]
Ministral-3-8B	Native $-$	$-0.25$	125/400	31.25	$-2.00$ [ $-5.25,1.75$ ]

Appendix

Table 28: Executed SODA trajectories with the fixed LUMEN interventions. Each arm has 400 tasks in 80 scenarios. Counts and rates use valid labels; deltas are equal-scenario means of paired changes, in percentage points. Intervals adjust over ten nonzero steering contrasts. Qwen has no selected negative intervention; its zero-strength aliases are not additional trajectories or suppression effects. Paired changes use available labels in both arms and can differ from marginal rate differences. Intervals condition on DeepSeek judgments and omit evaluator error.

Figure 10: SODA changes in executed unsafe trajectories relative to the same model’s baseline. Intervals adjust over ten steering or eight LoRA contrasts. The LoRA mean uses shared task pairs across all three replicas. No layer, intervention strength, or adapter is selected using these outcomes.

Adapter	Unsafe/labeled (%)	Pairs	$\Delta$ vs. baseline [adj. CI]	$\Delta$ vs. shadow [adj. CI]
Replica 1	161/400 (40.25%)	400	$+11.75$ [ $6.75,17.00$ ]	$-27.25$ [ $-35.50,-19.00$ ]
Replica 2	166/400 (41.50%)	400	$+13.00$ [ $8.25,18.25$ ]	$-26.00$ [ $-35.00,-17.50$ ]
Replica 3	173/400 (43.25%)	400	$+14.75$ [ $9.75,20.25$ ]	$-24.25$ [ $-32.75,-15.75$ ]
Mean	–	400	$+13.17$ [ $8.85,17.83$ ]	$-25.83$ [ $-34.08,-17.77$ ]

Appendix

Table 29: All three current Qwen9B LoRA replicas on SODA, with the same 400 tasks. Training seeds are 26082701, 26082702, and 26082703. Baseline and shadow

+8

counts appear in Table 28 . The mean comparison uses tasks with all three adapter labels and resamples shared scenarios. Intervals adjust over eight replica/mean contrasts. Changes are percentage points. This comparison measures unsafe trajectories, not benign SODA task preservation.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 442 papers

Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents
Oct 20, 2026Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang +1LLM Agent Safety Refusal Behavior in Language Models
Harvard University · Clarkson University · Carleton University
Efficient Refusal Ablation in LLM through Optimal Transport
Mar 4, 2026Geraldin Nanfack, Eugene Belilovsky, Elvis DohmatobAdversarial Attacks on LLMs Jailbreak Attacks
Concordia University, Montreal, Canada · Mila – Quebec AI Institute, Montreal, Canada
Low-Resource Safety Failures Are Action Failures, Not Representation Failures
May 31, 2026Rashad Aziz, Ikhlasul Akmal Hanif, Fajri KotoLanguage Model Calibration LLM Safety
Mohamed bin Zayed University of Artificial Intelligence

Oct 20, 2026J/K move · Enter open · S save

Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents

Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang, Adam Al Gharib

Harvard University · Clarkson University · Carleton University

LLM Agent Safety Refusal Behavior in Language Models LLM Safety Benchmarks

Open paper AskPDF arXiv

While large language models are increasingly deployed as tool-calling agents, their safety evaluations still focus primarily on their ability to refuse harmful requests in text. We investigate whether the refusal behavior in text transfers to tool use. We construct a benchmark of 2,304 prompts across four domains, consisting of matched triplets that present the same harmful or benign request in text-only, standard tool-enabled, and adversarially framed tool-enabled settings. We then assess tool-call safety using 20 predefined forbidden actions. Testing five models of varying sizes and families, we find that requests refused in text frequently result in unsafe tool calls. Under standard tool-enabled prompting, this divergence occurs for 10-52 percent of harmful requests refused in chat. To mechanistically investigate this gap, we identify a refusal-associated direction in the residual stream of each of the five models. We find that in tool-enabled contexts, the activations along this direction decrease in four of five models. Furthermore, lower activation predicts unsafe tool calls across all five models (AUC 0.76-0.86). However, restoring the signal to the respective text-only value through activation patching only prevents a subset of unsafe calls, primarily by suppressing tool use rather than by producing safer calls. Steering along the refusal direction nearly eliminates unsafe calls only at strengths that also suppress benign tool use or degrade the output quality. Our findings indicate that the identified refusal direction contributes to whether a model acts, but only partially explains the refusal divergence. Therefore, evaluating the safety of tool-calling agents requires directly examining their actions along with their textual responses.

Abstract

LLM Safety

Explore similar work

Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents

Efficient Refusal Ablation in LLM through Optimal Transport

Low-Resource Safety Failures Are Action Failures, Not Representation Failures