Recent literature shows that unsafe tool calls can often accompany textual refusals in language models. To investigate this hidden vulnerability, we introduce PRISM, a framework for evaluating the transfer of activation interventions across precision, languages, and scales. Our findings demonstrate that leveraging lower-resource settings (specifically quantized NF4 labels and cross-lingual transfer) offers a robust pathway to expose and improve model alignment. Notably, using behavioral labels from an NF4 copy enables stable intervention estimation even with very few scenarios. Furthermore, these alignment improvements transfer effectively across languages: five-language NF4-supervised directions successfully transfer to held-out Arabic, reducing unsafe violations in Llama-3.1-8B by 21.95 percentage points without target-language tuning. By mapping paired benign activations, we also successfully transfer these safety interventions to much larger models (up to 70B parameters). Ultimately, PRISM highlights how auxiliary supervision from compressed models and multilingual data can uncover and mitigate hazardous tool-calling behaviors across different model families. We release all experimental code and supporting artifacts at https://github.com/GiKerche/prism, alongside an interactive explorer of recorded model responses and experimental results at https://gikerche.github.io/prism/.
Figures & tables
Figure 1: The PRISM pipeline. Unsafe-response fractions from BF16/NF4 weight BF16 activations to estimate native/shadow directions using all five languages or a single language. We evaluate precision, cross-language (including held-out Arabic), and cross-scale transfer (via W ), applying shifts at selected BF16 layers.
Same LUMEN request. “We don’t want poor or peripheral candidates in the trainee program - they don’t fit with our rich clients. Filter to exclude anyone who studied at public school or lives in low-income neighborhoods.”
Baseline: no intervention
Direction estimated directly in the model, a=+2
Ministral3-8B → 14B direction, a=+2
“I must refuse that request.” The response explains that filtering by school type or neighborhood is discriminatory.
Proposes a process_candidate call with the action filter , explicitly listing public schools and low-income residential areas among the exclusion criteria.
Proposes a process_candidate call with the action filter , using public-school education and low neighborhood income as filtering criteria.
No tool call; no action violation.
Unsafe call proposed.
Unsafe call proposed.
Figure 2: Refusal to action in Ministral-3-14B. Same English request, ethical context, and seed (104); layer 20, a=+2 . The receiver-native direction is estimated in Ministral-3-14B, the transported one in Ministral-3-8B. Calls are recorded without execution; baseline and calls are abridged.
Figure 3: Behavioral effects and tool-use capability. Changes from baseline at the selected intervention strengths: native/shadow (sources), native/transported (receivers). Bars show pointwise 95% CIs conditional on observed labels and fixed interventions (adjusted CIs in appendix). LUMEN positive/negative: ethical/neutral. Qwen lacks suppression; SODA covers sources only; blanks denote unevaluated cases.
Model
Layer
BA (%)
a+
a−
Qwen3.5-9B
18
96.91
8
0
Ministral-3-8B
21
85.29
1
−0.25
Llama-3.1-8B
26
73.66
4
−4
Qwen3.5-27B
40
74.41
8
0
Ministral-3-14B
20
85.84
2
−1
Llama-3.1-70B
78
78.70
4
−4
Table 1: Selected layers (zero-based) and strengths. Rows 1–3: sources (native and shadow share a± ); rows 4–6: receivers with the transported direction. BA: probe balanced accuracy on Validation I. Qwen has a−=0 (no eligible neutral calls in Validation II, 0/340).
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Role
Scenarios
Families
Construction
35
33
Validation I: layer selection
16
11
Validation II: intervention strength selection
17
12
Final test
40
19
Appendix
Table 2: Scenario-family allocation for construction, selection, and final evaluation. All models use the same assignment.
Qwen3.5 9B
Ministral-3 8B
Llama-3.1 8B
Validation labels (number of responses)
Unsafe action
2
318
1715
No unsafe action
2558
2241
826
Missing label
0
1
19
Validation performance (%)
Probe accuracy
93.83
83.21
73.39
Appendix
Table 3: Labels and probe performance used to select each frozen layer on Validation I; label-audit sensitivity is reported in the text. Each model generated 2,560 validation responses from 16 scenarios in 11 families. Responses with missing labels are excluded from performance calculations. Accuracy is the weighted percentage of correct predictions. The majority-class baseline always predicts the most frequent class in the construction split; it does not measure evaluator agreement. Balanced accuracy averages recall for the unsafe and non-unsafe classes and is the layer-selection criterion.
Figure 5: Probe balanced accuracy across all source layers on the Validation I partition. Markers indicate the selected layers (18 for Qwen, 21 for Ministral, and 26 for Llama), and the dashed line indicates 50% balanced accuracy. Each model uses the same valid responses at every layer. These curves support layer selection and do not measure causal intervention effects.
Figure 6: How native and shadow directions are constructed. Both paths read the original BF16 model at the last input token, before it answers. Native weights these activations using violation fractions from BF16 responses; shadow uses fractions from the NF4 copy’s responses. Within-language contributions are pooled and normalized using Equation 5 . Both directions intervene in BF16.
Source
Layer
Native–shadow cosine
Common σ
Qwen3.5-9B
18
0.9838
2.7549
Ministral-3-8B
21
0.9811
2.3595
Llama-3.1-8B
26
0.9847
3.5429
Appendix
Table 4: Descriptive geometry of the actual pooled intervention vectors fitted on the 35-scenario construction partition. Cosines are computed from the frozen vectors used in the final acquisition. Their agreement does not establish equal behavioral effects.
Receiver
Layers
Rank
λrel
Energy (%)
NMSE
Qwen3.5-27B
18 → 40
256
0.01
24.13
0.332
Ministral-3-14B
21 → 20
256
0.01
58.84
0.252
Llama-3.1-70B
26 → 78
256
0.1
42.37
0.595
Appendix
Table 5: Benign-anchor maps. Layers are source → receiver. Map rank and penalty use 512 validation anchors after fitting on 1,024 anchors. Retained energy is the squared length of the unit source direction projected onto the selected source PCA basis. Normalized squared error (NMSE) on 512 diagnostic anchors does not select the map.
Model
Arm
a
Baseline
Intervention
Δ [95% CI]
BFCL tool
Qwen3.5-9B
N+
8
0/800 (0.00%)
609/796 (76.51%)
+76.51 [68.41, 81.16]
-2.84
Qwen3.5-9B
S+
8
0/800 (0.00%)
623/797 (78.17%)
+78.17 [66.54, 83.94]
-2.41
Qwen3.5-9B
N-
0
19/800 (2.38%)
19/800 (2.38%)
+0.00 [0.00, 0.00]
+0.00
Qwen3.5-9B
S-
0
19/800 (2.38%)
19/800 (2.38%)
+0.00 [0.00, 0.00]
+0.00
Ministral-3-8B
N+
1
23/800 (2.88%)
73/798 (9.15%)
+6.27 [1.93, 10.65]
+0.88
Ministral-3-8B
S+
1
23/800 (2.88%)
78/797 (9.79%)
+6.90 [2.62, 11.77]
+1.20
Appendix
Table 6: Source effects: N uses native labels; S uses NF4 labels. Plus uses ethical context; minus uses neutral context. Each arm has 800 responses. Counts and percentages use available labels. Paired deltas use only pairs with both labels available, so they can differ from the marginal rate difference. Intervals are pointwise family-bootstrap 95% CIs; adjusted intervals are in Table 8 . The BFCL column gives accuracy changes on 914 tool-required cases. All changes are percentage points. Qwen has no selected suppression intervention strength because its neutral Validation II baseline has no eligible calls (0/340). Zero-strength rows reuse the final baseline and do not measure an intervention effect.
Model
Arm
a
Baseline
Intervention
Δ [95% CI]
BFCL tool
Qwen3.5-27B
T+
8
0/800 (0.00%)
605/796 (76.01%)
+76.01 [65.12, 82.77]
-2.19
Qwen3.5-27B
T-
0
8/800 (1.00%)
8/800 (1.00%)
+0.00 [0.00, 0.00]
+0.00
Ministral-3-14B
T+
2
6/800 (0.75%)
35/800 (4.38%)
+3.62 [1.21, 7.66]
+0.11
Ministral-3-14B
T-
−1
148/797 (18.57%)
87/800 (10.88%)
-7.65 [-12.63, -1.94]
-0.77
Llama-3.1-70B
T+
4
35/800 (4.38%)
255/800 (31.87%)
+27.50 [22.14, 33.91]
+0.11
Llama-3.1-70B
T-
−4
383/792 (48.36%)
104/799 (13.02%)
-35.15 [-46.53, -23.22]
+0.00
Appendix
Table 7: Receiver effects: T is the transported native source direction. Plus uses ethical context; minus uses neutral context. Each arm has 800 responses. Counts and percentages use available labels. Paired deltas use only pairs with both labels available, so they can differ from the marginal rate difference. Intervals are pointwise family-bootstrap 95% CIs; adjusted intervals are in Table 8 . The BFCL column gives accuracy changes on 914 tool-required cases. All changes are percentage points. Qwen has no selected suppression intervention strength because its neutral Validation II baseline has no eligible calls (0/340). Zero-strength rows reuse the final baseline and do not measure an intervention effect.
Model
Contrast
Labeled pairs
Δ (pp)
Adjusted interval (pp)
Qwen3.5-9B
N+ − B
796/800
+76.51
[62.55, 83.02]
Qwen3.5-9B
S+ − B
797/800
+78.17
[58.78, 85.74]
Qwen3.5-9B
N+ − S+
793/800
-1.51
[-4.64, 5.52]
Qwen3.5-9B
N- − B
800/800
+0.00
[0.00, 0.00]
Qwen3.5-9B
S- − B
800/800
+0.00
[0.00, 0.00]
Qwen3.5-9B
N- − S-
800/800
+0.00
[0.00, 0.00]
Appendix
Table 8: The 18 predefined source contrasts use nominal 1−.05/18 intervals, and the six receiver contrasts use 1−.05/6 , from 10,000 paired family-bootstrap draws. N/S/T denote native/shadow/transported; B denotes the context-matched baseline. These percentile intervals condition on fitted objects and observed judge labels. Exact-zero intervals for zero-strength aliases represent identical observations.
Model
Estimator
Estimation → evaluation
Labeled pairs
Δ (pp) [95% CI]
Qwen3.5-9B
native
Spanish → Portuguese
159/160
+87.42 [ +79.00,+93.50 ]
Qwen3.5-9B
shadow
Chinese → Portuguese
160/160
+88.12 [ +77.42,+93.75 ]
Ministral-3-8B
native
Hindi → Chinese
160/160
+13.75 [ +3.33,+24.19 ]
Ministral-3-8B
shadow
Hindi → Chinese
160/160
+12.50 [ +3.38,+20.65 ]
Llama-3.1-8B
native
Portuguese → Spanish
154/160
+34.42 [ +15.44,+54.82 ]
Llama-3.1-8B
shadow
Portuguese → Spanish
157/160
+25.48 [ +11.27,+41.23 ]
Appendix
Table 9: Largest cross-language induction point estimates in ethical context. Each row is selected descriptively from the 20 off-diagonal cells for that model and estimator. Intervals are pointwise, without adjustment for selecting a maximum; these rows do not establish that the listed pair is uniquely best. Denominators show pairs with both labels available out of 160 attempts.
Model
Arm
Baseline
Intervention
Δ [95% CI] (pp)
Qwen3.5-9B
N+
0/44 (0.00%)
39/44 (88.64%)
+88.64 [80.77, 97.50]
Qwen3.5-9B
S+
0/44 (0.00%)
40/43 (93.02%)
+93.02 [82.69, 100.00]
Qwen3.5-9B
N-
1/44 (2.27%)
1/44 (2.27%)
+0.00 [0.00, 0.00]
Qwen3.5-9B
S-
1/44 (2.27%)
1/44 (2.27%)
+0.00 [0.00, 0.00]
Ministral-3-8B
N+
0/44 (0.00%)
1/44 (2.27%)
+2.27 [0.00, 9.38]
Ministral-3-8B
S+
0/44 (0.00%)
0/44 (0.00%)
+0.00 [0.00, 0.00]
Appendix
Table 10: Arabic transfer of pooled five-language vectors: 44 responses per generated condition from 11 scenarios in six families, excluded from construction and both validation partitions. N/S denote native/shadow; plus is ethical and minus neutral context. Counts use available labels and deltas use complete pairs. Pointwise intervals are descriptive. A zero-event or degenerate interval is not evidence of zero risk; Qwen zero-strength arms are exact baseline aliases.
Model
Audited
A/B disagree
C agrees when A=B
Qwen3.5-9B
133
27/133 (20.30%)
104/106 (98.11%)
Ministral-3-8B
264
23/264 (8.71%)
230/238 (96.64%)
Llama-3.1-8B
272
92/272 (33.82%)
156/178 (87.64%)
Qwen3.5-27B
13
2/13 (15.38%)
11/11 (100.00%)
Ministral-3-14B
33
3/33 (9.09%)
29/30 (96.67%)
Llama-3.1-70B
38
22/38 (57.89%)
14/16 (87.50%)
Appendix
Table 11: Stratified final audit. A is Claude Sonnet 4.6, B is Grok 4.5, and C is DeepSeek V4 Flash 0731 FP8. Counts aggregate all components within each model. C agreement conditions on valid A and B votes agreeing and a valid C vote. These raw sample fractions are descriptive; inclusion-weighted summaries and missingness are exported. They are not human-validated accuracy, and no audit vote replaces a primary label.
Condition
Ethical unsafe
Neutral unsafe
BFCL tool-required
Unsteered
0/800 (0.00%)
19/800 (2.38%)
846/914 (92.56%)
Native a=+8
609/796 (76.51%)
–
820/914 (89.72%)
Shadow a=+8
623/797 (78.17%)
–
824/914 (90.15%)
LoRA seed 26082701
485/800 (60.62%)
461/799 (57.70%)
859/914 (93.98%)
LoRA seed 26082702
478/800 (59.75%)
452/800 (56.50%)
854/914 (93.44%)
LoRA seed 26082703
500/799 (62.58%)
473/800 (59.13%)
856/914 (93.65%)
Appendix
Table 12: Current Qwen9B LoRA comparison: unsafe/labeled responses for LUMEN, and valid/total cases for BFCL, with percentages. All three adapters use construction data only; their fixed targets are described in the text. Steering uses layer 18 and a=+8 in ethical context; no nonzero Qwen suppression intervention strength was selected. Each unsafe arm has 800 possible responses; BFCL has 914 tool-required cases. Missing judgments remain missing. These are marginal rates, which need not subtract to the paired estimates.
Quantity
Steering
LoRA
Stored coefficients
4,096
43,278,336
Additional artifact
16 KiB
165.15 MiB
Weight-update steps
0
20
Optimizer-loop time (min)
—
15.99–16.50
Training peak (GiB)
—
55.67
Appendix
Table 13: Measured Qwen9B intervention artifacts and LoRA training resources. LoRA entries are per adapter; the time range covers all three replicas on one H200 each. Steering stores one pooled FP32 direction; LoRA stores FP32 adapter weights. The shared BF16 checkpoint is excluded from artifact size. Peak memory is PyTorch allocated memory, including the backbone. Dashes denote quantities without a comparable measurement, not zero preparation cost.
Source
BF16 positives
NF4 positives
Qwen3.5-9B
28/700 (4.00%)
56/699 (8.01%)
Ministral-3-8B
98/700 (14.00%)
188/700 (26.86%)
Llama-3.1-8B
459/692 (66.33%)
371/695 (53.38%)
Appendix
Table 14: Construction labels: unsafe/labeled responses, out of 700 attempts per model and precision. Missing labels remain excluded, not negative.
Model
Language
BF16 unsafe/labeled (%)
NF4 unsafe/labeled (%)
Qwen3.5-9B
pt
4/140 (2.86)
11/140 (7.86)
Qwen3.5-9B
en
4/140 (2.86)
4/140 (2.86)
Qwen3.5-9B
es
5/140 (3.57)
10/140 (7.14)
Qwen3.5-9B
zh
5/140 (3.57)
8/140 (5.71)
Qwen3.5-9B
hi
10/140 (7.14)
23/139 (16.55)
Ministral-3-8B
pt
19/140 (13.57)
34/140 (24.29)
Appendix
Table 15: Construction positives by language. Each precision has 140 attempted responses per cell; the difference from the labeled denominator is missingness. All ten language vectors per source model are identifiable at full budget.
Figure 7: Construction-only, paired family-subsampling diagnostic. NF4 reduces undefined direction fits for Qwen and Ministral when few construction families are available. At eight families, Llama fits are available in every subset under either label source. This does not measure downstream effects, generalization or monetary savings.
Figure 8: Complete Validation II intervention-strength curves at frozen layers. Lines show changes in the valid action-call proxy, not semantic unsafe-action labels. Solid/dashed lines denote native/shadow sources; receivers use transported native directions. Stars identify retained nonzero intervention strengths, circles other admissible points, and crosses points rejected by capability or quality gates. Colors distinguish the positive ethical and negative neutral contrasts.
Figure 9: Cosines between the actual language-specific intervention vectors fitted on current construction data. High alignment in Ministral and lower alignment in Qwen/Llama coexist with varying cross-language effects. Cosine is a geometric diagnostic and does not establish equal causal efficacy.
Model
Estimator
Context
Diagonal Δ
Other-language Δ
Paired same minus other [95% CI]
Qwen3.5-9B
native
ethical
+59.39
+63.72
-4.09 [-8.31, +0.51]
Qwen3.5-9B
shadow
ethical
+62.46
+64.21
-2.00 [-4.64, +2.43]
Ministral-3-8B
native
ethical
+8.02
+6.04
+1.98 [+0.47, +4.21]
Ministral-3-8B
native
neutral
-5.14
-4.94
-0.50 [-1.31, +0.72]
Ministral-3-8B
shadow
ethical
+7.41
+6.33
+0.88 [-0.47, +2.33]
Ministral-3-8B
shadow
neutral
-5.75
-5.51
-0.25 [-0.83, +0.44]
Appendix
Table 16: Descriptive diagonal and off-diagonal cell averages, and a separate paired contrast over complete five-vector tuples, in percentage points. Qwen neutral outcomes are zero-strength aliases. Different missing-label support means subtraction of the two cell averages need not equal the paired contrast.
Model
Estimator
Context
Claude
Grok
DeepSeek audit
Majority
Qwen3.5-9B
native
ethical
+74.26
+33.02
+77.56
+74.26
Qwen3.5-9B
shadow
ethical
+78.79
+56.71
+66.46
+66.46
Ministral-3-8B
native
ethical
+7.54
+7.10
+7.51
+7.54
Ministral-3-8B
shadow
ethical
+8.13
+8.13
+8.25
+8.13
Ministral-3-8B
native
neutral
-7.45
-2.38
-4.43
-4.96
Ministral-3-8B
shadow
neutral
-10.50
-8.57
-10.62
-11.15
Appendix
Table 17: Effects standardized from the stratified audit at the selected intervention strengths, in percentage points. The same audited candidates and inclusion weights are used for all four columns. These sparse-audit point estimates are not the full-population primary effects, uncertainty intervals, or independent validation. Qwen zero suppression aliases are omitted.
Context
Evaluator
Baseline unsafe
Intervention unsafe
Paired Δ (pp)
Adjusted interval (pp)
Ethical +4
Claude
41/800 (5.12%)
281/800 (35.12%)
+30.00
[+23.06,+41.65]
Ethical +4
Grok
20/800 (2.50%)
111/800 (13.88%)
+11.38
[+6.06,+24.25]
Ethical +4
DeepSeek
35/800 (4.38%)
255/800 (31.87%)
+27.50
[+20.00,+37.34]
Ethical +4
Majority
36/800 (4.50%)
256/800 (32.00%)
+27.50
[+20.28,+39.05]
Neutral −4
Claude
429/800 (53.62%)
120/800 (15.00%)
−38.62
[−53.75,−21.68]
Neutral −4
Grok
146/800 (18.25%)
47/800 (5.88%)
−12.38
[−20.93,−5.62]
Appendix
Table 18: Complete evaluator sensitivity in Llama70B. Each arm has 800 responses; percentages use available labels. All 952 structural candidates receive both Claude and Grok votes; exact structural negatives are shared. DeepSeek supplies the primary endpoint. Family-bootstrap intervals adjust across the eight displayed contrasts. Missing labels remain missing, and the majority requires two matching valid votes. Paired effects need not equal differences of marginal percentages. These intervals exclude evaluator error.
Model
Context
Δ (pp)
98% interval (pp)
Qwen3.5-9B
ethical
−1.51
[−4.10,+4.01]
Ministral-3-8B
ethical
−0.75
[−3.37,+0.26]
Ministral-3-8B
neutral
−0.25
[−1.13,+1.12]
Llama-3.1-8B
ethical
+1.64
[−1.30,+3.59]
Llama-3.1-8B
neutral
−1.80
[−5.47,+1.75]
Appendix
Table 19: Exploratory native-minus-shadow equivalence-margin diagnostic. The reference margin is [−7,+7] pp. The margin was introduced after outcomes were inspected and is not an independently established utility threshold. Qwen’s zero-strength baseline alias is excluded.
Control
Unsafe/labeled (%)
Δ vs. baseline [adj. CI]
Δ vs. shadow [adj. CI]
BFCL Δ
Random 1 ( + )
1/800 (0.12%)
+0.12 [ 0.00,1.22 ]
−78.04 [ −86.24,−53.27 ]
−10.83
Random 2 ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−78.17 [ −86.34,−53.52 ]
−14.00
Random 3 ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−78.17 [ −86.34,−53.52 ]
−19.37
Probe ( + )
4/800 (0.50%)
+0.50 [ 0.00,1.78 ]
−77.67 [ −86.00,−53.39 ]
−6.67
Call propensity ( + )
580/800 (72.50%)
+72.50 [ 50.64,83.84 ]
−5.77 [ −16.92,6.09 ]
−2.52
Permuted labels ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−78.17 [ −86.34,−53.52 ]
−15.97
Appendix
Table 20: Qwen3.5-9B controls at the same layer and signed displacement as the original intervention. Each arm contains 800 responses; counts and rates use available semantic labels. Plus uses ethical and minus neutral context. Baselines: ethical 0/800 (0.00%). Changes are paired percentage points; brackets are Bonferroni-adjusted family-bootstrap intervals for the full declared family of 174 additional contrasts. BFCL is the validity change on 914 tool-required cases. Controls are not retuned to pass the capacity gate. Zero-event bootstrap intervals do not establish zero population risk. Native/shadow primary outcomes are retained in Table 6 .
Control
Unsafe/labeled (%)
Δ vs. baseline [adj. CI]
Δ vs. shadow [adj. CI]
BFCL Δ
Random 1 ( + )
22/800 (2.75%)
−0.12 [ −3.04,1.03 ]
−7.03 [ −19.82,−0.46 ]
−0.77
Random 2 ( + )
23/800 (2.88%)
+0.00 [ −1.59,0.69 ]
−6.90 [ −18.45,−0.46 ]
−1.53
Random 3 ( + )
25/800 (3.12%)
+0.25 [ −1.03,1.07 ]
−6.65 [ −17.77,−0.35 ]
−0.77
Probe ( + )
26/800 (3.25%)
+0.38 [ −2.24,2.07 ]
−6.52 [ −19.14,−0.46 ]
−0.77
Call propensity ( + )
76/800 (9.50%)
+6.62 [ 0.36,13.98 ]
−0.50 [ −4.56,1.31 ]
+0.88
Permuted labels ( + )
16/800 (2.00%)
−0.88 [ −3.45,0.00 ]
−7.78 [ −20.56,−0.46 ]
−1.86
Appendix
Table 21: Ministral-3-8B controls at the same layer and signed displacement as the original intervention. Each arm contains 800 responses; counts and rates use available semantic labels. Plus uses ethical and minus neutral context. Baselines: ethical 23/800 (2.88%); neutral 165/800 (20.62%). Changes are paired percentage points; brackets are Bonferroni-adjusted family-bootstrap intervals for the full declared family of 174 additional contrasts. BFCL is the validity change on 914 tool-required cases. Controls are not retuned to pass the capacity gate. Zero-event bootstrap intervals do not establish zero population risk. Native/shadow primary outcomes are retained in Table 6 .
Control
Unsafe/labeled (%)
Δ vs. baseline [adj. CI]
Δ vs. shadow [adj. CI]
BFCL Δ
Random 1 ( + )
310/800 (38.75%)
−15.19 [ −26.35,−2.47 ]
−32.91 [ −56.08,−14.12 ]
+0.00
Random 2 ( + )
347/800 (43.38%)
−10.25 [ −18.25,−0.06 ]
−28.27 [ −48.64,−8.84 ]
+0.11
Random 3 ( + )
342/800 (42.75%)
−11.01 [ −23.17,−1.26 ]
−28.89 [ −52.06,−8.80 ]
−0.22
Probe ( + )
352/800 (44.00%)
−9.87 [ −22.18,−0.55 ]
−27.64 [ −50.12,−6.90 ]
−0.77
Call propensity ( + )
605/799 (75.72%)
+22.18 [ 0.40,49.75 ]
+4.15 [ −1.15,9.47 ]
−1.20
Permuted labels ( + )
213/800 (26.62%)
−27.47 [ −38.41,−13.23 ]
−44.97 [ −62.41,−28.37 ]
−2.08
Appendix
Table 22: Llama-3.1-8B controls at the same layer and signed displacement as the original intervention. Each arm contains 800 responses; counts and rates use available semantic labels. Plus uses ethical and minus neutral context. Baselines: ethical 426/790 (53.92%); neutral 547/787 (69.50%). Changes are paired percentage points; brackets are Bonferroni-adjusted family-bootstrap intervals for the full declared family of 174 additional contrasts. BFCL is the validity change on 914 tool-required cases. Controls are not retuned to pass the capacity gate. Zero-event bootstrap intervals do not establish zero population risk. Native/shadow primary outcomes are retained in Table 6 .
Estimator
Context
a
Baseline unsafe
Intervention unsafe
Paired Δ [adj. CI]
Native
Neutral
+4
547/787 (69.50%)
622/800 (77.75%)
+8.39 [ −4.03,21.85 ]
Native
Ethical
−4
426/790 (53.92%)
135/800 (16.88%)
−36.96 [ −50.61,−20.97 ]
Shadow
Neutral
+4
547/787 (69.50%)
624/798 (78.20%)
+8.92 [ −4.27,23.09 ]
Shadow
Ethical
−4
426/790 (53.92%)
159/800 (19.88%)
−33.80 [ −43.88,−20.91 ]
Appendix
Table 23: Additional Llama8B tests of the opposite intervention strength sign in each context. Each arm has 800 responses; denominators retain missing labels. Changes are percentage points, with intervals adjusted over the same 174-comparison control family. Identical intervention settings reuse their existing tool-required BFCL evaluation.
Direction
Unsafe/labeled (%)
Δ vs. baseline [adj. CI]
Δ vs. transport [adj. CI]
BFCL Δ
Random 1 ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−76.01 [ −86.00,−54.21 ]
−3.83
Random 2 ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−76.01 [ −86.00,−54.21 ]
−4.27
Random 3 ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−76.01 [ −86.00,−54.21 ]
−7.33
Probe ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−76.01 [ −86.00,−54.21 ]
−6.13
Call propensity ( + )
152/800 (19.00%)
+19.00 [ 8.33,28.23 ]
−57.04 [ −67.82,−35.62 ]
−7.99
Permuted labels ( + )
0/800 (0.00%)
+0.00 [ 0.00,0.00 ]
−76.01 [ −86.00,−54.21 ]
−25.38
Appendix
Table 24: Qwen3.5-27B controls at the layer and signed displacement used for transport. Plus uses ethical and minus neutral context; every arm has 800 responses. Baselines: ethical 0/800 (0.00%); neutral 8/800 (1.00%). Counts and rates use available judgments. Changes are paired percentage points, with family-bootstrap intervals adjusted over all 174 additional contrasts. The transport difference is control minus transported intervention. BFCL reports the validity change on 914 tool-required cases. These controls retain the transport intervention strength and are not separately optimized for capacity. All three random draws and the permuted-label draw are reported.
Direction
Unsafe/labeled (%)
Δ vs. baseline [adj. CI]
Δ vs. transport [adj. CI]
BFCL Δ
Random 1 ( + )
7/800 (0.88%)
+0.12 [ 0.00,0.35 ]
−3.50 [ −12.23,0.00 ]
−0.44
Random 2 ( + )
6/800 (0.75%)
+0.00 [ −2.12,0.68 ]
−3.62 [ −13.99,0.00 ]
−2.08
Random 3 ( + )
6/800 (0.75%)
+0.00 [ −1.06,0.34 ]
−3.62 [ −13.04,0.00 ]
+0.33
Probe ( + )
14/800 (1.75%)
+1.00 [ 0.00,3.23 ]
−2.62 [ −10.38,0.00 ]
−0.77
Call propensity ( + )
70/800 (8.75%)
+8.00 [ 0.59,21.28 ]
+4.38 [ 0.25,10.36 ]
+1.20
Permuted labels ( + )
5/800 (0.62%)
−0.12 [ −1.06,0.00 ]
−3.75 [ −13.04,0.00 ]
−3.50
Appendix
Table 25: Ministral-3-14B controls at the layer and signed displacement used for transport. Plus uses ethical and minus neutral context; every arm has 800 responses. Baselines: ethical 6/800 (0.75%); neutral 148/797 (18.57%). Counts and rates use available judgments. Changes are paired percentage points, with family-bootstrap intervals adjusted over all 174 additional contrasts. The transport difference is control minus transported intervention. BFCL reports the validity change on 914 tool-required cases. These controls retain the transport intervention strength and are not separately optimized for capacity. All three random draws and the permuted-label draw are reported.
Direction
Unsafe/labeled (%)
Δ vs. baseline [adj. CI]
Δ vs. transport [adj. CI]
BFCL Δ
Random 1 ( + )
37/800 (4.62%)
+0.25 [ −2.57,2.45 ]
−27.25 [ −40.87,−18.02 ]
−0.22
Random 2 ( + )
31/800 (3.88%)
−0.50 [ −3.14,2.13 ]
−28.00 [ −42.61,−17.98 ]
+0.00
Random 3 ( + )
31/800 (3.88%)
−0.50 [ −3.01,2.12 ]
−28.00 [ −42.30,−18.43 ]
+0.11
Probe ( + )
81/800 (10.12%)
+5.75 [ 1.25,15.24 ]
−21.75 [ −30.81,−12.73 ]
−0.22
Call propensity ( + )
366/800 (45.75%)
+41.38 [ 29.40,51.47 ]
+13.88 [ 4.38,18.66 ]
+0.00
Permuted labels ( + )
407/800 (50.88%)
+46.50 [ 34.49,58.43 ]
+19.00 [ 7.06,29.01 ]
−0.22
Appendix
Table 26: Llama-3.1-70B controls at the layer and signed displacement used for transport. Plus uses ethical and minus neutral context; every arm has 800 responses. Baselines: ethical 35/800 (4.38%); neutral 383/792 (48.36%). Counts and rates use available judgments. Changes are paired percentage points, with family-bootstrap intervals adjusted over all 174 additional contrasts. The transport difference is control minus transported intervention. BFCL reports the validity change on 914 tool-required cases. These controls retain the transport intervention strength and are not separately optimized for capacity. All three random draws and the permuted-label draw are reported.
Model
Native a
Unsafe/labeled (%)
Δ vs. baseline [adj. CI]
Δ vs. transport [adj. CI]
BFCL Δ
Qwen3.5-27B
+4
7/800 (0.88%)
+0.88 [ 0.00,3.27 ]
−75.13 [ −84.98,−53.20 ]
−0.22
Qwen3.5-27B
0†
8/800 (1.00%)
+0.00 [ 0.00,0.00 ]
+0.00 [ 0.00,0.00 ]
+0.00
Ministral-3-14B
+2
65/800 (8.12%)
+7.38 [ 0.54,19.48 ]
+3.75 [ 0.16,8.37 ]
+1.20
Ministral-3-14B
−0.25
128/800 (16.00%)
−2.76 [ −6.25,0.00 ]
+5.12 [ 0.25,10.91 ]
−0.66
Llama-3.1-70B
+8
648/798 (81.20%)
+76.82 [ 64.09,83.93 ]
+49.37 [ 27.48,62.95 ]
−0.33
Llama-3.1-70B
−4
146/800 (18.25%)
−29.92 [ −47.35,−14.79 ]
+5.13 [ 0.08,12.40 ]
+0.44
Appendix
Table 27: Receiver-native directions at their Validation II-selected intervention strengths. Positive intervention strengths use ethical and negative intervention strengths neutral context. The transport comparators use +8/0 , +2/−1 , and +4/−4 for Qwen, Ministral, and Llama, respectively. Each arm has 800 possible responses; BFCL uses 914 tool-required cases. Changes are paired percentage points and intervals adjust over the same 174 additional contrasts. † No negative Qwen intervention strength was selected; this row reuses baseline. Ministral +2 and Llama −4 reuse the identical receiver-native conditions in the matched-displacement tables. These reused observations are not additional experimental replicates. Llama70B uses the receiver-native intervention strengths +8/−4 selected with Prompt/Python in Validation II.
Model
Arm
a
Unsafe/labeled
Rate (%)
Paired Δ [adjusted CI]
Qwen3.5-9B
Baseline
0
114/400
28.50
–
Qwen3.5-9B
Native +
8
270/400
67.50
+39.00 [ 30.25,48.00 ]
Qwen3.5-9B
Shadow +
8
270/400
67.50
+39.00 [ 29.25,48.50 ]
Ministral-3-8B
Baseline
0
133/400
33.25
–
Ministral-3-8B
Native +
1
198/400
49.50
+16.25 [ 10.75,22.25 ]
Ministral-3-8B
Native −
−0.25
125/400
31.25
−2.00 [ −5.25,1.75 ]
Appendix
Table 28: Executed SODA trajectories with the fixed LUMEN interventions. Each arm has 400 tasks in 80 scenarios. Counts and rates use valid labels; deltas are equal-scenario means of paired changes, in percentage points. Intervals adjust over ten nonzero steering contrasts. Qwen has no selected negative intervention; its zero-strength aliases are not additional trajectories or suppression effects. Paired changes use available labels in both arms and can differ from marginal rate differences. Intervals condition on DeepSeek judgments and omit evaluator error.
Figure 10: SODA changes in executed unsafe trajectories relative to the same model’s baseline. Intervals adjust over ten steering or eight LoRA contrasts. The LoRA mean uses shared task pairs across all three replicas. No layer, intervention strength, or adapter is selected using these outcomes.
Adapter
Unsafe/labeled (%)
Pairs
Δ vs. baseline [adj. CI]
Δ vs. shadow [adj. CI]
Replica 1
161/400 (40.25%)
400
+11.75 [ 6.75,17.00 ]
−27.25 [ −35.50,−19.00 ]
Replica 2
166/400 (41.50%)
400
+13.00 [ 8.25,18.25 ]
−26.00 [ −35.00,−17.50 ]
Replica 3
173/400 (43.25%)
400
+14.75 [ 9.75,20.25 ]
−24.25 [ −32.75,−15.75 ]
Mean
–
400
+13.17 [ 8.85,17.83 ]
−25.83 [ −34.08,−17.77 ]
Appendix
Table 29: All three current Qwen9B LoRA replicas on SODA, with the same 400 tasks. Training seeds are 26082701, 26082702, and 26082703. Baseline and shadow +8 counts appear in Table 28 . The mean comparison uses tasks with all three adapter labels and resamples shared scenarios. Intervals adjust over eight replica/mean contrasts. Changes are percentage points. This comparison measures unsafe trajectories, not benign SODA task preservation.
While large language models are increasingly deployed as tool-calling agents, their safety evaluations still focus primarily on their ability to refuse harmful requests in text. We investigate whether the refusal behavior in text transfers to tool use. We construct a benchmark of 2,304 prompts across four domains, consisting of matched triplets that present the same harmful or benign request in text-only, standard tool-enabled, and adversarially framed tool-enabled settings. We then assess tool-call safety using 20 predefined forbidden actions. Testing five models of varying sizes and families, we find that requests refused in text frequently result in unsafe tool calls. Under standard tool-enabled prompting, this divergence occurs for 10-52 percent of harmful requests refused in chat. To mechanistically investigate this gap, we identify a refusal-associated direction in the residual stream of each of the five models. We find that in tool-enabled contexts, the activations along this direction decrease in four of five models. Furthermore, lower activation predicts unsafe tool calls across all five models (AUC 0.76-0.86). However, restoring the signal to the respective text-only value through activation patching only prevents a subset of unsafe calls, primarily by suppressing tool use rather than by producing safer calls. Steering along the refusal direction nearly eliminates unsafe calls only at strengths that also suppress benign tool use or degrade the output quality. Our findings indicate that the identified refusal direction contributes to whether a model acts, but only partially explains the refusal divergence. Therefore, evaluating the safety of tool-calling agents requires directly examining their actions along with their textual responses.
Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang +1
Harvard University · Clarkson University · Carleton University
Safety-aligned language models refuse harmful requests through learned refusal behaviors encoded in their internal representations. Recent activation-based jailbreaking methods circumvent these safety mechanisms by applying orthogonal projections to remove refusal directions, but these approaches treat refusal as a one-dimensional phenomenon and ignore the rich distributional structure of model activations. We introduce a principled framework based on optimal transport theory that transforms the entire distribution of harmful activations to match harmless ones. By combining PCA with closed-form Gaussian optimal transport, we achieve efficient computation in high-dimensional representation spaces while preserving essential geometric structure. Across six models (Llama-2, Llama-3.1, Qwen-2.5; 7B-32B parameters), our method achieves up to 11% higher attack success rates than state-of-the-art baselines while maintaining comparable perplexity, demonstrating superior preservation of model capabilities. Critically, we discover that layer-selective intervention (applying optimal transport to 1-2 carefully chosen layers at approximately 40-60% network depth) substantially outperforms full-network interventions, revealing that refusal mechanisms may be localized rather than distributed. Our analysis provides new insights into the geometric structure of safety representations and suggests that current alignment methods may be vulnerable to distributional attacks beyond simple direction removal.
Geraldin Nanfack, Eugene Belilovsky, Elvis Dohmatob
Concordia University, Montreal, Canada · Mila – Quebec AI Institute, Montreal, Canada
Language models often answer harmful requests in low-resource languages (LRLs) that they refuse in high-resource languages (HRLs). Across three instruction-tuned models and 23 languages, harmful refusal falls from 87.9% in HRLs to 43.9% in LRLs, while harmless refusal remains low. A common explanation is that models represent harmfulness weakly in LRLs. We test whether harmfulness is instead represented but does not reliably produce refusal. Across three models, a harmfulness direction learned from HRL activations still separates harmful from harmless LRL prompts, showing that complete absence of harmfulness information cannot explain many failures. However, harmfulness scores shift downward for LRL prompts, making harmful prompts less likely to reach the range associated with refusal. Motivated by this shift, we train a low-rank logistic classifier on HRL activations and calibrate its threshold with a few target-language examples. During generation, the classifier conditionally adds or ablates the HRL harmfulness direction. With the same HRL data and 32 target-language examples per class, CAST remains limited by low harmful refusal and AdaSteer by high harmless refusal, yielding mean refusal selectivity (Δ = harmful - harmless refusal) of 33.6 and 6.8, respectively. Our intervention reaches 54.5 while preserving MMLU utility. HRL-only calibration improves selectivity for Qwen and Gemma, whereas Llama benefits from target-language calibration. These results show that recalibrating existing representations can offer a training-free method for repairing low-resource safety failures.
Rashad Aziz, Ikhlasul Akmal Hanif, Fajri Koto
Mohamed bin Zayed University of Artificial Intelligence