cs.LGSep 4, 2026

Locating and Steering Refusal Beyond Attention

Authors: Preethi Carmel Bosco, Gopalakrishnan Srinivasan

Organizations: Department of Computer Science and Engineering, Indian Institute of Technology Madras

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Where inside a language model does refusal live, and does that place change when the architecture does? In a transformer, refusal is governed by a single direction in the residual stream, a finding that safety and interpretability tooling now depend on. State-space models (SSMs) route information through a recurrent update instead of attention, sharing no token-mixing mechanism with a transformer. Does the same safety representation survive this shift, or must it be rediscovered per architecture? It survives. A single rigid rotation, which can only reorient a space and not reshape it, aligns one model's representation space with another's, so the two genuinely share the representation. A harm probe trained on a transformer then flags an SSM's harmful inputs, and removing the aligned direction makes a model answer attacks it would otherwise refuse, while a random direction of the same size does far less. What is architecture-specific is not where the direction is steered but where it must be read. Each layer computes a fresh output that is then added into the residual stream, and harm is cleanly readable at this output, the write site, before the addition. A control that holds the intervention's strength fixed shows that what matters is where the direction is estimated, not where it is applied. Applied through a detector-triggered gate, this direction lowers jailbreak success in all four architecture families we test (SSM, transformer, recurrent, hybrid), and on the SSM it holds against an attacker that tunes its prompt against the defense. The gate only matches a trivial rule that returns a fixed refusal whenever the same detector fires, so what transfers across architectures is the direction itself, not defense strength. Safety tooling built on refusal therefore ports to a new architecture by re-estimating the direction at that architecture's write site, not by rebuilding it.

Figures & tables

Figure 1: The write site in the two reference architectures. Left: transformer block (Llama-3.1, Mistral). The write site is the attention output projection WOW_{O} , the fresh contribution attention writes into the residual stream. Right: Falcon-Mamba block (selective-scan SSM). The write site is the mixer’s out-projection, the block output the Δ\Delta -gated scan writes before the residual accumulates it, and Δ\Delta inside the scan is decodable but a readout only. In both, the probe and the refusal direction are estimated at the red site; a direction estimated in the accumulated residual (gray) degenerates.
ModelsCorporaAttacksJudges & metricsBaselines
Falcon-Mamba-7B (SSM anchor) Falcon3-Mamba-7B (2nd SSM) Llama-3.1-8B, Mistral-7B, Qwen2.5-7B (transformers) RWKV-6-World-7B (recurrent) Zamba2-7B (hybrid)AdvBench (harm) Alpaca (benign) XSTest (surface control, over-refusal)Prefix injection (fluent) Persona roleplay (DAN, fluent) Fictional framing (fluent) Dilution (adaptive) Optimized adversarial suffix (GCG; on the SSM both a gradient-free variant and GCG through the differentiable scan)Llama-Guard-3-8B (primary) +Llama-3.1-8B-Instruct, Mistral-7B-Instruct, ShieldGemma-9B MMLU, TruthfulQA (capability) Probe AUROC (decodability)Residual-subspace projection CAST Perplexity filter SmoothLLM Cross-architecture linear-map transfer
Table 1: Experimental setup at a glance. The minimal-pair corpus is matched to XSTest in template and length. Over-refusal is measured on XSTest’s benign-but-spicy prompts (safe requests worded to sound harmful). Attack success is the Llama-Guard-3-8B-judged unsafe fraction ( n=200n{=}200 harmful, 100 benign-but-spicy per condition, 95% CI, four-judge agreement 72.8%). Capability is accuracy with and without the gate.
Figure 2: The refusal direction is shared across architectures. Reading it in one architecture, mapping it into another, and removing it there (Transferred) raises attack success as much as removing the target’s own direction (Native), while an orthogonalized random direction through the same map (Random) does not, in all six transfer directions. A plot of Table 5 (Mamba ={=} Falcon-Mamba, Mamba3 ={=} Falcon3-Mamba).
Selective-scan componentAUROC (surface-controlled)
Δ\Delta (step size)0.95
BB (input selection)0.79
CC (readout selection)0.78
mixer output0.99
Table 2: Where harm lives in the selective scan. Surface-controlled held-out AUROC of a linear harm probe at each selective-scan component (Falcon-Mamba-7B), on the surface-matched XSTest contrast where a surface probe is near chance (0.54). Harm is decodable at the step size Δ\Delta and the mixer output, not at BB or CC .
Figure 3: The write site in the two further architecture classes. Left: RWKV-6 block (token-shift RNN). The write sites are the sub-block outputs, the time-mix output (primary) and the channel-mix output (secondary), while the block boundary is architecturally the residual and is inert. Right: Zamba2 (hybrid). A Mamba2 backbone taps shared attention blocks (two weight instances reused across depth) every kk -th layer, and both branch outputs are write sites.
Source →\to targetMean ownMean transp.Pearson rr [95% CI]Folds >> null (shuffled-category control)
Llama-3.1-8B →\to Falcon-Mamba0.950.950.99 [0.95, 1.00]8/8
Mistral-7B →\to Falcon-Mamba0.950.950.99 [0.97, 1.00]8/8
Table 3: Per-pair category-held-out cross-validation results. Mean own and mean transported probe scores are averaged across the 8 held-out categories. Pearson rr measures how closely they track each other. Full setup and interpretation in Section 4 , plotted in Figure 4 .
Target (site)SourceBaseRemovedRandom
Falcon-Mamba (block out)Llama10.5%44.0%9.0%
Falcon-Mamba (block out)Mistral10.5%45.0%9.5%
RWKV-6 (time-mix out)Llama3.0%54.7%32.3%
Zamba2 (Mamba out)Llama8.9%93.3%22.8%
Zamba2 (attn. out)Llama8.9%16.7%8.9%
Table 4: Removing the rotation-transported refusal direction raises attack success in every non-transformer architecture we test, while a matched random direction through the same map does much less (Llama-Guard-judged; RWKV-6 and Zamba2 pooled over three seeds). The margin is decisive on the two Mamba write sites, narrower on RWKV-6, whose random control alone jailbreaks a third of prompts, and not decisive at Zamba2’s attention site. Per-seed numbers, 95% confidence intervals (CIs), and coherence screens in Appendix A.9 , A.16 , and A.22 .
Figure 4: Category-held-out cross-validation. Each point is one held-out XSTest category, plotting the transported orthogonal probe’s AUROC against the target’s own-probe AUROC. Points follow the y=xy{=}x line as the own probe falls off ceiling, so the transport tracks the target’s harm structure category by category rather than sitting at a saturated ceiling.
Pair / directionBaseNativeTransferredRandom
Llama →\to Mamba10.5%41.5%36.5%9.0%
Mamba →\to Llama2.5%50.5%79.0%8.0%
Mistral →\to Mamba10.5%41.5%36.5%9.5%
Mamba →\to Mistral26.5%50.0%60.0%27.0%
Llama →\to Mamba30.0%42.0%25.5%0.0%
Mamba3 →\to Llama2.5%51.5%55.5%2.5%
Table 5: Cross-architecture transfer, ablation test (plotted in Figure 2 ). Removing the mapped direction (Transferred) raises attack success like removing the target’s own (Native). The Random control does not. Random is mapped through WW like Transferred, then orthogonalized to the target’s refusal axis (cosine ≈ 0{\approx}\,0 ), so it shares the map without inheriting the refusal direction. The effect holds in all six rows: three model pairs (Mamba ={=} Falcon-Mamba, Mamba3 ={=} Falcon3-Mamba), each tested in both directions.
ArchitectureExemplarWrite siteBase →\to gatedΔ\Delta over-ref.
Selective-scan SSMFalcon-Mambablock output98.0% →\to 5.0%+0.0+0.0 pt
TransformerMistral-7Battn. output98.0% →\to 14.0%+0.0+0.0 pt
TransformerLlama-3.1-8Battn. output25.0% →\to 0.0%+0.0+0.0 pt
Token-shift rec.RWKV-6time-mix output47.5% →\to 27.5%+0.0+0.0 pt
Hybrid (attn. site)Zamba2-7Battn. output31.7% →\to 0.0%+0.0+0.0 pt †
Hybrid (Mamba site)Zamba2-7BMamba output31.7% →\to 5.0%+0.0+0.0 pt ‡
Table 6: Write-site prefix-injection defense across four architecture families (base to gated attack success at the coherent gain and 64 new tokens; Wilson 95% CIs for every row in Appendix A.24 (transformers), Appendix A.20 (SSM), Appendix A.16 (RWKV), and Appendix A.21 (hybrid)). The last column is the benign-side cost, the change in over-refusal on benign-but-spicy prompts at the same operating point. Per-row nn : 200 (SSM), 100 (each transformer), 40 (RWKV), 100 (hybrid, shared base). No row adds any over-refusal, though this is a weak test, since the trigger was fitted to reject those prompts. Under benign prompts wearing a jailbreak template every trigger holds except the hybrid’s, †\dagger marking the site where training on wrapped negatives repairs the confound and ‡\ddagger the one where it does not (Appendix A.37 ). The hybrid’s two rows share one base, from a single closed-loop run whose always-on version instead costs 99.0% over-refusal (Appendix A.39 ). A causal-ablation test (Appendix A.22 ) finds the hybrid’s control concentrates at the Mamba site. The attention site’s own causal-ablation evidence is weaker, with overlapping confidence intervals, and is not decisive on its own.
Push siteDirection est. atFalcon-MambaMistral-7B
write sitewrite site6.0% [2.8%, 12.5%]15.0% [9.3%, 23.3%]
residualwrite site6.0% [2.8%, 12.5%]23.0% [15.8%, 32.2%]
residualresidual100.0%48.0%
Table 7: The push location is irrelevant, the direction-fit decides it (matched-absolute-magnitude control, Llama-Guard, Wilson 95% CI, n=100n{=}100 , a model’s three rows share the same absolute push at its coherent gain). Holding the direction fixed leaves the push site within CIs (rows 1–2). Estimating the direction in the accumulated residual instead degenerates it (row 3). RWKV-6 and Zamba2 in Appendix A.35 .
AttackBase ASRGated ASR, Llama-Guard (95% CI)Gated ASR, ShieldGemma
Prefix injection98.0%5.0% [2.7, 9.0]2.0%
Persona roleplay81.5%30.0% [24.1, 36.7]23.0%
Fictional framing87.0%16.5% [12.0, 22.3]10.0%
Table 8: The closed-loop gate on Falcon-Mamba-7B. Attack success rate (ASR) before and after amplifying the refusal direction at the selective-scan block output, decided once per prompt, across three jailbreak families, at 200 harmful prompts per condition with the Wilson 95% confidence interval on the gated rate. The primary ASR is the Llama-Guard-3-8B rate. The reduction holds under an out-of-family judge (ShieldGemma-9B, Gemma family, disjoint from all four targets), and across 4 judges spanning three model families the n-way verdict agreement is 72.8%.
DefensePrefixRoleplayFictionOver-ref.
None98.0%81.5%87.0%12.0%
Residual projection95.0%87.5%94.5%9.0%
CAST (residual loop)100.0%100.0%100.0%12.0%
Perplexity filter98.0%81.5%87.0%19.0%
SmoothLLM62.0%72.0%86.5%29.0%
Open-loop block output (no detector)5.0%30.0%16.5%86.0%
Table 9: Defenses on Falcon-Mamba-7B, attack success (lower is better) across three jailbreak families and over-refusal (plotted in Figure 5 ). CAST runs the same closed loop as our gate but reads its direction from the residual stream, and the open-loop gate runs the same direction with no detector. The direction gives the defense, the loop the selectivity, so the block-output gate reaches the lowest attack success while holding over-refusal at the undefended base rate. CAST’s near-total attack success is not a coherent jailbreak: when its closed loop fires on harm, the residual push breaks generation into repetitive text rather than producing either a refusal or usable harmful content, and the judge still scores that broken text as compliant. Only 10.5% (prefix injection), 7.0% (roleplay), and 11.0% (fictional framing) of its harmful completions pass our coherence screen (Appendix A.25 ), against 9.5% pooled ( n=600n{=}600 ). CAST’s benign completions, where the loop does not fire, remain fully coherent, so the breakdown is specific to the closed-loop residual push on harmful prompts, not a general fault in the model or the generation setup.
Figure 5: Defenses on Falcon-Mamba-7B under prefix injection (selected rows of Table 9 ). Only the block-output gate, which reads the refusal direction at the write site, drives attack success down to a low rate. CAST and projection, which read and steer the direction in the residual, barely move it. The perplexity filter has no effect, and randomized smoothing (SmoothLLM ( Robey et al. 2025 ) ) achieves a partial reduction well short of the gate’s.
Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: (Left) Harm is decodable at the step size Δ\Delta across base Mamba models from 130M to 7B parameters, while a surface probe stays near chance. (Right) Calibrating the gate strength α\alpha on the second SSM (Falcon3-Mamba), block-output gate at its Δ\Delta -ranked layers. Attack success falls monotonically with α\alpha where there is headroom (dotted lines are the undefended rates) and every point passes the coherence screen. With the anchor’s mixer-output layer ranking instead, the anchor’s gain over-steers Falcon3 (Appendix A.20 ).
MethodAttack successOver-refusal (XSTest)Over-refusal (OR-Bench)
Undefended base100%2.0%32.5%
Detect-then-refuse0%2.0%47.5%
Write-site gate ( α=4\alpha{=}4 )0%2.0%46.3%
Appendix
Table 10: Write-site steering against detect-then-refuse at the deployed detector threshold, under a lexical refusal check. Both share the detector and reach the same attack success and over-refusal. The gate has a single coherent gain (4). Higher gains degrade the generation rather than improving the tradeoff. Over-refusal on the harder OR-Bench-hard set is bounded by the base model’s own refusals (32.5%) and the detector’s false-positive rate (21%), which both methods inherit.
ModelPersona–refusal cosine
Falcon-Mamba-7B (selective-scan SSM)-0.30
Falcon3-Mamba-7B (selective-scan SSM)-0.30
RWKV-6 (token-shift recurrent)-0.49
Llama-3.1-8B (transformer, residual)+0.65
Mistral-7B (transformer, residual)+0.74
Appendix
Table 11: The roleplay wrapper’s injected persona direction versus the refusal direction (cosine at each model’s read site: the write site for the SSMs and RWKV, the residual stream for the transformers). It is anti-aligned with refusal in every non-transformer model tested and aligned in the transformers’ residual stream, so amplifying refusal alone cannot remove the persona in the non-transformers, and a second, persona-ablating direction is needed. At the transformers’ attention write site the cosine is near zero (Table 13 ), so the split is between sites as much as architectures.
Intervention on the roleplay attackAttack success
Base (no gate)81.5%
Amplify refusal only38.0%
Suppress persona only79.5%
Both (amplify refusal, suppress persona)17.5%
Appendix
Table 12: Roleplay attack success on Falcon-Mamba-7B under each block-output intervention (single-layer persona-suppression setting). Amplifying refusal alone helps but does not remove the attack, suppressing the persona alone barely moves it, and the two together close the gap. Over-refusal is unchanged from base (0.0% added).
ModelAttackResidualWrite site
Mistral-7Broleplay+0.74-0.11
Mistral-7Bcombo_strong+0.76-0.02
Llama-3.1-8Broleplay+0.52+0.08
Llama-3.1-8Bcombo_strong+0.55+0.17
Appendix
Table 13: Cosine between the refusal direction and the injected-persona direction, at the residual stream versus the attention write site, for two persona attacks and two transformers. The two directions are strongly aligned at the residual and near-orthogonal at the write site, so the persona can be suppressed independently of refusal only at the write site.
Intervention at the write siteLlama-GuardGemma
none (base)97.0%79.0%
amplify refusal only77.0%–
ablate persona only96.0%–
both (refusal and persona ablation)5.0%0.0%
Appendix
Table 14: Compositional defense against a strong persona attack (combo_strong) at the Mistral-7B attention write site. Neither refusal amplification nor persona suppression alone defends, while the two together drive attack success from 97.0% to 5.0% (0.0% under the out-of-family Gemma judge), with coherent refusals. The same combined gate at the residual, where the two directions are aligned, leaves attack success at 100.0%.
SiteNormrelRefusal ASRRandom ASR
Block output (residual)519447.5%47.5%
Time-mix output270.2517.5%82.5%
Channel-mix output540.257.5%15.0%
Appendix
Table 15: RWKV-6 norm-matched steering (native prompting, prefix injection, base attack success 47.5%). Each site is steered at a magnitude relative to its own activation norm, so the perturbation is matched across sites. The block output (the residual) is inert up to rel =4{=}4 , a push several times its norm, while the sub-block write points steer at a fraction of theirs and a matched random direction does not (cleanly for the time-mix output). The residual’s inertness is therefore direction-intrinsic, not a matter of scale. Both sub-blocks write a fresh contribution into the stream before the residual accumulates it, so both are write sites.
ModelDirectionBase ASRrelASRCoherent
Mistral-7Bresidual1.0000.021.0001.000
Mistral-7Bresidual1.0000.051.0001.000
Mistral-7Bresidual1.0000.10.9251.000
Mistral-7Bresidual1.0000.150.6750.875
Mistral-7Bresidual1.0000.21.0000.025
Mistral-7Bresidual1.0000.251.0000.000
Appendix
Table 16: The residual below the grid floor used elsewhere in this paper. Every residual row reported in the main sweeps is at a relative magnitude of 0.250.25 or above, which is the smallest magnitude those grids contain. Sweeping beneath it finds attack-success drops of at least 0.20.2 that stay ≥0.9\geq 0.9 coherent, in bold. The band closes quickly, and above it the model degenerates rather than resisting, which is why a grid starting at 0.250.25 reports the site as unmoved. “Direction” is the site the refusal direction was fitted at, which is the push site unless stated otherwise. n=40n{=}40 per cell, single seed. Coherence is the fraction of completions passing the distinct-bigram screen (Appendix A.25 ).
ModelSiteNormrelBase →\to gated ASRCoherent
Mistral-7BAttention0.791100.0% →\to 85.0%100.0%
Mistral-7BMLP1.330.5100.0% →\to 0.0%100.0%
Llama-3.1-8BAttention0.780.527.5% →\to 0.0%100.0%
Llama-3.1-8BMLP1.690.2527.5% →\to 2.5%100.0%
Appendix
Table 17: The MLP output, a non-mixing site, defends as well as or better than the mixing attention output, at a matched relative gain, on both transformer anchors. The residual (Table 15 gives the analogous RWKV numbers, and the transformer residual figures are in Appendix A.20 ) does not defend anywhere in this range. It defends only below it, over a band too narrow to deploy (Appendix A.17 ). Coherence is the fraction of completions passing the distinct-bigram screen (Appendix A.25 ). Each row reports the most attack-success-reducing relative gain at which output stayed at least 90% coherent.
relASROver-refusalCoherent
0.25100.0%20.0%100.0%
0.592.5%13.3%100.0%
1.077.5%20.0%100.0%
1.2565.0%33.3%100.0%
1.557.5%36.7%100.0%
1.7532.5%46.7%98.6%
Appendix
Table 18: The SSM’s non-mixing skip term DxtDx_{t} , gain sweep on Falcon-Mamba-7B (prefix injection, base ASR 100.0%, base over-refusal 13.3%). Coherence is the fraction passing the distinct-bigram screen (Appendix A.25 ).
relASROver-refusalCoherent
0.2587.5%46.7%100.0%
0.557.5%100.0%100.0%
1.0100.0%0.0%51.4%
2.0100.0%0.0%87.1%
4.0100.0%0.0%0.0%
Appendix
Table 19: The mixer’s raw input (before selection), gain sweep on Falcon-Mamba-7B (prefix injection, base ASR 100.0%, base over-refusal 13.3%).
GainTransported ASRTransported coh.Random ASRRandom coh.
113.3%100.0%13.3%100.0%
211.7%100.0%15.0%100.0%
420.0%100.0%13.3%100.0%
860.0%100.0%5.0%100.0%
1291.7%88.3%25.0%100.0%
16100.0%68.3%85.0%41.7%
Appendix
Table 20: Coherence-gated suppression-gain sweep at Zamba2’s Mamba mixer write site. Both the rotation- transported refusal direction and a matched orthogonalized-random control are suppressed at each gain; coherence is the fraction of outputs passing the distinct-bigram/repetition screen. The chosen operating point (bold) is the largest gain at which the transported direction stays at or above 80.0% coherence. Base attack success (gain 0) is 11.7%. Gains 8, 12 are rescored with Llama-Guard-3-8B (matching the operating-point number quoted in the main text). The remaining gains are the original lexical-heuristic score, not rejudged, and are shown only to establish the coherence frontier’s shape, not as headline attack-success numbers.
AttackBase ASRGated ASROver-ref. (ours)Over-ref. (DeTAM)
Llama-3.1-8B
Prefix injection28.0%2.0%10.0%87.0%
Persona roleplay0.5%0.0%10.0%87.0%
Fictional framing19.5%1.0%10.0%87.0%
Mistral-7B
Prefix injection96.0%15.0%11.0%69.0%
Appendix
Table 21: DeTAM comparison at the coherent gain ( α=2\alpha{=}2 , n=200n{=}200 harmful, 100100 benign per attack, an independent draw from the gain-sweep evaluation set in Table 22 , so base rates differ slightly by sampling variance between the two tables). The closed-loop gate and the open-loop DeTAM-style edit reach the same gated attack success on each attack (the detector fires on all harmful prompts), so the difference is over-refusal: the closed-loop trigger holds it at the base rate, the open-loop edit raises it sharply. On Mistral, single-direction amplification is not reliable against persona attacks: roleplay attack success rises from 63.5% to 78.0% (it backfires), and fiction falls only partially, from 96.5% to 81.5%. This is the same attack-dependence as the SSM, which the two-lever gate that also suppresses the persona defends (Appendices A.14 and A.15 ).
α\alphaGated ASRHarm coh.Benign coh.DeTAM benign
Llama-3.1-8B , base ASR 26.0%
0.57.0%100.0%100.0%100.0%
1.02.0%100.0%100.0%100.0%
1.51.0%100.0%100.0%100.0%
2.01.0%100.0%100.0%100.0%
3.03.0%80.0%100.0%100.0%
Appendix
Table 22: Attention-site gain sweep on two transformers (prefix injection). Gated ASR (Llama-Guard-3) is shared by the closed-loop gate and the open-loop DeTAM-style edit, since the detector fires on all harmful prompts. “Harm coh.” and “Benign coh.” are the coherent fractions under our gate. “DeTAM benign” is the open-loop edit’s benign coherence, which collapses at high gain.
Gain α\alphaPrefix injectionPersona roleplay
base (no gate)98.0% / 99.5%82.5% / 100.0%
199.0% / 99.0%89.5% / 99.5%
298.0% / 99.5%91.5% / 98.5%
4 (deployed at 64)17.5% / 10.5%55.0% / 23.5%
648.5% / 0.0%60.5% / 0.0%
Appendix
Table 23: The anchor’s gate at 256 new tokens (attack success / fraction coherent, n=200n{=}200 per cell, Llama-Guard judge). The gain calibrated at 64 tokens ( α=4\alpha{=}4 ) neither stays coherent nor fully defends at 256, and no swept gain does both.
Action on the refusal directionBaseAfter
Amplify at block output (ours), under attack98.0%5.0%
Project out of residual, under attack98.0%95.0%
Remove direction, plain harm10.5%41.5%
Appendix
Table 24: The action matters: the same refusal direction (all on Falcon-Mamba-7B-Instruct, n=n{=} 200). Amplifying it at the block output defends, projecting it out of the residual does little, and removing it from a model that was refusing jailbreaks the model. The first two rows use prefix-injected prompts (high base attack success, room to fall). The third uses plain harmful prompts (low base, room to rise).
kk labelsTransportFrom scratchPaired Δ\DeltaWins
40.780.75+0.032 [+0.008, +0.057]64%
80.830.79+0.042 [+0.011, +0.073]68%
160.870.83+0.038 [+0.019, +0.058]78%
320.900.85+0.040 [+0.006, +0.073]72%
640.920.87+0.046 [+0.013, +0.080]78%
Appendix
Table 25: Data-scarce payoff of the portable direction over a target direction fit from scratch on the same kk target labels, generalizing to an unseen harm category (Llama-3.1-8B to Falcon-Mamba, AUROC on ground-truth labels). Δ\Delta is the paired difference and its 95% tt interval over the 8 held-out categories, the independent unit. Wins is the fraction of the 120 individual draws favoring transport, which is why it can sit near half while the interval still excludes zero: the mean gain is carried by larger wins, not by winning more often. Positive Δ\Delta favors transport. The margin does not close over this range of budgets.
ModelWrite siteGate layers (decision)α\alpha final (tried)Transfer: read / write layersNotes
Falcon-Mamba-7B-Inst.mixer output{24,28,30,34,35,36,37,44} (35)4 (1,2,4,6,8; capability at 4,8,16)44 / {22,24,28,31,38,40,42,43}anchor; Δ\Delta -ranked set for transfer targets
Falcon3-Mamba-7B-Inst.mixer outputΔ\Delta -ranked {24,27,28,31,42,45,46,47} (42)4 (0.5,1,2,4; Fig. 6 )47 / same setblock-vs-residual control on this set at α=4\alpha{=}4 ; the output-ranked set {14,15,17,18,21,22,23,24} over-steers at α=4\alpha{=}4 (reported at α=1\alpha{=}1 )
Llama-3.1-8B-Inst.attention output{10,12,…,24} (18)2 (0.5,1,1.5,2,3,4)14 / {12,14,16,18,20}hardened trigger at the deployed point
Mistral-7B-Inst.-v0.2attention output{10,12,…,24} (18)2 (0.5,1,1.5,2,3,4)14 / {12,14,16,18,20}persona fix: layer 18, α=6\alpha{=}6 , β=6\beta{=}6
RWKV-6-World-7Btime-mix output{14,16,…,24} (20)16 (0,4,8,16,24,32,48; coh. ≥0.8\geq 0.8 )20 (target of Llama L14)defense run gating the block hook at α=6\alpha{=}6 is a separate operating point; the block boundary (residual) is inert
Zamba2-7B-Inst.attention out. / Mamba out.{36,42,48,54,60} (48) / {20,24,28,32} (28)8 / 16 (persona grid α∈{2,4,6,8,12}\alpha\in\{2,4,6,8,12\} , β∈{0,1,2}\beta\in\{0,1,2\} )–hybrid, both write sites gated
Appendix
Table 26: Final hyperparameters per model, with the values tried where a sweep was run. Gate layers are the top-8 by held-out probe AUROC on the SSMs (ranked by mixer-output AUROC on Falcon-Mamba, by Δ\Delta AUROC on Falcon3-Mamba, Section 2.5 ); the other architectures use fixed mid-network bands. The decision layer is the middle of the listed set and carries the per-prompt detector. Every gate uses θ=0.5\theta{=}0.5 on the probe’s harm probability, the mean refusal direction (mean harmful minus mean benign activation) at each gate layer, greedy decoding with 64 new tokens, and the gain α\alpha is the largest value that passes the coherence screen (Appendix A.25 ). Transfer rows give the layer the direction is read at in the source and the layers it is written at in the target.
More in this topic

Refusal Mechanisms in Language Models

See all 39 papers

Explore similar work

CardsList
Oct 9, 2026

Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs

Safety-aligned large language models (LLMs) remain vulnerable to jailbreaks and can over-refuse benign requests. Understanding these failures requires distinguishing harmfulness recognition from the computation that writes refusal. We reframe this as a pathway question and make it measurable on residual-stream updates in decoder-only Transformers, separating the harmfulness information a module update exposes from the change it produces in readable refusal evidence. Across 18 Base-Instruct pairs spanning Qwen, Llama, and Gemma, harmfulness is decoded more readily from attention updates, particularly in early layers, whereas MLP updates contribute more strongly to harm-selective refusal writing in aligned models. Post-training strengthens MLP refusal writing in all 18 pairs without systematically improving harmfulness decodability, while positive MLP writing becomes more concentrated in deeper layers after post-training. Across six reasoning-enabled models, harmfulness remains decodable during reasoning while strong refusal writing concentrates at answer onset, a pattern that persists with reasoning disabled. Causal interventions further support this pathway: resetting downstream MLP updates to their clean values substantially reduces the refusal shift caused by upstream attention interventions. Building on these findings, we introduce Sentinel, which combines an attention-derived perception score P with an MLP-derived decision score D. Their joint profiles characterize jailbreaks and over-refusals, and conditional routing with both signals achieves a lower aggregate error rate than single-signal routing and evaluated fixed reminders across eight models.

Zixuan Weng, Haoqi He, Weifei Jin +2

Georgia Institute of Technology · Independent Researcher · Duke University +1

Refusal Behavior in Language ModelsLanguage Model Safety EvaluationLLM InterpretabilityAdversarial Attacks on LLMs
AskPDF
Mar 4, 2026

Efficient Refusal Ablation in LLM through Optimal Transport

Safety-aligned language models refuse harmful requests through learned refusal behaviors encoded in their internal representations. Recent activation-based jailbreaking methods circumvent these safety mechanisms by applying orthogonal projections to remove refusal directions, but these approaches treat refusal as a one-dimensional phenomenon and ignore the rich distributional structure of model activations. We introduce a principled framework based on optimal transport theory that transforms the entire distribution of harmful activations to match harmless ones. By combining PCA with closed-form Gaussian optimal transport, we achieve efficient computation in high-dimensional representation spaces while preserving essential geometric structure. Across six models (Llama-2, Llama-3.1, Qwen-2.5; 7B-32B parameters), our method achieves up to 11% higher attack success rates than state-of-the-art baselines while maintaining comparable perplexity, demonstrating superior preservation of model capabilities. Critically, we discover that layer-selective intervention (applying optimal transport to 1-2 carefully chosen layers at approximately 40-60% network depth) substantially outperforms full-network interventions, revealing that refusal mechanisms may be localized rather than distributed. Our analysis provides new insights into the geometric structure of safety representations and suggests that current alignment methods may be vulnerable to distributional attacks beyond simple direction removal.

Geraldin Nanfack, Eugene Belilovsky, Elvis Dohmatob

Concordia University, Montreal, Canada · Mila – Quebec AI Institute, Montreal, Canada

Adversarial Attacks on LLMsJailbreak AttacksLLM SafetyRefusal Behavior in Language Models
AskPDF
Aug 26, 2026

Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks

Refusal training protects AI models from jailbreaks by training models to decline unsafe queries, reducing the risk of misuse. Recent work finds that refusal behavior in aligned language models can be mediated by a single activation direction or a low-dimensional refusal subspace shared across harmful prompts: ablating those directions suppresses refusals while largely preserves other model capabilities. Yet it remains unclear why safety-critical features in a wide range of models emerge in a concentrated, low-dimensional structure. In a case study of OLMo-2-0425-1B-Instruct we find that the refusal geometry reflects refusal training: activation updates resulting from refusal-completion first-token losses explain the resulting refusal direction and refusal subspace. We study refusal directions through the training dynamics across refusal datasets and reveal that their brittleness is associated with repetitive refusal starts, which in turn is linked to concentration of gradients and refusal features in a low-dimensional subspace. Across frozen-model analyses and controlled synthetic fine-tuning, we find evidence of a hardening lever: diverse refusal starts can raise stable ranks of gradients and activation changes, making refusals harder to remove with a vector ablation attack.

Andrey Labunets

UC San Diego

Adversarial Attacks on LLMsAdversarial Robustness of Language ModelsRefusal Behavior in Language ModelsRefusal Mechanisms in Language Models
AskPDF