cs.LGSep 4, 2026
SaveLocating and Steering Refusal Beyond Attention
Organizations: Department of Computer Science and Engineering, Indian Institute of Technology Madras
Abstract
Where inside a language model does refusal live, and does that place change when the architecture does? In a transformer, refusal is governed by a single direction in the residual stream, a finding that safety and interpretability tooling now depend on. State-space models (SSMs) route information through a recurrent update instead of attention, sharing no token-mixing mechanism with a transformer. Does the same safety representation survive this shift, or must it be rediscovered per architecture? It survives. A single rigid rotation, which can only reorient a space and not reshape it, aligns one model's representation space with another's, so the two genuinely share the representation. A harm probe trained on a transformer then flags an SSM's harmful inputs, and removing the aligned direction makes a model answer attacks it would otherwise refuse, while a random direction of the same size does far less. What is architecture-specific is not where the direction is steered but where it must be read. Each layer computes a fresh output that is then added into the residual stream, and harm is cleanly readable at this output, the write site, before the addition. A control that holds the intervention's strength fixed shows that what matters is where the direction is estimated, not where it is applied. Applied through a detector-triggered gate, this direction lowers jailbreak success in all four architecture families we test (SSM, transformer, recurrent, hybrid), and on the SSM it holds against an attacker that tunes its prompt against the defense. The gate only matches a trivial rule that returns a fixed refusal whenever the same detector fires, so what transfers across architectures is the direction itself, not defense strength. Safety tooling built on refusal therefore ports to a new architecture by re-estimating the direction at that architecture's write site, not by rebuilding it.
Figures & tables
Figure 1: The write site in the two reference architectures. Left: transformer block (Llama-3.1, Mistral). The write site is the attention output projection , the fresh contribution attention writes into the residual stream. Right: Falcon-Mamba block (selective-scan SSM). The write site is the mixer’s out-projection, the block output the -gated scan writes before the residual accumulates it, and inside the scan is decodable but a readout only. In both, the probe and the refusal direction are estimated at the red site; a direction estimated in the accumulated residual (gray) degenerates.
| Models | Corpora | Attacks | Judges & metrics | Baselines |
| Falcon-Mamba-7B (SSM anchor) Falcon3-Mamba-7B (2nd SSM) Llama-3.1-8B, Mistral-7B, Qwen2.5-7B (transformers) RWKV-6-World-7B (recurrent) Zamba2-7B (hybrid) | AdvBench (harm) Alpaca (benign) XSTest (surface control, over-refusal) | Prefix injection (fluent) Persona roleplay (DAN, fluent) Fictional framing (fluent) Dilution (adaptive) Optimized adversarial suffix (GCG; on the SSM both a gradient-free variant and GCG through the differentiable scan) | Llama-Guard-3-8B (primary) +Llama-3.1-8B-Instruct, Mistral-7B-Instruct, ShieldGemma-9B MMLU, TruthfulQA (capability) Probe AUROC (decodability) | Residual-subspace projection CAST Perplexity filter SmoothLLM Cross-architecture linear-map transfer |
Table 1: Experimental setup at a glance. The minimal-pair corpus is matched to XSTest in template and length. Over-refusal is measured on XSTest’s benign-but-spicy prompts (safe requests worded to sound harmful). Attack success is the Llama-Guard-3-8B-judged unsafe fraction ( harmful, 100 benign-but-spicy per condition, 95% CI, four-judge agreement 72.8%). Capability is accuracy with and without the gate.
Figure 2: The refusal direction is shared across architectures. Reading it in one architecture, mapping it into another, and removing it there (Transferred) raises attack success as much as removing the target’s own direction (Native), while an orthogonalized random direction through the same map (Random) does not, in all six transfer directions. A plot of Table 5 (Mamba Falcon-Mamba, Mamba3 Falcon3-Mamba).
| Selective-scan component | AUROC (surface-controlled) |
|---|---|
| (step size) | 0.95 |
| (input selection) | 0.79 |
| (readout selection) | 0.78 |
| mixer output | 0.99 |
Table 2: Where harm lives in the selective scan. Surface-controlled held-out AUROC of a linear harm probe at each selective-scan component (Falcon-Mamba-7B), on the surface-matched XSTest contrast where a surface probe is near chance (0.54). Harm is decodable at the step size and the mixer output, not at or .
Figure 3: The write site in the two further architecture classes. Left: RWKV-6 block (token-shift RNN). The write sites are the sub-block outputs, the time-mix output (primary) and the channel-mix output (secondary), while the block boundary is architecturally the residual and is inert. Right: Zamba2 (hybrid). A Mamba2 backbone taps shared attention blocks (two weight instances reused across depth) every -th layer, and both branch outputs are write sites.
| Source target | Mean own | Mean transp. | Pearson [95% CI] | Folds null (shuffled-category control) |
|---|---|---|---|---|
| Llama-3.1-8B Falcon-Mamba | 0.95 | 0.95 | 0.99 [0.95, 1.00] | 8/8 |
| Mistral-7B Falcon-Mamba | 0.95 | 0.95 | 0.99 [0.97, 1.00] | 8/8 |
Table 3: Per-pair category-held-out cross-validation results. Mean own and mean transported probe scores are averaged across the 8 held-out categories. Pearson measures how closely they track each other. Full setup and interpretation in Section 4 , plotted in Figure 4 .
| Target (site) | Source | Base | Removed | Random |
|---|---|---|---|---|
| Falcon-Mamba (block out) | Llama | 10.5% | 44.0% | 9.0% |
| Falcon-Mamba (block out) | Mistral | 10.5% | 45.0% | 9.5% |
| RWKV-6 (time-mix out) | Llama | 3.0% | 54.7% | 32.3% |
| Zamba2 (Mamba out) | Llama | 8.9% | 93.3% | 22.8% |
| Zamba2 (attn. out) | Llama | 8.9% | 16.7% | 8.9% |
Table 4: Removing the rotation-transported refusal direction raises attack success in every non-transformer architecture we test, while a matched random direction through the same map does much less (Llama-Guard-judged; RWKV-6 and Zamba2 pooled over three seeds). The margin is decisive on the two Mamba write sites, narrower on RWKV-6, whose random control alone jailbreaks a third of prompts, and not decisive at Zamba2’s attention site. Per-seed numbers, 95% confidence intervals (CIs), and coherence screens in Appendix A.9 , A.16 , and A.22 .
Figure 4: Category-held-out cross-validation. Each point is one held-out XSTest category, plotting the transported orthogonal probe’s AUROC against the target’s own-probe AUROC. Points follow the line as the own probe falls off ceiling, so the transport tracks the target’s harm structure category by category rather than sitting at a saturated ceiling.
| Pair / direction | Base | Native | Transferred | Random |
|---|---|---|---|---|
| Llama Mamba | 10.5% | 41.5% | 36.5% | 9.0% |
| Mamba Llama | 2.5% | 50.5% | 79.0% | 8.0% |
| Mistral Mamba | 10.5% | 41.5% | 36.5% | 9.5% |
| Mamba Mistral | 26.5% | 50.0% | 60.0% | 27.0% |
| Llama Mamba3 | 0.0% | 42.0% | 25.5% | 0.0% |
| Mamba3 Llama | 2.5% | 51.5% | 55.5% | 2.5% |
Table 5: Cross-architecture transfer, ablation test (plotted in Figure 2 ). Removing the mapped direction (Transferred) raises attack success like removing the target’s own (Native). The Random control does not. Random is mapped through like Transferred, then orthogonalized to the target’s refusal axis (cosine ), so it shares the map without inheriting the refusal direction. The effect holds in all six rows: three model pairs (Mamba Falcon-Mamba, Mamba3 Falcon3-Mamba), each tested in both directions.
| Architecture | Exemplar | Write site | Base gated | over-ref. |
|---|---|---|---|---|
| Selective-scan SSM | Falcon-Mamba | block output | 98.0% 5.0% | pt |
| Transformer | Mistral-7B | attn. output | 98.0% 14.0% | pt |
| Transformer | Llama-3.1-8B | attn. output | 25.0% 0.0% | pt |
| Token-shift rec. | RWKV-6 | time-mix output | 47.5% 27.5% | pt |
| Hybrid (attn. site) | Zamba2-7B | attn. output | 31.7% 0.0% | pt † |
| Hybrid (Mamba site) | Zamba2-7B | Mamba output | 31.7% 5.0% | pt ‡ |
Table 6: Write-site prefix-injection defense across four architecture families (base to gated attack success at the coherent gain and 64 new tokens; Wilson 95% CIs for every row in Appendix A.24 (transformers), Appendix A.20 (SSM), Appendix A.16 (RWKV), and Appendix A.21 (hybrid)). The last column is the benign-side cost, the change in over-refusal on benign-but-spicy prompts at the same operating point. Per-row : 200 (SSM), 100 (each transformer), 40 (RWKV), 100 (hybrid, shared base). No row adds any over-refusal, though this is a weak test, since the trigger was fitted to reject those prompts. Under benign prompts wearing a jailbreak template every trigger holds except the hybrid’s, marking the site where training on wrapped negatives repairs the confound and the one where it does not (Appendix A.37 ). The hybrid’s two rows share one base, from a single closed-loop run whose always-on version instead costs 99.0% over-refusal (Appendix A.39 ). A causal-ablation test (Appendix A.22 ) finds the hybrid’s control concentrates at the Mamba site. The attention site’s own causal-ablation evidence is weaker, with overlapping confidence intervals, and is not decisive on its own.
| Push site | Direction est. at | Falcon-Mamba | Mistral-7B |
|---|---|---|---|
| write site | write site | 6.0% [2.8%, 12.5%] | 15.0% [9.3%, 23.3%] |
| residual | write site | 6.0% [2.8%, 12.5%] | 23.0% [15.8%, 32.2%] |
| residual | residual | 100.0% | 48.0% |
Table 7: The push location is irrelevant, the direction-fit decides it (matched-absolute-magnitude control, Llama-Guard, Wilson 95% CI, , a model’s three rows share the same absolute push at its coherent gain). Holding the direction fixed leaves the push site within CIs (rows 1–2). Estimating the direction in the accumulated residual instead degenerates it (row 3). RWKV-6 and Zamba2 in Appendix A.35 .
| Attack | Base ASR | Gated ASR, Llama-Guard (95% CI) | Gated ASR, ShieldGemma |
|---|---|---|---|
| Prefix injection | 98.0% | 5.0% [2.7, 9.0] | 2.0% |
| Persona roleplay | 81.5% | 30.0% [24.1, 36.7] | 23.0% |
| Fictional framing | 87.0% | 16.5% [12.0, 22.3] | 10.0% |
Table 8: The closed-loop gate on Falcon-Mamba-7B. Attack success rate (ASR) before and after amplifying the refusal direction at the selective-scan block output, decided once per prompt, across three jailbreak families, at 200 harmful prompts per condition with the Wilson 95% confidence interval on the gated rate. The primary ASR is the Llama-Guard-3-8B rate. The reduction holds under an out-of-family judge (ShieldGemma-9B, Gemma family, disjoint from all four targets), and across 4 judges spanning three model families the n-way verdict agreement is 72.8%.
| Defense | Prefix | Roleplay | Fiction | Over-ref. |
|---|---|---|---|---|
| None | 98.0% | 81.5% | 87.0% | 12.0% |
| Residual projection | 95.0% | 87.5% | 94.5% | 9.0% |
| CAST (residual loop) | 100.0% | 100.0% | 100.0% | 12.0% |
| Perplexity filter | 98.0% | 81.5% | 87.0% | 19.0% |
| SmoothLLM | 62.0% | 72.0% | 86.5% | 29.0% |
| Open-loop block output (no detector) | 5.0% | 30.0% | 16.5% | 86.0% |
Table 9: Defenses on Falcon-Mamba-7B, attack success (lower is better) across three jailbreak families and over-refusal (plotted in Figure 5 ). CAST runs the same closed loop as our gate but reads its direction from the residual stream, and the open-loop gate runs the same direction with no detector. The direction gives the defense, the loop the selectivity, so the block-output gate reaches the lowest attack success while holding over-refusal at the undefended base rate. CAST’s near-total attack success is not a coherent jailbreak: when its closed loop fires on harm, the residual push breaks generation into repetitive text rather than producing either a refusal or usable harmful content, and the judge still scores that broken text as compliant. Only 10.5% (prefix injection), 7.0% (roleplay), and 11.0% (fictional framing) of its harmful completions pass our coherence screen (Appendix A.25 ), against 9.5% pooled ( ). CAST’s benign completions, where the loop does not fire, remain fully coherent, so the breakdown is specific to the closed-loop residual push on harmful prompts, not a general fault in the model or the generation setup.
Figure 5: Defenses on Falcon-Mamba-7B under prefix injection (selected rows of Table 9 ). Only the block-output gate, which reads the refusal direction at the write site, drives attack success down to a low rate. CAST and projection, which read and steer the direction in the residual, barely move it. The perplexity filter has no effect, and randomized smoothing (SmoothLLM ( Robey et al. 2025 ) ) achieves a partial reduction well short of the gate’s.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: (Left) Harm is decodable at the step size across base Mamba models from 130M to 7B parameters, while a surface probe stays near chance. (Right) Calibrating the gate strength on the second SSM (Falcon3-Mamba), block-output gate at its -ranked layers. Attack success falls monotonically with where there is headroom (dotted lines are the undefended rates) and every point passes the coherence screen. With the anchor’s mixer-output layer ranking instead, the anchor’s gain over-steers Falcon3 (Appendix A.20 ).
| Method | Attack success | Over-refusal (XSTest) | Over-refusal (OR-Bench) |
|---|---|---|---|
| Undefended base | 100% | 2.0% | 32.5% |
| Detect-then-refuse | 0% | 2.0% | 47.5% |
| Write-site gate ( ) | 0% | 2.0% | 46.3% |
Table 10: Write-site steering against detect-then-refuse at the deployed detector threshold, under a lexical refusal check. Both share the detector and reach the same attack success and over-refusal. The gate has a single coherent gain (4). Higher gains degrade the generation rather than improving the tradeoff. Over-refusal on the harder OR-Bench-hard set is bounded by the base model’s own refusals (32.5%) and the detector’s false-positive rate (21%), which both methods inherit.
| Model | Persona–refusal cosine |
|---|---|
| Falcon-Mamba-7B (selective-scan SSM) | -0.30 |
| Falcon3-Mamba-7B (selective-scan SSM) | -0.30 |
| RWKV-6 (token-shift recurrent) | -0.49 |
| Llama-3.1-8B (transformer, residual) | +0.65 |
| Mistral-7B (transformer, residual) | +0.74 |
Table 11: The roleplay wrapper’s injected persona direction versus the refusal direction (cosine at each model’s read site: the write site for the SSMs and RWKV, the residual stream for the transformers). It is anti-aligned with refusal in every non-transformer model tested and aligned in the transformers’ residual stream, so amplifying refusal alone cannot remove the persona in the non-transformers, and a second, persona-ablating direction is needed. At the transformers’ attention write site the cosine is near zero (Table 13 ), so the split is between sites as much as architectures.
| Intervention on the roleplay attack | Attack success |
|---|---|
| Base (no gate) | 81.5% |
| Amplify refusal only | 38.0% |
| Suppress persona only | 79.5% |
| Both (amplify refusal, suppress persona) | 17.5% |
Table 12: Roleplay attack success on Falcon-Mamba-7B under each block-output intervention (single-layer persona-suppression setting). Amplifying refusal alone helps but does not remove the attack, suppressing the persona alone barely moves it, and the two together close the gap. Over-refusal is unchanged from base (0.0% added).
| Model | Attack | Residual | Write site |
|---|---|---|---|
| Mistral-7B | roleplay | +0.74 | -0.11 |
| Mistral-7B | combo_strong | +0.76 | -0.02 |
| Llama-3.1-8B | roleplay | +0.52 | +0.08 |
| Llama-3.1-8B | combo_strong | +0.55 | +0.17 |
Table 13: Cosine between the refusal direction and the injected-persona direction, at the residual stream versus the attention write site, for two persona attacks and two transformers. The two directions are strongly aligned at the residual and near-orthogonal at the write site, so the persona can be suppressed independently of refusal only at the write site.
| Intervention at the write site | Llama-Guard | Gemma |
|---|---|---|
| none (base) | 97.0% | 79.0% |
| amplify refusal only | 77.0% | – |
| ablate persona only | 96.0% | – |
| both (refusal and persona ablation) | 5.0% | 0.0% |
Table 14: Compositional defense against a strong persona attack (combo_strong) at the Mistral-7B attention write site. Neither refusal amplification nor persona suppression alone defends, while the two together drive attack success from 97.0% to 5.0% (0.0% under the out-of-family Gemma judge), with coherent refusals. The same combined gate at the residual, where the two directions are aligned, leaves attack success at 100.0%.
| Site | Norm | rel | Refusal ASR | Random ASR |
|---|---|---|---|---|
| Block output (residual) | 519 | 4 | 47.5% | 47.5% |
| Time-mix output | 27 | 0.25 | 17.5% | 82.5% |
| Channel-mix output | 54 | 0.25 | 7.5% | 15.0% |
Table 15: RWKV-6 norm-matched steering (native prompting, prefix injection, base attack success 47.5%). Each site is steered at a magnitude relative to its own activation norm, so the perturbation is matched across sites. The block output (the residual) is inert up to rel , a push several times its norm, while the sub-block write points steer at a fraction of theirs and a matched random direction does not (cleanly for the time-mix output). The residual’s inertness is therefore direction-intrinsic, not a matter of scale. Both sub-blocks write a fresh contribution into the stream before the residual accumulates it, so both are write sites.
| Model | Direction | Base ASR | rel | ASR | Coherent |
|---|---|---|---|---|---|
| Mistral-7B | residual | 1.000 | 0.02 | 1.000 | 1.000 |
| Mistral-7B | residual | 1.000 | 0.05 | 1.000 | 1.000 |
| Mistral-7B | residual | 1.000 | 0.1 | 0.925 | 1.000 |
| Mistral-7B | residual | 1.000 | 0.15 | 0.675 | 0.875 |
| Mistral-7B | residual | 1.000 | 0.2 | 1.000 | 0.025 |
| Mistral-7B | residual | 1.000 | 0.25 | 1.000 | 0.000 |
Table 16: The residual below the grid floor used elsewhere in this paper. Every residual row reported in the main sweeps is at a relative magnitude of or above, which is the smallest magnitude those grids contain. Sweeping beneath it finds attack-success drops of at least that stay coherent, in bold. The band closes quickly, and above it the model degenerates rather than resisting, which is why a grid starting at reports the site as unmoved. “Direction” is the site the refusal direction was fitted at, which is the push site unless stated otherwise. per cell, single seed. Coherence is the fraction of completions passing the distinct-bigram screen (Appendix A.25 ).
| Model | Site | Norm | rel | Base gated ASR | Coherent |
|---|---|---|---|---|---|
| Mistral-7B | Attention | 0.79 | 1 | 100.0% 85.0% | 100.0% |
| Mistral-7B | MLP | 1.33 | 0.5 | 100.0% 0.0% | 100.0% |
| Llama-3.1-8B | Attention | 0.78 | 0.5 | 27.5% 0.0% | 100.0% |
| Llama-3.1-8B | MLP | 1.69 | 0.25 | 27.5% 2.5% | 100.0% |
Table 17: The MLP output, a non-mixing site, defends as well as or better than the mixing attention output, at a matched relative gain, on both transformer anchors. The residual (Table 15 gives the analogous RWKV numbers, and the transformer residual figures are in Appendix A.20 ) does not defend anywhere in this range. It defends only below it, over a band too narrow to deploy (Appendix A.17 ). Coherence is the fraction of completions passing the distinct-bigram screen (Appendix A.25 ). Each row reports the most attack-success-reducing relative gain at which output stayed at least 90% coherent.
| rel | ASR | Over-refusal | Coherent |
|---|---|---|---|
| 0.25 | 100.0% | 20.0% | 100.0% |
| 0.5 | 92.5% | 13.3% | 100.0% |
| 1.0 | 77.5% | 20.0% | 100.0% |
| 1.25 | 65.0% | 33.3% | 100.0% |
| 1.5 | 57.5% | 36.7% | 100.0% |
| 1.75 | 32.5% | 46.7% | 98.6% |
Table 18: The SSM’s non-mixing skip term , gain sweep on Falcon-Mamba-7B (prefix injection, base ASR 100.0%, base over-refusal 13.3%). Coherence is the fraction passing the distinct-bigram screen (Appendix A.25 ).
| rel | ASR | Over-refusal | Coherent |
|---|---|---|---|
| 0.25 | 87.5% | 46.7% | 100.0% |
| 0.5 | 57.5% | 100.0% | 100.0% |
| 1.0 | 100.0% | 0.0% | 51.4% |
| 2.0 | 100.0% | 0.0% | 87.1% |
| 4.0 | 100.0% | 0.0% | 0.0% |
Table 19: The mixer’s raw input (before selection), gain sweep on Falcon-Mamba-7B (prefix injection, base ASR 100.0%, base over-refusal 13.3%).
| Gain | Transported ASR | Transported coh. | Random ASR | Random coh. |
|---|---|---|---|---|
| 1 | 13.3% | 100.0% | 13.3% | 100.0% |
| 2 | 11.7% | 100.0% | 15.0% | 100.0% |
| 4 | 20.0% | 100.0% | 13.3% | 100.0% |
| 8 | 60.0% | 100.0% | 5.0% | 100.0% |
| 12 | 91.7% | 88.3% | 25.0% | 100.0% |
| 16 | 100.0% | 68.3% | 85.0% | 41.7% |
Table 20: Coherence-gated suppression-gain sweep at Zamba2’s Mamba mixer write site. Both the rotation- transported refusal direction and a matched orthogonalized-random control are suppressed at each gain; coherence is the fraction of outputs passing the distinct-bigram/repetition screen. The chosen operating point (bold) is the largest gain at which the transported direction stays at or above 80.0% coherence. Base attack success (gain 0) is 11.7%. Gains 8, 12 are rescored with Llama-Guard-3-8B (matching the operating-point number quoted in the main text). The remaining gains are the original lexical-heuristic score, not rejudged, and are shown only to establish the coherence frontier’s shape, not as headline attack-success numbers.
| Attack | Base ASR | Gated ASR | Over-ref. (ours) | Over-ref. (DeTAM) |
|---|---|---|---|---|
| Llama-3.1-8B | ||||
| Prefix injection | 28.0% | 2.0% | 10.0% | 87.0% |
| Persona roleplay | 0.5% | 0.0% | 10.0% | 87.0% |
| Fictional framing | 19.5% | 1.0% | 10.0% | 87.0% |
| Mistral-7B | ||||
| Prefix injection | 96.0% | 15.0% | 11.0% | 69.0% |
Table 21: DeTAM comparison at the coherent gain ( , harmful, benign per attack, an independent draw from the gain-sweep evaluation set in Table 22 , so base rates differ slightly by sampling variance between the two tables). The closed-loop gate and the open-loop DeTAM-style edit reach the same gated attack success on each attack (the detector fires on all harmful prompts), so the difference is over-refusal: the closed-loop trigger holds it at the base rate, the open-loop edit raises it sharply. On Mistral, single-direction amplification is not reliable against persona attacks: roleplay attack success rises from 63.5% to 78.0% (it backfires), and fiction falls only partially, from 96.5% to 81.5%. This is the same attack-dependence as the SSM, which the two-lever gate that also suppresses the persona defends (Appendices A.14 and A.15 ).
| Gated ASR | Harm coh. | Benign coh. | DeTAM benign | |
| Llama-3.1-8B , base ASR 26.0% | ||||
| 0.5 | 7.0% | 100.0% | 100.0% | 100.0% |
| 1.0 | 2.0% | 100.0% | 100.0% | 100.0% |
| 1.5 | 1.0% | 100.0% | 100.0% | 100.0% |
| 2.0 | 1.0% | 100.0% | 100.0% | 100.0% |
| 3.0 | 3.0% | 80.0% | 100.0% | 100.0% |
Table 22: Attention-site gain sweep on two transformers (prefix injection). Gated ASR (Llama-Guard-3) is shared by the closed-loop gate and the open-loop DeTAM-style edit, since the detector fires on all harmful prompts. “Harm coh.” and “Benign coh.” are the coherent fractions under our gate. “DeTAM benign” is the open-loop edit’s benign coherence, which collapses at high gain.
| Gain | Prefix injection | Persona roleplay |
|---|---|---|
| base (no gate) | 98.0% / 99.5% | 82.5% / 100.0% |
| 1 | 99.0% / 99.0% | 89.5% / 99.5% |
| 2 | 98.0% / 99.5% | 91.5% / 98.5% |
| 4 (deployed at 64) | 17.5% / 10.5% | 55.0% / 23.5% |
| 6 | 48.5% / 0.0% | 60.5% / 0.0% |
Table 23: The anchor’s gate at 256 new tokens (attack success / fraction coherent, per cell, Llama-Guard judge). The gain calibrated at 64 tokens ( ) neither stays coherent nor fully defends at 256, and no swept gain does both.
| Action on the refusal direction | Base | After |
|---|---|---|
| Amplify at block output (ours), under attack | 98.0% | 5.0% |
| Project out of residual, under attack | 98.0% | 95.0% |
| Remove direction, plain harm | 10.5% | 41.5% |
Table 24: The action matters: the same refusal direction (all on Falcon-Mamba-7B-Instruct, 200). Amplifying it at the block output defends, projecting it out of the residual does little, and removing it from a model that was refusing jailbreaks the model. The first two rows use prefix-injected prompts (high base attack success, room to fall). The third uses plain harmful prompts (low base, room to rise).
| labels | Transport | From scratch | Paired | Wins |
|---|---|---|---|---|
| 4 | 0.78 | 0.75 | +0.032 [+0.008, +0.057] | 64% |
| 8 | 0.83 | 0.79 | +0.042 [+0.011, +0.073] | 68% |
| 16 | 0.87 | 0.83 | +0.038 [+0.019, +0.058] | 78% |
| 32 | 0.90 | 0.85 | +0.040 [+0.006, +0.073] | 72% |
| 64 | 0.92 | 0.87 | +0.046 [+0.013, +0.080] | 78% |
Table 25: Data-scarce payoff of the portable direction over a target direction fit from scratch on the same target labels, generalizing to an unseen harm category (Llama-3.1-8B to Falcon-Mamba, AUROC on ground-truth labels). is the paired difference and its 95% interval over the 8 held-out categories, the independent unit. Wins is the fraction of the 120 individual draws favoring transport, which is why it can sit near half while the interval still excludes zero: the mean gain is carried by larger wins, not by winning more often. Positive favors transport. The margin does not close over this range of budgets.
| Model | Write site | Gate layers (decision) | final (tried) | Transfer: read / write layers | Notes |
| Falcon-Mamba-7B-Inst. | mixer output | {24,28,30,34,35,36,37,44} (35) | 4 (1,2,4,6,8; capability at 4,8,16) | 44 / {22,24,28,31,38,40,42,43} | anchor; -ranked set for transfer targets |
| Falcon3-Mamba-7B-Inst. | mixer output | -ranked {24,27,28,31,42,45,46,47} (42) | 4 (0.5,1,2,4; Fig. 6 ) | 47 / same set | block-vs-residual control on this set at ; the output-ranked set {14,15,17,18,21,22,23,24} over-steers at (reported at ) |
| Llama-3.1-8B-Inst. | attention output | {10,12,…,24} (18) | 2 (0.5,1,1.5,2,3,4) | 14 / {12,14,16,18,20} | hardened trigger at the deployed point |
| Mistral-7B-Inst.-v0.2 | attention output | {10,12,…,24} (18) | 2 (0.5,1,1.5,2,3,4) | 14 / {12,14,16,18,20} | persona fix: layer 18, , |
| RWKV-6-World-7B | time-mix output | {14,16,…,24} (20) | 16 (0,4,8,16,24,32,48; coh. ) | 20 (target of Llama L14) | defense run gating the block hook at is a separate operating point; the block boundary (residual) is inert |
| Zamba2-7B-Inst. | attention out. / Mamba out. | {36,42,48,54,60} (48) / {20,24,28,32} (28) | 8 / 16 (persona grid , ) | – | hybrid, both write sites gated |
Table 26: Final hyperparameters per model, with the values tried where a sweep was run. Gate layers are the top-8 by held-out probe AUROC on the SSMs (ranked by mixer-output AUROC on Falcon-Mamba, by AUROC on Falcon3-Mamba, Section 2.5 ); the other architectures use fixed mid-network bands. The decision layer is the middle of the listed set and carries the per-prompt detector. Every gate uses on the probe’s harm probability, the mean refusal direction (mean harmful minus mean benign activation) at each gate layer, greedy decoding with 64 new tokens, and the gain is the largest value that passes the coherence screen (Appendix A.25 ). Transfer rows give the layer the direction is read at in the source and the layers it is written at in the target.