Where Is Rank-1 DiM Refusal Intervenable in Multimodal LM Derivatives?
Organizations: NICT · Yonsei University
Abstract
Open-weight model hubs now ship not only chat checkpoints but a long tail of derivatives---fine-tunes, safety guards, vision--language models (VLMs), audio wrappers, and generative heads---yet residual-stream safety audits still often intervene at a single site, usually the text decoder . We ask a localization question under one shared mean / rank-1 difference-in-means (DiM) extractintervene protocol, with matched-norm random and empty-rate controls: does causal chat attack lift concentrate at , or do perceptual encoders, projectors, and fusion paths (//) supply a comparable bypass? On the mixed-input settings we stress-test---HADES adversarial images paired with harmful text prompts, and spoken AdvBench via text-to-speech (TTS) or silence---text DiM at raises attack success rate (ASR) while the same mean DiM at // does not; attacks track harmful text rather than image or audio content. Margin-rank and leverage probes on a Vision leaf show that this recipe is extremal at and near chance at /, which pressures a weak-extractor reading of those flat cells for the DiM family we study. Catalog breadth and mapper-free basederivative transfer when residual widths match provide supporting coverage under the same gates; generative U-Nets are a separate band under generation judges. The primary claim is this recipe-scoped vs.\ non-text intervenability asymmetry---intervention accessibility under mean / rank-1 DiM, not a claim that refusal is absent or fully represented outside (text vs.\ perceptual contrasts and pooling differ by site). On VLGuard Unsafe prompts (; Llama-Guard), the same vs.\ / directionality holds at modest absolute ASR ( at ). An intervention-oriented Classsite ledger records where this protocol still hooks as supporting coverage, not as a universal multimodal bypass.
Figures & tables
| (A) Recipe axes | ||
| Axis | : @ | : @ |
| Contrast variable | text prompt | perceptual input (image , audio , …) |
| Contrast pools | (matched-format benign) | |
| Hooked tensor | decoder residual | : e.g. / / / / |
| Pool | : often last text position | : mean over non-text tokens (patches, VT, Q-Former); at fusion use image/audio positions, not last text |
| Path to | ||
| Claim | Evidence cell | Site | steer (Nemo) | Control / note |
| (1) Coordinates: mixed-input at ; non-text flat | ||||
| Mixed@ | Vision HADES test500 | rand | ||
| Mixed@ | Qwen2-Audio TTS Adv | held-out Adv text | ||
| flat | Vision @ / | / | matched-norm rand | |
| flat | Qwen2-Audio @ | benign-TTS contrast | ||
| ABL_POS | LLaVA @ | defence role; Tab. 11 | ||
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Corpus | Mod. | Pool | Extract | Eval | Role |
| AdvBench | text | 520 | 128 | 392 | Harmful requests; disjoint / |
| Alpaca | text | 31k | 128 | — | Benign contrast for only |
| HarmBench | text | 320 | — | 320 | Eval-only |
| StrongREJECT | text | 313 | — | 313 | Eval-only |
| HADES (step 5) | image | 750 | — | 500 | conditional-input |
| AdvBench TTS | audio | 520 | — | 392 | conditional-input |
| Ax. | Class (model) | Extract | Self OP | Cross OP |
| A1 | Root (Llama-3.1-8B-Instruct) | Text DiM @ | L16 | same |
| A1 | Wild FT (Hermes / Tulu-3 / UltraMedical) | Text DiM @ | L16 / / | / self-donor / |
| A1 | Size / Distill (3B / SuperNova) | Text DiM @ | L14 / L16 | / |
| A1 | PEFT S5 (Instruct LoRA) | Text DiM @ | L16 | |
| A1 | MoE Mixtral-8x7B | Text DiM @ | L16 | Mistral-family |
| A1 | Compress (Sheared-LLaMA) | Text DiM @ | L16 (ineff) | — / |
| Judge | Acc | Prec | Rec | Note |
| Keyword (ref.) | 0.83 | 0.73 | 1.00 | excluded from median/IQR |
| Llama-Guard | 0.91 | 0.86 | 0.94 | panel |
| Nemotron SG 8B v3 | 0.91 | 0.90 | 0.89 | panel |
| MD-Judge-v0.1 | 0.91 | 0.84 | 0.97 | panel |
| BeaverDam-7B | 0.89 | 0.84 | 0.94 | panel |
| Open-weight LLM judge (26B) | 0.89 | 0.92 | 0.83 | panel |
| Approach | Role | Extract | Intervene | Random |
| Arditi ( Arditi et al., 2024 ) | attack | text DiM | (add) | |
| CMRM ( Liu et al., 2025 ) | defence | MM vs text | multi | — |
| ShiftDC ( Zou et al., 2025 ) | defence | modality shift | — | |
| Yin ABL_POS ( Yin et al., 2026 ) | defence | cond@ | yes | |
| ASTRA ( Wang et al., 2025 ) | defence | visual attr | — | |
| ECSO ( Gou et al., 2024 ) | defence | — (I T) | pipeline | — |
| Task | Out | Model (site) | Judge | Status |
| Chat / label @ | ||||
| text | Llama-lineage / Qwen @ | N/HBC | effective | |
| label | Guard-3 @ | safe/unsafe | effective (miss ) | |
| Vision mixed-input | ||||
| text | Vision / LLaVA @ | N/HBC | effective | |
| text | Vision / LLaVA @ / / | N/HBC | ineffective | |
| Task | Out | Model (site) | Judge | Status |
| Ceiling / KW-only (no Class gate) | ||||
| text | Qwen-VL / LLaVA-1.5 @ | KW | KW only (not N/HBC-gated) | |
| text | Vision/LLaVA FigStep @ / | N/HBC | ceiling / no lift | |
| image | SD-Turbo img2img down2/mid | NudeNet+CLIP | flat (ineffective) | |
| Steer-N/A / literature / path check | ||||
| text | LLaDA / Dream @ | N/HBC | ineffective / collapse | |
| Donor recipient | Class | rand | Cross | |||
| Llama UltraMedical | Domain FT | 0.822 | 0.066 | 0.821 | 0.84 | effective |
| Llama LLaVA-Mini | Comm. VLM | 0.768 | 0.069 | 0.766 | 1.00 | effective |
| Mistral Mixtral ∗ | Official MoE | 0.630 | 0.067 | 0.616 | 1.00 | effective |
| Llama Tulu-3 † | Wild FT | 0.626 | 0.068 | 0.616 | 0.28 | effective |
| Llama Guard-3 | Guard | 0.168 | 0.066 | 0.127 | 0.91 | effective |
| Llama CodeLlama | Code FT | 0.048 | 0.065 | 0.034 | 0.00 | ineffective |
| Recipe | Role | Nemo | HBC | |
| base / random@ | floor | / | / | 176 |
| Ours abl@ | attack † | 176 | ||
| Yin ABL_POS vs. image | defence ‡ | 104 |
| Class | Model | Tried / outcome | Why closed (or unlock) |
| Compress | Sheared / Sparse* | pure RFA; grids | KW false+ / Nemo anti-lift; HBC 0; RFA |
| Audio | Qwen2-Audio @ | Adv/HB/SR; TTS; family cross | unlock (Class effective; @ still ineffective) |
| Speculative | EAGLE3 | draft verifier | draft-only infeasible |
| MTP | V3 | not run | no chat- residual ( N/A ) |
| AR diff | Dream / LLaDA | mid/multi ; empty re-sweep | empty collapse / Nemo |
| Attn replace | DS-V2 | /layer/RFA; HB | HBC ; quality collapse |