cs.LGNov 15, 2026

Where Is Rank-1 DiM Refusal Intervenable in Multimodal LM Derivatives?

Authors: Chansu Han, Seok-Hwan Choi, Takeshi Takahashi

Organizations: NICT · Yonsei University

Abstract

Open-weight model hubs now ship not only chat checkpoints but a long tail of derivatives---fine-tunes, safety guards, vision--language models (VLMs), audio wrappers, and generative heads---yet residual-stream safety audits still often intervene at a single site, usually the text decoder TT. We ask a localization question under one shared mean / rank-1 difference-in-means (DiM) extract==intervene protocol, with matched-norm random and empty-rate controls: does causal chat attack lift concentrate at TT, or do perceptual encoders, projectors, and fusion paths (EE/PP/FF) supply a comparable bypass? On the mixed-input settings we stress-test---HADES adversarial images paired with harmful text prompts, and spoken AdvBench via text-to-speech (TTS) or silence---text DiM at TT raises attack success rate (ASR) while the same mean DiM at EE/PP/FF does not; attacks track harmful text rather than image or audio content. Margin-rank and leverage probes on a Vision leaf show that this recipe is extremal at TT and near chance at PP/EE, which pressures a weak-extractor reading of those flat cells for the DiM family we study. Catalog breadth and mapper-free base→\toderivative transfer when residual widths match provide supporting coverage under the same gates; generative U-Nets are a separate band under generation judges. The primary claim is this recipe-scoped TT vs.\ non-text intervenability asymmetry---intervention accessibility under mean / rank-1 DiM, not a claim that refusal is absent or fully represented outside TT (text vs.\ perceptual contrasts and pooling differ by site). On VLGuard Unsafe prompts (n=200n{=}200; Llama-Guard), the same TT vs.\ PP/EE directionality holds at modest absolute ASR (0.030→0.1500.030{\to}0.150 at TT). An intervention-oriented Class×\timessite ledger records where this protocol still hooks as supporting coverage, not as a universal multimodal bypass.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. From Detection to Refusal: Safer LLMs via Circuit-Guided Weight Scaling

    Aug 30, 2026Kuan-Lin Chu, Chung-En Sun, Tsui-Wei WengLLM SafetyRefusal Behavior in Language Models

  2. Safety Reconstructed: Generative Modeling via Masked Diffusion Builds Strong Safety Guardrails

    Sep 27, 2026Gert Lek, Abele Malan, Chaoyi Zhu +3LLM Safety BenchmarksLLM Guardrails