Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

Jinhu Fu; Yihang Lou; Qingyi Si; Shudong Zhang; Yan Bai; Sen Su

cs.CVMar 28, 2026

Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection

Authors: Jinhu Fu, Yihang Lou, Qingyi Si, Shudong Zhang, Yan Bai, Sen Su

Organizations: Beijing University of Posts and Telecommunications · Huawei Technologies Ltd. · Peking University · Chongqing University of Posts and Telecommunications

Abstract

Large Vision-Language Models (LVLMs) have achieved impressive performance across multimodal understanding and reasoning tasks, yet their internal safety mechanisms remain opaque and poorly controlled. In this work, we present a comprehensive framework for diagnosing and repairing unsafe channels within LVLMs (CARE). We first perform causal mediation analysis to identify neurons and layers that are causally responsible for unsafe behaviors. Based on these findings, we introduce a dual-modal safety subspace projection method that learns generalized safety subspaces for both visual and textual modalities through generalized eigen-decomposition between benign and malicious activations. During inference, activations are dynamically projected toward these safety subspaces via a hybrid fusion mechanism that adaptively balances visual and textual corrections, effectively suppressing unsafe features while preserving semantic fidelity. Extensive experiments on multiple safety benchmarks demonstrate that our causal-subspace repair framework significantly enhances safety robustness without degrading general multimodal capabilities, outperforming prior activation steering and alignment-based baselines. Additionally, our method exhibits good transferability, defending against unseen attacks.

Figures & tables

Figure 1: We introduce the diagnosing-and-repairing framework for VLM safety. By precisely identifying safety-critical components, our method avoids disrupting unrelated model abilities. Leveraging both visual and textual token attribution, we construct a dual-modal safety subspace and project activations onto its safe direction, enabling targeted, training-free correction.

Figure 2: Causal tracing of security components through layered blocking methods.

Figure 3: Quantifying the security differentiation capability of different layers through clustering metrics.

Figure 4: Changes in ASR when blocking FFN and MHSA.

Figure 5: Comparison of Pairwise Correlations between MHSA and FFN.

Figure 6: Overview of our CARE. We attribute multimodal activations via RBF kernels, identify harmful directions through generalized eigen-decomposition, and project activations onto the benign safety subspace to enhance model safety.

Model	Method	Benchmarks		PGD-Toxic				PGD-Jailbreak
Model	Method	JailBreakV	MMSafety	UnConstrain	$\kappa=16/255$	$\kappa=32/255$	$\kappa=64/255$	UnConstrain	$\kappa=16/255$	$\kappa=32/255$	$\kappa=64/255$
LLaVA	Original model	45.71	36.48	67.34	55.98	57.13	60.38	67.37	62.98	63.73	65.15
	JailGuard [ 54 ]	18.46	30.17	40.48	38.50	36.72	39.84	33.73	31.48	42.82	44.07
	Refusal Pairs [ 41 ]	32.81	32.13	61.05	53.44	55.67	56.30	62.63	50.23	50.60	51.74
	CAST	25.03	27.83	48.90	40.22	46.80	49.10	50.12	42.68	45.37	47.78
	SPO-VLM [ 51 ]	10.37	16.26	18.34	25.70	21.05	17.90	15.44	23.92	20.01	17.38

Table 1: Comparison of different defense methods across two VLM backbones (LLaVA-onevision and Qwen-2.5VL) under various jailbreak and PGD attack settings. Lower values indicate better safety performance.

\kappa

denotes perturbation radius.

Model	Method	Benchmarks / Attacks
Model	Method	JailbreakV	MMSafety	Toxic64	Jail64
Qwen-VL	Original	20.00	36.29	61.11	73.18
	CARE-w/o text	14.30	17.46	15.51	27.45
	CARE-w/o visual	10.71	26.73	46.13	30.17
	CARE-w random	22.30	36.10	64.35	73.10
	CARE	6.55	8.72	4.60	10.68

Table 2: Ablation results on Qwen2.5-VL and LLaVA-OneVision under jailbreak and PGD attack benchmarks (%). Lower is better.

Figure 7: General ability of CARE against baselines.

Figure 8: Defensive effectiveness at different layers.

Figure 9: Defensive effectiveness across different domains.

Figure 10: Quantifying the security differentiation capability of different layers through clustering metrics in Qwen2.5VL.

Figure 11: Comparison of Pairwise Correlations between MHSA and FFN in Qwen2.5VL.

Figure 12: Changes in ASR when blocking FFN and MHSA in Qwen2.5VL.

Model	Method	APGD-Toxic				APGD-JailBreak
Model	Method	unconstrain	$\kappa=16/255$	$\kappa=32/255$	$\kappa=64/255$	unconstrain	$\kappa=16/255$	$\kappa=32/255$	$\kappa=64/255$
Qwen2.5-VL	Original model	67.71	59.38	61.03	70.62	69.37	60.63	64.45	73.32
	ASTRA	7.13	16.08	13.73	10.12	9.47	17.54	13.35	12.26
	CARE	4.43	9.37	9.62	8.18	3.31	11.17	8.82	6.13
LLaVA-OneVision	Original model	71.11	56.16	60.60	62.90	72.11	63.18	66.43	70.70
	ASTRA	3.28	12.40	9.75	6.62	7.26	13.30	12.13	10.24

Table 3: Comparison of defense methods under APGD-Toxic and APGD-JailBreak attacks on two VLM backbones. Lower ASR indicates better safety.

\kappa

denotes perturbation radius.

Model	Method	MI-FGSM-Toxic				MI-FGSM-JailBreak
Model	Method	unconstrain	$\kappa=16/255$	$\kappa=32/255$	$\kappa=64/255$	unconstrain	$\kappa=16/255$	$\kappa=32/255$	$\kappa=64/255$
Qwen2.5VL	Original model	82.78	72.17	77.30	79.63	86.50	76.66	80.87	82.80
	ASTRA	15.54	18.29	18.78	19.36	16.67	21.25	20.20	19.44
	CARE	9.97	13.34	11.56	10.43	10.11	14.94	14.33	12.31
LLaVA-OneVision	Original model	83.64	80.66	82.18	82.80	88.60	83.15	84.17	85.36
	ASTRA	18.97	23.41	19.86	20.10	19.36	24.45	20.20	22.63

Table 4: Comparison of defense methods under MI-FGSM-Toxic and MI-FGSM-JailBreak attacks on two VLM backbones. Lower ASR indicates better safety.

\kappa

denotes perturbation radius.

Figure 13: Defensive effectiveness across different domains in Qwen2.5VL.

Figure 14: Some cases of image tokens attribution.

Figure 15: Some cases of text tokens attribution.

Methods	ScienceQA	MMBench	MM-Vet
$\text{CARE}_{\text{COCO}}$	81.81%	82.24%	61.40%
$\text{CARE}_{\text{COCO+Flickr}}$	82.33%	83.50%	63.79%

Table 5: Effect of benign anchor construction on general multimodal benchmarks.

Model	ScienceQA	MM-Vet	MMBench	OR-Bench-1k
Original	0	0	0	15.09%
CARE	1.6%	2.5%	0	21.9%

Table 6: Safety over-rejection rate (SARR) on benign multimodal benchmarks. Lower is better.

Method	MM-Safety	JailBreakV	$\mathrm{PGD}_{\mathrm{T64}}$	$\mathrm{PGD}_{\mathrm{J64}}$
ShiftDC [ 57 ]	10.31%	11.17%	15.7%	19.73%
ProFS [ 47 ]	13.39%	16.30%	22.64%	31.20%
CARE	8.72 %	6.55 %	4.60 %	8.46 %

Table 7: Comparison with additional baselines on Qwen2.5-VL. Lower is better.

VLM Safety

See all 29 papers

Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan
SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning
Aug 11, 2026Caoyuan Ma, Wenpu Liu, Weichu Xie +12VLM Robustness Adversarial Attacks on VLMs
The University of Tokyo · Peking University · Wuhan University +4
GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
Feb 27, 2026Xingyu Zhu, Beier Zhu, Junfeng Fang +4VLM Safety Alignment VLM Safety
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore +1

Aug 31, 2026J/K move · Enter open · S save

Do VLMs Share Safety Neurons Across Modalities?

Jiaxuan Li, Jiahao Zhang, Duc Minh Vo, Huy H. Nguyen+2

SB Intuitions Corp., Tokyo, Japan

VLM Evaluation VLM Interpretability VLM Safety VLM Safety Alignment

Open paper AskPDF arXiv

Vision-language models (VLMs) can comply with harmful requests delivered through images, even when their LLM backbones would refuse the same content in text. While prior work characterizes these jailbreaks empirically or at the representation level, how visual inputs perturb safety pathways at the neuron level remains uncharted. We close this gap with a causal, neuron-level analysis of safety mechanisms in 10 VLMs. We propose a two-stage detection pipeline with iterative ablation that accounts for self-repair, and introduce two modality-isolated benchmarks, ViSafe-Detect and ViSafe-Eval, which decouple visual and textual safety signals. Our analysis reveals: (i) Text safety in VLMs is localizable:

\sim

88 neurons (

<

0.01%) whose targeted ablation substantially reduces refusal. (ii) Text safety neurons constitute the dominant refusal pathway: ablating them is the only intervention that consistently and substantially reduces refusal across all models. (iii) Visual safety is high-dimensional and diffuse at the single-neuron level: text safety concentrates in

\sim

5 subspace directions while visual safety requires

\geq

50. This gap holds across architectures, explaining why current alignment has not closed the visual safety gap. Project page is at: https://jiaxuan-li.github.io/vlm-safety-neuron/ Warning: this paper may include examples of harmful content.

Abstract

VLM Safety

Explore similar work

Do VLMs Share Safety Neurons Across Modalities?

SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models