Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models

Enyi Shi; Fei Shen; Chuancheng Shi; Shuyi Miao; Linxia Zhu; Pengyang Shao; Jinhui Tang; Tat-Seng Chua

cs.CVApr 10, 2026

Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models

Authors: Enyi Shi, Fei Shen, Chuancheng Shi, Shuyi Miao, Linxia Zhu, Pengyang Shao, Jinhui Tang, Tat-Seng Chua

Organizations: Nanjing University of Science and Technology · National University of Singapore · The University of Sydney · Beihang University ·

Abstract

With the widespread deployment of vision-language large models (VLLMs), their safety alignment faces dual challenges across languages and modalities. Existing methods model multilingual and multimodal safety separately, overlooking coupled risks between low-resource-language instructions and visual contexts, which hinders the detection of cross-lingual and cross-modal harmful intent and the formation of robust safety boundaries. To address this, we propose a neuron-level interpretable safety alignment framework that identifies safety neurons and performs neuron-targeted safety tuning to jointly mitigate multilingual and multimodal risks. Specifically, we compare FFN representations elicited by harmful requests and benign inputs to identify neuron activation strengths associated with safety refusals. Next, we jointly model neuron activations and corresponding down-projection columns to derive neuron-level saliency, separating general multilingual and multimodal neurons from safety neurons responsible for model defense. Finally, neuron-targeted gradient masking restricts parameter updates to the safety subspace spanned by the identified neurons, enabling precise and interpretable safety enhancement. Extensive experiments show that our method enhances multilingual and multimodal safety by tuning only a few safety neurons, while preserving general capabilities.

Figures & tables

Figure 1. Existing methods make VLLMs fairly robust to isolated monolingual–multimodal or multilingual–unimodal risks, but vulnerable to joint multilingual–multimodal risks. Our method addresses this by probing and enhancing safety neurons against multilingual–multimodal attacks while preserving general capabilities.

Figure 2. Pipeline of our framework. §3.1 Input Scenarios. We use multilingual–multimodal harmful requests in 10 languages, covering Image-Dominant and Text-Dominant Risk, to identify safety neurons across languages and modalities. §3.2 Safety Neuron Probing. Neuron importance is measured by activation strength and the corresponding column norm of $W_{\text{down}}$ , while generic neurons are removed through set difference. §3.3 Neuron-Targeted Safety Fine-Tuning. Gradients of non-safety-neuron rows in the LoRA $B$ -matrix are masked, enabling parameter-targeting and interpretable safety tuning.

Figure 3. Impact of the number of safety neurons on model safety. Under both Image-Dominant Risk and Text-Dominant Risk settings, the number of safety neurons shows a strong negative correlation with the ASR across all languages.

a: Safety Neurons Masking					b: Training Strategy Comparison
Setting	Safety $\downarrow$		General $\uparrow$		Setting	Safety $\downarrow$
Setting	IR	TR	MGSM	MM-Vet	Setting	IR	TR
Vanilla	30.71	19.57	67.7	66.6	Vanilla	30.71	19.57
M-RN	26.11	20.74	69.2	65.3	Random	7.66	7.34
M-SN	51.23	58.44	69.2	66.1	Ours	6.19	5.47

Table 1. Safety specificity of identified neurons. Masking them collapses safety with limited impact on general capabilities, whereas random-neuron training yields less gains.

Figure 4. Impact of different masking strategies on model safety. M-RN and M-SN denote masking Random Neurons and Safety Neurons, respectively. Masking Safety Neurons causes a substantial ASR increase under both risk types, highlighting their critical role in model safety.

Setting	XSTest		Benign Refusal Rate
Setting	Safe Refusal $\downarrow$	Safe Compliance $\uparrow$	MGSM $\downarrow$	MM-Vet $\downarrow$
Vanilla	0.00	100.00	0.00	6.80
Ours	2.00	98.00	0.00	7.34

Table 2. Over-refusal evaluation on benign inputs. Targeted safety-neuron tuning maintains high compliance with only marginal changes in refusal rates.

Image-Dominant Risk	Method	AR	ZH	EN	FI	FR	DE	JA	NO	RU	ES	AVG
	Llama ( Grattafiori et al., 2024 )	24.21	17.86	21.83	28.17	17.46	21.43	37.30	9.52	30.16	28.17	23.61
	XSAFETY ( Wang et al., 2024 )	26.59	16.67	19.44	27.38	9.13	18.65	31.75	9.92	14.29	26.19	20.00
	ESCO ( Gou et al., 2024 )	24.21	17.46	22.22	27.78	17.46	21.03	37.30	9.52	29.76	28.17	23.49
	ASTRA ( Wang et al., 2025 )	24.60	21.21	16.27	25.79	21.83	15.87	26.19	15.87	27.78	23.02	21.84
	Self Defense ( Phute et al., )	22.22	14.68	10.32	23.81	15.08	17.06	25.79	6.35	29.76	21.03	18.61
	MLC ( Bu et al., 2026 )	15.48	15.48	12.70	22.22	6.35	14.68	25.79	6.75	14.68	17.86	15.20

Table 3. ASR on the evaluation set across ten languages under Image-Dominant and Text-Dominant Risk conditions. Our method achieves the lowest ASR in most cases and outperforms all baselines on average.

Setting	Llama ( Grattafiori et al., 2024 )			LLaVA ( An et al., 2025 )			Qwen ( Yang et al., 2025 )
Setting	Para	IR	TR	Para	IR	TR	Para	IR	TR
Vanilla	–	23.61	16.21	–	20.87	22.31	–	30.71	19.57
LoRA	0.11	5.64	8.55	0.11	8.77	9.96	0.11	7.22	6.88
Ours	0.03	4.21	7.15	0.03	6.07	4.45	0.03	6.19	5.47

Table 4. Average ASR across 10 languages comparing our method with traditional LoRA fine-tuning under Image-Dominant (IR) and Text-Dominant (TR) Risks. Our method updates fewer parameters (%) while achieving lower ASR.

Setting	Llama ( Grattafiori et al., 2024 )		LLaVA ( An et al., 2025 )		Qwen ( Yang et al., 2025 )
Setting	MM-Vet	MGSM	MM-Vet	MGSM	MM-Vet	MGSM
Vanilla	55.2	67.7	47.1	59.5	66.6	67.7
LoRA	46.8	67.1	30.0	48.0	66.7	69.4
Ours	54.0	66.9	51.0	63.1	67.5	69.4

Table 5. Impact of our method on the multimodal and multilingual capabilities of VLLMs, evaluated by accuracy on MM-Vet (

\uparrow

) and MGSM (

\uparrow

). Performance remains stable or slightly improves, with these capabilities preserved.

Method	Multimodal		MultiJail				Attack
Method	FigStep	SPA-VL	ZH	IT	AR	KO	GCG	ImgJP	BAP
Vanilla	6.40	3.02	6.03	1.59	1.90	5.71	10.38	11.73	12.69
MLC	1.00	3.77	3.81	4.13	6.98	6.35	0.40	9.23	11.73
Ours	0.80	0.75	1.59	0.32	0.95	2.54	0.33	0.19	4.04

Table 6. ASR on OOD benchmarks and attacks. Our method generalizes well across OOD multilingual and jailbreak scenarios and open-world attacks.

Figure 5. Safety neuron distribution across layers. HRLs and NHRLs denote the numbers of safety neurons in high- and non-high-resource languages, respectively, while

\Delta

denotes their difference. Across modalities, safety neurons first increase and then decrease, peaking in the mid-to-late layers, where most HRL–NHRL differences are also concentrated.

Figure 6. Safety neuron overlap across languages and risk modalities. Safety neurons show moderate overlap across languages and risk modalities.

Panel of Figure 9 — Figure 9. Case visualization. Our method successfully rejects multilingual–multimodal attacks that easily bypass VLLMs.

Method	Llama ( Grattafiori et al., 2024 )		LLaVA ( An et al., 2025 )		Qwen ( Yang et al., 2025 )
Method	IR	TR	IR	TR	IR	TR
MLC	6.03	4.65	15.48	14.33	2.46	1.52
Ours	2.26	4.61	0.75	1.64	1.47	1.44

Table 7. Additional Llama-Guard evaluation results. Results averaged across 10 languages show that our method consistently outperforms the recent SOTA baseline MLC and that the gains are not specific to a single LLM-based judge.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10. Sensitivity analysis of the intervention ratio. Our method remains relatively robust across values, while the selected (0.03) achieves the best safety performance.

Configuration	LLaVA	Llama	Qwen
Computing Device	$1\times$ A800	$1\times$ A800	$1\times$ A800
Global Batch Size	32	32	32
Training Epochs	5	5	5
Optimizer	AdamW	AdamW	AdamW
Learning Rate	$6\mathrm{e}{-4}$	$6\mathrm{e}{-4}$	$6\mathrm{e}{-4}$
Warmup Ratio	0.05	0.05	0.05

Appendix

Table 8. Implementation details across VLLMs.

Language	Arabic	Chinese	English	Finnish	French
Accuracy (%)	91.41	92.19	90.63	95.31	96.09
Language	German	Japanese	Norwegian	Russian	Spanish
Accuracy (%)	92.97	94.53	90.63	94.53	90.63
Average Accuracy (%)					92.89

Appendix

Table 9. Qwen-Guard Judgment Accuracy (%). Qwen-Guard demonstrates strong and consistent alignment with human judgments across 10 diverse languages.

Figure 11. Image-Dominant Risk visualization cases across multiple languages.

Figure 12. Text-Dominant Risk visualization cases across multiple languages.

Figure 13. Layer-wise feature space shifts under Image-Dominant Risk. Our method gradually separates harmful-input representations from those of the vanilla model in deeper layers, steering them toward a safer representation space.

Figure 14. Layer-wise feature space shifts under Text-Dominant Risk. Our method induces an earlier and progressively clearer separation of harmful-input representations across layers, steering them toward a safer representation space.

LLM Safety Alignment

Large language model safety alignment trains and evaluates systems to follow human policies while resisting harmful requests and misuse.

See all 344 papers

One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs
Jul 30, 2026Enyi Shi, Fei Shen, Chuancheng Shi +4VLM Robustness Multimodal Large Language Models
1Nanjing University of Science and Technology · 2National University of Singapore · 3The University of Sydney +2
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Jun 2, 2026Hashmat Shadab Malik, Muzammal Naseer, Salman KhanMLLM Safety Multilingual Language Model Evaluation
Mohamed Bin Zayed University of AI, UAE · Khalifa University, UAE · Australian National University, Australia
Safety Kernel: Mechanistic Alignment for Cross-Lingual Defense in LLMs
Oct 17, 2026Chengyu Xie, Fei Shen, Shuyi Miao +5LLM Safety Language Model Robustness
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China · Nanjing University of Science and Technology, Nanjing, China · School of Artificial Intelligence, Beihang University, Beijing, China +3

Jul 30, 2026J/K move · Enter open · S save

One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs

Enyi Shi, Fei Shen, Chuancheng Shi, Linxia Zhu+3

1Nanjing University of Science and Technology · 2National University of Singapore · 3The University of Sydney+2

VLM Robustness Multimodal Large Language Models LLM Safety Alignment

Open paper AskPDF arXiv

As large vision-language models (LVLMs) are deployed globally, the combination of multilingual instructions and visual information makes malicious attacks more covert and sophisticated than ever before. However, existing methods isolate language and modality defenses, which, coupled with the scarcity of safety data and high fine-tuning costs, makes it difficult for models to defend against compound attacks. To address this severe challenge, we propose a neuron-level cross-dimensional safety alignment framework driven by modality- and language-shared safety neurons (MLS-Neurons). First, we identify monolingual and unimodal safety neurons by comparing responses to harmful and benign samples, quantifying functional saliency through activation strength and downstream impact. Then, by intersecting these unimodal neurons within each language, we extract modality-shared safety neurons (MS-Neurons) responsive to both visual and textual risks, bridging the safety representation gap between modalities. Furthermore, using English as a semantic anchor, we intersect MS-Neurons across languages to identify modality- and language-shared safety neurons (MLS-Neurons), serving as key defenses against compound attacks. Finally, we update only this minimal subset of shared neurons (~0.03% of parameters), transferring English-only safety supervision to multilingual and multimodal scenarios. Extensive experiments show that our method significantly outperforms state-of-the-art approaches across diverse multilingual and multimodal safety benchmarks while preserving general utility.

Abstract

LLM Safety Alignment

Explore similar work

One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs

Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models

Safety Kernel: Mechanistic Alignment for Cross-Lingual Defense in LLMs