GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models

Xingyu Zhu; Beier Zhu; Junfeng Fang; Shuo Wang; Yin Zhang; Xiang Wang; Xiangnan He

cs.CVFeb 27, 2026

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models

Authors: Xingyu Zhu, Beier Zhu, Junfeng Fang, Shuo Wang, Yin Zhang, Xiang Wang, Xiangnan He

Organizations: MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore · Tianjin University

Abstract

Large vision-language models (LVLMs) have achieved remarkable progress in vision-language reasoning tasks, yet ensuring their safety remains a critical challenge. Recent input-side defenses detect unsafe images with CLIP and prepend safety prefixes to prompts, but they still suffer from inaccurate detection in complex scenes and unstable safety signals during decoding. To address these issues, we propose GuardAlign, a training-free defense framework that integrates two strategies. First, OT-enhanced safety detection leverages optimal transport to measure distribution distances between image patches and unsafe semantics, enabling accurate identification of malicious regions without additional computational cost. Second, cross-modal attentive calibration strengthens the influence of safety prefixes by adaptively reallocating attention across layers, ensuring that safety signals remain consistently activated throughout generation. Extensive evaluations on six representative MLLMs demonstrate that GuardAlign reduces unsafe response rates by up to 39% on SPA-VL, while preserving utility, achieving an improvement on VQAv2 from 78.51% to 79.21%.

Figures & tables

Figure 1: Comparison between existing strategies ( Ding et al., 2025 ) and ours. (a) Similarity scores overlap between safe and unsafe images, while OT-based transport costs yield clear separation for reliable detection. (b) Prefix attention decays in the existing strategies (orange) but remains stable with ours (blue). (c) Example from SPA-VL ( Zhang et al., 2025a ) showing that existing methods generate harmful content, whereas ours maintains safety. (d) Our method achieves consistent safety gains across diverse benchmarks.

Figure 2: Framework of GuardAlign . OT-Enhanced Safety Detection : image patches and predefined unsafe prompt categories are jointly encoded, and optimal transport is used to identify patches that align with harmful semantics. The most suspicious patches are masked to produce a sanitized image. Cross-Modal Attention Calibration : a lightweight safety prefix is added to the query, and the multimodal model attends over the sanitized visual tokens. This design guides the model toward safe evidence and prevents unsafe generations.

Method	SPA-VL	MM-SafetyBench	FigStep		Adv. Image+Text
Method	Harm $\downarrow$	SD+TYPO $\downarrow$	Vanilla $\downarrow$	Suffix $\downarrow$	Unconstrained $\downarrow$
LLaVA-1.5-7B	46.04	40.46	58.60	62.00	97.50
+ ECSO	23.40	15.89	37.40	59.00	95.00
+ ETA	16.98	15.83	7.80	22.60	22.50
+ GuardAlign	10.31	9.65	3.40	15.30	15.00
LLaVA-1.5-13B	40.75	41.01	61.60	66.40	100.00

Table 1: USR evaluation results across multiple safety benchmarks.

Model	Method	Fine-tuned	Comprehensive Benchmark $(\uparrow)$				General VQA ( $\uparrow$ )
Model	Method		MME P	MME C	MMB	SQA I	TextVQA	VQAv2
LLaVA-1.5-7B	Vanilla MLLM		1505.88	357.86	64.60	69.51	58.20	78.51
	+ Posthoc-LoRA	✓	1420.66	332.50	63.32	67.33	55.99	76.87
	+ Mixed-LoRA	✓	1483.00	267.14	68.04	68.42	57.88	79.18
	+ ECSO	✗	1495.88	360.00	63.83	69.36	58.15	78.39
	+ ETA	✗	1506.13	357.86	64.69	69.51	58.15	78.51

Table 2: General performance of different methods.

Model	OT-enhanced	Cross-modal	SPA-VL
Model	Safety Detection	Attention Calibration	Harm $\downarrow$ (USR )	Helpful Score $\uparrow$
LLaVA-1.5-7B	✗	✗	46.04	7.64
	✗	✓	32.12	8.05
	✓	✓	10.31	8.63

Table 3: Ablation study of GuardAlign on the SPA-VL test set using LLaVA-1.5-7B, where we ablate two alignment components: malicious detection via OT and mask-guided alignment .

Panel of Figure 3 — Figure 3: Comparison of safe and unsafe patch feature distributions using different distance metrics. (a): OT distance effectively separates the two distributions. (b): Cosine distance provides a less distinct separation between safe and unsafe patches.

Panel of Figure 4 — Figure 4: Analysis of factors affecting safety and utility. (a) Varying $\tau$ shows a trade-off: smaller values lower USR with higher helpfulness, while larger ones improve robustness at some cost. (b) More patches reduce USR while keeping helpfulness stable, as finer partitioning exposes hidden malicious semantics. (c) Across CLIP backbones from RN50 to SigLIP, our method lowers harm rates while preserving VQA accuracy, showing robustness to encoder variations.

Model	SPA-VL (Harm)		MM-SafetyBench
Model	Running Time $\downarrow$	USR $\downarrow$	Running Time $\downarrow$	USR $\downarrow$
LLaVA-1.5-7B	37 min	46.04	5 h 02 min	40.46
+ ETA	2h 30 min	16.98	13h 40min	15.89
+ GuardAlign	42 min	10.31	5h 28 min	9.65

Table 4: Comparison of inference cost and safety on SPA-VL and MM-SafetyBench.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	SPA-VL	MM-SafetyBench	FigStep		Adv. Image+Text
Method	Harm $\downarrow$	SD+TYPO $\downarrow$	Vanilla $\downarrow$	Suffix $\downarrow$	Unconstrained $\downarrow$
LLaVA-1.5-7B	72.45	84.46	86.40	85.80	85.00
+ ECSO	53.96	72.44	79.29	82.20	67.50
+ ETA	38.87	53.39	32.40	25.00	17.50
+ GuardAlign	32.71	45.52	25.40	20.00	13.00
LLaVA-1.5-13B	66.79	87.98	90.20	87.40	80.00

Appendix

Table 5: ASR performance across multiple safety benchmarks.

Model	Comparison Methods		Win-Tie (%) $\uparrow$
LLaVA-1.5-7B	GuardAlign v.s.	Vanilla VLM	93.6
		Posthoc-LoRA	53.2
		Mixed-LoRA	55.7
		ECSO	85.3
		ETA	79.8
LLaVA-1.5-13B	GuardAlign v.s.	Vanilla VLM	90.2

Appendix

Table 6: Helpfulness evaluation on the SPA-VL Help.

Figure 5: Examples of mask-guided alignment. Unsafe regions in the original image are automatically detected and masked.

Methods	Patch-level PGD		Auto-PGD
Methods	$\epsilon=16/255$	$\epsilon=32/255$	$\epsilon=16/255$	$\epsilon=32/255$
LLaVA-v1.5-7B	69.09	74.55	80.60	90.30
+ ETA	5.78	5.93	4.27	4.48
+ GuardAlign	2.15	2.36	3.23	3.65

Appendix

Table 7: Results on MM-SafetyBench with adaptive adversaries.

	Harm(USR) $\downarrow$	Helpful Score $\uparrow$
$\gamma=1.05$	10.63	8.26
$\gamma=1.10$	10.42	8.47
$\gamma=1.15$	10.33	8.56
$\gamma=1.20$	10.31	8.63
$\gamma=1.25$	10.35	8.58

Appendix

Table 8: Effect of

\gamma

on safety and helpfulness.

Model	SPA-VL( $\downarrow$ )	TextVQA( $\uparrow$ )
LLaVA-1.5-7B	46.04	58.20
LLaVA-1.5-7B (Posthoc-SFT)	12.96	57.80
+ GuardAlign	9.37	58.00

Appendix

Table 9: Comparison of safety and utility with post-hoc SFT and GuardAlign.

	MM-SafetyBench
	SD+TYPO	TYPO	SD+TYPO
InternVL2.5-38B	7.10	8.09	6.78
+ GuardAlign	7.45	8.51	7.23
InternVL2.5-78B	7.43	7.22	6.60
+ GuardAlign	8.25	8.16	7.58

Appendix

Table 10: Results on MM-SafetyBench for larger LVLMs (InternVL2.5-38B and 78B).

Model	SPA-VL	MM-SafetyBench
Gemini-2.5-Flash	7.36	7.15
+ GuardAlign (OT-ENHANCED DETECTION)	7.41	7.28
GPT-4V	7.02	6.97
+ GuardAlign (OT-ENHANCED DETECTION)	7.23	7.06

Appendix

Table 11: Safety evaluation on the black-box model.

VLM Safety Alignment

See all 15 papers

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images
Mar 9, 2026Qishun Yang, Shu Yang, Lijie Hu +1VLM Safety Alignment VLM Safety
King Abdullah University of Science and Technology · Provable Responsible AI and Data Analytics Lab · China University of Petroleum-Beijing at Karamay +1
VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts
Apr 7, 2026Peigui Qi, Kunsheng Tang, Yanpu Yu +7VLM Safety VLM Robustness
University of Science and Technology of China · Ant Group · University of Washington
SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Models
Sep 3, 2026Caoyuan Ma, Tian Gu, Wenpu Liu +11Vision-Language Models VLM Safety Alignment
The University of Tokyo · Wuhan University · Shanghai AI Laboratory +4

Mar 9, 2026J/K move · Enter open · S save

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

Qishun Yang, Shu Yang, Lijie Hu, Di Wang

King Abdullah University of Science and Technology · Provable Responsible AI and Data Analytics Lab · China University of Petroleum-Beijing at Karamay+1

VLM Safety Alignment VLM Safety Vision-Language Models VLM Adaptation VLM Robustness

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs. To address this, existing methods require explicit safety labels or contrastive data; yet, threat-related concepts are concrete and visually depictable, while safety concepts, like helpfulness, are abstract and lack visual referents. Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA). VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels. Through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution, shaping safety-oriented personas. Experiments across multiple VLMs and safety benchmarks demonstrate that VSFA reduces the attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. Our work extends the self-fulfilling mechanism from text to visual modalities, offering a label-free approach to VLMs alignment.

Abstract

VLM Safety Alignment

Explore similar work

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts

SafeRI: Recognition and Intervention for Token-Level Safety Intervention in Large Vision Language Models