Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

Qishun Yang; Shu Yang; Lijie Hu; Di Wang

cs.CVMar 9, 2026

Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

Authors: Qishun Yang, Shu Yang, Lijie Hu, Di Wang

Organizations: King Abdullah University of Science and Technology · Provable Responsible AI and Data Analytics Lab · China University of Petroleum-Beijing at Karamay · Mohamed bin Zayed University of Artificial Intelligence

Abstract

Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs. To address this, existing methods require explicit safety labels or contrastive data; yet, threat-related concepts are concrete and visually depictable, while safety concepts, like helpfulness, are abstract and lack visual referents. Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA). VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels. Through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution, shaping safety-oriented personas. Experiments across multiple VLMs and safety benchmarks demonstrate that VSFA reduces the attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. Our work extends the self-fulfilling mechanism from text to visual modalities, offering a label-free approach to VLMs alignment.

Figures & tables

Figure 1: Overview of the VSFA framework. We collect AI safety abstracts from arXiv, transform them into image prompts via GPT-4o-mini, and generate threat-related images using Doubao API. Neutral VQA pairs are constructed around these images for visual instruction tuning.

Model	Method	FigStep		MM-SafetyBench		SPA-VL		Avg.
Model	Method	ASR $\downarrow$	CS $\uparrow$	ASR $\downarrow$	CS $\uparrow$	ASR $\downarrow$	CS $\uparrow$	ASR $\downarrow$	CS $\uparrow$
Qwen3-VL-8B	No Defense	32.40	0.12	38.63	0.09	45.28	0.11	38.77	0.11
	AdaShield †	2.40	0.04	8.57	0.03	32.45	0.05	14.47	0.04
	VLGuard ‡	3.80	0.32	10.83	0.29	28.49	0.31	14.37	0.31
	VSFA (Ours)	5.60	0.51	14.29	0.48	22.64	0.52	14.18	0.50
Qwen2.5-VL-7B	No Defense	35.60	0.14	42.26	0.11	48.49	0.13	42.12	0.13

Table 1: Safety performance comparison across different defense methods. We report Attack Success Rate (ASR

\downarrow

) and Constructive Score (CS

\uparrow

) for each benchmark. CS measures the balance between safety compliance and user-centric helpfulness Duan et al. (2025) . Best in bold , second best underlined .

Model	Defense	Multimodal Capabilities $\uparrow$							Refusal Rate $\downarrow$
Model	Defense	Rec	OCR	Know	Gen	Spat	Math	Total	Refusal Rate $\downarrow$
Qwen3-VL-8B	AdaShield	46.2	60.5	28.8	33.5	57.9	14.2	44.8	24.82
	VLGuard	54.5	63.2	40.8	42.6	57.5	28.5	51.7	8.95
	VSFA (Ours)	53.2	62.8	39.5	41.2	59.5	27.2	51.0	2.62
Qwen2.5-VL-7B	AdaShield	43.7	58.6	25.6	31.1	58.1	12.5	42.7	26.15
	VLGuard	50.2	60.8	37.1	39.5	56.3	25.8	48.6	10.82

Table 2: Over-refusal evaluation on MM-Vet. We report multimodal capabilities (

\uparrow

) and refusal rate on benign queries (

\downarrow

Figure 2: Ablation study on fine-tuning modality. We compare three variants: Image (threat-related images with neutral VQA), Text (text-only safety data), and Mixed (combination of both). (a) Attack Success Rate across four models. (b) Constructive Score across four models.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Search Terms
AI safety alignment	AI risk existential
artificial intelligence threat	AI alignment problem
AI safety research	machine learning safety
AI control problem	AGI risk
AI alignment failure	AI safety measures

Appendix

Table 3: The complete arXiv search terms used for text collection. We retrieve up to 5 papers per term from categories cs.AI, cs.LG, cs.CY, and cs.CR.

Visual Styles
Photorealistic	Digital art
Concept art	Technical illustration
Documentary style	Abstract representation
Cinematic	Artistic
Professional	Casual
Futuristic	Vintage

Appendix

Table 4: The 12 visual styles applied during image generation. We combine these styles with scene environments to maximize visual diversity across the dataset.

Scene Environments
Office/Workplace	Laboratory	Public space
Home environment	Industrial setting	Educational
Medical	Research facility	Urban
Rural	Indoor	Outdoor
Virtual	Mixed reality	Studio

Appendix

Table 5: The 15 scene environments used for image composition. Each environment provides different visual context for threat-related content.

Statistic	Value
Total images	700
Total VQA pairs	4,200
VQA pairs per image	6
Image resolution	1024 $\times$ 1024
Question categories	4
Question templates	16

Appendix

Table 6: Summary statistics of the VSFA dataset. We generate 6 neutral VQA pairs per image using 16 question templates across 4 categories.

Metric	Words
Mean	85
Median	78
Min	25
Max	180
Std	32

Appendix

Table 7: Answer length statistics in word count. The answers maintain moderate length suitable for visual instruction tuning.

Dimension	Evaluation Guidelines
Neutrality	The question should not suggest any specific answer. We check for value-laden words like “dangerous”, “harmful”, or “threatening”. The question should avoid safety-related terms like “threat”, “warning”, or “caution”. A neutral question asks about facts in the image. It does not ask for opinions or value judgments. VSFA relies on implicit learning from visual content. Explicit safety terms would confound this learning process.
Clarity	The question should have one clear meaning. Ambiguous questions create confusion in training. The answer should directly address the question. Both should use correct grammar with clear subjects and verbs. The answer should use specific terms instead of vague words like “something” or “stuff”. Clear QA pairs provide strong training signals.
Consistency	The answer should not contradict itself. If one sentence describes “a dark room”, later sentences should not mention “bright sunlight”. All descriptions should match what appears in the image. The model should not describe objects that are not visible. The reasoning should follow a logical order. In multi-turn dialogues, answers should stay consistent across turns.

Appendix

Table 8: Evaluation criteria for QA quality control. Each dimension is scored on a 0-10 scale. We discard samples with overall score below 6.0.

Figure 3: Bidirectional steering effects of top SAE latents. Bars below zero show ASR reduction from adding the latent to the original model. Bars above zero show ASR increase from removing it from the VSFA model. Latent #12 shows the strongest effect in both directions (

-

18%/+14%), confirming it as the primary safety-oriented persona latent.

Teacher Model	Images	QA Pairs	Avg. Length	ASR (%) $\downarrow$
Baseline (no fine-tuning)	–	–	–	35.6
GPT-4o-mini (ours)	60	360	659 chars	12.0
Claude 4.5 Sonnet	60	360	1,322 chars	22.2
Gemini-3-pro	60	360	2,467 chars	28.8
GPT-5	60	360	769 chars	22.8

Appendix

Table 9: FigStep ASR across teacher models on Qwen2.5-VL-7B. All models use 60 images and about 360 QA pairs. Lower ASR means safer.

Visual Style	Images	FigStep ASR (%) $\downarrow$
No Defense (baseline)	0	35.6
Photorealistic	50	12.8
Digital art	50	13.1
Concept art	50	12.4
Technical illustration	50	13.7
Documentary style	50	11.9

Appendix

Table 10: Visual style ablation on Qwen2.5-VL-7B. Each row is a separate training run using 50 images of one style. Lower ASR means safer.

Benchmark	Baseline	VSFA	Diff
MMLU	68.52%	68.13%	$-$ 0.39%
MMMU	50.56%	50.11%	$-$ 0.45%

Appendix

Table 11: General capability evaluation on Qwen2.5-VL-7B before and after VSFA training.

Model	Size	Baseline ASR	VSFA ASR	Diff
Gemma 3 IT	4B	37.0%	7.2%	$-$ 29.8%
Llama 3.2 Vision	11B	49.8%	17.6%	$-$ 32.2%

Appendix

Table 12: VSFA on additional model families. Each model is trained with only 60 images. Lower ASR means safer.

VLM Safety Alignment

See all 15 papers

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models
Feb 27, 2026Xingyu Zhu, Beier Zhu, Junfeng Fang +4VLM Safety Alignment VLM Safety
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore +1
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning
Aug 11, 2026Caoyuan Ma, Wenpu Liu, Weichu Xie +12VLM Robustness Adversarial Attacks on VLMs
The University of Tokyo · Peking University · Wuhan University +4

Feb 27, 2026J/K move · Enter open · S save

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models

Xingyu Zhu, Beier Zhu, Junfeng Fang, Shuo Wang+3

MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore+1

VLM Safety Alignment VLM Safety VLM Robustness

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have achieved remarkable progress in vision-language reasoning tasks, yet ensuring their safety remains a critical challenge. Recent input-side defenses detect unsafe images with CLIP and prepend safety prefixes to prompts, but they still suffer from inaccurate detection in complex scenes and unstable safety signals during decoding. To address these issues, we propose GuardAlign, a training-free defense framework that integrates two strategies. First, OT-enhanced safety detection leverages optimal transport to measure distribution distances between image patches and unsafe semantics, enabling accurate identification of malicious regions without additional computational cost. Second, cross-modal attentive calibration strengthens the influence of safety prefixes by adaptively reallocating attention across layers, ensuring that safety signals remain consistently activated throughout generation. Extensive evaluations on six representative MLLMs demonstrate that GuardAlign reduces unsafe response rates by up to 39% on SPA-VL, while preserving utility, achieving an improvement on VQAv2 from 78.51% to 79.21%.

Abstract

VLM Safety Alignment

Explore similar work

GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models

Jailbreaking Vision-Language Models Through the Visual Modality

SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning