OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences

Ming Wen; Kun Yang; Jingyu Zhang; Yuxuan Liu; shiwen cui; Shouling Ji; Xingjun Ma

cs.AIMar 10, 2026

OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences

Authors: Ming Wen, Kun Yang, Jingyu Zhang, Yuxuan Liu, shiwen cui, Shouling Ji, Xingjun Ma

Organizations: Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Ant Group, Hangzhou, China · Zhejiang University, Hangzhou, China

Abstract

While safety alignment for Multimodal Large Language Models (MLLMs) has gained significant attention, current paradigms primarily target malicious intent or situational violations. We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents. To formalize this shift, we introduce OOD-MMSafe, a benchmark comprising 455 curated query-image pairs designed to evaluate a model's ability to identify latent hazards within context-dependent causal chains. Our analysis reveals a pervasive causal blindness among frontier models, with the highest 67.5% failure rate in high-capacity closed-source models, and identifies a preference ceiling where static alignment yields format-centric failures rather than improved safety reasoning as model capacity grows. To address these bottlenecks, we develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards. Experimental results demonstrate that CASPO significantly enhances consequence projection, reducing the failure ratio of risk identification to 7.3% for Qwen2.5-VL-7B and 5.7% for Qwen3-VL-4B while maintaining overall effectiveness.

Figures & tables

Figure 1: Comparison of MLLM safety paradigms and reasoning depth. (Left) Comparison across intent-, situational-, and consequence-driven dimensions through representative scenarios. (Right) Evolution of safety depth from intent detection to causal projection.

Category	Samples	Ratio (%)
I. Violent Content	128	28.13
$\bullet$ Animal Abuse	44	9.67
$\bullet$ Child Abuse and Neglect	36	7.91
$\bullet$ Property Vandalism	30	6.59
$\bullet$ Psychological Abuse	18	3.96
II. Self-Harm	116	25.49

Table 1: Data Distribution of OOD-MMSafe.

Closed-source Models
Model	Standard Mode						Malicious Mode						Constitution Mode
Model	$R_{A}\uparrow$	$R_{0}\downarrow$	$S_{A}\uparrow$	$S_{0}\downarrow$	$E_{A}\uparrow$	$E_{0}\downarrow$	$R_{A}\uparrow$	$R_{0}\downarrow$	$S_{A}\uparrow$	$S_{0}\downarrow$	$E_{A}\uparrow$	$E_{0}\downarrow$	$R_{A}\uparrow$	$R_{0}\downarrow$	$S_{A}\uparrow$	$S_{0}\downarrow$	$E_{A}\uparrow$	$E_{0}\downarrow$
Gemini-3-Pro-Think	1.12	40.9	1.09	42.4	1.71	1.1	1.85	5.7	1.87	5.4	1.83	1.8	1.70	12.7	1.70	13.0	1.91	0.2
Gemini-3-Pro	1.35	29.7	1.31	30.8	1.79	0.7	1.88	3.5	1.93	3.1	1.73	2.9	1.79	9.2	1.80	8.6	1.94	0.0
GPT-5.1	1.18	34.5	1.17	37.8	1.85	0.4	1.96	1.5	1.96	1.3	2.00	0.0	1.82	7.7	1.83	7.5	1.98	0.0
Doubao-1.6	0.69	61.5	0.67	64.0	1.58	2.9	1.80	6.8	1.82	7.3	1.90	1.1	1.42	24.0	1.48	22.4	1.78	0.7

Table 2: Comprehensive evaluation results on OOD-MMSafe. Standard uses raw queries directly; Malicious employs queries rewritten with explicit malicious intent; Constitution utilizes category-specific safety policies. Metrics

R,S,\text{ and }E

respectively assess risk identification, safety of consequence, and effectiveness.

X_{A}

and

X_{0}

denote average scores (

\uparrow

) and zero-score percentages (

\downarrow

). Within each group (Closed-source and Open-source models), bold and underline highlight the best and the second-best results, respectively.

Models	SIUO		MSS-Bench		OOD-MMSafe
Models	Safe $\uparrow$	Eff. $\uparrow$	Safe $\uparrow$	Eff. $\uparrow$	$R_{A}\uparrow$	$R_{0}\downarrow$	$S_{A}\uparrow$	$S_{0}\downarrow$	$E_{A}\uparrow$	$E_{0}\downarrow$
Qwen2.5-VL-7B	34.33 +00.0	78.44 +00.0	66.10 +00.0	91.94 +00.0	0.24 +00.0	82.6 +00.0	0.29 +00.0	82.4 +00.0	1.10 +00.0	3.5 +00.0
+ SPAVL	36.14 +1.8	80.84 +2.4	52.36 -13.7	92.28 +0.3	0.23 -.01	84.4 +1.8	0.27 -.02	83.3 +0.9	1.07 -.03	4.2 +0.7
+ Beavertails-V	58.43 +24.1	89.22 +10.8	73.02 +6.9	89.75 -2.2	0.41 +0.17	70.5 -12.1	0.50 +0.21	69.9 -12.5	1.18 +0.08	1.3 -2.2
+ CASPO (OutR)	72.41 +38.1	88.02 +9.6	79.63 +13.5	90.83 -1.1	1.72 +1.48	10.3 -72.3	1.83 +1.54	6.8 -75.6	1.93 +0.83	0.2 -3.3
+ CASPO (TokR)	83.23 +48.9	91.61 +13.2	92.26 +26.2	88.08 -3.9	1.80 +1.56	6.8 -75.8	1.82 +1.53	6.2 -76.2	1.98 +0.88	0.2 -3.3

Table 3: Comparison of safety alignment performance. OOD-MMSafe evaluation dimensions consist of Risk Appraisal (

R

), Safety of Consequences (

S

), and Effectiveness (

E

\cdot_{A}

denotes average score (

\uparrow

\cdot_{0}

is zero-score sample percentage (

\downarrow

). Main numbers in bold and underline are the best and second-best per group. Subscripts represent the relative gain/loss compared to the base model. The variants OutR, TokR, and Hybrid represent CASPO optimized with outcome-based, token-level, and combined rewards respectively.

Figure 6: Ablation study on constitution internalization and distillation models. We evaluate the internalization of safety constitutions using two distillation targets: the Original Model (OMC) and the SFT Model (SMC). Compared to the SFT baseline, both CASPO-w.-OMC and CASPO-w.-SMC show significant improvements in $R_{A},S_{A}$ , and $E_{A}$ .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Condition	Dimension R				Dimension S				Dimension E
Model	Condition	$A\uparrow$	$0(\%)\downarrow$	$1(\%)$	$2(\%)\uparrow$	$A\uparrow$	$0(\%)\downarrow$	$1(\%)$	$2(\%)\uparrow$	$A\uparrow$	$0(\%)\downarrow$	$1(\%)$	$2(\%)\uparrow$
	Standard	0.69	61.5	8.4	30.1	0.67	64.0	5.5	30.5	1.58	2.9	36.5	60.7
	Caption	0.62	64.0	9.7	26.4	0.61	66.4	6.6	27.0	1.47	2.4	48.4	49.2
	Constitution	1.42	24.0	9.9	66.2	1.48	22.4	6.8	70.8	1.78	0.7	21.1	78.2
Doubao-1.6	Malicious	1.80	6.8	5.9	87.3	1.82	7.3	3.1	89.7	1.90	1.1	7.5	91.4
	Standard	1.35	29.7	5.7	64.6	1.31	30.8	7.5	61.8	1.79	0.7	19.8	79.6

Appendix

Table 5: Detailed evaluation results of Closed-source Models on OOD-MMSafe.

R,S,E

represent Reward, Safety, and Ethical dimensions. For each, we report the Average Score (

A\uparrow

) and percentage of samples scoring 0 (

0\downarrow

), 1, and 2. All values are averaged across GPT and Qwen evaluators.

Model	Condition	Dimension R				Dimension S				Dimension E
Model	Condition	$A\uparrow$	$0\downarrow$	$1$	$2\uparrow$	$A\uparrow$	$0\downarrow$	$1$	$2\uparrow$	$A\uparrow$	$0\downarrow$	$1$	$2\uparrow$
	Standard	0.88	51.0	9.9	39.1	0.82	55.6	7.0	37.4	1.70	2.9	24.2	73.0
	Caption	0.88	50.1	11.6	38.2	0.79	56.5	7.7	35.8	1.68	1.8	28.6	69.7
	Constitution	1.58	17.6	6.6	75.8	1.62	17.6	3.1	79.3	1.85	0.2	14.5	85.3
Qwen3-VL-32B	Malicious	1.85	4.8	5.7	89.5	1.90	4.0	1.8	94.3	1.90	3.5	2.9	93.6
	Standard	0.87	52.5	8.4	39.1	0.91	50.8	7.9	41.3	1.61	3.7	31.2	65.1

Appendix

Table 6: Detailed evaluation results of Open-source Models on OOD-MMSafe.

R,S,E

represent Reward, Safety, and Ethical dimensions. For each, we report the Average Score (

A\uparrow

) and percentage of samples scoring 0 (

0\downarrow

), 1, and 2. All values are averaged across GPT and Qwen evaluators.

Model	Alignment	Mode	Dimension R				Dimension S				Dimension E
Model	Alignment	Mode	$A\uparrow$	$0\downarrow$	$1$	$2\uparrow$	$A\uparrow$	$0\downarrow$	$1$	$2\uparrow$	$A\uparrow$	$0\downarrow$	$1$	$2\uparrow$
		Std	0.09	92.3	5.9	1.8	0.15	89.7	6.2	4.2	0.86	16.5	81.3	2.2
	Base	Mal	0.64	54.5	27.0	18.5	0.57	70.1	2.9	27.0	1.02	22.9	52.7	24.4
		Std	0.30	78.2	13.4	8.4	0.36	76.9	9.7	13.4	1.08	3.1	86.2	10.8
	Beavertails	Mal	1.49	15.6	20.0	64.4	1.49	20.9	8.8	70.3	1.55	6.2	33.2	60.7
		Std	0.19	84.0	13.2	2.9	0.34	79.3	7.3	13.4	1.05	3.7	87.3	9.0

Appendix

Table 7: Comparative evaluation of models before and after safety alignment using Beavertails and SPAVL datasets. Results are shown for Standard (Std) and Malicious (Mal) modes.

A

denotes Average Score (

\uparrow

), while

0,1,2

represent the percentage of samples in each score category. All metrics are averaged across GPT and Qwen evaluators.

Figure 7: Per-category Results for four frontier close-source models.

Figure 8: Per-category Results for four frontier open-source models.

Panel of Figure 9 — Figure 9: Model evaluation results (Part I)

Panel of Figure 10 — Figure 10: Model evaluation results (Part II)

Major Policy	Beavertails (BT)	OOD-MMSafe	Total	Ratio (%)
Violent Content	2023	480	2503	38.02%
Illegal Activity	1209	311	1520	23.09%
Hate Speech	504	281	785	11.92%
Sexual Content	506	105	611	9.28%
Privacy Violation	505	94	599	9.10%
Self-Harm	253	312	565	8.58%

Appendix

Table 8: Composition and distribution of the training dataset across six major safety categories.

Models	R-Metrics				S-Metrics				E-Metrics
Models	Avg	0%	1%	2%	Avg	0%	1%	2%	Avg	0%	1%	2%
Qwen2.5-VL-7B
Original-Model (Policy)	0.52	64.4	18.9	16.7	0.60	66.8	6.8	26.4	1.24	2.4	70.8	26.8
CASPO-w.-OMP (Standard)	0.55	64.8	14.9	20.2	0.58	65.5	11.4	23.1	1.60	0.0	39.6	60.4
SFT-Model (Standard)	1.02	46.6	5.3	48.1	1.07	42.4	7.7	49.9	1.85	1.5	11.4	87.0
SFT-Model (Policy)	1.71	9.7	9.5	80.9	1.76	9.2	5.9	84.8	1.95	0.7	3.3	96.0

Appendix

Table 9: Detailed evaluation results of safety alignment across different metrics.

R,S,E

denote three evaluation dimensions. For each dimension, we report the average score (Avg) and the percentage of samples with scores 0%, 1%, and 2%.

Models	SIUO		MSS-Bench		OOD-MMSafe
Models	Safe $\uparrow$	Eff. $\uparrow$	Safe $\uparrow$	Eff. $\uparrow$	$R_{A}\uparrow$	$R_{0}\downarrow$	$S_{A}\uparrow$	$S_{0}\downarrow$	$E_{A}\uparrow$	$E_{0}\downarrow$
Qwen2.5-VL-7B
CASPO ( $\lambda=0$ )	72.41	88.02	79.63	90.83	1.72	10.3	1.83	6.8	1.93	0.2
CASPO ( $\lambda=0.3$ )	88.02	92.81	89.91	92.95	1.80	7.3	1.82	7.0	1.98	0.0
CASPO ( $\lambda=0.6$ )	84.93	91.01	89.76	88.42	1.73	7.9	1.74	9.5	1.97	0.4
CASPO ( $\lambda=1.0$ )	80.84	91.02	88.76	88.59	1.77	8.1	1.80	7.3	1.97	0.7

Appendix

Table 10: Sensitivity analysis of the correction strength parameter

\lambda

across different benchmarks. For SIUO and MSS-Bench, we report Safe and Effective Rates (%). For OOD-MMSafe,

R,S,E

denote Risk, Safety, and Effectiveness dimensions, where Avg is the average score and 0% is the zero-score sample ratio.

Figure 11: Per-category Results for Alignment results of Qwen-2-vl-7B.

Figure 12: Per-category Results for Alignment results of Qwen-3-vl-4B.

Figure 13: Training indicators during grpo RL process.

Figure 14: Visualization case of CASPO results on Qwen2.5-VL (Part I)

Figure 15: Visualization case of CASPO results on Qwen2.5-VL (Part II)

Figure 16: Visualization case of CASPO results on Qwen3-VL (Part I)

Figure 17: Visualization case of CASPO results on Qwen3-VL (Part II)

MLLM Safety

See all 28 papers

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
Aug 8, 2026Yilian Shi, Yueming Lyu, Haoxiang Tan +5LLM Safety Benchmarks Multimodal Robustness
Nanjing University
ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Models
Sep 29, 2026Ruochen Zhang, Yao Huang, Yitong Sun +5Multimodal Reasoning LLM Safety
Beihang University Beijing, China · Tsinghua University Beijing, China · Institute of Artificial Intelligence Beihang University Beijing, China State Key Laboratory of AI Safety
Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory
Mar 16, 2026Ce Zhang, Jinxi He, Junyi He +2Multimodal Large Language Models Language Model Safety Evaluation
Robotics Institute, Carnegie Mellon University

Aug 8, 2026J/K move · Enter open · S save

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs

Yilian Shi, Yueming Lyu, Haoxiang Tan, Linzhuang Zou+4

Nanjing University

LLM Safety Benchmarks Multimodal Robustness Multimodal Large Language Models LLM Safety Evaluation

Open paper AskPDF arXiv

While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easily bypass unimodal filters. Existing benchmarks lack fine-grained intent-related annotations and rely on unidimensional metrics, hindering comprehensive robustness evaluation. To address this, we propose MME-Safety, a rigorously verified benchmark featuring a unique four-dimensional annotation schema that categorizes risk scenarios, harm severity, and modality-specific stealth levels. Furthermore, we introduce a hierarchical evaluation framework to assess fundamental response reliability, actual risk exposure, and the structural integrity of defensive behaviors. Extensive zero-shot evaluations across 17 state-of-the-art MLLMs provide a comprehensive safety profile of current multimodal systems. Our analysis systematically investigates cross-modal input configurations and uncovers safety implications associated with Chain-of-Thought (CoT) reasoning. These multifaceted findings underscore the urgent need for robust, reasoning-aware safety alignment in the multimodal landscape.

Abstract

MLLM Safety

Explore similar work

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs

ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language Models

Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory