Accurate rejection of sensitive or harmful visual content, i.e., harmful image guardrail, is critical in many application scenarios. This task must continuously adapt to the evolving safety policies and content across various domains and over time. However, traditional classifiers, confined to fixed categories, require frequent retraining when new policies are introduced. Vision-language models (VLMs) offer a more adaptable and generalizable foundation for dynamic safety guardrails. Despite this potential, existing VLM-based safeguarding methods are typically trained and evaluated under only a fixed safety policy. We find that these models are heavily overfitted to the seen policy, fail to generalize to unseen policies, and even lose the basic instruction-following ability and general knowledge. To address this issue, in this paper we make two key contributions. First, we benchmark the cross-policy generalization performance of existing VLMs with SafeEditBench, a new evaluation suite. SafeEditBench leverages image-editing models to convert unsafe images into safe counterparts, producing policy-aligned datasets where each safe-unsafe image pair remains visually similar except for localized regions violating specific safety rules. Human annotators then provide accurate safe/unsafe labels under five distinct policies, enabling fine-grained assessment of policy-aware generalization. Second, we introduce SafeGuard-VL, a reinforcement learning-based method with verifiable rewards (RLVR) for robust unsafe-image guardrails. Instead of relying solely on supervised fine-tuning (SFT) under fixed policies, SafeGuard-VL explicitly optimizes the model with policy-grounded rewards, promoting verifiable adaptation across evolving policies. Extensive experiments verify the effectiveness of our method for unsafe image guardrails across various policies.
Figures & tables
Figure 1: High-level illustration of our SafeGuard-VL. Unlike prior guardrails that fit only the fixed safety policy, SafeGuard-VL is designed from the perspective of cross-policy adaptability and robustness . In Stage 1 (SFT), the model learns general unsafe-related visual and textual semantics through data constructed using our self-recaption mechanism. In Stage 2 (RL), the model is optimized to perform policy-aware safe/unsafe discrimination, adapting its decisions to different policy definitions rather than relying on a single fixed rule set. This two-stage framework enables SafeGuard-VL to generalize to unseen or shifting safety policies during testing.
Figure 2: Examples from the proposed SafeEditBench dataset. Our key innovation lies in constructing semantically aligned safe-unsafe image pairs where the global visual semantics remain unchanged , while only the minimal unsafe regions are locally edited using precise image-editing operations. This produces safe counterparts that preserve the original scene, composition, and objects, altering solely the safety-violating content. Such fine-grained, locality-preserving edits make SafeEditBench highly challenging: models must accurately identify and reason about the specific unsafe elements rather than relying on coarse, scene-level cues.
Figure 3: The proposed novel self-recaption mechanism that lets the model generate and refine its own captions. Specifically, the baseline model (Qwen-VL) first produces a high-level description with less unsafe details, sampled from its own distribution. The recaption model (Gemma 27B) then performs minimal edits to this caption by recovering the suppressed unsafe semantics, producing a caption with more unsafe details that preserves the original structure while adding explicit harmful descriptions.
Figure 4: The statistics of the five policy levels in SafeEditBench , showing how the same image set is labeled differently under varying safety policies. From L1 (most permissive) to L5 (most restrictive), each policy defines different categories of violation. Policies L3 and L4 reflect widely accepted societal norms, while L1 and L5 represent most counterintuitive regimes designed to test policy adherence.
Figure 5: Examples showing that “safety” is fundamentally policy-dependent rather than common-sense–dependent. The same image may be judged “Safe” or “Unsafe” under different policies, especially when the policies adopt counterintuitive or non–common-sense definitions of safety (e.g., prohibiting ordinary affection while allowing sexually suggestive content). These examples highlight the core challenge: safety labels are not intrinsic to the image but are also determined by the specific policy applied.
Method / Benchmark
Policy Source
#Categories
Policy Adaptation Mechanism
Llama Guard [ 14 ]
Meta textual hazards
Fixed (14)
Category exemption; Structural changes need retraining
LlavaGuard [ 16 ]
O1–O9 visual taxonomy
Fixed (9)
Category exemption; adjust rules within fixed taxonomy; no new categories/entries
ShieldGemma [ 50 ]
Google’s responsible AI toolkit
Fixed (6)
Prompt modification; threshold tuning
OpenAI Mod [ 27 ]
US law-focused
Fixed (hierarchical)
Not user-customizable; designed as a single, powerful model
SafeWatch [ 5 ]
Laws & platform rules
Policy-specific
Accepts natural language policy descriptions (via PEPE/PAP); unreleased yet
AIR-BENCH [ 51 ]
Aggregated real-world policies
Fixed (314 blocks)
Select from 314 predefined blocks, cannot handle unseen risks
Table 1: Comparison of policy adaptation mechanisms across existing safety guardrails and benchmarks. Existing methods rely on fixed taxonomies or pre-defined blocks with limited adaptation flexibility, whereas our method supports arbitrary natural language policies with zero-shot cross-policy generalization.
Model
Hate
Violence
Self-Harm
Sexual
Shocking
Illegal
Deception
Political
Spam
Overall
Traditional Classifier
NudeNet
–
–
–
62.4
–
–
–
–
–
–
NSFW_Detector
–
–
–
73.8
–
–
–
–
–
–
MultiHeaded
29.2
42.6
–
75.7
74.9
–
–
60
–
–
SD_Filter
–
–
–
78.5
–
–
–
–
–
–
General Purpose
Qwen2.5-7B
24.5
69.1
55.3
35.5
47.2
37.5
33.9
23.3
23
41.7
LLaVA-V1.6-7B
25.3
57
57.9
41.4
72.2
52.1
54.9
66.7
6.5
52
Table 2: Cross-policy generalization performance comparison on UnsafeBench [ 34 ] across 9 harmful categories. Results show significant improvements over general-purpose models and the safety-focused Qwen-Guard-7B baseline. Results of other baselines are directly cited.
Policy Level
L1
L2
L3
L4
L5
Qwen2.5-7B
47.46
20.59
37.36
70.87
70.34
SFT on L1
100
0
0
0
0
RL on L1
50
20.59
35
70.97
65.69
SFT on L4
62.71
14.55
41.03
73.68
58.41
RL on L4
43.22
19.18
38.64
75.2
73.61
SFT on L5
40.68
19.18
40.96
73.02
84.35
Table 3: Policy adaptability analysis on our challenging SafeEditBench. The model is trained at a single policy level (L1-L5) and evaluated at all five levels. Training on extreme policies (e.g., L1 or L5) results in a significant performance drop on other policies, revealing a key limitation: current safety guardrail methods lack basic cross-policy generalization ability.
Safety Bench
General Bench
Model
LlavaGuard
Unsafebench
SafeEditBench
Overall
MMMU
RealWorldQA
BLINK
MMT
Overall
Qwen2.5-7B
57.08
41.71
48.68
49.16
45
68.5
54.66
59.55
56.92
QwenGuard-7B
84.57
43.56
32.76
53.63
36
57
12.05
38.89
35.98
Ours (RL)
71.78
62.39
45.59
59.92
45.33
68.37
53.6
60.76
57.02
Table 4: Performance comparison across safety and general VQA benchmarks. QwenGuard-7B achieves high scores on its own LlavaGuardBench but suffers significant degradation on other safety (UnsafeBench) and general benchmarks. In contrast, with the same training data, simply changing to RL training improves performance on both safety and general benchmarks, demonstrating better generalization and avoiding the drawbacks of over-specialization in existing safety models.
Figure 6: Comparison of safety vs. general capability trade-off. Left: QwenGuard exhibits a large gap between its proprietary benchmark (84.6) and other safety/general benchmarks (43.6, 36.0). Right: Our SafeGuard-VL-RL maintains balanced performance across safety (71.8, 62.4) and general tasks (57.0), demonstrating superior safety ability without sacrificing general capacity. The general score is the average of MMMU, RealWorldQA, BLINK, and MMT-Bench.
Figure 7: Qualitative comparison highlighting two key advantages of SafeGuard-VL-RL over the existing method QwenGuard [ 16 ] . (1) Policy-aware safety judgment: Under Policy L2, which explicitly allows historical or educational firearm displays, QwenGuard incorrectly marks a museum exhibit as unsafe, failing to incorporate policy context. In contrast, SafeGuard-VL-RL correctly interprets the image within the allowed educational setting and labels it as safe. (2) Robust instruction following: When given a simple multiple-choice question, QwenGuard ignores the user instruction and outputs a long JSON-style safety rationale. SafeGuard-VL-RL, however, adheres strictly to the required format and returns only the correct option (“B”), demonstrating reliable reasoning and faithful instruction compliance.
Variant
Recap
RL
Unsafebench
General
Qwen2.5-7B
–
–
41.71
56.92
w/o Recap (SFT)
×
×
53.22
54.51
Ours (SFT)
✓
×
66.96
53.37
Ours (Full)
✓
✓
72.16
53.09
Table 5: Ablation study on the effectiveness of recaption and RL training. Removing recaption (w/o Recap) leads to a drop in safety performance, confirming that our carefully designed captions help the model learn fine-grained harmful patterns. Further applying RL after SFT yields the best performance on UnsafeBench (+5.2 over SFT-only), validating our two-stage training strategy. General capability remains stable across variants.
Model
Policy L1
Policy L2
Policy L3
Policy L4
Policy L5
Overall
Qwen2-VL-7B
65.25
15.38
24
60.87
44
36.65
GLM-4V-9B
0
14.68
27.54
67.95
73.37
41.16
Qwen2.5-VL-7B
47.46
20.59
37.36
70.87
70.34
48.68
QwenGuard-3B
0
0
1.63
22.39
0
5.19
Llama Guard
100
0
0
42.37
0
22.73
ShieldGemma2
44.92
5.63
14.12
37.29
51.06
27.50
Table 6: Performance (F1-score %) on SafeEditBench under five policy levels. Our method trained on the SafeEdited data outperforms both general-purpose models and QwenGuard models.
Figure 8: Qualitative comparison. We visualize the progression of SafeGuard-VL. The SafeGuard-VL-SFT stage reveals the model’s ability to comprehend harmful semantics through detailed description, while SafeGuard-VL-Full successfully translates this understanding into a precise refusal. In contrast, general-purpose models fail to detect risks. (Sensitive images are replaced by Nano Banana generations).
Figure 9: Performance comparison of the cross-policy generalization , where all methods are evaluated on UnsafeBench [ 34 ] across 10 harmful categories with previously unseen data and policy. Scores reflect the F1-score for binary classification (safe vs. unsafe). Higher scores indicate better performance.
Image guardrails are typically trained and evaluated under a fixed safety policy, implicitly treating safety as an intrinsic property of an image. Real deployments are different: the same image may be allowed in one product, restricted in another, and newly disallowed when a policy boundary changes. We study policy-adaptive image guardrailing, where a model must decide whether an image violates the currently supplied policy and generalize to held-out policy definitions. We introduce PolicyShiftBench, a comprehensive benchmark with 2,000 policy-discriminative instances over 265 images, where each image is paired with 7.55 policy-conditioned prompts on average to test whether models adapt to the active policy rather than relying on image-level safety priors. We then propose PolicyShiftGuard, a compact policy-conditioned guardrail trained with a two-stage training recipe that combines Randomized Policy SFT (RP-SFT) with Boundary-Pair Policy Adaptation (BP-Adapt). BP-Adapt trains matched prompts for the same image and risk category using standard label supervision and a pairwise comparison loss that separates blocking policies from passing policies. Experiments show that existing VLMs and specialized guardrails remain brittle under policy shifts, while PolicyShiftGuard substantially improves policy-sensitive performance. The 7B model achieves SOTA performance of 76.9 Avg. F1 and 72.1 Avg. PSS on PolicyShiftBench, transfers well to UnSafeBench and SafeEditBench, and improves the latency-performance trade-off with a concise output format. Ablations confirm that matched pass/block boundary pairs are essential for stable policy adaptation.
Mingyang Song, Luxin Xu, Haoyu Sun +3
1Fudan University · 2Tongji University · 3Virtue AI +3
Vision-language models (VLMs) are increasingly deployed in consumer, medical, financial, and enterprise applications. This broad deployment expands the safety surface: risks can arise from multimodal question answering, assistant responses, and cross-modal composition, while moderation policies may vary across products, regions, and deployment stages. Most existing guardrails either rely on fixed taxonomies or target only a narrow set of interaction settings, which limits their adaptability when safety rules change at deployment time. We present SingGuard, a policy-adaptive multimodal guardrail model family for safety assessment in multimodal conversations. SingGuard treats the active policy as a runtime input: given natural-language rules, it checks the target content against the active policy rule by rule and predicts both the safety label and the triggered rule. To balance efficiency and interpretability, SingGuard supports fast, hybrid, and slow inference regimes along a fast-to-slow reasoning spectrum, ranging from direct safety judgments to policy-grounded deliberation. We further optimize this behavior with fast--slow decoupled reinforcement learning. We also introduce SingGuard-Bench, a multimodal guardrail benchmark with 56{,}340 examples spanning 80+ fine-grained risk types across multimodal QA, adversarial attack, and dynamic-rule evaluation settings, including cross-modal joint-risk cases where each modality is harmless in isolation but their composition implies unsafe intent. Across six benchmark families (35 datasets), SingGuard achieves state-of-the-art average F1 in every family. Dynamic-rule evaluation further shows improved policy-following accuracy from 0.6465 to 0.7415 under runtime policy shifts. Our code is available at https://github.com/inclusionAI/Sing-Guard.
Large vision-language models (LVLMs) have achieved remarkable progress in vision-language reasoning tasks, yet ensuring their safety remains a critical challenge. Recent input-side defenses detect unsafe images with CLIP and prepend safety prefixes to prompts, but they still suffer from inaccurate detection in complex scenes and unstable safety signals during decoding. To address these issues, we propose GuardAlign, a training-free defense framework that integrates two strategies. First, OT-enhanced safety detection leverages optimal transport to measure distribution distances between image patches and unsafe semantics, enabling accurate identification of malicious regions without additional computational cost. Second, cross-modal attentive calibration strengthens the influence of safety prefixes by adaptively reallocating attention across layers, ensuring that safety signals remain consistently activated throughout generation. Extensive evaluations on six representative MLLMs demonstrate that GuardAlign reduces unsafe response rates by up to 39% on SPA-VL, while preserving utility, achieving an improvement on VQAv2 from 78.51% to 79.21%.
Xingyu Zhu, Beier Zhu, Junfeng Fang +4
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University · National University of Singapore +1