Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation

Kaichao Jiang; Changtao Miao; Baiqi Wu; Zhiyuan Lu; Kang Yang; Peiwei Zhao; Junchi Chen; Yunfeng Diao

cs.AIAug 27, 2026

Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation

Authors: Kaichao Jiang, Changtao Miao, Baiqi Wu, Zhiyuan Lu, Kang Yang, Peiwei Zhao, Junchi Chen, Yunfeng Diao, +3 more

Organizations: University of Science and Technology of China · Independent Researcher · Zhejiang University · Hefei University of Technology

Abstract

Recent audio-video generators increasingly support joint conditioning on text, images, audio, and video. These capabilities also enable attacks that exploit cross-modal interactions or obscure harmful intent to bypass safeguards and induce harmful audio-video outputs. However, existing generation-safety benchmarks have not kept pace with these advances, providing limited coverage of multimodal input combinations and obscured attack intents. To address these gaps, we introduce Multi2AV-Safety, the first full-coverage red-team benchmark for multimodal-to-audio-video generation, comprising 11,024 attack instances across all 11 non-singleton T/I/A/V conditioning configurations, 4 attack-intent categories, and 5 harm categories. Our evaluation of recent state-of-the-art models, including four multimodal-conditioned audio-video generators and eight safety guards, reveals substantial vulnerabilities in both generation and safeguarding, with multimodal compositional risk and obscured attack-intent risk emerging as two complementary challenges. Guided by these findings, we introduce PerceptGuard, an omni-modal guard integrating compositional-risk and attack-intent supervision through structured risk perception learning. By jointly training rationale generation and safety classification, it learns shared risk representations that enable a safety head to make efficient predictions at inference without rationale decoding, while retaining the ability to generate explanations on demand. Across 34 safety benchmarks, PerceptGuard combines SOTA multimodal safety detection with highly competitive unimodal performance, strengthening input-side safeguards against multimodal attacks on omni models. In particular, it improves safeguarding against the above risks, achieving an overall recall of 86.06% on Multi2AV-Safety and outperforming GuardReasoner-Omni by 14.56%.

Figures & tables

Input

Attack intent

Harm structure

Benchmark / Dataset

Output

Direct

Jail.

Adv.

Temp.

Full

Diluted

Composed

I2P ( Schramowski et al., 2023 )

\checkmark

–

\checkmark

–

N/A

T2ISafety ( Li et al., 2025 )

\checkmark

–

\checkmark

–

N/A

T2I-RiskyPrompt ( Zhang et al., 2026 )

\checkmark

–

\checkmark

\checkmark

\checkmark

–

N/A

JailbreakDiffBench ( Jin et al., 2025 )

\checkmark

–

I/V

–

\checkmark

\checkmark

–

N/A

MPJ ( Liu et al., 2025 )

\checkmark

–

\checkmark

–

N/A

Table 1: Comparison of representative safety benchmarks for generative models. Full, Diluted, and Composed denote harmful multimodal requests with all, some but not all, or none of their individual modality inputs harmful in isolation, respectively.

Figure 2: Construction pipeline of Multi2AV-Safety . Seeds for four attack-intent categories and their benign counterparts are realized across all 11 conditioning configurations, then validated through model screening and expert review.

Config.	Attack intent counts (#)					Generator ASR (%) $\downarrow$				Input-side Guard Recall (%) $\uparrow$
Direct	Jail.	Adv.	Temp.	$N$		[-1pt]	[-1pt]	[-1pt] davinci	Qwen3-Omni	GR-Omni	Ours

T+I	900	749	100	100	1,849	86.75	37.37	76.85	81.48	58.46	71.07	85.18
T+A	900	400	300	100	1,700	89.24	52.59	–	70.07	66.82	70.00	83.12
T+V	900	200	100	100	1,300	92.00	41.85	76.08	–	76.00	68.85	80.38
I+A	900	400	100	100	1,500	99.07	20.93	64.87	85.36	55.53	78.67	96.53

Table 2: Multimodal-conditioned audio-video generation safety and omni-guard detection across multimodal input combinations in Multi2AV-Safety .

N

denotes the number of requests, and Ours denotes PerceptGuard. Non-omni guard results are in Appendix E.6 .

Figure 3: Guard detection of compositional risks. Across all panels, the two-, three-, and four-modal requests are derived from a shared pool of seed scenarios. (a) Recall on Composed Harm ( $k=0$ ) across input modality counts. (b–d) Recall on Direct attacks with two, three, and four modalities, respectively, grouped by the number of individually harmful modality inputs $k$ . In (b–d), $0<k<|S|$ denotes Diluted Harm , while $k=|S|$ provides the Full Harm reference.

Figure 4: Overview of PerceptGuard , which learns shared safety representations through compositional risk and attack intent supervision, enabling direct prediction and on-demand explanations.

Training objectives				Supervised outputs		32-task F1 (%) $\uparrow$		Omni recall (%) $\uparrow$
$\mathcal{L}_{\mathrm{cls}}$	$\mathcal{L}_{\mathrm{rel}}$	$\mathcal{L}_{\mathrm{gen}}$	$\mathcal{L}_{\mathrm{intent}}$	Rationale generation	Intent recognition	Request	Response	Omni- SafetyBench	Multi2AV- Safety
$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	87.44	74.05	99.35	71.43
$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\times$	88.60	73.39	99.13	76.24
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$	89.01	72.56	99.37	82.18
[0pt][0.2pt] $\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	88.98	74.83	99.12	86.06

Table 5: Ablation of PerceptGuard. F1 is sample-weighted by task; recall uses full-benchmark denominators. All variants use safety-head inference without rationale decoding.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Axis	Types and variations
Modality	T: existing safety datasets, adapted and refined with Claude Opus 4.6; I: Stable Diffusion 1.5 ( Rombach et al., 2022 ) / HiDream-O1-Image ( Cai et al., 2026 ) ; A: TTS by CosyVoice2 ( Du et al., 2024 ) / TTS-1 ( OpenAI, 2023 ) ; V: LTX-2 ( HaCohen et al., 2026 ) / daVinci-MagiHuman ( SII-GAIR et al., 2026 ) .
Conditioning	2-modal: T+I, T+A, T+V, I+A, I+V, A+V; 3-modal: T+I+A, T+I+V, T+A+V, I+A+V; 4-modal: T+I+A+V.
Attack intent	Direct: explicit harmful intent; Obscured: Jailbreak, Adversarial, and Temporal intents.
Harm evidence	Composed Harm ( $k=0$ ): individually benign inputs convey harm in the complete multimodal request; Diluted Harm ( $1\leq k<\|S\|$ ): explicit harmful evidence is embedded in benign context; Full Harm ( $k=\|S\|$ ): all active inputs are harmful in isolation.

Appendix

Table B.1: Construction taxonomy of Multi2AV-Safety , covering modalities, conditioning, attack intents, and harm evidence.

Figure 5: A successful attack scenario across four audio-video generators. Harmful text is combined with benign media, forming Diluted Harm . The shared attack scenario elicits harmful outputs from all four generators under their respective conditioning configurations. Representative output frames are shown with sensitive regions masked.

Table 10

Table 11

Explore similar work

Sep 7, 2026cs.CV

The Price of Consistency: Exploiting Visual Anchors for Multimodal Jailbreaking in Video Generation

The rapid evolution of video generation has shifted the paradigm from pure text-driven to multi-conditional controllable generation, with reference images now widely adopted as conditional inputs to achieve superior spatiotemporal consistency. While these reference images serve as powerful visual anchors that significantly enhance controllability, their impact on safety remains largely unexplored. In this work, we reveal the visual anchoring effect: by enforcing consistency, the mechanism prevents the generated content from drifting away from the original harmful intent, thereby eliminating the model's natural safety escape route from harmful to benign content. Consequently, visual anchors inherently increase the safety risk---this is the price of consistency. Building on this insight, we propose Decoupling Intent via Visual Anchors (DIVA), a training-free multimodal jailbreak framework for video generation that exploits this vulnerability. DIVA decouples harmful intent into a static visual anchor image and a dynamic motion text prompt, and employs dual-criteria selection to balance attack stealthiness with semantic preservation. Extensive experiments across various leading commercial platforms and mainstream open-source video generation models demonstrate that DIVA achieves a substantially higher Attack Success Rate than existing text-only methods. To facilitate future research, we additionally contribute TI2VSafetyBench, the first safety benchmark for multi-conditional video generation.

Peng Li, Qianqian Xu, Yangbangyan Jiang +2

School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China +1

Image-to-Video Generation Multimodal Jailbreak Attacks Controllable Video Generation

Ask PDF

Jun 24, 2026cs.CV

VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks

Recent advancements in Image-to-Video (I2V) generation have transformed input images from simple appearance references into interactive control interfaces where visual cues such as arrows, sketches, and emojis orchestrate complex video dynamics with unprecedented controllability. However, these seemingly innocuous static cues can be interpreted by models as executable temporal instructions, unfolding into harmful actions in the generated videos. Despite the severity of this threat, existing safety benchmarks remain predominantly focused on text-based and content-only image-based jailbreaks, leaving implicit visual prompt attacks insufficiently explored. To bridge this gap, we present VVA-Bench, the first systematic benchmark for evaluating video generation safety under categorized vision-centric prompt attacks. Extensive experiments on VVA-Bench demonstrate that state-of-the-art models are highly susceptible to such attacks, with Attack Success Rates (ASR) reaching 100.0% on Wan 2.7 and 74.8% on Veo 3.1. To mitigate these risks, we propose VPA-Guard, a retrieval-augmented and self-evolving defense framework. By leveraging few-shot reasoning to identify latent malicious intents, our method reduces the attack ASR by 44.2% and the harmfulness score by 73.4% on average, while maintaining the model's utility for legitimate user edits. Our work provides both a rigorous benchmark and an effective defense strategy to advance safe and socially responsible multimodal generation.

Yining Sun, Haoyu Kang, Jiajun Wu +7

1Tsinghua University · 3Central South University · 4South China Normal University +3

Image-to-Video Generation Video Generation Video Generation Evaluation

Ask PDF

May 31, 2026cs.CV

SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation

With the rapid advancements in text-to-image diffusion models, generative video models (T2V models) like Sora can now produce short synthetic videos from a text prompt or an initial image. However, synthetic video generation -- especially when guided by an initial image -- often poses risks, including the potential creation of illegal, politically sensitive, or unethical content. Existing benchmarks have started to consider the safety of generated videos, but they primarily focus on testing models with malicious text prompts, ignoring the scenario where text prompt and image combination may still lead to harmful video content. In practice, this is a common and challenging issue: videos generated from safe text and image inputs can nonetheless convey harmful information. To bridge this gap, we introduce SafeGen-Bench, a benchmark specifically designed to evaluate the safety of conditional T2V models. Our benchmark defines 10 malicious categories, concentrating on risks related to both temporal sequences and depicted behaviors. SafeGen-Bench consists of carefully selected start frames from diverse image and video sources, paired with corresponding text prompts to simulate realistic inputs. We evaluate a variety of conditional T2V models on SafeGen-Bench, and the results indicate that current models struggle to consistently avoid generating malicious content with unsafety scores reaching up to 44.5, especially under conditions requiring high quality. Furthermore, we assess the effectiveness of both text-based and image-based guardrails on our benchmark, finding that unimodal guardrails alone were insufficient to provide a robust defense, with an 80% failure rate across seven malicious categories. We hope that SafeGen-Bench will foster the development of safer and more controllable conditional T2V models.

Yingzi Ma, Xiaogeng Liu, Yawen Zheng +1

University of Wisconsin-Madison, Madison · Tsinghua University · Johns Hopkins University

Image-to-Video Generation Text-to-Video Generation Video Generation Video Generation Evaluation

Ask PDF