Omni-UBench: Benchmarking Omni-Modal LLM-based Guardrails Across Safety, Utility, and Reliability

Junhao Wang; Yidou Chen; Li Lu; Hongwei Yao; Xiaodan Li; Mengjie Wu; Yuefeng Chen; Zhan Qin

cs.CROct 22, 2026

Omni-UBench: Benchmarking Omni-Modal LLM-based Guardrails Across Safety, Utility, and Reliability

Authors: Junhao Wang, Yidou Chen, Li Lu, Hongwei Yao, Xiaodan Li, Mengjie Wu, Yuefeng Chen, Zhan Qin,

Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University · Alibaba Group

Abstract

Advances in Omni-modal Large Language Models (OLLMs) have enabled increasingly seamless human-AI interaction, but their unified processing of heterogeneous modalities also expands the attack surface, allowing adversaries to exploit richer input compositions to circumvent safety alignment. To counter such risks, omni-modal guardrails have been proposed to detect and filter unsafe queries across arbitrary modalities. However, their capabilities and limitations remain insufficiently understood due to the absence of a comprehensive evaluation benchmark. In this paper, we present Omni-UBench, the first comprehensive framework for omni-modal safety assessment, evaluating guardrail models along three critical dimensions, i.e., safety, utility, and reliability. Omni-UBench curates over 183k samples drawn from 35+ heterogeneous sources spanning text, image, audio, video, and their cross-modal combinations, annotated with safety labels under a unified risk taxonomy. Benchmarking under Omni-UBench on 15 representative OLLMs and omni-modal guardrails yields several key findings, including: (1) modern OLLMs exhibit strong content moderation potential even without guardrail-specific tuning; (2) existing guardrails still struggle to identify cross-modal queries that conceal harmful intent or involve sophisticated jailbreaks; (3) they further exhibit substantial safety hallucination and over-sensitivity to textual patterns, revealing a deficient understanding of cross-modal safety semantics. Our findings provide valuable insights for developing more trustworthy omni-modal guardrails in future research.

Figures & tables

Figure 1: Leaderboard of omni-modal safety guardrails on Omni-UBench . The balanced score provides a holistic view of guardrail performance across safety, utility, and reliability.

Benchmark	Venue	Modalities				Target	Include Jailbreak (#)	#Risks	Unsafe (%)	Data Scale
Benchmark	Venue	T	I	A	V	Target	Include Jailbreak (#)	#Risks	Unsafe (%)	Data Scale
OpenAI Moderation ( Markov et al. 2023 )	AAAI’23	✓	✗	✗	✗	Guardrail	NO	5	39	$\sim$ 1.6k
JailbreakBench ( Chao et al. 2024 )	NIPS’24	✓	✗	✗	✗	LLM	YES (5)	10	100	1.8k
GuardBench ( Bassani and Sanchez 2024 )	EMNLP’24	✓	✗	✗	✗	Guardrail	NO	NaN	59	$\sim$ 146k
VLGuard ( Zong et al. 2024 )	ICML’24	✓	✓	✗	✗	MLLM	NO	9	43	3k
MM-Safetybench ( Liu et al. 2024 )	ECCV’24	✓	✓	✗	✗	MLLM	YES (3)	13	100	$\sim$ 5k

Table 1: Comparison with representative safety benchmarks across modalities ( T : Text, I : Image, A : Audio, V : Video).

Figure 2: Overview of Omni-UBench . It unifies benign, unsafe, and jailbreak queries from 35+ public datasets spanning different modalities, annotated under a unified 10-class risk taxonomy via an LLM-assisted pipeline. It provides five evaluation modules assessing guardrail models along Safety, Utility, and Reliability to comprehensively reveal their capabilities and deficiencies.

Data Source	Metric	Q3	Q7	Q30	CPM	PHI	GEM	MIM	Q30 *	CPM *	PHI *	MG	GR3	GR7	OG3	OG7	Avg
TruthfulQA	Acc	0.38	0.91	0.96	0.99	0.83	0.91	0.99	0.86	0.97	0.88	0.87	0.97	0.99	0.98	0.98	0.90
MME	Acc	0.07	0.97	0.88	1.00	0.07	0.98	0.99	1.00	1.00	1.00	1.00	0.80	0.79	0.98	0.97	0.83
VoiceBench	Acc	0.37	0.95	0.96	0.99	0.69	0.97	0.98	0.95	0.99	0.99	0.98	0.77	0.63	0.97	0.98	0.88
MMBench-Video	Acc	0.56	0.96	0.96	0.99	/	0.86	0.98	0.99	0.98	/	1.00	0.41	0.53	0.93	0.95	0.85
$S_{\text{utility}}$		34.5	94.8	94.0	99.3	53.0	93.0	98.5	95.0	98.5	95.7	96.3	73.8	73.5	96.5	97.0	-
Beavertails	F1	0.90	0.92	0.92	0.85	0.92	0.94	0.94	0.94	0.92	0.93	0.92	0.92	0.90	0.89	0.91	0.91

Table 2: Benchmarking results of different guardrails across different evaluation modules. Model abbreviations: Q3 / Q7 / Q30 (Qwen2.5-Omni-3B/7B, Qwen3-Omni-30B), CPM (MiniCPM-o-4.5), PHI (Phi4-Multimodal), GEM (Gemini-2.5-Flash), MIM (MiMo-V2.5), GR3/7 (GuardReasoner-Omni-3B/7B), OG3/7 (OmniGuard-3B/7B). Bold values indicate the top-3 results.

Setting	Modality	Q3	Q7	Q30	CPM	PHI	GEM	MIM	Q30 *	CPM *	PHI *	MG	GR3	GR7	OG3	OG7	avg
Uni-modal	T	0.98	0.88	0.88	0.80	0.44	0.92	0.89	0.96	0.93	0.93	0.96	0.87	0.87	0.80	0.86	0.86
	A	0.95	0.88	0.95	0.81	0.35	0.93	0.89	0.99	0.92	0.98	0.97	0.99	0.99	0.93	0.95	0.90
	I	0.98	0.89	0.91	0.80	0.48	0.95	0.96	0.97	0.91	0.73	0.99	0.98	0.97	0.94	0.93	0.89
	V	1.00	0.38	0.13	0.15	/	0.31	0.65	0.46	0.51	/	0.98	0.18	0.08	0.31	0.26	0.42
Dual-modal	T-A	1.00	0.92	0.89	0.62	0.44	0.96	0.93	0.96	0.84	0.90	0.97	0.93	0.95	0.86	0.86	0.87
	T-I	0.96	0.85	0.82	0.71	0.75	0.95	0.90	0.96	0.91	0.88	0.98	0.91	0.98	0.46	0.44	0.83

Table 3: Guardrail performance (Recall) under more modality combinations. Bold values indicate the top-3 results.

Figure 3: Performance comparison in hallucination analysis. We report the recall for the original unsafe content and the FPR after input masking.

Figure 4: Accuracy comparison across risk categories.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: An example of an over-sensitivity evaluation sample. A benign image is paired with an extracted textual pattern, resulting in a completely benign multimodal query that tests for false refusals.

Figure 6: Sample distribution across risk categories.

Model Family	Version/Snapshot	Usage
Qwen-Max	qwen3-max-2025-09-23	Annotation
DeepSeek-Chat	DeepSeek-V3.2	Annotation
Mistral-Medium	mistral-medium-2508	Annotation
Seed-1.6	doubao-seed-1.6-251015	Annotation
Qwen3-VL-Plus	qwen3-vl-plus-2025-12-19	Caption
Qwen3-Omni-Flash	qwen3-omni-flash-2025-12-01	ASR

Appendix

Table 4: Expert Models and Version Specifications.

Modality	Data Source	Mod. Comb.	Size
Utility Module
Text	TruthfulQA	1 (T)	817
Image	MME	1 (T-I)	2,374
Audio	VoiceBench (Alpacaeval)	1 (T-A)	636
Video	MMBench-Video	1 (T-V)	1,998
Basic Safety Module

Appendix

Table 5: Dataset composition for each evaluation module. (A part of the supported datasets is not used for evaluation.)

Hyperparameter	Value
training type	LoRA
LoRA Rank ( $r$ )	8
LoRA Alpha ( $\alpha$ )	32
target modules	all-linear
freeze modules	vision & audio encoders
learning rate	$5\times 10^{-5}$

Appendix

Table 6: Guardrail Training hyperparameters.

Modality	Dataset	Label Type	Count
Text	MT-Bench	Safe	80
	VoiceBench	Safe	2436
	*DAN	Unsafe	600
	*Beavertails	Safe/Unsafe	400
	*Redteam-2K	Unsafe	200
	MM-SafetyBench	Unsafe	200

Appendix

Table 7: Training data details for guardrail fine-tuning. (* denotes those appearing in evaluation; while training data is disjoint from evaluation).

Figure 7: The complete leadearboard of Omni-UBench .

Data Source	Metric	Q3 *	Q7 *	GPT
TruthfulQA	Acc	0.92	0.90	0.93
MME	Acc	1.00	1.00	0.94
VoiceBench	Acc	0.97	0.99	/
MMBench-Video	Acc	0.98	0.98	/
$S_{\text{utility}}$		96.8	96.8	93.5
Beavertails	F1	0.92	0.94	0.95

Appendix

Table 8: Benchmarking results of More OLLMs/Guardrails across different evaluation modules. Model abbreviations: Q3 * / Q7 * (Qwen2.5-Omni-3B/7B * ), GPT (GPT-4o-mini).

ID	Benchmarks	Modalities				Safety Labels	#Risk Topic	Scale	Huggingface Dataset ID ( ) Github Repository ID ( )
ID	Benchmarks	T	I	A	V	Safety Labels	#Risk Topic	Scale	Huggingface Dataset ID ( ) Github Repository ID ( )
Text Modality
1	MT-Bench	✓	✗	✗	✗	S	NaN	80	HuggingFaceH4/mt_bench_prompts
2	TruthfulQA	✓	✗	✗	✗	S	NaN	817	domenicrosati/TruthfulQA
3	AdvBench	✓	✗	✗	✗	U	NaN	520	walledai/AdvBench
4	DAN	✓	✗	✗	✗	U	13	$\sim$ 20k	verazuo/jailbreak-llms

Appendix

Table 9: Complete Benchmark List Supported by Omni-UBench .

Risk Category	Description	Representative Examples
Illegal Activity and Crimes (IA)	Content that promotes, facilitates, or meaningfully supports illegal or criminal acts prohibited by law.	Violent crimes (e.g., terrorism, murder, kidnapping, human trafficking); financial crimes (e.g., embezzlement, money laundering, bribery); property crimes and theft; sexual offenses; illegal manufacturing, distribution, or abuse of drugs, controlled substances, or weapons prohibited by law; other serious criminal activities (e.g., bootlegging, child abuse).
Physical Harm (PH)	Content that enables, encourages, or normalizes behaviors that may cause physical or psychological harm to individuals, animals, or critical infrastructure, regardless of legality.	Self-harm and suicide encouragement; dangerous activities causing physical or mental injury; critical infrastructure sabotage (e.g., power, water, transportation); instructions for creating physical, biological, or chemical weapons that can cause harm to people; offensive military or warfare operations; animal cruelty and environmental harm.
Economic Harm (EH)	Content that may lead to significant financial loss, economic exploitation, or unfair market outcomes, even if not explicitly illegal.	Gambling and betting schemes; multi-level marketing and pyramid schemes; payday lending and predatory loans; unauthorized automated decision-making affecting credit, employment, housing, or public services; market manipulation.
Fraud and Deceptive Activities (FA)	Content involving deception, misrepresentation, or manipulation intended to mislead individuals or groups for unfair or harmful gain.	Financial fraud and scams; academic dishonesty and plagiarism; astroturfing and fake grassroots campaigns; misinformation and disinformation (historical, scientific, medical, or safety-related); false or misleading advertising; defamation; identity theft and impersonation; other deceptive behaviors (e.g., lying or cheating for gain).
Hateful Speech and Toxic Content (HS)	Content that expresses hatred, discrimination, harassment, or dehumanization toward individuals or groups based on protected or non-protected attributes.	Hate speech based on identity; discrimination, bias, and stereotyping; glorification of violence against individuals, groups, or animals; harassment, threats, intimidation, bullying, and humiliation.
Sexual Content (SC)	Sexually explicit, exploitative, or non-consensual content, including any form of sexual exploitation or abuse.	Child sexual abuse material (CSAM); pornographic content; explicit sexual conversations; non-consensual intimate content.

Appendix

Table 10: Detailed definitions and representative examples for each unified risk category used in Omni-UBench .

LLM Safety Benchmarks

Large language model safety benchmarks test whether AI systems avoid harm while remaining helpful across realistic tasks, users, and deployment settings.

See all 284 papers

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
Apr 17, 2026Manh Luong, Tamas Abraham, Junae Kim +6LLM Safety Benchmarks Language Model Safety Evaluation
Monash University, Australia · Defence Science and Technology Group, Australia
Does Unification Come at a Cost? Uni-SafeBench: A Safety Benchmark for Unified Multimodal Large Models
Apr 1, 2026Zixiang Peng, Yongxiu Xu, Qin-Yi Zhang +5Unified Multimodal Models LLM Safety Benchmarks
Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences Beijing, China · Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences Beijing, China
MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
Aug 8, 2026Yilian Shi, Yueming Lyu, Haoxiang Tan +5LLM Safety Benchmarks Multimodal Robustness
Nanjing University

Apr 17, 2026J/K move · Enter open · S save

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

Manh Luong, Tamas Abraham, Junae Kim, Amar Kaur+5

Monash University, Australia · Defence Science and Technology Group, Australia

LLM Safety Benchmarks Language Model Safety Evaluation Multimodal Large Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text. We introduce MCBench, a benchmark with 1196 scenarios spanning four safety categories that require integrating multiple modalities for accurate safety assessment. Each unsafe scenario is paired with a minimally different safe counterpart to assess model sensitivity. Our evaluations of state-of-the-art models reveal significant challenges. Omni LLMs struggle with subtle or non-physical risks but perform better when salient visual or acoustic cues are present. Analysis of reasoning traces shows that, although models can extract modality-specific information, they often fail to integrate these cues effectively for safety judgments. Our findings reveal that current Omni LLMs lack robust cross-modal reasoning in safety-critical settings, underscoring the need for improved architectures and training strategies for multimodal safety.

Abstract

LLM Safety Benchmarks

Explore similar work

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

Does Unification Come at a Cost? Uni-SafeBench: A Safety Benchmark for Unified Multimodal Large Models

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs