Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention

Zabir Al Nazi; Shubhashis Roy Dipta; Md Rizwan Parvez

cs.LGMar 28, 2026

Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention

Authors: Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan Parvez

Organizations: University of California, Riverside · University of Maryland, Baltimore County · Qatar Computing Research Institute (QCRI)

Abstract

Existing omni-modal benchmarks attempt to measure modality-specific contributions, but their measurements are confounded: naturally co-occurring modalities carry correlated yet unequal information, making it unclear whether results reflect true modality reliance or information asymmetry. We introduce OMD-Bench, where all modalities are initially congruent - each presenting the same anchor, an object or event independently perceivable through video, audio, and text - which we then systematically corrupt to isolate each modality's contribution. We also evaluate calibrated abstention: whether models appropriately refrain from answering when evidence is conflicting. The benchmark comprises 4,080 instances spanning 27 anchors across eight corruption conditions. Evaluating ten omni-modal models under zero-shot and chain-of-thought prompting, we find that models over-abstain when two modalities are corrupted yet under-abstain severely when all three are, while maintaining high confidence (~60-100%) even under full corruption. Chain-of-thought prompting improves abstention alignment with human judgment but amplifies overconfidence rather than mitigating it. OMD-Bench provides a diagnostic benchmark for diagnosing modality reliance, robustness to cross-modal inconsistency, and uncertainty calibration in omni-modal systems.

Figures & tables

	Zero-Shot								Chain-of-Thought
	Synthetic				Real				Synthetic				Real
Model	$k{=}0$	$\bar{k}{=}1$	$\bar{k}{=}2$	$k{=}3$	$k{=}0$	$\bar{k}{=}1$	$\bar{k}{=}2$	$k{=}3$	$k{=}0$	$\bar{k}{=}1$	$\bar{k}{=}2$	$k{=}3$	$k{=}0$	$\bar{k}{=}1$	$\bar{k}{=}2$	$k{=}3$
Gemini 2.5	96.5	91.0	70.5	60.0	96.8	91.2	69.3	59.4	97.2	94.8	73.8	58.3	97.2	93.4	71.4	53.2
Qwen3o-Think	93.7	86.7	68.1	61.4	92.2	88.0	68.9	60.0	91.8	88.6	64.5	69.0	92.9	85.1	66.3	71.8
VideoLLaMA2	91.0	83.8	68.2	16.1	90.2	82.9	66.1	14.1	91.2	85.4	70.0	14.2	93.4	84.2	67.2	13.2
Qwen3o-Inst	91.0	83.9	63.8	65.1	88.2	81.2	63.9	66.7	95.3	88.7	67.9	57.1	92.2	87.5	67.5	57.1

Table 1: Accuracy (%) across corruption levels. At

k{\leq}1

, values reflect factual correctness (higher = better). At

k{\geq}2

, values reflect the rate of correct response per the human annotation protocol (higher = better).

\bar{k}{=}1,\!2

averages over the three single/double-corruption conditions. Models sorted by

k{=}0

(ZS, Synthetic). ( green = better).

	Zero-Shot								Chain-of-Thought
	Synthetic				Real				Synthetic				Real
Model	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$
Human	0.0	0.0	1.2	97.6	0.0	0.0	1.8	98.4	0.0	0.0	1.2	97.6	0.0	0.0	1.8	98.4
Gemini 2.5	1.2	5.5	20.0	59.6	1.2	4.6	18.8	58.5	2.0	2.9	17.5	56.2	0.4	3.4	18.2	51.9
Qwen3o-Think	2.0	6.1	22.6	61.0	3.5	5.9	19.0	58.8	2.4	6.8	28.4	69.0	3.1	10.2	26.7	71.4
Qwen3o-Inst	3.9	10.5	28.5	63.9	7.8	13.2	26.8	66.3	2.4	6.9	22.4	57.5	3.9	7.1	22.6	56.7

Table 2: Abstention rates (%) across corruption levels. The Human row shows evidence-based target rates from our annotation protocol: abstention is near-zero at

k{\leq}2

(the remaining clean modalities suffice) and near-universal at

k{=}3

. Cell shading reflects proximity to human targets ( green = close, red = far). ACE measures the average gap between model and human rates across levels (Appendix Table 8 ). Models sorted by ZS ACE (Synthetic).

	Zero-Shot								Chain-of-Thought
	Synthetic				Real				Synthetic				Real
Model	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}3$	M.
Gemini 2.5	99.7	99.3	98.5	96.9	99.8	99.3	98.5	98.1	99.8	99.6	99.0	98.3	99.5	99.5	99.2	98.8	TP
GPT-4o-mini ${\dagger}$	99.1	97.6	96.1	95.8	98.8	97.5	96.3	95.9	99.9	99.7	99.5	99.2	99.6	99.7	99.7	99.0	TP
VITA-1.5	93.1	88.1	81.0	75.5	92.1	86.4	80.0	72.2	84.5	78.8	72.3	63.8	79.1	75.0	68.1	60.7	TP
Qwen3o-Think	99.0	98.0	96.2	93.6	98.6	98.2	95.2	93.1	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	RS

Table 3: Mean predictive confidence (%) across corruption levels. Persistent high confidence as

k

increases indicates overconfidence. Models sorted by

k{=}0

confidence (ZS, Synthetic) within each group. M. : confidence extraction method - TP = token probability (full vocabulary); RS = restricted softmax (5 answer options). TP and RS are on different scales and are not directly comparable. Gemini 2.0 and MiniCPM omitted (no token-level data available). Aggregate ECE and RC-AUC are in Appendix Table 9 . ( green = higher confidence).

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

	Synthetic			Real
Model	V	A	T	V	A	T
VideoLLaMA2	.28 37%	.14 18%	.33 44%	.27 35%	.14 18%	.35 47%
Phi-4	.24 37%	.06 9%	.36 55%	.25 36%	.05 7%	.40 57%
Uni-MoE-2	.14 31%	.02 5%	.29 64%	.16 32%	.02 4%	.31 64%
VITA-1.5	.14	$-$ .04	.32	.12	$-$ .06	.36
Gemini 2.5	.18	$-$ .07	.26	.14	$-$ .04	.27

Appendix

Table 4: Shapley values (zero-shot). Gray percentages show each modality’s share of total value. Bold : dominant modality. Shares omitted when negative values are present.

	Synthetic			Real
Model	$\Delta_{V}$	$\Delta_{A}$	$\Delta_{T}$	$\Delta_{V}$	$\Delta_{A}$	$\Delta_{T}$
GPT-4o-mini ${\dagger}$	.03	.04	.22	.02	.01	.20
Gemini 2.0	.08	$-$ .00	.31	.14	$-$ .00	.30
Gemini 2.5	.04	.00	.13	.01	.02	.14
MiniCPM	.10	$-$ .00	.29	.07	$-$ .00	.34
Phi-4	.09	.00	.24	.11	.00	.29

Appendix

Table 5: Normalized reliance

\Delta_{m}^{\mathrm{norm}}

(zero-shot). Higher = more sensitive to that modality’s corruption.

		$k{=}1$			$k{=}2$
Model	C000	C100(V)	C010(A)	C001(T)	C110(VA)	C101(VT)	C011(AT)	C111
Gemini 2.5	96.5	92.8	96.4	83.7	90.7	37.2	83.5	60.0
Qwen3o-Think	93.7	88.6	91.0	80.4	80.8	44.3	79.2	61.4
VideoLLaMA2	91.0	83.9	89.4	78.0	78.8	51.4	74.5	16.1
Qwen3o-Inst	91.0	85.1	91.4	75.3	79.2	41.2	71.0	65.1
Gemini 2.0	89.8	82.4	90.2	62.0	80.0	10.2	65.5	81.8

Appendix

Table 6: Per-condition accuracy (%, synthetic, zero-shot). C

xyz

= corruption vector

(c_{v},c_{a},c_{t})

; 1 = corrupted.

	Clean acc.		Abstention
Model	ZS	$\Delta$	ACE ZS	$\Delta$
Gemini 2.0	0.898	+0.058	0.211	$-$ 0.016
GPT-4o-mini ${\dagger}$	0.894	+0.043	0.195	$-$ 0.011
MiniCPM	0.875	+0.043	0.226	$-$ 0.015
Qwen3o-Inst	0.910	+0.043	0.189	$-$ 0.012
Gemini 2.5	0.965	+0.007	0.158	$-$ 0.002

Appendix

Table 7: CoT effect (synthetic).

\Delta

: CoT

-

ZS. Positive

\Delta

C000 = CoT helps accuracy. Positive

\Delta

ACE = CoT worsens calibration.

	Zero-Shot		Chain-of-Thought
Model	Syn	Real	Syn	Real
Gemini 2.5	15.8	15.7	15.6	16.6
Qwen3o-Think	16.5	16.5	16.3	16.3
Qwen3o-Inst	18.9	19.5	17.7	18.4
GPT-4o-mini ${\dagger}$	19.5	21.3	18.4	18.7
Gemini 2.0	21.1	21.5	19.5	18.6

Appendix

Table 8: Abstention Calibration Error (ACE, %,

\downarrow

) across all four settings.

\mathrm{ACE}=\frac{1}{4}\sum_{k=0}^{3}|\mathrm{Abs}_{k}-\mathrm{Abs}^{*}_{k}|

, where

\mathrm{Abs}^{*}_{k}

is the human-annotated abstention rate at corruption level

k

; 0 = perfectly aligned with human abstention behavior. Models sorted by ZS Synthetic ACE.

	Zero-Shot				Chain-of-Thought
	Syn		Real		Syn		Real
Model	ECE	RC	ECE	RC	ECE	RC	ECE	RC
Restricted Softmax (RS)
Qwen3o-Inst	13.9	8.5	15.1	8.8	22.2	21.0	23.3	19.0
Uni-MoE-2	12.3	11.0	13.7	11.1	29.3	36.3	20.3	27.2
Qwen3o-Think	19.5	6.7	19.1	6.6	20.4	20.9	23.4	19.2

Appendix

Table 9: Aggregate ECE (%,

\downarrow

) and RC-AUC (%,

\downarrow

) computed over all corruption levels pooled. Lower = better calibrated. Models grouped by extraction method.

Model	Scores	LogP	Method
VideoLLaMA2	✓	✓	RS
Phi-4	✓	✓	RS
Qwen3o-Inst	✓	✓	RS
Qwen3o-Think	✓	✓	RS
Uni-MoE-2	✓	✓	RS
Gemini 2.5	✗	✓	TP

Appendix

Table 10: Token-level data availability across models. RS requires per-step logits; TP requires scalar log-probabilities.

Figure 2: MCQ vs. open-ended accuracy across corruption levels. Shaded area shows the ZS performance gap. MCQ values are from Table 1 ; open-ended values are averaged across judges and splits.

Figure 3: Open-ended accuracy by corruption level, split by judge. Each line is a (model, prompt) configuration averaged across both data splits.

Figure 4: Per-condition open-ended accuracy (%) averaged across judges and splits. Rows show each (model, prompt) configuration. White lines separate corruption levels.

	Overall	$k$ =0	$k$ =1	$k$ =2	$k$ =3
Agreement	96.6	94.6	95.5	98.4	96.8

Appendix

Table 11: Inter-judge agreement (%) between GPT-4.1-mini and GPT-5-mini.

Condition	$c_{v}$	$c_{a}$	$c_{t}$	$k$	Description
$C_{000}$	0	0	0	0	Fully congruent (baseline)
$C_{100}$	1	0	0	1	Video corrupted
$C_{010}$	0	1	0	1	Audio corrupted
$C_{001}$	0	0	1	1	Text corrupted
$C_{110}$	1	1	0	2	Video + audio corrupted
$C_{101}$	1	0	1	2	Video + text corrupted

Appendix

Table 12: Complete corruption conditions.

Level ( $k$ )	Corrupted Modalities	Abstention (%)
Level ( $k$ )	Corrupted Modalities	Real	Synthetic
$k{=}0$	None (baseline)	0.00	0.00
$k{=}1$	Text only	0.00	0.00
	Audio only	0.00	0.00
	Visual only	0.00	0.00
$k{=}2$	Audio + Text	0.78	0.39

Appendix

Table 13: Human-annotated abstention distribution across corruption levels (

k

) and splits. Abstention is near-zero for

k\leq 1

, marginal for

k{=}2

except when text and visual are jointly corrupted, and near-universal (

{\approx}98\%

) under full tri-modal corruption (

k{=}3

). Cell shading reflects abstention magnitude: x low → x high.

	Text vs. Video		Text vs. Audio
Model	$\bar{\Delta}$ (pp)	$p$	$\bar{\Delta}$ (pp)	$p$
Gemini 2.0	$+$ 17.5	.006**	$+$ 28.8	${<}.001$ ***
Gemini 2.5	$+$ 6.5	.021*	$+$ 8.5	.006**
GPT-4o-mini ${\dagger}$	$+$ 15.9	.001**	$+$ 8.6	.015*
MiniCPM	$+$ 10.7	.026*	$+$ 20.8	${<}.001$ ***
Uni-MoE-2	$+$ 22.3	${<}.001$ ***	$+$ 27.7	${<}.001$ ***

Appendix

Table 14: Wilcoxon signed-rank tests for modality reliance (synthetic, zero-shot).

\bar{\Delta}

: mean per-anchor accuracy drop difference (positive = text corruption hurts more). Text–audio significance holds for all ten models; text–video for six.

{}^{*}p{<}.05

;

{}^{**}p{<}.01

;

{}^{***}p{<}.001

	Zero-Shot		Chain-of-Thought
Model	ACE	95% CI	ACE	95% CI
Gemini 2.5	15.9	[14.1, 17.8]	15.6	[13.9, 17.4]
Qwen3o-Think	16.5	[14.7, 18.3]	16.2	[14.4, 18.0]
Qwen3o-Inst	18.8	[17.0, 20.7]	17.7	[15.8, 19.5]
GPT-4o-mini ${\dagger}$	19.5	[17.6, 21.4]	18.4	[16.7, 20.1]
Gemini 2.0	21.1	[19.3, 22.9]	19.6	[17.7, 21.5]

Appendix

Table 15: Abstention Calibration Error (ACE, %,

\downarrow

) with 95% bootstrap CIs (synthetic split, 10,000 resamples). CIs confirm that the top two models (Gemini 2.5, Qwen3o-Think) are separated from the bottom cluster (VideoLLaMA2, Phi-4, VITA-1.5) by non-overlapping intervals. Models sorted by ZS ACE.

Anchor	Acc.	Anchor	Acc.
bubble_wrap_pop	80.0	kettle_whistle	100.0
kid_laughing	80.0	kid_crying	100.0
train	84.2	match	100.0
clock	85.0	opera	100.0
piano	95.0	popcorn_pop	100.0
bee	100.0	rain	100.0

Appendix

Table 16: Per-anchor accuracy (%) at

k{=}0

for Gemini 2.5 (synthetic, zero-shot). 22 of 27 anchors are solved perfectly; the five below 100% mainly reflect harder knowledge questions.

Figure 5: E1: Wrong factual answer - piano ( $k{=}0$ , Gemini 2.5, real).

Figure 6: E1: Wrong factual answer - bubble wrap ( $k{=}1$ , MiniCPM, synthetic).

Figure 7: E1: Wrong factual answer - gun ( $k{=}1$ , GPT-4o-Mini, synthetic).

Figure 8: E1: Wrong factual answer – train ( $k{=}0$ , MiniCPM, synthetic).

Figure 9: E1: Wrong factual answer - car ( $k{=}1$ , Gemini 2.5, synthetic).

Figure 10: E2: Wrong abstention - horse ( $k{=}2$ , Gemini 2.5, real).

Figure 11: E2: Wrong abstention - tiger ( $k{=}2$ , Gemini 2.5, synthetic).

Figure 12: E2: False abstention - typewriter ( $k{=}0$ , GPT-4o-Mini, synthetic).

Figure 13: E3: Missed abstention - piano ( $k{=}3$ , Qwen3o-Instruct, real).

Figure 14: E3: Missed abstention - stapler ( $k{=}2$ , GPT-4o-Mini, real).

Figure 15: E3: Missed abstention - typewriter ( $k{=}3$ , Qwen3o-Instruct, real).

Figure 16: E3: Missed abstention - cat ( $k{=}3$ , MiniCPM, synthetic).

Modality Reliance in Multimodal Models

See all 23 papers

Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning
Aug 29, 2026Zhaolu Kang, Meixin Wu, Yu Xue +6Multimodal Robustness Multimodal Large Language Models
Tencent · Peking University · Zhejiang University
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Cross-Modal Learning Multimodal Large Language Models
StepFun-Audio Team
OmniConfess: Eliciting Token Confessions to Mitigate Omni-Modal Hallucination
Oct 2, 2026Huiqiang Rong, Haoran Luo, Hui Feng +4Multimodal Large Language Models Multimodal Hallucination
Beijing University of Posts and Telecommunications · Nanyang Technological University, Singapore

Aug 29, 2026J/K move · Enter open · S save

Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning

Zhaolu Kang, Meixin Wu, Yu Xue, Yingjie He+5

Tencent · Peking University · Zhejiang University

Multimodal Robustness Multimodal Large Language Models Multimodal Reasoning Multimodal Model Evaluation

Open paper AskPDF arXiv

Omni-modal large language models are increasingly evaluated on clean text--vision--audio inputs, where every channel is present, synchronized, and readily interpretable. Such scores are often taken as evidence of robust cross-modal fusion, but clean evaluation cannot tell whether success depends on stable cross-modal structure or on cues sufficient only in intact inputs. To address this gap, we define a modality fault line: a boundary at which model behavior becomes unstable when a modality remains present and human-interpretable, but its internal evidence structure is perturbed. We introduce SCEval (Structure-Corruption Evaluation) a diagnostic evaluation protocol that keeps the question, answer space, and modality channels fixed while applying controlled structural corruptions to text, vision, and audio individually and jointly. Built from

273

human-verified tri-modal examples from Social-IQ, OmniBench, and VALOR, SCEval evaluates

15

proprietary and open-source omni-modal systems. The results show that structural corruption lowers clean accuracy, text--vision damage forms the most stable shared fault line, and multi-modal degradation is non-additive rather than a simple function of the number of corrupted modalities. Clean omni-modal accuracy therefore does not establish that a model will remain reliable when cross-modal evidence becomes structurally unreliable.

Abstract

Modality Reliance in Multimodal Models

Explore similar work

Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning

Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

OmniConfess: Eliciting Token Confessions to Mitigate Omni-Modal Hallucination