Existing omni-modal benchmarks attempt to measure modality-specific contributions, but their measurements are confounded: naturally co-occurring modalities carry correlated yet unequal information, making it unclear whether results reflect true modality reliance or information asymmetry. We introduce OMD-Bench, where all modalities are initially congruent - each presenting the same anchor, an object or event independently perceivable through video, audio, and text - which we then systematically corrupt to isolate each modality's contribution. We also evaluate calibrated abstention: whether models appropriately refrain from answering when evidence is conflicting. The benchmark comprises 4,080 instances spanning 27 anchors across eight corruption conditions. Evaluating ten omni-modal models under zero-shot and chain-of-thought prompting, we find that models over-abstain when two modalities are corrupted yet under-abstain severely when all three are, while maintaining high confidence (~60-100%) even under full corruption. Chain-of-thought prompting improves abstention alignment with human judgment but amplifies overconfidence rather than mitigating it. OMD-Bench provides a diagnostic benchmark for diagnosing modality reliance, robustness to cross-modal inconsistency, and uncertainty calibration in omni-modal systems.
Figures & tables
Zero-Shot
Chain-of-Thought
Synthetic
Real
Synthetic
Real
Model
k=0
kˉ=1
kˉ=2
k=3
k=0
kˉ=1
kˉ=2
k=3
k=0
kˉ=1
kˉ=2
k=3
k=0
kˉ=1
kˉ=2
k=3
Gemini 2.5
96.5
91.0
70.5
60.0
96.8
91.2
69.3
59.4
97.2
94.8
73.8
58.3
97.2
93.4
71.4
53.2
Qwen3o-Think
93.7
86.7
68.1
61.4
92.2
88.0
68.9
60.0
91.8
88.6
64.5
69.0
92.9
85.1
66.3
71.8
VideoLLaMA2
91.0
83.8
68.2
16.1
90.2
82.9
66.1
14.1
91.2
85.4
70.0
14.2
93.4
84.2
67.2
13.2
Qwen3o-Inst
91.0
83.9
63.8
65.1
88.2
81.2
63.9
66.7
95.3
88.7
67.9
57.1
92.2
87.5
67.5
57.1
Table 1: Accuracy (%) across corruption levels. At k≤1 , values reflect factual correctness (higher = better). At k≥2 , values reflect the rate of correct response per the human annotation protocol (higher = better). kˉ=1,2 averages over the three single/double-corruption conditions. Models sorted by k=0 (ZS, Synthetic). ( green = better).
Zero-Shot
Chain-of-Thought
Synthetic
Real
Synthetic
Real
Model
k=0
k=1
k=2
k=3
k=0
k=1
k=2
k=3
k=0
k=1
k=2
k=3
k=0
k=1
k=2
k=3
Human
0.0
0.0
1.2
97.6
0.0
0.0
1.8
98.4
0.0
0.0
1.2
97.6
0.0
0.0
1.8
98.4
Gemini 2.5
1.2
5.5
20.0
59.6
1.2
4.6
18.8
58.5
2.0
2.9
17.5
56.2
0.4
3.4
18.2
51.9
Qwen3o-Think
2.0
6.1
22.6
61.0
3.5
5.9
19.0
58.8
2.4
6.8
28.4
69.0
3.1
10.2
26.7
71.4
Qwen3o-Inst
3.9
10.5
28.5
63.9
7.8
13.2
26.8
66.3
2.4
6.9
22.4
57.5
3.9
7.1
22.6
56.7
Table 2: Abstention rates (%) across corruption levels. The Human row shows evidence-based target rates from our annotation protocol: abstention is near-zero at k≤2 (the remaining clean modalities suffice) and near-universal at k=3 . Cell shading reflects proximity to human targets ( green = close, red = far). ACE measures the average gap between model and human rates across levels (Appendix Table 8 ). Models sorted by ZS ACE (Synthetic).
Zero-Shot
Chain-of-Thought
Synthetic
Real
Synthetic
Real
Model
k=0
k=1
k=2
k=3
k=0
k=1
k=2
k=3
k=0
k=1
k=2
k=3
k=0
k=1
k=2
k=3
M.
Gemini 2.5
99.7
99.3
98.5
96.9
99.8
99.3
98.5
98.1
99.8
99.6
99.0
98.3
99.5
99.5
99.2
98.8
TP
GPT-4o-mini †
99.1
97.6
96.1
95.8
98.8
97.5
96.3
95.9
99.9
99.7
99.5
99.2
99.6
99.7
99.7
99.0
TP
VITA-1.5
93.1
88.1
81.0
75.5
92.1
86.4
80.0
72.2
84.5
78.8
72.3
63.8
79.1
75.0
68.1
60.7
TP
Qwen3o-Think
99.0
98.0
96.2
93.6
98.6
98.2
95.2
93.1
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
RS
Table 3: Mean predictive confidence (%) across corruption levels. Persistent high confidence as k increases indicates overconfidence. Models sorted by k=0 confidence (ZS, Synthetic) within each group. M. : confidence extraction method - TP = token probability (full vocabulary); RS = restricted softmax (5 answer options). TP and RS are on different scales and are not directly comparable. Gemini 2.0 and MiniCPM omitted (no token-level data available). Aggregate ECE and RC-AUC are in Appendix Table 9 . ( green = higher confidence).
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Synthetic
Real
Model
V
A
T
V
A
T
VideoLLaMA2
.28 37%
.14 18%
.33 44%
.27 35%
.14 18%
.35 47%
Phi-4
.24 37%
.06 9%
.36 55%
.25 36%
.05 7%
.40 57%
Uni-MoE-2
.14 31%
.02 5%
.29 64%
.16 32%
.02 4%
.31 64%
VITA-1.5
.14
− .04
.32
.12
− .06
.36
Gemini 2.5
.18
− .07
.26
.14
− .04
.27
Appendix
Table 4: Shapley values (zero-shot). Gray percentages show each modality’s share of total value. Bold : dominant modality. Shares omitted when negative values are present.
Synthetic
Real
Model
ΔV
ΔA
ΔT
ΔV
ΔA
ΔT
GPT-4o-mini †
.03
.04
.22
.02
.01
.20
Gemini 2.0
.08
− .00
.31
.14
− .00
.30
Gemini 2.5
.04
.00
.13
.01
.02
.14
MiniCPM
.10
− .00
.29
.07
− .00
.34
Phi-4
.09
.00
.24
.11
.00
.29
Appendix
Table 5: Normalized reliance Δmnorm (zero-shot). Higher = more sensitive to that modality’s corruption.
Table 8: Abstention Calibration Error (ACE, %, ↓ ) across all four settings. ACE=41∑k=03∣Absk−Absk∗∣ , where Absk∗ is the human-annotated abstention rate at corruption level k ; 0 = perfectly aligned with human abstention behavior. Models sorted by ZS Synthetic ACE.
Zero-Shot
Chain-of-Thought
Syn
Real
Syn
Real
Model
ECE
RC
ECE
RC
ECE
RC
ECE
RC
Restricted Softmax (RS)
Qwen3o-Inst
13.9
8.5
15.1
8.8
22.2
21.0
23.3
19.0
Uni-MoE-2
12.3
11.0
13.7
11.1
29.3
36.3
20.3
27.2
Qwen3o-Think
19.5
6.7
19.1
6.6
20.4
20.9
23.4
19.2
Appendix
Table 9: Aggregate ECE (%, ↓ ) and RC-AUC (%, ↓ ) computed over all corruption levels pooled. Lower = better calibrated. Models grouped by extraction method.
Model
Scores
LogP
Method
VideoLLaMA2
✓
✓
RS
Phi-4
✓
✓
RS
Qwen3o-Inst
✓
✓
RS
Qwen3o-Think
✓
✓
RS
Uni-MoE-2
✓
✓
RS
Gemini 2.5
✗
✓
TP
Appendix
Table 10: Token-level data availability across models. RS requires per-step logits; TP requires scalar log-probabilities.
Figure 2: MCQ vs. open-ended accuracy across corruption levels. Shaded area shows the ZS performance gap. MCQ values are from Table 1 ; open-ended values are averaged across judges and splits.
Figure 3: Open-ended accuracy by corruption level, split by judge. Each line is a (model, prompt) configuration averaged across both data splits.
Figure 4: Per-condition open-ended accuracy (%) averaged across judges and splits. Rows show each (model, prompt) configuration. White lines separate corruption levels.
Overall
k =0
k =1
k =2
k =3
Agreement
96.6
94.6
95.5
98.4
96.8
Appendix
Table 11: Inter-judge agreement (%) between GPT-4.1-mini and GPT-5-mini.
Condition
cv
ca
ct
k
Description
C000
0
0
0
0
Fully congruent (baseline)
C100
1
0
0
1
Video corrupted
C010
0
1
0
1
Audio corrupted
C001
0
0
1
1
Text corrupted
C110
1
1
0
2
Video + audio corrupted
C101
1
0
1
2
Video + text corrupted
Appendix
Table 12: Complete corruption conditions.
Level ( k )
Corrupted Modalities
Abstention (%)
Real
Synthetic
k=0
None (baseline)
0.00
0.00
k=1
Text only
0.00
0.00
Audio only
0.00
0.00
Visual only
0.00
0.00
k=2
Audio + Text
0.78
0.39
Appendix
Table 13: Human-annotated abstention distribution across corruption levels ( k ) and splits. Abstention is near-zero for k≤1 , marginal for k=2 except when text and visual are jointly corrupted, and near-universal ( ≈98% ) under full tri-modal corruption ( k=3 ). Cell shading reflects abstention magnitude: x low → x high.
Text vs. Video
Text vs. Audio
Model
Δˉ (pp)
p
Δˉ (pp)
p
Gemini 2.0
+ 17.5
.006**
+ 28.8
<.001 ***
Gemini 2.5
+ 6.5
.021*
+ 8.5
.006**
GPT-4o-mini †
+ 15.9
.001**
+ 8.6
.015*
MiniCPM
+ 10.7
.026*
+ 20.8
<.001 ***
Uni-MoE-2
+ 22.3
<.001 ***
+ 27.7
<.001 ***
Appendix
Table 14: Wilcoxon signed-rank tests for modality reliance (synthetic, zero-shot). Δˉ : mean per-anchor accuracy drop difference (positive = text corruption hurts more). Text–audio significance holds for all ten models; text–video for six. ∗p<.05 ; ∗∗p<.01 ; ∗∗∗p<.001 .
Zero-Shot
Chain-of-Thought
Model
ACE
95% CI
ACE
95% CI
Gemini 2.5
15.9
[14.1, 17.8]
15.6
[13.9, 17.4]
Qwen3o-Think
16.5
[14.7, 18.3]
16.2
[14.4, 18.0]
Qwen3o-Inst
18.8
[17.0, 20.7]
17.7
[15.8, 19.5]
GPT-4o-mini †
19.5
[17.6, 21.4]
18.4
[16.7, 20.1]
Gemini 2.0
21.1
[19.3, 22.9]
19.6
[17.7, 21.5]
Appendix
Table 15: Abstention Calibration Error (ACE, %, ↓ ) with 95% bootstrap CIs (synthetic split, 10,000 resamples). CIs confirm that the top two models (Gemini 2.5, Qwen3o-Think) are separated from the bottom cluster (VideoLLaMA2, Phi-4, VITA-1.5) by non-overlapping intervals. Models sorted by ZS ACE.
Anchor
Acc.
Anchor
Acc.
bubble_wrap_pop
80.0
kettle_whistle
100.0
kid_laughing
80.0
kid_crying
100.0
train
84.2
match
100.0
clock
85.0
opera
100.0
piano
95.0
popcorn_pop
100.0
bee
100.0
rain
100.0
Appendix
Table 16: Per-anchor accuracy (%) at k=0 for Gemini 2.5 (synthetic, zero-shot). 22 of 27 anchors are solved perfectly; the five below 100% mainly reflect harder knowledge questions.
Omni-modal large language models are increasingly evaluated on clean text--vision--audio inputs, where every channel is present, synchronized, and readily interpretable. Such scores are often taken as evidence of robust cross-modal fusion, but clean evaluation cannot tell whether success depends on stable cross-modal structure or on cues sufficient only in intact inputs. To address this gap, we define a modality fault line: a boundary at which model behavior becomes unstable when a modality remains present and human-interpretable, but its internal evidence structure is perturbed. We introduce SCEval (Structure-Corruption Evaluation) a diagnostic evaluation protocol that keeps the question, answer space, and modality channels fixed while applying controlled structural corruptions to text, vision, and audio individually and jointly. Built from 273 human-verified tri-modal examples from Social-IQ, OmniBench, and VALOR, SCEval evaluates 15 proprietary and open-source omni-modal systems. The results show that structural corruption lowers clean accuracy, text--vision damage forms the most stable shared fault line, and multi-modal degradation is non-additive rather than a simple function of the number of corrupted modalities. Clean omni-modal accuracy therefore does not establish that a model will remain reliable when cross-modal evidence becomes structurally unreliable.
Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision. Project page: https://cheliu-computation.github.io/omni/
Omni-modal large language models (OmniLLMs) unify text, images, audio, and video, yet hallucinate when generation relies on the wrong evidence. Existing inference-time methods can reduce hallucinations, but rarely reveal which evidence sustains a generated commitment. We introduce OmniConfess, a training-free method for mitigating omni-modal hallucinations. It fixes a candidate response and re-scores it at token resolution under controlled channel-wise evidence interventions, producing a structured token-by-channel confession that reveals the response's evidential dependence. OmniConfess uses this confession to preserve grounded content and correct commitments driven by irrelevant or contradictory evidence. To evaluate OmniConfess, we construct OmniHalluBench, a 3,540-example benchmark built from six datasets spanning text, image, audio, and video settings and both judgment and free-form generation. Experiments show that OmniConfess mitigates hallucinations across heterogeneous modality and task settings. Our code and benchmark are publicly available at https://github.com/RongHuiQiang/OmniConfess.
Huiqiang Rong, Haoran Luo, Hui Feng +4
Beijing University of Posts and Telecommunications · Nanyang Technological University, Singapore