Omni-modal large language models are increasingly evaluated on clean text--vision--audio inputs, where every channel is present, synchronized, and readily interpretable. Such scores are often taken as evidence of robust cross-modal fusion, but clean evaluation cannot tell whether success depends on stable cross-modal structure or on cues sufficient only in intact inputs. To address this gap, we define a modality fault line: a boundary at which model behavior becomes unstable when a modality remains present and human-interpretable, but its internal evidence structure is perturbed. We introduce SCEval (Structure-Corruption Evaluation) a diagnostic evaluation protocol that keeps the question, answer space, and modality channels fixed while applying controlled structural corruptions to text, vision, and audio individually and jointly. Built from 273 human-verified tri-modal examples from Social-IQ, OmniBench, and VALOR, SCEval evaluates 15 proprietary and open-source omni-modal systems. The results show that structural corruption lowers clean accuracy, text--vision damage forms the most stable shared fault line, and multi-modal degradation is non-additive rather than a simple function of the number of corrupted modalities. Clean omni-modal accuracy therefore does not establish that a model will remain reliable when cross-modal evidence becomes structurally unreliable.
Figures & tables
Figure 1: Teaser illustration of modality fault lines. SCEval keeps text, vision, and audio present while damaging their internal evidence structure, revealing failures that clean all-modality evaluation can hide.
Figure 2: The fourteen structural corruption operators used by SCEval (4 text, 7 vision, 3 audio). The taxonomy is curated rather than exhaustive: it targets cross-modal evidence assembly (word identity/order, dense visual signal, audio presence) and excludes trivially recoverable perturbations; per-operator definitions, severity parameterisations, and rejection criteria are in Appendix E (Table 28 ).
Source
Modalities
# Examples
SRC Social-IQ
video, audio, text
100
OmniBench
image, audio, text
77
VALOR
video, audio, text
96
Total
—
VERIFY 273
Table 1: Per-source composition of SCEval . From a 300-candidate pool (100 per source), third-party verification of example validity, tri-modal answerability, and corrupted-variant interpretability retains 273 examples as the base set.
Figure 3: SCEval case illustration. One base example with its three clean modality channels (question/options, sampled frames, audio waveform) and one structurally corrupted variant per modality. Every variant keeps the gold answer, options, and modality channels fixed and damages only the internal evidence structure—distinguishing the fault-line probe from missing-modality ablations.
Text
Vision
Audio
Model
Clean
Typo
Drop
Shuffle
Break
Noise
Occ.
LowRes
MBlur
DBlur
Expo.
Bright.
Remove
Mute
Distort
Proprietary / API omni-modal models
Gemini 3.1 Pro
84.15
79.85 ↓ 4.30
72.57 ↓ 11.58
75.54 ↓ 8.61
80.72 ↓ 3.43
73.57 ↓ 10.58
82.88 ↓ 1.27
85.07 ↑ –0.92
82.29 ↓ 1.86
83.70 ↓ 0.45
85.08 ↑ –0.93
84.88 ↑ –0.73
78.89 ↓ 5.26
80.62 ↓ 3.53
86.17 ↑ –2.02
Gemini 3 Pro
82.40
76.72 ↓ 5.68
71.30 ↓ 11.10
73.16 ↓ 9.24
80.27 ↓ 2.13
73.00 ↓ 9.40
80.09 ↓ 2.31
83.80 ↑ –1.40
81.41 ↓ 0.99
82.89 ↑ –0.49
80.49 ↓ 1.91
83.88 ↑ –1.48
76.36 ↓ 6.04
75.82 ↓ 6.58
82.51 ↑ –0.11
Gemini 3 Flash
80.95
71.06 ↓ 9.89
67.40 ↓ 13.55
70.33 ↓ 10.62
72.89 ↓ 8.06
67.79 ↓ 13.16
80.33 ↓ 0.62
80.97 ↑ 0.02
77.59 ↓ 3.36
78.95 ↓ 2.00
79.37 ↓ 1.58
80.30 ↓ 0.65
74.30 ↓ 6.65
73.24 ↓ 7.71
79.17 ↓ 1.78
Gemini 3.5 Flash
78.39
72.53 ↓ 5.86
67.03 ↓ 11.36
68.50 ↓ 9.89
73.63 ↓ 4.76
62.75 ↓ 15.64
75.14 ↓ 3.25
78.96 ↑ –0.57
75.56 ↓ 2.83
77.47 ↓ 0.92
79.30 ↑ –0.91
78.31 ↓ 0.08
73.45 ↓ 4.94
71.00 ↓ 7.39
77.83 ↓ 0.56
Table 2: Severity- 70 single-modality corruption matrix on the 14 -model panel (clean accuracy in the second column; subsequent columns report accuracy under each operator with the absolute drop from clean as a subscript, ↓ drop / ↑ gain). Columns are grouped by modality (4 text, 7 vision, 3 audio). The four operators with the steepest panel-mean drops ( drop_words , word_shuffle , noise , mute / remove ) define the joint-corruption building blocks of § 3.2 . Table 3 reports the coverage-aware operator grouping.
Figure 4: Per-model severity curves for all 14 systems on the fourteen single-modality operators (clean and severities 10/30/50/70 ). Modality is encoded by line style (text solid, vision dashed, audio dotted), operator by colour; per-panel y -axis is auto-scaled. drop_words , word_shuffle , and noise steepen past severity 30 on every panel, while several cosmetic vision operators and audio distortion remain closest to clean.
Operator group
Mean drop (pp)
Primary (10 operators)
↓ 5.31
All operators (14)
↓ 5.14
High-rejection stress set (4)
↓ 4.72
Table 3: Coverage-aware severity-70 panel-mean drops by operator group (14 models).
Figure 5: Severity-stacking scatter for joint corruption ( 29 cells per deep-dive model). x is the total corruption budget st+sv+sa , y is the drop from the model’s clean baseline; markers encode the combination: ∙ T+V, ■ T+A, ▲ V+A, ⧫ T+V+A. The worst point is asymmetric (mildly corrupted text t30× heavily corrupted vision v70 ), the trimodal staircase climbs with the vision component, and points below Δ=0 are coverage-induced apparent gains (§ 3.1 ).
Table 4: Design-axis comparison of SCEval against the most closely related multimodal evaluation benchmarks (§ 4 ). Struct. corruption means operator-level structural damage that keeps the modality present, as opposed to missing-modality ablation or semantic perturbation; Joint corruption requires applying corruption to two or more modalities of the same example; Audio/vision variant verification means that every corrupted audio or visual variant (not just every clean example) is human-judged for interpretability. Deterministic text corruptions are assessed separately in the blind answer-preservation audit; Sample-paired protocol means clean and corrupted versions share the same base example and gold answer. “—” indicates that the axis is not part of the original benchmark’s evaluation surface.
Table 5: Full operator-level joint-corruption breakdown for the four deep-dive models. Each row is one of the 29 operator × severity cells defined in Section 3.2 : the four canonical T+V cells ( drop_words × noise at {30,70}×{30,70} ) and two replacement cells; the four canonical T+A cells and two replacements; the four canonical V+A cells and two replacements; and the eight canonical T+V+A cells (cube over t∈{30,70},v∈{30,70},a∈{30,50} ) and three replacements. Cells marked r substitute a different operator family ( occlusion for noise , remove for mute , or word_shuffle for drop_words ) at the strongest severity inside that combination. Each model cell shows the mean-variant accuracy together with the absolute change from the model’s clean all-modality baseline ( ↓ red = drop, ↑ green = relative gain, ±0.00 = unchanged). All four deep-dive panels are reported here; Gemini 3 Flash and Qwen3.5-Omni-Plus provide the canonical reference points used in Figure 5 , while Gemini 3.1 Pro and MiniCPM-o 4.5 are paired in the same table so that the proprietary vs. open-API comparison is preserved cell-by-cell.
Mod.
Operator
Qwen3.5-Omni-Plus
Gemini 3 Flash
s10
s30
s50
s70
s10
s30
s50
s70
Text severity grid
Text
typo_ocr
72.91
72.93
71.16
69.60
79.40
75.40
74.04
71.06
Text
drop_words
71.57
72.21
67.20
64.10
81.85
71.98
71.60
67.40
Text
word_shuffle
71.58
69.04
65.63
65.20
80.64
77.37
74.47
70.33
Text
sentence_break
74.33
72.54
71.27
69.60
80.82
78.03
73.33
72.89
Appendix
Table 6: Full single-modality severity grid. Values are mean-variant accuracies at severity 10/30/50/70 (averaged across the three random variants of each stochastic operator). This table exposes the full 56-cell single-modality suite rather than only the weakest cell per operator.
Model
Group
Valid/expected
Trials
Invalid
Acc.
Coverage and invalid-output pressure
Qwen3.5-Omni-Plus
Bimodal
4,368/4,368
4,368
53
67.42
Qwen3.5-Omni-Plus
Trimodal
6,552/6,552
6,552
124
64.78
Gemini 3 Flash
Bimodal
4,368/4,368
4,368
41
74.16
Gemini 3 Flash
Trimodal
6,552/6,552
6,552
98
72.18
Appendix
Table 7: Coverage statistics for combined-corruption runs. The invalid column is decomposed by failure type in Table 16 .
Model
Clean
TV
TA
VA
TVA
Panel fault
Casym
Strongest cell
Proprietary / API omni-modal models
Gemini 3.1 Pro
84.15
78.62
79.97
81.17
78.04
↓ 4.70
+1.15
drop_words @sev30
Gemini 3 Pro
82.40
75.72
78.13
79.65
75.19
↓ 5.23
+0.92
drop_words @sev30
Gemini 3 Flash
80.95
73.50
76.53
77.30
72.27
↓ 6.05
+1.29
noise @sev70
Gemini 3.5 Flash
78.39
71.04
73.75
75.59
70.14
↓ 5.76
+1.71
drop_words @sev30
Gemini 2.5 Pro
79.12
71.92
74.71
76.50
71.03
↓ 5.58
+0.48
drop_words @sev30
Appendix
Table 8: All-model lightweight panel results. TV, TA, VA, and TVA denote mean accuracy over the selected text–vision, text–audio, vision–audio, and tri-modal cells (each cell itself already averaged over its three random variants); Panel fault is the mean clean-to-panel drop across the four combination blocks. Casym=Acc(t70/v70)−Acc(t30/v70) ; positive values mean that the mildly corrupted-text / heavily corrupted-vision cell is worse. The Strongest cell column flags the condition responsible for the largest single-model drop.
Model
No-text
No-vision
No-audio
Mt
Mv
Ma
Q+Opt
Text-only
VA-only
Proprietary / API omni-modal models
Gemini 3.1 Pro
63.15
70.98
78.49
↓ 2.04
↑ 1.18
↑ 1.90
34.48
40.77
21.37
Gemini 3 Pro
67.43
67.24
78.29
↓ 2.56
↑ 0.67
↑ 2.70
32.80
43.21
29.26
Gemini 3 Flash
66.24
72.01
74.59
↓ 2.84
↑ 2.78
↑ 3.79
29.36
37.70
27.38
Gemini 3.5 Flash
56.46
62.76
72.17
↓ 3.21
↑ 0.68
↑ 2.39
29.34
39.06
22.79
Gemini 2.5 Pro
65.68
67.89
71.52
↓ 2.96
↑ 2.25
↑ 2.97
37.01
40.54
19.06
Appendix
Table 9: Required controls for modality absence and shortcut reliance. No-X columns report accuracy when modality X is removed entirely; positive Mm means a corrupted-but-present modality is more harmful than removing that modality (so the joint corruption is not equivalent to a missing-modality ablation). Q+Opt provides only the question and answer options (no media), Text-only keeps just the textual context, and VA-only drops the textual context beyond the answer interface; these three columns measure the shortcut-only ceiling that any structural-corruption result must beat.
Model
Std. JSON
CoT
Open-form
Invalid shift
Proprietary / API omni-modal models
Gemini 3.1 Pro
85.30
83.34
80.85
↓ 2.83
Gemini 3 Pro
81.59
81.68
81.79
↓ 3.81
Gemini 3 Flash
82.18
79.11
83.57
↓ 2.41
Gemini 3.5 Flash
77.52
78.50
78.26
↓ 3.27
Gemini 2.5 Pro
77.47
73.32
73.34
↓ 2.37
Appendix
Table 10: Prompt-robustness results on the diagnostic subset that contains clean, the weakest single-modality cells, the strongest text–vision cell, and the strongest trimodal cell. Std. JSON is the canonical structured prompt used everywhere else; CoT forces a chain-of-thought scratchpad before the JSON; Open-form replaces the JSON schema with free-text answers parsed post-hoc. Invalid shift is the drop in valid-output coverage between Std. JSON and the worst alternative prompt for each model.
Model
Source
Clean
TV
TVA
Strongest drop
Proprietary / API omni-modal models
Gemini 3.1 Pro
Social-IQ
85.75
81.98
77.03
↓ 8.32
OmniBench
83.51
79.03
74.28
↓ 7.24
VALOR
82.76
78.72
73.99
↓ 9.73
Gemini 3 Pro
Social-IQ
85.62
81.44
78.33
↓ 11.37
OmniBench
78.95
72.91
69.54
↓ 7.72
Appendix
Table 11: Per-source breakdown for each model. Each block reports the model’s clean accuracy on each of the three source benchmarks (Social-IQ, OmniBench, VALOR), the mean text–vision joint accuracy, the mean trimodal accuracy, and the strongest source-level drop. These rows test whether the main text–vision and trimodal fault lines are stable across sources or driven by a single benchmark.
Comparison
Δ
95% CI
McNemar
Holm p
Interpretation
Statistical reliability
Clean → TV, Qwen3.5-Omni-Plus
↓ 5.97
[+4.97, +6.97]
<.001
<.001
Significant clean-to-fault-line drop
Clean → TV, Gemini 3 Flash
↓ 7.32
[+6.32, +8.32]
<.001
<.001
Significant clean-to-fault-line drop
Weakest single → strongest joint, Qwen3.5-Omni-Plus
↓ 0.73
[-0.27, +1.73]
0.012
0.034
Tests Δsingle>0
Weakest single → strongest joint, Gemini 3 Flash
↑ 2.45
[+1.45, +3.45]
<.001
<.001
Tests Δsingle>0
Appendix
Table 12: Statistical reliability for the headline comparisons. Δ is the paired drop (or gain) reported elsewhere in the paper; the 95% CI is computed by sample-level paired bootstrap on the per-item correctness vectors after mean-variant aggregation; McNemar reports the paired McNemar exact p -value on the same vectors; Holm p is the Holm-Bonferroni-corrected p -value inside each comparison family. All comparisons remain significant after correction.
Estimand
Definition
Comparison cohort
Confirmatory estimands and cohort alignment
Paired corruption drop
Accclean(Ic)−Accc(Ic)
Same human-valid base examples Ic
Excess model drop
Model drop minus human drop
Same gold-preserved cohort for all terms
Asymmetric TV contrast
Acc(t70/v70)−Acc(t30/v70)
Common sample × seed cohort
Vision / text main effect
Mean at v30−v70 / t30−t70
Common four-cell cohort
TV interaction
[A70,70−A30,70]−[A70,30−A30,30]
Common four-cell cohort
Appendix
Table 13: Confirmatory estimands and cohort alignment. Each contrast is evaluated within the cohort named in the final column; clean and corrupted accuracies are never compared across unmatched retained pools. Fine-grained grids are descriptive unless a paired contrast is stated explicitly.
Condition group
N
Human acc.
Agreement
Gold valid
Human answerability and gold-preservation check
Clean
273/273
96.7
95.2
99.3
Worst text single
264/273
90.5
88.6
94.7
Worst vision single
228/273
87.3
86.4
92.4
Worst audio single
210/273
89.6
88.2
95.1
Strongest TV joint
198/273
80.4
79.7
88.5
Appendix
Table 14: Sampled human answerability and original-gold validity audit. N is the number of audited variants, Human acc. is accuracy against the original gold answer, Agreement is pre-adjudication agreement, and Gold valid is the fraction of variants for which the original gold remains defensible. The sampled audit motivates the gold-preserved analysis in Table 15 .
Condition
Sampled gold valid
Model drop on Gc
Human drop
Excess drop
95% CI
Gold-preserved, human-normalized results
Worst text single
94.7%
↓ 10.68
↓ 1.80
↓ 8.88
[7.4, 10.4]
Worst vision single
92.4%
↓ 10.97
↓ 2.20
↓ 8.77
[7.2, 10.2]
Worst audio single
95.1%
↓ 7.02
↓ 1.50
↓ 5.52
[4.1, 6.9]
Strongest TV joint
88.5%
↓ 12.00
↓ 3.20
↓ 8.80
[7.2, 10.4]
Strongest TVA joint
84.8%
↓ 15.50
↓ 4.60
↓ 10.90
[9.0, 12.8]
Appendix
Table 15: Gold-preserved, human-normalized results on the gated cohort Gc . Sampled gold valid is the reference rate from Table 14 . Model and human drops are both computed on Gc relative to the paired clean condition; Excess drop is the model drop minus the human drop. Confidence intervals are 95% paired bootstrap intervals for the excess drop.
Model
Group
API
Parse
Refusal
Empty
Valid cov.
Proprietary / API omni-modal models
Gemini 3.1 Pro
Bimodal
0
3
0
2
266/273
Trimodal
1
4
2
1
263/273
Gemini 3 Pro
Bimodal
0
4
1
1
266/273
Trimodal
2
1
1
1
265/273
Gemini 3 Flash
Bimodal
2
1
4
3
259/273
Appendix
Table 16: Invalid-output decomposition for the combined-corruption runs. Each model is reported separately for bimodal and trimodal conditions; columns count the number of base examples (out of 273) whose response was invalidated for each failure mode ( API = API/network errors, Parse = unparseable output, Refusal = explicit refusal, Empty = empty response). Valid cov. is the number of items that returned a parseable answer at all; the headline accuracy elsewhere in the paper is reported only over these items.
Table 17: Mechanism-probe diagnostics. These analyses examine why a fault line appears after the core robustness result has been established.
Model
Mod.
Weakest cell
Mean acc
Worst-variant
Mean − worst
Std (3 var)
nvar
Deep-dive models with full single-modality severity grids
Qwen3.5-Omni-Plus
Text
drop_words @sev70
64.10
62.55
1.55
0.87
3
Vision
occlusion @sev50
70.29
67.69
2.60
1.72
3
Audio
mute @sev70
69.72
67.38
2.34
1.71
3
Gemini 3 Flash
Text
drop_words @sev70
67.40
65.69
1.71
0.88
3
Vision
noise @sev70
67.79
65.23
2.56
1.31
3
Appendix
Table 18: Seed-stability diagnostic at each model’s weakest single-modality cell. Mean acc is the headline mean-variant accuracy used everywhere else in the paper; Worst-variant is Accworst(c)=minrAccr(c) , the accuracy of the worst of the three random variants at the same condition; Mean − worst is their absolute gap; Std is the across-variant standard deviation. A small mean − worst gap and low std (heuristically ≲2 pp) certify that the reported headline drop is not driven by one unlucky seed. nvar records the effective number of random variants that survived the per-variant human filtering at that cell (Appendix D , Table 24 ).
Model
St
Sv
Sa
Steepest channel
Proprietary / API omni-modal models
Gemini 3.1 Pro
−0.096
−0.023
−0.029
Text
Gemini 3 Pro
−0.102
−0.025
−0.058
Text
Gemini 3 Flash
−0.159
−0.044
−0.080
Text
Gemini 3.5 Flash
−0.110
−0.043
−0.051
Text
Gemini 2.5 Pro
−0.125
−0.023
−0.056
Text
Appendix
Table 19: Fragility slope Sm in percentage points per severity unit (negative = accuracy decreases as severity increases). For each (model, modality) pair we OLS-regress the modality-averaged accuracy on severity s∈{0,10,30,50,70} where s=0 is clean accuracy, averaging over the operators inside that modality (4 text, 7 vision, 3 audio). The Steepest channel column flags the modality that decays fastest for each model. Text is consistently the steepest channel across the 14-model panel, even though vision and audio carry larger isolated-operator drops at severity 70.
Model family
n
Mean clean
Δtext
Δvision
Δaudio
Ftv
Fta
Fva
Ftva
Steepest channel
Aggregated by model family (mean across rows, severity 70 single-modality and headline joint cells)
Proprietary / API
6
80.07
↓ 8.33
↓ 2.34
↓ 4.58
↓ 6.99
↓ 4.65
↓ 3.06
↓ 7.94
Text
Open / open-API
8
69.17
↓ 9.09
↓ 3.63
↓ 5.49
↓ 8.88
↓ 5.93
↓ 4.23
↓ 10.16
Text
Gap (Prop − Open)
—
+10.90
+0.76
+1.29
+0.91
+1.89
+1.28
+1.17
+2.21
—
Appendix
Table 20: Family-level summary across the 14-model panel. Δmod is the mean severity-70 drop on modality mod averaged over all operators inside that modality and all models inside that family. FS is the mean clean-to-fault drop on the S∈{tv,ta,va,tva} block of the 12-cell lightweight panel (Table 31 ), averaged over the models in that family. The bottom row reports the proprietary − open gap (positive means proprietary is better) so the reader can tell at a glance whether the open band trails the proprietary band uniformly or is more channel-specific.
Model
Combo
ncells
Mean actual
Mean additive pred.
Mean Δint
% sub-additive
Paired bootstrap p
Deep-dive models: canonical joint grids only (replacement cells excluded; see Table 5 )
Qwen3.5-Omni-Plus
T+V
4
66.81
65.87
+0.94
50%
0.082
T+A
4
72.26
66.47
+5.79
75%
0.283
V+A
4
75.12
69.66
+5.46
100%
<.001
T+V+A
8
66.76
64.18
+2.58
50%
0.172
Gemini 3 Flash
T+V
4
71.41
59.90
+11.50
100%
<.001
Appendix
Table 21: Sub-additivity diagnostic for joint corruption on the canonical grids of Table 5 . Mean additive pred. is Accclean−∑mΔm averaged across the canonical cells in that combination type, where each Δm is the single-modality drop at the same severity, read from the regenerated severity curves of Figure 4 . Mean Δint is the across-cell average of the interaction term Δint=Acc(joint)−Accadd ; positive means corruptions overlap (joint is gentler than additive); negative means corruptions compound (joint is harsher than additive). % sub-additive is the fraction of cells with Δint>0 . All 16 point estimates are positive, but five paired-bootstrap tests have p>.05 ; Qwen3.5-Omni-Plus is near the additive baseline and has two mildly super-additive cells in each of its T+V and T+V+A grids.
Model
ECE (lower is better)
ρoc : % wrong answers emitted with high confidence
Clean
Single-TV worst
Joint-TV worst
Joint-TVA worst
Clean
Single-TV worst
Joint-TV worst
Joint-TVA worst
Deep-dive panel calibration
Qwen3.5-Omni-Plus
6.40
7.49
8.98
11.12
18.6
22.0
28.1
33.7
Gemini 3 Flash
5.10
6.96
9.13
10.87
14.8
21.4
25.4
29.2
Gemini 3.1 Pro
4.20
5.28
5.86
8.29
11.5
16.5
23.2
28.0
MiniCPM-o 4.5
7.60
9.16
10.68
11.88
23.4
25.8
31.0
35.2
Appendix
Table 22: Calibration diagnostic on the prompt-robustness diagnostic subset (clean; single-TV worst = drop_words @sev30 single-modality; joint-TV worst = drop_words × noise @ t30/v70 ; joint-TVA worst = the strongest cell in the eight canonical T+V+A grid). ECE bins the confidence ratings into the three categorical levels emitted by the JSON schema and reports the standard expected calibration error in pp. ρoc reports the proportion of wrong answers that were nonetheless emitted with high confidence; a rising ρoc between clean and joint-TVA is the silent-failure signal.
Condition
Mean Jaccard
Shared-hard ( ≥10/14 wrong)
Idiosyncratic ( =1/14 wrong)
Total wrong (union)
Failure-set overlap across the 14-model panel
Clean baseline
0.17
17/273
44/273
95/273
drop_words @sev30 (single-T worst)
0.31
59/273
47/273
172/273
noise @sev70 (single-V worst)
0.36
75/273
41/273
185/273
mute @sev50 (single-A worst)
0.28
52/273
53/273
165/273
Joint T+V worst ( drop_words × noise @ t30/v70 )
0.42
103/273
35/273
210/273
Appendix
Table 23: Cross-model failure-overlap diagnostic on the 14-model panel. Mean Jaccard is the average ∣A∩B∣/∣A∪B∣ over all (214)=91 pairs of model failure sets; values near 1 mean every model misses essentially the same items, values near 0 mean each model has its own idiosyncratic failure mode. The Shared-hard column counts base examples failed by at least 10 of the 14 models and the Idiosyncratic column counts items failed by exactly one model; the gap between these counts and the Total wrong (union) tells the reader whether the corruption opens a shared “structurally impossible” core or just amplifies model-specific weaknesses.
Operator
Annotated
Retained
Retain
Audio operators (3)
mute
3,324
2,362
71.1%
remove
3,324
2,478
74.5%
distortion
1,108
1,074
96.9%
audio subtotal
7,756
5,914
76.3%
Vision operators (7, image + video unified)
Appendix
Table 24: Per-operator pool sizes and human interpretability filtering on corrupted variants sent to the third-party annotation team (task ID 1246075789726052352 ). One row per corruption operator; the Annotated column counts every (sample × severity {10,30,50,70}× random-variant) cell sent for verification, and the Retained column counts cells that pass the corrupted-input interpretability check. Vision counts merge the image-source pool (308 cells per severity grid) and the video-source pool (800 cells per severity grid) into the seven canonical vision operators of Figure 2 . Text operators are not included in this media-interpretability table; their answer-preservation status is evaluated in the separate blind audit in Appendix G.7 . Cells failing the interpretability check are excluded from every accuracy aggregate in the main paper.
Mod.
Operator
sev 10
sev 30
sev 50
sev 70
Confound zone: drop rate climbs steeply with severity
Vision
occlusion
0 7.8%
32.7%
64.9%
82.7%
Vision
brightness
0 3.2%
0 7.2%
20.9%
51.6%
Audio
mute
0 6.0%
18.8%
35.4%
55.6%
Audio
remove
0 5.9%
16.8%
31.5%
47.5%
Stable zone: drop rate stays low across all severities (confound-free)
Appendix
Table 25: Per-operator human-filtering drop rate as a function of corruption severity. Each cell is the fraction of (sample × severity × random-variant) variants that the annotation team rejected as no longer perceptually interpretable. Colour-graded by magnitude: white <10% , sand 10 – 25% , rose 25 – 50% , light red 50 – 75% , bold red ≥75% . Four operators (vision occlusion , vision brightness , audio mute , audio remove ) account for essentially all of the rejection mass at severities ≥50 ; on these operators the retained pool is selection-biased toward easier base examples, which is the most plausible cause of the negative drops (apparent accuracy gains under corruption) seen in Section 3.2 . The remaining six operators retain ≥88% of their variants even at severity 70 and are confound-free across the entire severity grid. Text operators are deterministic and not annotated per variant; see Table 24 .
Rank
Operator
Panel-mean drop (pp)
Group
Most damaging severity-70 operators (14-model panel)
1
noise
↓ 11.39
Primary
2
drop_words
↓ 11.28
Primary
3
word_shuffle
↓ 11.17
Primary
4
mute
↓ 7.46
Coverage-aware stress
5
remove
↓ 7.20
Coverage-aware stress
Appendix
Table 26: Most damaging operators by panel-mean severity-70 drop, recomputed from Table 2 . The three largest drops are all in the primary set.
Table 27: Inter-annotator agreement (IAA) on every annotation field used during data verification. % full agree is the fraction of items where both independent annotators emitted the same label before senior-annotator adjudication. Cohen κ is reported for fields with exactly two annotators; Fleiss κ is reported for fields where additional annotators (a third annotator on the answer-preservation re-annotation, or the senior adjudicator) entered labels on the same items. % adjudicated is the fraction of items that the senior annotator had to break ties on. The headline aggregate row weighs the per-field agreement and κ by item count and is the single number a reviewer can cite when asking “what is the IAA on this benchmark”.
Modality
Operator family
Condition key
Structure being damaged
Severity / variants
Text structure corruptions
Text
Word dropping
drop_words
Removes lexical evidence while preserving the question/options format.
10/30/50/70; stochastic
Text
Word shuffling
word_shuffle
Breaks local word order and phrase composition without deleting all tokens.
10/30/50/70; stochastic
Text
OCR-like typos
typo_ocr
Simulates recognition noise through character-level substitutions and distortions.
10/30/50/70; stochastic
Text
Sentence breaking
sentence_break
Fragments sentence and phrase boundaries, weakening syntactic structure.
10/30/50/70; stochastic
Visual structure corruptions
Appendix
Table 28: Detailed corruption operator taxonomy. All operators keep the modality channel present and only damage its internal evidence structure; this is the key distinction from missing-modality ablations. Severity is reported on a 0 – 100 scale; stochastic operators ship up to three random variants per severity, aggregated by the mean-variant rule of Section 2.4 .
Mod.
Operator
Parameter
s10
s30
s50
s70
Text operators
Text
drop_words
Token drop rate (%)
10
30
50
70
Text
word_shuffle
Shuffle window
2
4
6
8
Text
typo_ocr
Char. subst. rate (%)
10
30
50
70
Text
sentence_break
Fragment prob. (%)
10
30
50
70
Vision operators
Appendix
Table 29: Severity-to-parameter mapping for all fourteen corruption operators. For stochastic operators ( drop_words , word_shuffle , occlusion , remove , mute ) the parameter value is the expected magnitude; random variants differ in which tokens, pixels, or segments are selected. The distortion clipping threshold decreases with severity.
Model
Group
Interface
Clean
Panel fault
Frame-input compatibility
Gemini 3.1 Pro
PROP Proprietary
IFACE OK
84.15
↓ 4.70
Multi-image prompt through Gemini-style content parts
Gemini 3 Pro
PROP Proprietary
IFACE OK
82.40
↓ 5.23
Multi-image prompt through Gemini-style content parts
Gemini 3 Flash
PROP Proprietary
IFACE OK
80.95
↓ 6.05
Multi-image prompt through Gemini-style content parts
Gemini 3.5 Flash
PROP Proprietary
IFACE OK
78.39
↓ 5.76
Multi-image prompt through Gemini-style content parts
Gemini 2.5 Pro
PROP Proprietary
IFACE OK
79.12
↓ 5.58
Multi-image prompt through Gemini-style content parts
Gemini 2.5 Flash
PROP Proprietary
IFACE OK
75.40
↓ 6.65
Multi-image prompt through Gemini-style content parts
Appendix
Table 30: Model roster, interface compatibility, and all-model panel summary. Clean matches the all-modality clean baseline used in Table 2 ; Panel fault is the mean clean-to-panel drop on the 12-cell lightweight panel of Table 31 . Models that use a non-matched native video interface are marked in the interface column rather than mixed silently with frame-based runs; the panel-fault numbers are computed uniformly over the 12-cell panel for all fourteen models.
Comparable fault-line score without running the full 29-cell matrix
Appendix
Table 31: Lightweight multimodal fault-line panel for all-model expansion. The panel preserves the strongest observed pattern while keeping the cross-model budget small enough to run on every interface-passing model.
Table 32: Full enumeration of the 46-cell extended representative-model joint suite. The 29-cell-suite column indicates whether a cell is part of the canonical/replacement grid in Table 5 ( yes , 29 cells) or one of the 17 additional extended cells ( no , 17 cells: 4 T+V, 4 T+A, 3 V+A, 6 T+V+A). The extended rows fill in the severity-50 mid-pressure row and the light/heavy asymmetric corners of each combination’s severity grid, so the 46-cell suite covers (i) every canonical {30,70}×{30,70} grid corner, (ii) every operator-replacement cell at the strongest severity, (iii) every severity-50 mid-pressure cell, and (iv) every {10,70}×{30/50/70} light/heavy asymmetric cell. Reading: 29+17=46 .
Control
Setup
Purpose
Task-format and option-shortcut controls
No-option generation
Remove answer options; the model writes a short answer and evidence. Two blinded scorers assess correctness, partial correctness, and answerability after clean and corrupted items are verified as answerable and gold-valid.
Transfer beyond fixed options
Option order (Latin square)
Evaluate all four option orders, with every option occupying every position once and the gold key remapped. Conditions include clean, drop_words @70, noise @70, TV t30/v70 , TV t70/v70 , and TVA t30/v70/a30 .
Position bias and parser stability
Options only
Provide only the four option strings, with no question or media.
Answer-text prior
Appendix
Table 33: Additional controls for task format and option shortcuts. The no-option control is evaluated only on a shared gold-preserved subset; option manipulation is never mixed into headline accuracy tables.
Control
Metric
Value
Option-shortcut results
Options only
Panel accuracy
29.96
Option order (clean)
Mean-order accuracy
73.16
Option order (clean)
Worst-order accuracy
70.86
Option order (clean)
Prediction consistency
94.1%
Option order ( noise @70)
Mean–worst-order gap
3.20 pp
Appendix
Table 34: Options-only baseline and option-order sensitivity. Prediction consistency is the fraction of examples with the same remapped answer across all four orders.
Condition
Existing MC drop (pp)
No-option drop (pp)
Transfer Δ (pp)
Clean-to-corrupted drop on the shared gold-preserved subset
drop_words @70
11.28
13.64
+2.36
noise @70
11.39
14.22
+2.83
TV t30/v70
10.01
12.25
+2.24
TV t70/v70
8.28
9.80
+1.52
TVA t30/v70/a30
6.36
7.95
+1.59
Appendix
Table 35: No-option generation control. The model produces a short answer rather than selecting an option; two blinded scorers evaluate the output on the shared gold-preserved subset. Existing multiple-choice drops for the single-modality rows are 14-model panel means; the joint rows are means over the four deep-dive models. Transfer Δ is the no-option drop minus the multiple-choice drop.
Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision. Project page: https://cheliu-computation.github.io/omni/
Building an omni-modal foundation model means evaluating it across text, image, video, and audio. Excellent evaluation toolkits exist for each modality, but their inference engines, prompt conventions, and metric implementations are mutually incompatible, so practitioners end up maintaining separate environments for every toolchain and still struggle to compare results across them. OmniEvaluator grew out of this need in our own model development: rather than reimplementing benchmarks, it connects existing inference engines and curated evaluation libraries at a higher level, exposing four inference backends, four evaluation frameworks, and over a thousand benchmarks through a single interface. Every run is recorded as an artifact capturing the full configuration for exact reproduction, and results flow into a shared dashboard for cross-model comparison. A federated mode shares GPU inference servers across concurrent evaluations, and a built-in verifier, small enough to run on CPU, keeps its score stable across engines and prompts where rule-based scoring fluctuates under configuration mismatch, matching cost-efficient commercial LLM judges without their recurring API cost. The system, demo video, and dashboard are publicly available. (https://github.com/naver-ai/omni-evaluator)
Omni-modal large language models (OmniLLMs) unify text, images, audio, and video, yet hallucinate when generation relies on the wrong evidence. Existing inference-time methods can reduce hallucinations, but rarely reveal which evidence sustains a generated commitment. We introduce OmniConfess, a training-free method for mitigating omni-modal hallucinations. It fixes a candidate response and re-scores it at token resolution under controlled channel-wise evidence interventions, producing a structured token-by-channel confession that reveals the response's evidential dependence. OmniConfess uses this confession to preserve grounded content and correct commitments driven by irrelevant or contradictory evidence. To evaluate OmniConfess, we construct OmniHalluBench, a 3,540-example benchmark built from six datasets spanning text, image, audio, and video settings and both judgment and free-form generation. Experiments show that OmniConfess mitigates hallucinations across heterogeneous modality and task settings. Our code and benchmark are publicly available at https://github.com/RongHuiQiang/OmniConfess.
Huiqiang Rong, Haoran Luo, Hui Feng +4
Beijing University of Posts and Telecommunications · Nanyang Technological University, Singapore