TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

Zhaolu Kang; Yidi Wang; Yile Li; Siqi Zeng; Yingjie He; Richeng Xuan; Zhichao hu

cs.CLOct 22, 2026

TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models

Authors: Zhaolu Kang, Yidi Wang, Yile Li, Siqi Zeng, Yingjie He, Richeng Xuan, Zhichao hu

Organizations: Hunyuan, Tencent · Peking University

Abstract

Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current multimodal benchmarks mostly test perception, recognition, or reasoning over already determinate inputs, leaving unclear whether omni-modal large language models can resolve ambiguity distributed across text, vision, and audio. We introduce TRIAD, a benchmark for omni-modal ambiguity resolution. Each item is a text--image--audio triplet paired with a question and a set of answer options, and is constructed so that the full triplet determines a unique answer while removing any one modality makes the set of consistent options non-unique. TRIAD contains $327$ hand-written queries in $86$ scene groups and covers $18$ ambiguity categories across the three modalities. Evaluating $14$ omni-modal systems reveals that strong item-level performance does not translate into robust scene-level disambiguation: the best model still trails humans by over $20$ points in group accuracy. Removing answer options widens the gap further, driving model group accuracy to the single digits while humans remain robust. Leave-one-out and transcript/description counterfactuals further show that current systems often underuse audio cues when text and image are present. TRIAD reframes omni-modal evaluation around disambiguation rather than recognition, and provides a taxonomy, benchmark, and diagnostic protocol for measuring this capability.

Figures & tables

Figure 1: Overview of Triad . Each query is constructed so that text, image, and audio mutually disambiguate one another: only the full triplet determines a unique answer, while any two-modality view remains ambiguous. The benchmark contrasts human-like cross-modal disambiguation with model behavior under partial modality cues, and organizes examples into checked scene groups.

Figure 2: TRIAD distribution overview. The inner disc shows ambiguity tiers, and the outer rings show the leaf-category distribution for text, image, and audio.

Figure 3: Case examples across ambiguity tiers. The dataset contains items with varying levels of internal ambiguity across modalities. Each case shows how different modality combinations create distinct disambiguation challenges.

	Full input						Group-accuracy controls
	Std		CoT		Open		None		Text only		Image only		Audio only		No-Text		No-Image		No-Audio
Model	q	g	q	g	q	g	Std	Open	Std	Open	Std	Open	Std	Open	Std	Open	Std	Open	Std	Open
Random	24.3	2.2	–	–	–	–	2.2	–	2.2	–	2.2	–	2.2	–	2.2	–	2.2	–	2.2	–
Human	95.1	91.6	–	–	88.4	79.2	2.4	0.6	18.2	6.3	15.6	5.0	11.8	4.2	22.9	8.4	70.5	44.9	60.7	36.1
Proprietary
Gemini 3.1 Pro	90.4	71.2	91.1	72.5	18.4	4.6	0.8	0.2	42.6	8.1	8.9	1.3	3.7	0.6	16.4	2.4	57.2	12.4	51.8	10.7

Table 1: Main diagnostic results on Triad . The table reports the full diagnostic matrix used in our analysis: full text–image–audio performance under Standard, Chain-of-thought, and Open-ended prompts, plus group accuracy under no-input, single-modality, and leave-one-out controls. Query accuracy measures isolated answers, while group accuracy measures consistency across sister queries.

Panel of Figure 4 — Figure 4: Two compact diagnostics kept in the main text because they support the main claims beyond Table 1 . Left: the strongest model remains high under Text-only and leave-one-out views, revealing shortcut dependence. Right: per-query accuracy is nearly stable across group sizes, but group accuracy drops sharply. Full tables appear in Appendix E .

	Ambiguity tier			Selected ablation probes
Model	1-amb.	2-amb.	3-amb.	Full	None	Text	No-T	No-I	No-A
Gemini 3.1 Pro	95.4	94.1	87.8	90.4	29.1	85.7	55.4	87.1	82.9
Gemini 3 Pro	94.8	92.6	86.2	89.2	28.7	83.9	53.6	86.1	81.4
Gemini 3 Flash	87.9	87.7	80.6	82.5	27.4	81.1	43.6	82.1	82.0
Qwen3.5-Omni-Plus	93.9	82.3	74.4	78.0	21.1	74.5	43.8	77.5	75.5
GPT-4o	87.5	80.9	72.1	76.3	25.9	69.7	44.3	74.0	70.8

Table 2: Model-aligned Standard-prompt diagnostics. The first block reports

\mathrm{Acc}_{\mathrm{q}}

stratified by the number of individually ambiguous modalities; a

k

-ambiguous item has

k

modalities that are each individually ambiguous, while the full triplet resolves the answer. The second block reuses the same model rows to report selected query-level ablation probes: Full input, no input, text only, and leave-one-out removal of text, image, or audio.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	Global		Single modality			Leave-one-out
Model	Full	None	Text	Image	Audio	No-T	No-I	No-A
Gemini 3.1 Pro	90.4	29.1	85.7	46.2	33.5	55.4	87.1	82.9
Gemini 3 Pro	89.2	28.7	83.9	44.6	32.6	53.6	86.1	81.4
Gemini 3 Flash	82.5	27.4	81.1	36.9	30.9	43.6	82.1	82.0
Gemini 2.5 Pro	80.7	26.1	74.8	37.4	31.2	46.0	78.2	72.1
Qwen3.5-Omni-Plus	78.0	21.1	74.5	38.3	28.4	43.8	77.5	75.5

Appendix

Table 3: Modality ablation under the Standard prompt. Each cell reports

\mathrm{Acc}_{\mathrm{q}}

on the full

327

-query benchmark. Full uses all three modalities; None uses only the question stem and option list; single-modality conditions use one modality plus the question and options; and leave-one-out conditions remove the named modality.

Model	Full (T+I+A)	T+I+A+	T+I + +A	T+I + +A+
Gemini 3 Pro	89.2	83.6 ( $-5.6$ )	85.4 ( $-3.8$ )	79.8 ( $-9.4$ )
Gemini 3 Flash	82.5	77.1 ( $-5.4$ )	79.3 ( $-3.2$ )	74.0 ( $-8.5$ )
Qwen3.5-Omni-Plus	78.0	71.4 ( $-6.6$ )	74.2 ( $-3.8$ )	68.1 ( $-9.9$ )
GPT-4o	76.3	69.9 ( $-6.4$ )	72.8 ( $-3.5$ )	65.7 ( $-10.6$ )
Gemini 2.5 Flash	73.2	66.8 ( $-6.4$ )	69.5 ( $-3.7$ )	62.3 ( $-10.9$ )
MiniCPM-o-4.5	66.2	58.9 ( $-7.3$ )	62.0 ( $-4.2$ )	54.7 ( $-11.5$ )

Appendix

Table 4: Replacing non-text modalities with human-written descriptions. Each cell reports

\mathrm{Acc}_{\mathrm{q}}

under the Standard prompt on the same

327

queries. A + denotes audio replaced by a transcript with prosody and non-speech-event notes; I + denotes image replaced by a human-written scene description. Numbers in parentheses are deltas from the model’s Full score in Table 3 .

Group size	2	3	4	8
Number of groups	23	11	42	10
Number of queries	46	33	168	80
Share of queries	14.1%	10.1%	51.4%	24.5%
Per-query $\mathrm{Acc}_{\mathrm{q}}$ (in-group)	70.5	70.1	69.6	68.0
$\mathrm{Acc}_{\mathrm{g}}$	51.8	38.7	27.3	8.9

Appendix

Table 5: Group-size distribution and the relationship between group size, in-group per-query accuracy, and group-wise accuracy. Per-query and

\mathrm{Acc}_{\mathrm{g}}

numbers are averaged across all 14 models under the Standard prompt.

Benchmark	Text Ambig.	Image Ambig.	Audio Ambig.	Tri-modal	Irred.	Group	Evaluator
MME	✗	✗	✗	✗	✗	✗	Metrics
HallusionBench	✗	✓	✗	✗	✗	✗	Metrics
CODIS	✗	✓	✗	✗	✗	✓	Human / GPT
VAGUE	✓	✗	✗	✗	✗	✗	Metrics
3AM	✓	✗	✗	✗	✗	✗	Metrics
MUCAR	✓	✓	✗	✗	✓ †	✓	Human / GPT

Appendix

Table 6: Comparison of TRIAD with prior multimodal benchmarks. “Ambig.” columns mark whether the benchmark explicitly induces ambiguity in that modality. “Irred.” = whether the benchmark guarantees that no two-modality subquery is sufficient. “Group” = whether items are organized into perturbation groups.

	Standard		CoT		Open-ended
	$\mathrm{Acc}_{\mathrm{q}}$	$\mathrm{Acc}_{\mathrm{g}}$	$\mathrm{Acc}_{\mathrm{q}}$	$\mathrm{Acc}_{\mathrm{g}}$	$\mathrm{Acc}_{\mathrm{q}}$	$\mathrm{Acc}_{\mathrm{g}}$
Random	24.3	2.2	–	–	–	–
Best model (Gemini 3.1 Pro)	90.4	71.2	91.1	72.5	18.4	4.6
Human	95.1	91.6	–	–	88.4	79.2
Gap to human reference	$-4.7$	$-20.4$	$-4.0$	$-19.1$	$-70.0$	$-74.6$

Appendix

Table 7: Human and random baselines versus the best model on TRIAD. Random

\mathrm{Acc}_{\mathrm{q}}

uses the expected value under uniform guessing given per-item option counts; Random

\mathrm{Acc}_{\mathrm{g}}

uses the expected product-of-reciprocal-option-counts within each group. Human scores are the mean of three raters on a cost-controlled 100-query stratified subset; the same subset is used for Standard multiple choice and Open-ended answering. Model and random rows are computed on the full 327-query benchmark unless otherwise stated, so human rows should be read as a reference rather than a paired full-benchmark comparison.

Model	3-prompt agr. [0,1]	Std-vs-CoT agr. [0,1]	Std $\mathrm{Acc}_{\mathrm{q}}$	CoT $\mathrm{Acc}_{\mathrm{q}}$	Open $\mathrm{Acc}_{\mathrm{q}}$	Std spread
Gemini 3.1 Pro	0.17	0.88	90.4	91.1	18.4	1.5
Gemini 3 Pro	0.15	0.87	89.2	90.0	16.1	1.9
Gemini 3 Flash	0.16	0.84	82.5	82.0	17.6	1.6
Qwen3.5-Omni-Plus	0.16	0.81	78.0	78.2	17.9	1.8
GPT-4o	0.14	0.83	76.3	78.1	15.4	2.3
Gemini 2.5 Flash	0.14	0.79	73.2	74.1	15.8	5.4

Appendix

Table 8: Prompt robustness. 3-prompt agreement is the fraction of queries on which Standard, CoT, and Open-ended all return the same option (Open-ended is mapped to an option by lexical match to the option text; items with no match are counted as disagreement). This number is bounded above by Open-ended accuracy and is low across the pool, reflecting the multiple choice to Open-ended collapse. Std-vs-CoT agreement is the fraction of queries on which the two multiple choice prompts agree, which is a cleaner prompt sensitivity measure. Spread is the max-min range of

\mathrm{Acc}_{\mathrm{q}}

across three paraphrased Standard prompts on the Full condition.

	$\mathrm{Acc}_{\mathrm{q}}$				$\mathrm{Acc}_{\mathrm{g}}$
Model	Standard	Open (mean)	Open (strict)	$\Delta$	Standard	Open (mean)	$\Delta$
Human	95.1	88.4	82.7	$+6.7$	91.6	79.2	$+12.4$
Gemini 3.1 Pro	90.4	18.4	9.4	$+72.0$	71.2	4.6	$+66.6$
Gemini 3 Pro	89.2	16.1	8.8	$+73.1$	68.9	3.1	$+65.8$
Gemini 3 Flash	82.5	17.6	9.2	$+64.9$	59.4	3.5	$+55.9$
Gemini 2.5 Pro	80.7	16.9	8.6	$+63.8$	55.3	3.3	$+52.0$

Appendix

Table 9: Standard multiple choice versus Open-ended accuracy. Model rows use the same

327

queries in both settings; the human row is measured on the cost-controlled 100-query stratified subset described in Appendix E , using the same subset for Standard and Open-ended. Open-ended

\mathrm{Acc}_{\mathrm{q}}

is the mean of two independent judges, GPT-5.1 and Claude-4.5, grading short free-form answers where no option list is shown; Strict is the fraction of items both judges mark correct.

\Delta

is Standard minus Open-ended. The collapse from Standard to Open-ended is an order-of-magnitude effect across the model pool, while the human reference remains substantially more robust in both query and group accuracy.

Model	$N$	Parseable-only Acc (GPT-5.1)	Parseable-only Acc (Claude-4.5)	Agreement	$\kappa$	Human-on- disagreement
Gemini 3.1 Pro	308	24.1	12.7	0.811	0.573	0.71
Gemini 3 Pro	306	20.9	11.3	0.816	0.562	0.68
Gemini 3 Flash	316	22.9	12.3	0.810	0.574	0.70
Qwen3.5-Omni-Plus	315	24.1	12.2	0.808	0.562	0.69
Gemini 2.5 Flash	319	21.2	10.3	0.821	0.552	0.67
GPT-4o	311	20.0	9.8	0.827	0.548	0.66

Appendix

Table 10: Per-model agreement between the two Open-ended judges, GPT-5.1 and Claude-4.5, on parseable short free-form answers (no option list shown). The

N

column counts parseable responses only; these judge-specific accuracies are used for agreement analysis and are not directly averaged to obtain the full-denominator Open-ended scores in Table 1 .

\kappa

is Cohen’s kappa on the binary correct/wrong verdict. Judge agreement is the fraction of items for which the two judges return the same verdict. Human-on-disagreement is the fraction of disagreement items where the averaged judge score matches an adjudicating human grader.

Model	Short ( $<5$ s)	Medium (5–10s)	Long ( $>10$ s)
Gemini 3 Pro	90.1	88.7	88.4
Gemini 3 Flash	83.2	82.0	81.9
Qwen3.5-Omni-Plus	79.3	77.8	76.4
GPT-4o	77.1	76.0	75.2
Gemini 2.5 Flash	74.5	72.8	71.6
MiniCPM-o-4.5	67.2	65.9	65.1

Appendix

Table 11:

\mathrm{Acc}_{\mathrm{q}}

under the Standard prompt, binned by audio duration. Duration bins are broad diagnostics over the

141

clips with available duration metadata; the released benchmark contains

143

unique audio clips in total. They are included only to check whether long clips explain the main failure pattern.

	Text leaves			Image leaves			Audio leaves
Model	Prag.	Temp.	Synt.	Semiot.	Kinem.	Ground.	Prosod.	Symbol.	Refer.
Gemini 3.1 Pro	87.3	89.0	93.1	84.2	86.7	92.8	79.6	81.7	88.1
Gemini 3 Pro	85.6	87.4	91.7	82.8	84.9	91.4	77.4	79.8	86.9
Gemini 3 Flash	77.4	79.5	85.0	75.1	77.8	85.6	68.2	72.1	79.5
Qwen3.5-Omni-Plus	73.2	75.1	82.7	71.8	74.0	82.4	65.8	70.4	77.9
GPT-4o	71.6	73.2	80.5	70.4	72.5	81.2	61.3	66.7	73.8

Appendix

Table 12: Representative leaf-level

\mathrm{Acc}_{\mathrm{q}}

under the Standard prompt. Columns are selected leaves from the TRIAD taxonomy; the full

14\times 18

leaf table is released in machine-readable form. Lower audio prosody and symbolic scores match the main finding that acoustic cues are harder to use as ambiguity resolvers than text or image cues.

Family	Model	Size / Tier	$\mathrm{Acc}_{\mathrm{g}}$
Qwen omni	Qwen2.5-Omni-3B	3B	22.1
	Qwen2.5-Omni-7B	7B	30.2
	Qwen3-Omni-30B	30B	41.7
	Qwen3.5-Omni-Plus	Plus tier	52.8
Gemini	Gemini 2.5 Flash	Flash tier	45.6
	Gemini 3 Flash	Flash tier	59.4

Appendix

Table 13: Scaling within two model families on TRIAD.

\mathrm{Acc}_{\mathrm{g}}

under the Standard prompt. Parameter counts for the proprietary models are not disclosed; we list them by release tier.

Category	Share	Typical trigger
Transcript-only listening	35.0%	Audio treated as text; prosody ignored
Image-anchored commitment	25.5%	Early decision from image, post-hoc justification
Option priors	20.0%	Answer from question stem alone
Cross-modal contradiction	19.5%	Text and audio disagree; model sides with text

Appendix

Table 14: Manual categorization of 200 sampled incorrect predictions, pooled across models and prompts. Shares are out of the 200 sampled errors.

Figure 5: Qualitative case 1 with selected model responses.

Figure 6: Qualitative case 2 with selected model responses.

Figure 7: Qualitative case 3 with selected model responses.

Figure 8: Qualitative case 4 with selected model responses.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 772 papers

Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention
Mar 28, 2026Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan ParvezModality Reliance in Multimodal Models Multimodal Model Evaluation
University of California, Riverside · University of Maryland, Baltimore County · Qatar Computing Research Institute (QCRI)
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Cross-Modal Learning Multimodal Large Language Models
StepFun-Audio Team
Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization
May 11, 2026Yeongtak Oh, Dongwook Lee, Sangkwon Park +2Multimodal Grounding Multimodal Large Language Models
Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University · Department of Artificial Intelligence, University of Seoul

Mar 28, 2026J/K move · Enter open · S save

Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention

Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan Parvez

University of California, Riverside · University of Maryland, Baltimore County · Qatar Computing Research Institute (QCRI)

Modality Reliance in Multimodal Models Multimodal Model Evaluation Multimodal Robustness LLM Abstention

Open paper AskPDF arXiv

Existing omni-modal benchmarks attempt to measure modality-specific contributions, but their measurements are confounded: naturally co-occurring modalities carry correlated yet unequal information, making it unclear whether results reflect true modality reliance or information asymmetry. We introduce OMD-Bench, where all modalities are initially congruent - each presenting the same anchor, an object or event independently perceivable through video, audio, and text - which we then systematically corrupt to isolate each modality's contribution. We also evaluate calibrated abstention: whether models appropriately refrain from answering when evidence is conflicting. The benchmark comprises 4,080 instances spanning 27 anchors across eight corruption conditions. Evaluating ten omni-modal models under zero-shot and chain-of-thought prompting, we find that models over-abstain when two modalities are corrupted yet under-abstain severely when all three are, while maintaining high confidence (~60-100%) even under full corruption. Chain-of-thought prompting improves abstention alignment with human judgment but amplifies overconfidence rather than mitigating it. OMD-Bench provides a diagnostic benchmark for diagnosing modality reliance, robustness to cross-modal inconsistency, and uncertainty calibration in omni-modal systems.

Abstract

Multimodal Model Evaluation

Explore similar work

Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention

Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization