Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models

Rohit Patel; Dieuwke Hupkes; Sloan Strader

cs.CVAug 26, 2026

Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models

Authors: Rohit Patel, Dieuwke Hupkes, Sloan Strader

Organizations: Meta Superintelligence Labs

Abstract

Frontier language models are increasingly marketed as omni systems that can perceive and respond across modalities. Existing evaluation frameworks, however, focus almost exclusively on bimodal understanding, typically text plus one other modality. We propose the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video and document) and combinations of up to three modalities in both inputs and outputs. MMI consists of 893 questions, each carefully crafted to require the model to demonstrate its understanding of multiple input modalities and to generate responses that incorporate various output formats. The questions are designed to be self-contained, with clear expectations for the correct modality or mix of modalities required for an accurate response. Every MMI prompt carries human-authored rubric criteria for each output modality expected in the response; a model's MMI Value expresses the average of the per-modality scores for each prompt. Because low scores can reflect either failure to generate a modality (lack of presence) or failure to generate correct content, we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities. Applying MMI to five frontier multimodal models, we find that the MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the low availability of returned modalities to even grade, we report MPS as our main result pending model improvements. To assess the viability of judging output correctness with LLM judges and rubrics, we run a separate experiment with custom generation tools. On the assets that generates, we find that an LLM judge applying the rubrics agrees with rubric-blind human annotators (who score the outputs directly and never see the criteria) on 70.8% of judgments.

Figures & tables

Figure 1: Per-model precision, recall and F1 (MPS) on MMI.

Benchmark	T	I	A	V	D	T	I	A	V	D	modal out?
	Inputs					Outputs					Cross-
MME, MMBench, MMMU ( Fu et al., 2023 ; Liu et al., 2024a ; Yue et al., 2024 )	✓	✓	✗	✗	✗	✓	✗	✗	✗	✗	✗
Video-MME, MVBench ( Fu et al., 2024a ; Li et al., 2024d )	✓	✗	$\bm{\sim}$	✓	✗	✓	✗	✗	✗	✗	✗
AudioBench, MMAU ( Wang et al., 2025a ; Sakshi et al., 2024 )	✓	✗	✓	✗	✗	✓	✗	✗	✗	✗	✗
DocVQA, MMLongBench-Doc ( Mathew et al., 2021 ; Ma et al., 2024 )	✓	✗	✗	✗	✓	✓	✗	✗	✗	✗	✗
T2I-CompBench, GenAI-Bench ( Huang et al., 2023 ; Li et al., 2024a )	✓	✗	✗	✗	✗	✗	✓	✗	✗	✗	✗

Table 1: Modality coverage of related academic benchmarks. For each benchmark, we indicate whether it includes Text (T), Image (I), Audio (A), Video (V) and Document (D) on the input and output side, and whether it evaluates the model’s choice of output modality across multiple non-text options (“Cross-modal out?”). ✓ indicates support, ✗ indicates no support, and

\bm{\sim}

indicates partial support (e.g. a separate single-modality generation track that is not jointly evaluated with the understanding side).

Prompt	Input	Output
What’s the average salary for my employees?	Document	Text
Omelet recipe, can you talk me through the steps as I go?	Image	Audio
Can you help me remove the people from this photo and then turn it into a painting? Please make the sky pink!	Text, Image	Image
Can you tell me what animal this is and pronounce its name slowly?	Audio, Video	Audio
I would like to recreate this look. Can you list out all the products that I will need and write out a very detailed instruction in a .txt file?	Text, Audio, Video	Document
Can you show me how to make soft chocolate chip cookies? A list of ingredients and a tutorial video would be nice.	Text	Text, Video

Table 2: Example prompts from the benchmark, showing the input assets provided to the model and the desired output modalities.

Model	In	Out	In	Out	In	Out	In	Out	In	Out
	Text		Image		Audio		Video		Document
GPT-5.4	✓	✓	✓	✓	✗	✗	✗	✗	$\bm{\sim}$	✗
Gemini 3.1	✓	✓	✓	✗	✓	✗	✓	✗	✓	✗
Claude Opus 4.6	✓	✓	✓	✗	✗	✗	✗	✗	$\bm{\sim}$	✗
Llama 4 Maverick	✓	✓	✓	✗	✗	✗	✓	✗	$\bm{\sim}$	✗

Table 3: Natively supported modalities. Modalities natively supported by the different models. ✓ indicates the model supports the modality, ✗ indicates it does not;

\bm{\sim}

indicates that the modality is partly supported, e.g. a model accepts .txt and .pdf documents, but not .docx .

Model	F1	Recall	Precision	Pass rate	Input failure rate
GPT-5.4	34.9	33.8	99.8	31.0	30.1%
Gemini 3.1 Flash	27.2	25.4	99.4	20.3	8.2%
Gemini 3.1 Pro	25.4	23.6	99.7	18.6	8.0%
Llama 4 Maverick	16.3	15.1	100	12.0	36.5%
Claude Opus 4.6	15.6	14.5	100	11.3	37.0%

Table 4: Summary model scores. Per-model sample-level averages of F1 (the MPS), recall, precision, pass rate and input failure rate. Models are sorted by MPS.

Figure 13: Detection method per modality. Models can return assets of the right modality natively or via a URL. As a fallback, we ask a judge to parse the answer and detect whether an asset of the correct modality is returned.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 17: Full modality grid. We show the distribution of input and output modalities of the full Modality Maturity Index benchmark.

	MMI Value		MMI Value by modality
Model	All	Excl. doc.	Text	Image	Audio	Video	Doc.
Gemini 3.5 Flash	0.51	0.63	0.69	0.78	0.53	0.47	0.00
Claude Opus 4.8	0.36	0.44	0.51	0.46	0.42	0.32	0.00
GPT-5.4	0.29	0.35	0.59	0.39	0.20	0.22	0.00

Appendix

Table 5: MMI Values for the three tool-scaffolded models of section 4.3 . All is the MMI Value as defined in section 3.4.1 : the mean over the 893 prompts of each prompt’s mean across its output modalities of the mean criterion score within that modality. Excl. doc. is the same quantity with document criteria removed, over the prompts that retain at least one non-document output modality. The five modality columns apply that same within-modality mean, averaged over the prompts for which the modality in question is expected; because different combinations of modalities are expected across prompts, the columns do not average to the MMI Value. Document is zero throughout because no document-generation tool was provided.

Explore similar work

May 23, 2025cs.AI

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation, especially for complex tasks that involve multiple modalities. We present MMMG, the first benchmark to bring the verifiable-task paradigm to multimodal generation, spanning 4 modality combinations (image, audio, interleaved text and image, interleaved text and audio). As few multimodal outputs can be checked by programs alone, MMMG targets tasks that are either verifiable or near-verifiable: by providing references, and constraining model judges with explicit rubrics. We keep tasks challenging for generation models while enabling reliable automatic evaluation through a combination of models and programs. MMMG encompasses 55 tasks (including 31 newly developed ones), each with a carefully designed evaluation pipeline, and 1288 instructions to systematically assess reasoning, controllability, and other key capabilities of multimodal generation models. Extensive validation demonstrates that MMMG is highly aligned with human judgment, achieving an average agreement of 94.4%. Benchmarking results on 29 models reveal that even though the state-of-the-art model, GPT Image, achieves 70.7% accuracy for image generation, it falls short on interleaved generation. Furthermore, results suggest considerable improvement space in audio generation, highlighting an important future direction.

Jihan Yao, Yushi Hu, Wenyuan Wang +10

University of Washington · Allen Institute for AI

Multimodal Generation Multimodal Model Evaluation

Ask PDF

Jun 24, 2026cs.AI

What We are Missing in Multimodal LLM Evaluation?

Multimodal large language models (MLLMs) can process diverse inputs, e.g., text, images, audio, and video, and generate textual responses. While their capabilities have advanced rapidly, evaluation of such models has not kept pace. Most existing evaluation benchmarks are limited to isolated tasks and reveal little about whether a model integrates information across modalities. We examine current means for evaluating MLLMs and review the existing benchmark taxonomy to identify gaps, including temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention. Addressing these gaps is essential for measuring real progress in multimodal intelligence and exposing capability boundaries.

Po-han Li, Shenghui Chen, Sandeep Chinchali +1

The University of Texas at Austin, USA

Spatiotemporal Reasoning Multimodal Large Language Models Multimodal Reasoning Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Aug 25, 2026cs.AI

OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses

Multimodal understanding models that can jointly judge text-to-image (T2I), text-to-video (T2V) and text-to-speech (TTS) generation are increasingly used as "OmniJudges" for evaluation and automatic annotation. How reliably they understand what they score remains unclear, since existing benchmarks and training data tend to overemphasize positive examples and to conflate distinct failure modes, so a judge may score well without recognizing failures while its capability gaps stay hidden. Motivated by this, we introduce D3-Omni, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding, covering 53 orthogonal binary dimensions (17/22/14) and 10,671 samples (3,526/1,998/5,147) across the three tasks. Rather than re-generating outputs, which may leak information across dimensions, we fix verified fully positive seeds and derive negatives through controlled prompt rewriting and atomic, dimension-isolating perturbations. The resulting D3 design is Dual-balanced, which helps alleviate negative-sample scarcity and per-dimension label imbalance; Decoupled, so that each error is attributable to a single capability; and Dynamic, steering construction toward under-represented regions of the label distribution as generative models improve.The suite reaches near 1:1 per-dimension parity and a uniform distribution over all total-score levels. Under this balanced view, even strong OmniJudges tend to struggle on modality-related dimensions, to confirm satisfied requirements far more reliably than they detect violated ones, and to treat nominally distinct attributes as largely a single decision, suggesting that aggregate accuracy may hide systematic blind spots that a balanced and decoupled lens can help expose and, in turn, address.

Guangzheng Hu, Ziyue Jiang, Weixu Qiao +14

Alibaba Group · University of Melbourne · Qwen Team +2

Multimodal Model Evaluation Multimodal Understanding Vision-Language Model-as-a-Judge

Ask PDF