MIBench: Evaluating LMMs on Multimodal Interaction
Authors: Yu Miao, Zequn Yang, Yake Wei, Ziheng Chen, Haotian Ni, Haodong Duan, Kai Chen, Di Hu
Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
In different multimodal scenarios, it needs to integrate and utilize information across modalities in a specific way based on the demands of the task. Different integration ways between modalities are referred to as "multimodal interaction". How well a model handles various multimodal interactions largely characterizes its multimodal ability. In this paper, we introduce MIBench, a comprehensive benchmark designed to evaluate the multimodal interaction capabilities of Large Multimodal Models (LMMs), which formulates each instance as a (con_v , con_t, task) triplet with contexts from vision and text, necessitating that LMMs employ correct forms of multimodal interaction to effectively complete the task. MIBench assesses models from three key aspects: the ability to source information from vision-centric or text-centric cues, and the ability to generate new information from their joint synergy. Each interaction capability is evaluated hierarchically across three cognitive levels: Recognition, Understanding, and Reasoning. MIBench comprises over 10,000 vision-text context pairs spanning 32 distinct tasks. Evaluation of state-of-the-art LMMs show that: (1) LMMs' ability on multimodal interaction remains constrained, despite the scaling of model parameters and training data; (2) they are easily distracted by textual modalities when processing vision information; (3) they mostly possess a basic capacity for multimodal synergy; and (4) natively trained multimodal models show noticeable deficits in fundamental interaction ability. We expect that these observations can serve as a reference for developing LMMs with more enhanced multimodal ability in the future.
Figures & tables
Figure 1: (Left) Different multimodal tasks require different types of modality interaction. Some tasks depend on information mostly from one modality (visual or textual context), while others require synergistic collaboration between them. (Middle) We introduce MIBench to systematically evaluate how well Large Multimodal Models (LMMs) handle multimodal interactions, which is structured around three fundamental interaction patterns (Vision-centric, Text-centric, and Synergy) across three cognitive levels (Recognition, Understanding, and Reasoning). (Right) Most current LMMs demonstrate regrettably limited capabilities in multimodal interaction (especially in synergy), struggling to selectively utilize cues from centric modality and achieve cross-modal collaboration.
Figure 2: Overview of the framework and samples. MIBench covers three interaction forms (Vision-centric, Text-centric, and Synergy) and three hierarchical levels of ability: Recognition, Understanding, and Reasoning. Each sample is composed of a textual context, a visual context, and a task (Q&A). For the vision-centric and text-centric samples, multiple types of contexts from another modality are prepared in our evaluation, though only one is shown here for illustrative purposes.
Figure 3: Overview of MIBench sample formats. Each instance consists of both visual and textual contexts and the corresponding task, formulated as ( conv , cont , task ). For vision- and text-centric tasks (Left and Middle) , they can be resolved by leveraging cues from the centric modality. We introduce various contexts from another modality to evaluate the model’s ability to selectively utilize cues from the target modality, which range from helpful contexts (e.g., correct guidance, concept visualization) to misleading guidance and unrelated contents. For the synergy part (Right) , the model is presented with one coupled visual-textual pair with complementary cues, necessitating effective cross-modal collaboration.
Figure 4: Overview of the sample annotation pipeline.
MIBench
MIBench-mini
Model
Vision
Text
Synergy
Overall
Vision
Text
Synergy
Overall
Open-source
InternVL2-8B
53.29
62.53
45.79
54.23
49.60
59.00
45.00
51.20
InternVL2.5-8B
59.61
65.03
47.43
57.62
58.80
63.20
45.00
55.67
InternVL3-8B
61.93
65.82
48.05
58.82
59.20
65.20
45.00
56.47
InternVL3.5-8B
57.52
67.28
51.03
58.98
57.60
67.00
53.00
59.20
Table 1: Performance of state-of-the-art open and closed-source models on MIBench and MIBench-mini. The best score is bolded and the second-best is underlined, within the open-source and closed-source groups separately.
Figure 5: Performance of open-source LMMs across three progressively cognitive levels: Recognition (Level 1), Understanding (Level 2), and Reasoning (Level 3).
Figure 6: Impact of version iteration ( Left ) and parameter scaling ( Right ) on multimodal interaction capabilities.
Figure 7: A comparison of LMMs’ modality source selection consistency evaluated on MIBench-mini, with dot sizes scaled according to model parameter counts.
Figure 8: Impact of parameter scaling on synergy capabilities (on Qwen2.5-VL).
Figure 9: Comparison of LMMs’ performance within different series. ( Left: LLaVA; Right: Qwen-VL)
Figure 10: Left: non-native vs. native LMMs. Right: Performance of SOTA open-source and closed-source LMMs.
Figure 11: LMM’s performance on different task types.
Figure 12: The left table shows more statistics of MIBench. The right graph shows proportion of sources for collected images.
Model
Source
Version
URL
Closed-source Models
Gemini-3-pro-preview
Google
stable
https://ai.google.dev/gemini-api/docs/models
Gemini-2.5-pro
Google
stable
https://ai.google.dev/gemini-api/docs/models
Gemini-2.5-flash
Google
stable
https://ai.google.dev/gemini-api/docs/models
GPT-5.1
OpenAI
2025-11-13
https://platform.openai.com/docs/models/gpt-5.1
GPT-5.2
OpenAI
2025-12-11
https://platform.openai.com/docs/models/gpt-5.2
Table 2: Detailed source information of the LMMs evaluated in this study.
Figure 13: Prompt used to add textual contexts for a vision sample.
Figure 14: Prompt used to obtain image captions for generating various visual contexts.
Figure 15: Prompt used to annotate newly-built samples based on the given task and image, here taking temporal simulation as an example
Figure 16: Data annotation tool interface for human filtering and quality checks. The left panel shows annotation statistics, while the right view displays the visual and textual contexts. Human annotators decide whether to discard, retain, or locally revise each instance based on its question quality.
Figure 17: A case where o3 fails in counting when faced with a misleading description of the visual context.
Figure 18: A case where Gemini 2.5 Pro fails in attribute comparison when faced witha misleading description of the visual context.
Figure 19: A case that GPT 5.1 fails in spatial reasoning when faced with a totally irrelevant textual context, indicating its shortcomings in deep visual reasoning.
Figure 20: A case that GPT-4o fails in checking for grammatical errors when facing a visual context with related concepts, indicating that visual context affects the model’s understanding of the textual context.
Figure 21: A case that Claude Sonnet 4.5 fails in determining the similarity relation between sentences when facing a visual context with misleading visual guidance.
Figure 22: A case that GPT 5.1 fails in solving a textual chemistry problem when faced with an unrelated image.
Figure 23: A case that Gemini 3 Pro Preview fails in basic position alignment, indicating a strong textual bias and limited attention to visual content.
Figure 24: A case that o3 fails in rule execution tasks, indicating its deficiencies in handling frequent complex cross-modal interactions in natural scenes.
Figure 25: A case that o3 fails in solving business problem.
Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.
Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5
Kyoto University · CyberAgent. · The Institute of Statistical Mathematics +1
Multimodal large language models (MLLMs) can process diverse inputs, e.g., text, images, audio, and video, and generate textual responses. While their capabilities have advanced rapidly, evaluation of such models has not kept pace. Most existing evaluation benchmarks are limited to isolated tasks and reveal little about whether a model integrates information across modalities. We examine current means for evaluating MLLMs and review the existing benchmark taxonomy to identify gaps, including temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention. Addressing these gaps is essential for measuring real progress in multimodal intelligence and exposing capability boundaries.
Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M3Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M3Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.
Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4