Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

Shunki Uebayashi; Kento Masui; Kyohei Atarashi; Han Bao; Hisashi Kashima; Naoto Inoue; Mayu Otani; Koh Takeuchi

cs.CLMar 3, 2026

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

Authors: Shunki Uebayashi, Kento Masui, Kyohei Atarashi, Han Bao, Hisashi Kashima, Naoto Inoue, Mayu Otani, Koh Takeuchi

Organizations: Kyoto University · CyberAgent. · The Institute of Statistical Mathematics · Tohoku University

Abstract

Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Questions with the highest or lowest cross-modal difficulty $b_{j}^{\rm cross}$ detected by M3IRT. Questions with high cross-modal difficulty require both modalities to find the correct answer. However, those with low difficulty allow us to solve using only the image or text.

Figure 2: M2IRT investigates the modality-specific and cross-modal difficulties of questions that enables to contract a tailored, compact, and high-quality subset for evaluating a new MLLM.

Panel of Figure 3 — Figure 3: Distributions of $\theta$ estimated by M3IRT sorted in descending order.

Panel of Figure 4 — Figure 4: The average and standard deviation of Spearman’s rank correlations between model rankings on the original benchmark and those estimated on extracted question subsets with different sizes.

Panel of Figure 5 — Figure 5: The average and standard deviation of the proportions of the low-quality questions in extracted question subsets $\gamma$ with different sizes.

Panel of Figure 6 — Figure 6: ROC-AUC on predicting answers of the noisy benchmarks containing the different size of the low-quality questions inserted into the original benchmark.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

	MMMU	MathVista	SEED-Bench
$\text{Decline}_{max}$	0.032	-0.23	0.18
Corr. between $\text{Decline}_{max}$ and $b_{j}^{\rm cross}$	0.090	-0.15	0.35
$\text{Decline}_{mean}$	0.14	-0.11	0.31
Corr. between $\text{Decline}_{mean}$ and $b_{j}^{\rm cross}$	0.077	-0.16	0.35
Fleiss’ $\kappa$	0.25	0.28	0.44
Accuracy of Crowd workers	0.41	0.23	0.62

Appendix

Table 1: Relationships between VLMs and human answer patterns

Panel of Figure 7 — Figure 7: Distributions of $\theta$ estimated by M2IRT sorted in descending order.

Figure 8: Distributions of

\theta

estimated by M3IRT sorted in descending order.

Panel of Figure 9 — Figure 9: MMMU : Questions with the high cross-modal difficulties $b_{j}^{\rm cross}$

Panel of Figure 10 — Figure 10: MMMU : Questions with the low cross-modal difficulties $b_{j}^{\rm cross}$ .

Panel of Figure 11 — Figure 11: MathVista : Questions with the high cross-modal difficulties $b_{j}^{\rm cross}$ . Each caption is the ID of the problem in MathVista

Panel of Figure 12 — Figure 12: MathVista : Questions with the low cross-modal difficulties $b_{j}^{\rm cross}$ . Each caption is the ID of the problem in MathVista

Panel of Figure 13 — Figure 13: SEED-Bench : Questions with the high cross-modal difficulties $b_{j}^{\rm cross}$ .

Panel of Figure 14 — Figure 14: VQAAT : Questions with the low cross-modal difficulties $b_{j}^{\rm cross}$

model	$\theta_{\rm base}$	$\theta_{\rm image}$	$\theta_{\rm text}$	$\theta_{\rm cross}$	total
Gemini-1.5-flash-8b	0.00	1.2	0.00	4.0	5.2
Claude-3.7-sonnet	0.03	0.78	4.0	0.00	4.8
Claude-3.5-sonnet	0.06	0.68	4.0	0.00	4.7
Gemini-2.0-flash	0.09	1.4	0.68	2.1	4.3
Gemini-1.5-pro	0.10	1.0	0.29	2.8	4.2
GPT-4.1-mini	0.09	1.8	0.70	1.4	3.9

Appendix

Table 2: Estimated

\theta

on MMMU

model	$\theta_{\rm base}$	$\theta_{\rm image}$	$\theta_{\rm text}$	$\theta_{\rm cross}$	total
Gemini-2.0-flash	0.00	0.93	0.99	4.0	5.9
Claude-3.7-sonnet	0.00	0.63	2.0	3.1	5.7
Nova-Pro	0.10	0.36	4.0	0.56	5.0
Grok-2-Vision	0.10	0.89	3.4	0.57	5.0
Gemini-1.5-pro	0.00	0.97	1.4	2.6	4.9
Gemini-1.5-flash	0.00	0.60	0.84	2.7	4.1

Appendix

Table 3: Estimated

\theta

on Mathvista

model	$\theta_{\rm base}$	$\theta_{\rm image}$	$\theta_{\rm text}$	$\theta_{\rm cross}$	total
GPT-4.1	0.10	2.0	1.9	2.0	6.0
GPT-4o	0.00	2.0	0.51	2.0	4.5
GPT-4.1-mini	0.10	2.0	2.0	0.00	4.1
Gemini-2.0-flash	0.10	2.0	2.0	0.00	4.1
Gemini-1.5-pro	0.10	2.0	2.0	0.00	4.1
Qwen-2.5-VL-72b	0.10	2.0	2.0	0.00	4.1

Appendix

Table 4: Estimated

\theta

on SeedBench

Figure 15: An illustration of our benchmark refinement and testing processes.

Panel of Figure 16 — Figure 16: Comparisons of parameters estimated by M2IRT between the original and artificial questions.

Panel of Figure 17 — Figure 17: Comparisons of parameters estimated by M3IRT between the original and artificial questions.

Benchmark	Method	10%	20%	30%	40%	50%
MMMU	M2IRT	0.96 $\pm$ 0.061	0.96 $\pm$ 0.057	0.96 $\pm$ 0.047	0.97 $\pm$ 0.042	0.96 $\pm$ 0.035
	M3IRT	0.89 $\pm$ 0.046	0.92 $\pm$ 0.030	0.92 $\pm$ 0.033	0.93 $\pm$ 0.023	0.93 $\pm$ 0.023
	IRT	0.50 $\pm$ 0.21	0.36 $\pm$ 0.25	0.38 $\pm$ 0.20	0.35 $\pm$ 0.23	0.12 $\pm$ 0.28
	MIRT	0.16 $\pm$ 0.089	0.33 $\pm$ 0.091	0.47 $\pm$ 0.11	0.61 $\pm$ 0.076	0.69 $\pm$ 0.055
	TinyBenchmarks	0.43 $\pm$ 0.13	0.50 $\pm$ 0.12	0.56 $\pm$ 0.11	0.61 $\pm$ 0.11	0.67 $\pm$ 0.082
	FlashEval	0.79 $\pm$ 0.029	0.79 $\pm$ 0.025	0.79 $\pm$ 0.024	0.80 $\pm$ 0.017	0.80 $\pm$ 0.015

Appendix

Table 5: The average and standard deviation of Spearman’s rank correlations between model rankings on the original benchmark and those estimated on extracted question subsets with 10%, 20%, 30%, 40%, and 50% of whole dataset.

Benchmark	Method	10%	20%	30%	40%	50%
MMMU	M2IRT	0.038 $\pm$ 0.015	0.090 $\pm$ 0.014	0.12 $\pm$ 0.014	0.16 $\pm$ 0.014	0.19 $\pm$ 0.013
	M3IRT	0.091 $\pm$ 0.017	0.11 $\pm$ 0.013	0.14 $\pm$ 0.012	0.16 $\pm$ 0.011	0.19 $\pm$ 0.0070
	IRT	0.32 $\pm$ 0.040	0.34 $\pm$ 0.035	0.36 $\pm$ 0.029	0.37 $\pm$ 0.023	0.38 $\pm$ 0.020
	MIRT	0.42 $\pm$ 0.020	0.40 $\pm$ 0.016	0.38 $\pm$ 0.012	0.37 $\pm$ 0.012	0.36 $\pm$ 0.011
	TB	0.38 $\pm$ 0.023	0.39 $\pm$ 0.021	0.38 $\pm$ 0.022	0.37 $\pm$ 0.021	0.36 $\pm$ 0.020
	FE	0.28 $\pm$ 0.014	0.29 $\pm$ 0.017	0.29 $\pm$ 0.021	0.27 $\pm$ 0.017	0.29 $\pm$ 0.010

Appendix

Table 6: The average and standard deviation of the proportions of the low-quality questions in extracted question subsets

\gamma

with 10%, 20%, 30%, 40%, and 50% of whole dataset. "TB" means TinyBenchmarks, and "FE" means FlashEval.

Panel of Figure 18 — Figure 18: The average and standard deviation of Spearman’s rank correlations on extracted question subsets of VQAAT and the proportions of the low-quality questions $\gamma$ in extracted question subsets with different sizes.

Panel of Figure 19 — Figure 19: The average and standard deviation of Spearman’s rank correlations between model rankings on the original benchmark and those estimated on extracted question subsets with different sizes.M2IRT and M3IRT are trained with sparse-response matrix.

Panel of Figure 20 — Figure 20: The average and standard deviation of the proportions of the low-quality questions in extracted question subsets $\gamma$ with different sizes. M2IRT and M3IRT are trained with sparse-response matrix.

Comparison	5% subset		10% subset		30% subset		50% subset
	$p$ -value	$W$	$p$ -value	$W$	$p$ -value	$W$	$p$ -value	$W$
vs FlashEval (Rank corr.)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0
vs TinyBench (Rank corr.)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0
vs FlashEval (Shuffle ratio)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0
vs TinyBench (Shuffle ratio)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0

Appendix

Table 7: Wilcoxon signed-rank test on MMMU comparing FlashEval and TinyBench against M3IRT.

Comparison	5% subset		10% subset		30% subset		50% subset
	$p$ -value	$W$	$p$ -value	$W$	$p$ -value	$W$	$p$ -value	$W$
vs FlashEval(Rank Corr.)	0.0197	222.0	0.0004	262.0	0.0019	251.5	$<0.0001$	293.0
vs TinyBench(Rank Corr.)	$<0.0001$	300.0	$<0.0001$	300.0	$<0.0001$	300.0	$<0.0001$	300.0
vs FlashEval(Shuffle ratio)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0
vs TinyBench(Shuffle ratio)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0

Appendix

Table 8: Wilcoxon signed-rank test on MathVista comparing FlashEval and TinyBench against M3IRT.

Comparison	5% subset		10% subset		30% subset		50% subset
	$p$ -value	$W$	$p$ -value	$W$	$p$ -value	$W$	$p$ -value	$W$
vs FlashEval (Rank corr.)	$<0.0001$	231.0	$<0.0001$	231.0	$<0.0001$	231.0	$<0.0001$	231.0
vs TinyBench (Rank corr.)	$<0.0001$	231.0	$<0.0001$	231.0	$<0.0001$	231.0	$<0.0001$	231.0
vs FlashEval (Shuffle ratio)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0
vs TinyBench (Shuffle ratio)	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0	$<0.0001$	0.0

Appendix

Table 9: Wilcoxon signed-rank test on SEEDBench comparing FlashEval and TinyBench against M3IRT.

Component	Specification
Operating System	Ubuntu 20.04 LTS
CPU	AMD EPYC Milan 7763 DP/UP (64C/128T, 2.45GHz) $\times$ 2
Memory	2048GB
python version	3.12.9
torch version	2.6.0

Appendix

Table 10: Computer Specifications Used for Experiments

Model Name	Type	License or Terms
GPT-4-turbo	Closed	OpenAI Terms of Use
GPT-4o ( OpenAI, 2024b )	Closed	OpenAI Terms of Use
GPT-4o-mini ( OpenAI, 2024a )	Closed	OpenAI Terms of Use
GPT-4.1 ( OpenAI, 2025 )	Closed	OpenAI Terms of Use
GPT-4.1-mini ( OpenAI, 2025 )	Closed	OpenAI Terms of Use
GPT-4.1-nano ( OpenAI, 2025 )	Closed	OpenAI Terms of Use

Appendix

Table 11: Overview of AI Models Used

Explore similar work

MIBench: Evaluating LMMs on Multimodal Interaction
Mar 13, 2026Yu Miao, Zequn Yang, Yake Wei +5Multimodal Large Language Models Multimodal Model Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China +1
CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
Aug 31, 2026Jiacheng Cai, Zijin Hong, Zheng Yuan +3Multimodal RAG Multimodal QA
The Hong Kong Polytechnic University · Jilin University
MMGist: A Comprehensive Multimodal Benchmark for 2027
Jun 21, 2026Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong +3VLM Evaluation Large Vision-Language Models
1Shanghai Jiao Tong University · 2Sichuan University

Mar 13, 2026J/K move · Enter open · S save

MIBench: Evaluating LMMs on Multimodal Interaction

Yu Miao, Zequn Yang, Yake Wei, Ziheng Chen+4

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China+1

Multimodal Large Language Models Multimodal Model Evaluation Multimodal Understanding Multimodal Reasoning

Open paper AskPDF arXiv

In different multimodal scenarios, it needs to integrate and utilize information across modalities in a specific way based on the demands of the task. Different integration ways between modalities are referred to as "multimodal interaction". How well a model handles various multimodal interactions largely characterizes its multimodal ability. In this paper, we introduce MIBench, a comprehensive benchmark designed to evaluate the multimodal interaction capabilities of Large Multimodal Models (LMMs), which formulates each instance as a (con_v , con_t, task) triplet with contexts from vision and text, necessitating that LMMs employ correct forms of multimodal interaction to effectively complete the task. MIBench assesses models from three key aspects: the ability to source information from vision-centric or text-centric cues, and the ability to generate new information from their joint synergy. Each interaction capability is evaluated hierarchically across three cognitive levels: Recognition, Understanding, and Reasoning. MIBench comprises over 10,000 vision-text context pairs spanning 32 distinct tasks. Evaluation of state-of-the-art LMMs show that: (1) LMMs' ability on multimodal interaction remains constrained, despite the scaling of model parameters and training data; (2) they are easily distracted by textual modalities when processing vision information; (3) they mostly possess a basic capacity for multimodal synergy; and (4) natively trained multimodal models show noticeable deficits in fundamental interaction ability. We expect that these observations can serve as a reference for developing LMMs with more enhanced multimodal ability in the future.

Abstract

Explore similar work

MIBench: Evaluating LMMs on Multimodal Interaction

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

MMGist: A Comprehensive Multimodal Benchmark for 2027