Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration

Shiqi Zhang; Weixin Zeng; Ziheng Zhang; Jiuyang Tang; Xiang Zhao

cs.AIOct 10, 2026

Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration

Authors: Shiqi Zhang, Weixin Zeng, Ziheng Zhang, Jiuyang Tang, Xiang Zhao

Organizations: National Key Laboratory of Information Systems Engineering, National University of Defense Technology, Changsha, China · National Key Laboratory of Big Data and Decision, National University of Defense Technology, Changsha, China

Abstract

Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has revealed that some benchmarks may contain shortcut items that can be answered from a single modality or from answer-option regularities alone, thus failing to reflect models' true cross-modal inference ability. Existing methods for assessing benchmarks typically rely either on intrinsic item structure or extrinsic model response behavior, which are not adequate for a comprehensive diagnosis. To fill in this gap, we propose SRCoD, a structure-response co-calibrated diagnosis framework for item-level diagnosis of modality dependence in multimodal benchmarks. SRCoD estimates sample-wise multimodal information structure from benchmark content and target answers, and uses this signal as a soft structural anchor for modality-conditioned item response modeling. The learned model produces a calibrated item-level diagnostic profile that reflects how strongly an item depends on different forms of modality evidence, enabling fine-grained benchmark analysis beyond aggregate accuracy. SRCoD also provides interpretable attribution of the diagnosis results. Extensive experiments under two complementary protocols, i.e., benchmark-refinement oriented validation and human-labeled item validation, demonstrate that SRCoD outperforms state-of-the-art benchmark diagnosis methods.

Figures & tables

Figure 1: Overview of SRCoD. SRCoD diagnoses item-level modality dependence by co-calibrating intrinsic item structure and extrinsic model responses. It first derives a structural prior from partial information decomposition, then anchors a modality-conditioned response model fitted to VLM responses under controlled modality conditions.

Figure 2: Benchmark-refinement oriented validation under contaminated candidate pools. Each method selects compact subsets from candidate pools containing equal numbers of clean benchmark items and synthetic low-quality items. We report the Spearman rank correlation between the model ranking induced by the selected subset and the ranking induced by the full clean benchmark.

Method	ScienceQA		MMMU		SEED-Bench		Average
Method	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1
Random	0.251	0.233	0.249	0.241	0.250	0.242	0.250	0.239
Raw	0.353	0.334	0.503	0.435	0.388	0.355	0.415	0.375
LSMI	0.464	0.255	0.265	0.171	0.309	0.189	0.346	0.205
IRT	0.507	0.476	0.384	0.382	0.494	0.474	0.462	0.444
MIRT	0.506	0.476	0.371	0.370	0.494	0.473	0.457	0.440

Table 1: Human-labeled item validation. Accuracy and macro-F1 measure agreement with human evidence-sufficiency references in the four-factor diagnostic space. Bold and underline denote the best and second-best results, respectively.

Figure 3: Ablation study of SRCoD. We compare the full model with variants that remove entropy-based confidence weighting or the profile-level structural alignment loss.

Figure 4: Robustness to model-pool subsampling. For each pool size, we randomly sample diagnostic models from the full model pool and repeat the experiment 10 times. We report performance retention relative to the full-pool setting. Shaded regions denote one standard deviation.

Figure 5: Qualitative example of the diagnostic output produced by SRCoD. The item is diagnosed as Redundant , indicating that both the image and the text provide sufficient evidence for answering the question.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Distribution of human-annotated modality-dependence labels across ScienceQA, MMMU, and SEED-Bench. Each pie chart shows the proportion of Redundant, Visual-dominant, Textual-dominant, and Synergistic items in the corresponding benchmark.

Component	Specification
CPU	12th Gen Intel(R) Core(TM) i7-12700KF @ 3.60 GHz
RAM	32 GB
GPU	NVIDIA GeForce RTX 4090, 24 GB
Python version	3.10.1
PyTorch version	2.7.0
CUDA version	12.6

Appendix

Table 2: Computational resources used for experiments.

Model Name	Type	License or Terms
gpt-4.1 [ 20 ]	Closed	OpenAI Terms of Use
gpt-4.1-nano [ 20 ]	Closed	OpenAI Terms of Use
gpt-4o-mini [ 20 ]	Closed	OpenAI Terms of Use
gemini-3.1-pro-preview [ 25 ]	Closed	Gemini API Additional Terms of Service
gemini-3.1-flash-lite [ 25 ]	Closed	Gemini API Additional Terms of Service
gemini-2.5-flash-lite [ 25 ]	Closed	Gemini API Additional Terms of Service

Appendix

Table 3: Overview of AI Models Used

Condition	Image	Question text	Answer options
Full multimodal	Yes	Yes	Yes
Visual-conditioned	Yes	No	Yes
Textual-conditioned	No	Yes	Yes
No-modality	No	No	Yes

Appendix

Table 4: Input information provided under each modality condition.

Figure 7: Examples of redundant items from ScienceQA, MMMU, and SEED-Bench.

Figure 8: Examples of textual-dominant items from ScienceQA, MMMU, and SEED-Bench.

Figure 9: Examples of visual-dominant items from ScienceQA, MMMU, and SEED-Bench.

Figure 10: Examples of synergistic items from ScienceQA, MMMU, and SEED-Bench.

Dataset	Accuracy			Macro-F1			McNemar
Dataset	$\Delta$	95% CI	$p_{\mathrm{rand}}$	$\Delta$	95% CI	$p_{\mathrm{rand}}$	McNemar
ScienceQA	+0.037	[0.015, 0.060]	$<0.001$	+0.049	[0.026, 0.075]	$<0.001$	$87/50,\ p=0.002$
MMMU	+0.061	[0.035, 0.089]	$<0.001$	+0.054	[0.025, 0.079]	$<0.001$	$129/68,\ p<0.001$
SEED-Bench	+0.067	[0.042, 0.091]	$<0.001$	+0.070	[0.043, 0.095]	$<0.001$	$115/48,\ p<0.001$
Average	+0.055	[0.040, 0.069]	$<0.001$	+0.058	[0.044, 0.072]	$<0.001$	$331/166,\ p<0.001$

Appendix

Table 5: Paired statistical significance analysis of SRCoD over M3IRT on the human-labeled diagnostic validation set. We report paired performance differences, 95% bootstrap confidence intervals, approximate-randomization

p

-values, and McNemar’s test for accuracy.

Method	ScienceQA		MMMU		SEED-Bench		Average
Method	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1	Acc.	Macro-F1
$\alpha=0.25$	0.52	0.44	0.46	0.46	0.61	0.60	0.53	0.50
$\alpha=0.50$	0.47	0.36	0.45	0.44	0.60	0.59	0.51	0.46
$\alpha=0.75$	0.44	0.30	0.45	0.44	0.60	0.58	0.50	0.44
SRCoD	0.61	0.59	0.55	0.54	0.69	0.67	0.61	0.60

Appendix

Table 6: Comparison with post-hoc profile fusion baselines. We report accuracy and macro-F1 on the human-labeled validation set.

Figure 11: Diagnostic performance under a small-scale model pool. We compare M3IRT and SRCoD using responses from 3B-12B MLLMs and report accuracy and macro-F1. SRCoD consistently outperforms M3IRT and shows smaller degradation on the subset, indicating stronger robustness when response-side evidence is obtained from smaller and noisier models.

Held-out Family	Benchmark	$G_{\mathrm{vis}}$	$G_{\mathrm{text}}$	$G_{\mathrm{red}}$	$G_{\mathrm{syn}}$
Qwen	SEED-Bench	+0.28	+0.11	+0.26	+0.15
Qwen	ScienceQA	+0.37	+0.40	+0.37	+0.11
Ministral	SEED-Bench	+0.44	+0.20	+0.18	+0.23
Ministral	ScienceQA	+0.30	+0.35	+0.27	+0.13

Appendix

Table 7: Behavioral validation on held-out model families. Values are absolute accuracy margins computed within the corresponding SRCoD-predicted diagnostic groups. Positive values indicate agreement with the expected behavioral patterns.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 772 papers

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
Mar 3, 2026Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5MLLM Evaluation Multimodal Model Evaluation
Kyoto University · CyberAgent. · The Institute of Statistical Mathematics +1
Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation
Mar 13, 2026Wayner Barrios, SouYoung JinReasoning Evaluation MLLM Reasoning
Dartmouth College, Hanover, NH, USA
Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention
Mar 28, 2026Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan ParvezModality Reliance in Multimodal Models Multimodal Model Evaluation
University of California, Riverside · University of Maryland, Baltimore County · Qatar Computing Research Institute (QCRI)

Mar 3, 2026J/K move · Enter open · S save

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

Shunki Uebayashi, Kento Masui, Kyohei Atarashi, Han Bao+4

Kyoto University · CyberAgent. · The Institute of Statistical Mathematics+1

MLLM Evaluation Multimodal Model Evaluation Multimodal Reasoning Item Response Theory

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.

Abstract

Multimodal Model Evaluation

Explore similar work

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation

Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention