Organizations: National Key Laboratory of Information Systems Engineering, National University of Defense Technology, Changsha, China · National Key Laboratory of Big Data and Decision, National University of Defense Technology, Changsha, China
Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has revealed that some benchmarks may contain shortcut items that can be answered from a single modality or from answer-option regularities alone, thus failing to reflect models' true cross-modal inference ability. Existing methods for assessing benchmarks typically rely either on intrinsic item structure or extrinsic model response behavior, which are not adequate for a comprehensive diagnosis. To fill in this gap, we propose SRCoD, a structure-response co-calibrated diagnosis framework for item-level diagnosis of modality dependence in multimodal benchmarks. SRCoD estimates sample-wise multimodal information structure from benchmark content and target answers, and uses this signal as a soft structural anchor for modality-conditioned item response modeling. The learned model produces a calibrated item-level diagnostic profile that reflects how strongly an item depends on different forms of modality evidence, enabling fine-grained benchmark analysis beyond aggregate accuracy. SRCoD also provides interpretable attribution of the diagnosis results. Extensive experiments under two complementary protocols, i.e., benchmark-refinement oriented validation and human-labeled item validation, demonstrate that SRCoD outperforms state-of-the-art benchmark diagnosis methods.
Figures & tables
Figure 1: Overview of SRCoD. SRCoD diagnoses item-level modality dependence by co-calibrating intrinsic item structure and extrinsic model responses. It first derives a structural prior from partial information decomposition, then anchors a modality-conditioned response model fitted to VLM responses under controlled modality conditions.
Figure 2: Benchmark-refinement oriented validation under contaminated candidate pools. Each method selects compact subsets from candidate pools containing equal numbers of clean benchmark items and synthetic low-quality items. We report the Spearman rank correlation between the model ranking induced by the selected subset and the ranking induced by the full clean benchmark.
Method
ScienceQA
MMMU
SEED-Bench
Average
Acc.
Macro-F1
Acc.
Macro-F1
Acc.
Macro-F1
Acc.
Macro-F1
Random
0.251
0.233
0.249
0.241
0.250
0.242
0.250
0.239
Raw
0.353
0.334
0.503
0.435
0.388
0.355
0.415
0.375
LSMI
0.464
0.255
0.265
0.171
0.309
0.189
0.346
0.205
IRT
0.507
0.476
0.384
0.382
0.494
0.474
0.462
0.444
MIRT
0.506
0.476
0.371
0.370
0.494
0.473
0.457
0.440
Table 1: Human-labeled item validation. Accuracy and macro-F1 measure agreement with human evidence-sufficiency references in the four-factor diagnostic space. Bold and underline denote the best and second-best results, respectively.
Figure 3: Ablation study of SRCoD. We compare the full model with variants that remove entropy-based confidence weighting or the profile-level structural alignment loss.
Figure 4: Robustness to model-pool subsampling. For each pool size, we randomly sample diagnostic models from the full model pool and repeat the experiment 10 times. We report performance retention relative to the full-pool setting. Shaded regions denote one standard deviation.
Figure 5: Qualitative example of the diagnostic output produced by SRCoD. The item is diagnosed as Redundant , indicating that both the image and the text provide sufficient evidence for answering the question.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Distribution of human-annotated modality-dependence labels across ScienceQA, MMMU, and SEED-Bench. Each pie chart shows the proportion of Redundant, Visual-dominant, Textual-dominant, and Synergistic items in the corresponding benchmark.
Component
Specification
CPU
12th Gen Intel(R) Core(TM) i7-12700KF @ 3.60 GHz
RAM
32 GB
GPU
NVIDIA GeForce RTX 4090, 24 GB
Python version
3.10.1
PyTorch version
2.7.0
CUDA version
12.6
Appendix
Table 2: Computational resources used for experiments.
Model Name
Type
License or Terms
gpt-4.1 [ 20 ]
Closed
OpenAI Terms of Use
gpt-4.1-nano [ 20 ]
Closed
OpenAI Terms of Use
gpt-4o-mini [ 20 ]
Closed
OpenAI Terms of Use
gemini-3.1-pro-preview [ 25 ]
Closed
Gemini API Additional Terms of Service
gemini-3.1-flash-lite [ 25 ]
Closed
Gemini API Additional Terms of Service
gemini-2.5-flash-lite [ 25 ]
Closed
Gemini API Additional Terms of Service
Appendix
Table 3: Overview of AI Models Used
Condition
Image
Question text
Answer options
Full multimodal
Yes
Yes
Yes
Visual-conditioned
Yes
No
Yes
Textual-conditioned
No
Yes
Yes
No-modality
No
No
Yes
Appendix
Table 4: Input information provided under each modality condition.
Figure 7: Examples of redundant items from ScienceQA, MMMU, and SEED-Bench.
Figure 8: Examples of textual-dominant items from ScienceQA, MMMU, and SEED-Bench.
Figure 9: Examples of visual-dominant items from ScienceQA, MMMU, and SEED-Bench.
Figure 10: Examples of synergistic items from ScienceQA, MMMU, and SEED-Bench.
Dataset
Accuracy
Macro-F1
McNemar
Δ
95% CI
prand
Δ
95% CI
prand
ScienceQA
+0.037
[0.015, 0.060]
<0.001
+0.049
[0.026, 0.075]
<0.001
87/50,p=0.002
MMMU
+0.061
[0.035, 0.089]
<0.001
+0.054
[0.025, 0.079]
<0.001
129/68,p<0.001
SEED-Bench
+0.067
[0.042, 0.091]
<0.001
+0.070
[0.043, 0.095]
<0.001
115/48,p<0.001
Average
+0.055
[0.040, 0.069]
<0.001
+0.058
[0.044, 0.072]
<0.001
331/166,p<0.001
Appendix
Table 5: Paired statistical significance analysis of SRCoD over M3IRT on the human-labeled diagnostic validation set. We report paired performance differences, 95% bootstrap confidence intervals, approximate-randomization p -values, and McNemar’s test for accuracy.
Method
ScienceQA
MMMU
SEED-Bench
Average
Acc.
Macro-F1
Acc.
Macro-F1
Acc.
Macro-F1
Acc.
Macro-F1
α=0.25
0.52
0.44
0.46
0.46
0.61
0.60
0.53
0.50
α=0.50
0.47
0.36
0.45
0.44
0.60
0.59
0.51
0.46
α=0.75
0.44
0.30
0.45
0.44
0.60
0.58
0.50
0.44
SRCoD
0.61
0.59
0.55
0.54
0.69
0.67
0.61
0.60
Appendix
Table 6: Comparison with post-hoc profile fusion baselines. We report accuracy and macro-F1 on the human-labeled validation set.
Figure 11: Diagnostic performance under a small-scale model pool. We compare M3IRT and SRCoD using responses from 3B-12B MLLMs and report accuracy and macro-F1. SRCoD consistently outperforms M3IRT and shows smaller degradation on the subset, indicating stronger robustness when response-side evidence is obtained from smaller and noisier models.
Held-out Family
Benchmark
Gvis
Gtext
Gred
Gsyn
Qwen
SEED-Bench
+0.28
+0.11
+0.26
+0.15
Qwen
ScienceQA
+0.37
+0.40
+0.37
+0.11
Ministral
SEED-Bench
+0.44
+0.20
+0.18
+0.23
Ministral
ScienceQA
+0.30
+0.35
+0.27
+0.13
Appendix
Table 7: Behavioral validation on held-out model families. Values are absolute accuracy margins computed within the corresponding SRCoD-predicted diagnostic groups. Positive values indicate agreement with the expected behavioral patterns.
Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.
Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5
Kyoto University · CyberAgent. · The Institute of Statistical Mathematics +1
We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning through verifiable intermediate steps. We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which further penalizes disordered reasoning chains. References are constructed through a Delphi-inspired pipeline in which four independent MLLMs generate trajectories, which are then aggregated via semantic clustering and validated through human quality gates. Evaluation of 20 MLLMs, including commercial frontier systems not used during benchmark construction, reveals systematic failures that are invisible to answer accuracy: universal cherry-picking (precision far exceeds recall), non-monotonic scaling trade-offs, and disordered reasoning in which no competitive model preserves more than 60% of matched steps in the correct order. Beyond evaluation, we propose the Causal Process Reward (CPR), a multiplicative reward that couples answer correctness with step-level alignment, and CPR-Curriculum, which progressively increases reasoning difficulty during training. CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
Existing omni-modal benchmarks attempt to measure modality-specific contributions, but their measurements are confounded: naturally co-occurring modalities carry correlated yet unequal information, making it unclear whether results reflect true modality reliance or information asymmetry. We introduce OMD-Bench, where all modalities are initially congruent - each presenting the same anchor, an object or event independently perceivable through video, audio, and text - which we then systematically corrupt to isolate each modality's contribution. We also evaluate calibrated abstention: whether models appropriately refrain from answering when evidence is conflicting. The benchmark comprises 4,080 instances spanning 27 anchors across eight corruption conditions. Evaluating ten omni-modal models under zero-shot and chain-of-thought prompting, we find that models over-abstain when two modalities are corrupted yet under-abstain severely when all three are, while maintaining high confidence (~60-100%) even under full corruption. Chain-of-thought prompting improves abstention alignment with human judgment but amplifies overconfidence rather than mitigating it. OMD-Bench provides a diagnostic benchmark for diagnosing modality reliance, robustness to cross-modal inconsistency, and uncertainty calibration in omni-modal systems.
Zabir Al Nazi, Shubhashis Roy Dipta, Md Rizwan Parvez
University of California, Riverside · University of Maryland, Baltimore County · Qatar Computing Research Institute (QCRI)