cs.CLApr 15, 2026
SaveMedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging
Organizations: School of Data Science, Fudan University, China · Shanghai Innovation Institute, China · School of Integrated Circuits, Zhejiang University, China · School of Software Engineering, Huazhong University of Science and Technology, China · School of Computer Science, Northwestern Polytechnical University, China
Abstract
The potential of Multimodal Large Language Models (MLLMs) in domain of medical imaging raise the demands of systematic and rigorous evaluation frameworks that are aligned with the real-world medical imaging practice. Existing practices that report single or coarse-grained metrics are lack the granularity required for specialized clinical support and fail to assess the reliability of reasoning mechanisms. To address this, we propose a paradigm shift toward multidimensional, fine-grained and in-depth evaluation. Based on a two-stage systematic construction pipeline designed for this paradigm, we instantiate it with MedRCube. We benchmark 33 MLLMs, Lingshu-32B achieve top-tier performance. Crucially, MedRCube exposes a series of pronounced insights inaccessible under prior evaluation settings. Furthermore, we introduce a credibility evaluation subset to quantify reasoning credibility, uncover a highly significant positive association between shortcut behavior and diagnostic task performance, raising concerns for clinically trustworthy deployment. The resources of this work can be found at https://github.com/F1mc/MedRCube.
Figures & tables
Figure 1: Paradigm shift required for evaluating multimodal large language models in medical imaging.
Figure 2: Overview of MedRCube. The framework evaluates models along Anatomical, Modality, and Task Hierarchy axes. The pipeline (right) aligns datasets via Stage I and generates rigorous items via Stage II with expert verification.
| Model | Overall | Task Type | Modality Type | Part Type | ||||||||||||||
| MR | VR | IPR | OR | RG | DD | AD | SG | CT | X-Ray | MRI | US | Brain | Chest | Heart | Lungs | Breast | ||
| Random | 25.64 | 24.22 | 23.67 | 24.57 | 24.37 | 23.00 | 28.63 | 21.40 | 28.52 | 24.65 | 27.48 | 24.61 | 22.96 | 26.41 | 26.49 | 22.62 | 24.55 | 29.13 |
| Proprietary MLLMs | ||||||||||||||||||
| Qwen3-VL-Plus | 51.14 | 82.66 | 50.00 | 23.14 | 70.01 | 42.06 | 47.03 | 30.74 | 44.72 | 60.54 | 59.81 | 37.63 | 40.45 | 38.81 | 62.70 | 45.71 | 54.79 | 46.32 |
| GPT-5.1 | 53.36 | 85.66 | 61.00 | 33.71 | 74.70 | 35.20 | 52.84 | 21.79 | 37.50 | 69.61 | 58.86 | 43.29 | 34.62 | 41.33 | 62.01 | 53.33 | 62.18 | 38.42 |
| Claude Opus 4.5 | 53.38 | 85.37 | 64.33 | 33.29 | 73.76 | 54.30 | 44.05 | 20.23 | 44.54 | 61.86 | 61.34 | 43.42 | 35.34 | 38.49 | 63.79 | 59.05 | 54.79 | 42.16 |
Table 1: Performance comparison on MedRCube. The best scores are highlighted in bold, and the second-best are underlined. All scores are reported in percentage (%). Abbreviations for subtasks and corresponding full terminology are detailed in the Table 6 .
Figure 3: Correlation Analysis. (a) Hierarchical clustering reveals three functional blocks. (b) Anatomical ordering highlights the brain isolated island effect, where brain tasks show minimal correlation with other regions, reflecting high task heterogeneity.
Figure 4: The left panel shows the distribution of models with a quadrant of Rationality score and accuracy. The right panel reveals a strong positive correlation ( ) between Shortcut Probability and accuracy.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Part | Collected Datasets |
|---|---|
| Brain | Curious 2022 ( Xiao et al., 2019 ) , CT-ICH ( Hssayeni et al., 2020 ) , BraTS 2020 ( Menze et al., 2014 ; Bakas et al., 2017 ; Bakas et al., 2018 ) |
| Chest and Lungs | COVIDxCXR-4 ( Wu et al., 2023 ) , NIH Chest X-ray ( Wang et al., 2017 ) , SegTHOR ( Lambert et al., 2020 ) , COVID-19 Xray Dataset ( Cohen et al., 2020 ; Kermany, 2018 ) , JSRT ( Shiraishi et al., 2000 ) , Covid-19 Image Dataset ( Raikokte, 2020 ) , SIIM-ACR Pneumothorax Segmentation ( Zawacki et al., 2019 ) , VinDr-PCXR ( Pham et al., 2023 ) , COVID-QU-Ex ( Tahir et al., 2021b ; Tahir et al., 2021a ; Rahman et al., 2021 ; Degerli et al., 2021 ; Chowdhury et al., 2020 ) , Pulmonary Chest X-Ray Abnormalities ( Jaeger et al., 2013 ; Candemir et al., 2013 ) , COVIDGR ( Tabik et al., 2020 ) , Chest X-Ray Images (Pneumonia) ( Kermany et al., 2018 ) , COVID-19 Radiography Database ( Rahman et al., 2021 ; Chowdhury et al., 2020 ) , Chest X-Ray Images with Pneumothorax Masks ( Zawacki et al., 2019 ) , CoronaHack ( Govi, 2020 ) , RSNA Pneumonia Detection Challenge ( Radiological Society of North America, 2018 ) , COVID19 Pneumonia and Normal Chest X-ray PA Dataset ( Asraf and Islam, 2021 ) , Shenzhen Chest X-ray Set ( Jaeger et al., 2014 ) , Montgomery County CXR Set ( Jaeger et al., 2014 ) , ChestX-Det ( Liu et al., 2020 ) , COVID-CT ( Yang et al., 2020 ) , COVID-19-20 ( Roth et al., 2022 ) , Learn2Reg2022 ( Hering et al., 2022 ) , Finding and Measuring Lungs in CT Data ( Mader, 2017 ) , CAD-PE ( Serrano, 2019 ) |
| Breast | BUSI ( Al-Dhabyani et al., 2020 ) , MIAS Mammography ( Suckling et al., 2015 ) , CBIS-DDSM ( Sawyer-Lee et al., 2016 ) |
| Heart | MSD-Heart ( Simpson et al., 2019 ) , M&Ms ( Campello et al., 2021 ; Martín-Isla et al., 2023 ) , HeartSegMRI ( Tobon-Gomez et al., 2015 ) , M&Ms-2 ( Martín-Isla et al., 2023 ) |
Table 2: List of collected medical imaging datasets across different parts used for the benchmark. Total 36 datasets including ROCOv2 used for supplementary analysis, see Appendix B .
| Name | # Samples | Name | # Samples |
|---|---|---|---|
| Curious 2022 | 759 | RSNA Pneumonia Detection Challenge | 79 |
| CT-ICH | 235 | COVID19 Pneumonia and Normal Chest X-ray PA Dataset | 67 |
| BraTS 2020 | 235 | Shenzhen Chest X-ray Set | 79 |
| COVIDxCXR-4 | 79 | Montgomery County CXR Set | 46 |
| NIH Chest X-Ray | 324 | ChestX-Det | 750 |
| SegTHOR | 105 | COVID-CT | 305 |
Table 3: Detailed sample statistics of data sources in MedRCube (total: 7,626 samples).
Figure 5: Statistical representation of samples in MedRCube.
Figure 6: Illustrative examples of perceptual and semantic tasks in MedRCube. (a) Modality Recognition: Classifies the fundamental imaging modality of a given scan (e.g., CT, MRI) to establish the prerequisite context. (b) View Recognition: Determines the anatomical imaging plane or projection angle (e.g., axial, sagittal). (c) Imaging Protocols Recognition: Identifies specific image acquisition protocols and technical parameters, probing the depth of physical perception. (d) Organ Recognition: Identifies and names salient anatomical structures or specific organs within the image.
| Model | Overall | Task Type | Modality Type | Part Type | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MR | VR | IPR | OR | RG | DD | AD | SG | CT | X-Ray | MRI | US | Brain | Chest | Heart | Lungs | Breast | ||
| GPT-5.1 | 31.52 (21.84 ) | 37.98 | 17.33 | 14.00 | 47.20 | 40.10 | 31.34 | 18.12 | 25.43 | 40.70 | 38.41 | 21.58 | 22.33 | 30.57 | 40.14 | 22.14 | 35.55 | 24.13 |
| Gemini-3-Pro | 30.36 (28.99 ) | 17.76 | 19.00 | 19.14 | 39.63 | 60.00 | 25.71 | 17.63 | 30.13 | 25.81 | 44.29 | 15.66 | 16.20 | 18.41 | 46.44 | 18.10 | 22.84 | 21.50 |
| InternVL3.5-38B | 34.15 (23.99 ) | 24.45 | 27.67 | 24.86 | 41.85 | 51.40 | 35.39 | 10.14 | 32.91 | 38.45 | 41.24 | 27.50 | 19.02 | 18.84 | 42.18 | 31.43 | 40.57 | 28.85 |
| Qwen3-VL-32B | 29.77 (24.66 ) | 31.15 | 14.00 | 17.71 | 41.33 | 40.60 | 28.27 | 18.36 | 31.20 | 32.02 | 37.89 | 22.24 | 20.86 | 26.26 | 37.72 | 28.33 | 28.63 | 26.35 |
| HuatuoGPT-Vision-34B | 28.49 (25.53 ) | 20.90 | 8.67 | 13.29 | 45.24 | 33.00 | 34.58 | 4.83 | 29.06 | 34.96 | 30.23 | 36.32 | 25.15 | 30.14 | 30.37 | 28.10 | 32.42 | 37.03 |
Table 4: Text-only baseline results (no images). Scores are accuracy (%), and the value in parentheses denotes the absolute drop in Overall accuracy relative to the standard multimodal evaluation.
| Principle | Sample-level | 1 | 1.1 | 1.2 | 1.3 | 1.4 | 2 | 2.1 | 2.2 | 2.3 |
|---|---|---|---|---|---|---|---|---|---|---|
| Consistency | 0.84 | 0.94 | 0.98 | 0.98 | 0.94 | 0.98 | 0.90 | 0.94 | 0.96 | 0.96 |
Table 5: Consistency between GPT-5.1 and a clinical expert on 50 randomly sampled question–answer pairs. Principle 1 covers question integrity (1.1 Clinical & Anatomical Accuracy, 1.2 Logical Coherence, 1.3 Unambiguous Phrasing, 1.4 VQA Suitability); Principle 2 covers option set effectiveness (2.1 Conceptual & Grammatical Homogeneity, 2.2 Mutual Exclusivity, 2.3 Plausibility of Distractors). Sample-level consistency requires the all judgments on a sample to match.
| Abbreviation | Full Name |
|---|---|
| DD | Disease Diagnosis |
| MR | Modality Recognition |
| AD | Abnormality Diagnosis |
| SG | Severity Grading |
| OR | Organ Recognition |
| RG | Region of Interest Grounding |
Table 6: Abbreviations and Full Names of Tasks
| Prompt Templates for Disease-Oriented Task | |
|---|---|
| Type | Prompt Content |
| Similar Distractors | Prompt: Generate {num_distractors} diseases similar to ‘{disease}’ but distinct and incorrect. Focus on diseases with similar symptoms or pathophysiology. Purpose: Leverage the Human Phenotype Ontology (HPO) to generate difficult, clinically confusing options requiring fine-grained differentiation. |
| Normal Distractors | Prompt: Generate {num_distractors} diseases from the same anatomical region as ’{disease}’ with similar disease category. Purpose: Serve as a control group using cross-region diseases to regulate difficulty and validate basic anatomical alignment. |
Table 7: Prompt templates for disease-oriented task. Braced placeholders are dynamic variables instantiated based on sample metadata: {disease} denote the ground-truth diagnosis; and {num_distractors} controls the required number of options.
| Prompt Templates for Quality Review |
| SYSTEM_PROMPT = """ You are a Senior Medical Content Analyst. You are performing a TEXT-ONLY pre-validation of a medical VQA dataset. You will NOT be provided with any images. Your evaluation must be based solely on the textual content (question, options, etc.) provided. Your objective is to ensure the clinical integrity and educational value of the VQA samples by performing a critical review of their textual components. You will evaluate each sample based on two stringent principles. You MUST evaluate and provide a 0 (Fail) or 1 (Pass) for EACH individual sub-principle (e.g., 1.1, 1.2, … 2.3). Principle 1: Question Integrity (Text-Only Focus) * 1.1. Clinical & Anatomical Accuracy: All terminology (e.g., anatomical structures, pathologies, findings) in the question must be accurate, standard, and free of error. * 1.2. Logical Coherence: The question must be internally consistent and free of factual or logical self-contradictions. * 1.3. Unambiguous Phrasing: The question must be concise, grammatically sound, and semantically unambiguous. * 1.4. CRITICAL - VQA Suitability (Image-Dependence & Objectivity): The question’s phrasing must be structured as a valid VQA query. This means it must satisfy TWO conditions: * a) Image-Dependent: The question MUST require visual information (from the hypothetical image) to be answered. (FAIL if the question is general knowledge, e.g., "What is pneumonia?" or "Define lung."). * b) Objective & Factual: The question must seek a factual, objective answer based on visual content. (FAIL if the question is subjective, e.g., "Is this a high-quality image?", "What do you think of this finding?"). * ALLOWED: Questions that directly reference the (hypothetical) image or its parts are valid, even if they seem open-ended. (PASS example: "Please observe this ultrasound image, what problem is shown in the area marked by the red box?") (PASS example: "What abnormality is present in the lower right quadrant?") Principle 2: Option Set Effectiveness (Text-Only Focus) * 2.1. Conceptual & Grammatical Homogeneity: All options (A, B, C, D) must belong to the same category (e.g., all diagnoses, all anatomical parts, all ’yes/no’ judgments) and share a parallel grammatical structure. * 2.2. Mutual Exclusivity: Options must be distinct and not overlap (e.g., "Pneumonia" and "Bacterial Pneumonia" as separate options constitutes a failure). * 2.3. Plausibility of Distractors: All incorrect options (distractors) must be plausible, reasonable "foils" given the clinical context (modality, anatomical regions). The ‘gt_answer‘ must be unequivocally the best and most correct answer among the choices. You MUST return your evaluation as a single, minified, valid JSON object. NO explanatory text, markdown (“json … ”), or any other characters outside the JSON structure are permitted. """ |
Table 10: Prompt of Quality Review. This prompt is adapted from the National Board of Medical Examinations (NBME) guidelines for item writing and has been tailored for multimodal medical imaging tasks to provide rigorous review of question items and options.