MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

Jiyao Liu; Junzhi Ning; Chenglong Ma; Wanying Qu; Jianghan Shen; Siqi Luo; Jinjie Wei; Jin Ye

cs.CVMar 8, 2026

MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

Authors: Jiyao Liu, Junzhi Ning, Chenglong Ma, Wanying Qu, Jianghan Shen, Siqi Luo, Jinjie Wei, Jin Ye, +9 more

Organizations: Fudan University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Shanghai Jiaotong University, Shanghai, China · Imperial College London, London, UK

Abstract

Despite impressive performance on standard benchmarks, multimodal large language models (MLLMs) face critical challenges in real-world clinical environments where medical images inevitably suffer various quality degradations. Existing benchmarks exhibit two key limitations: (1) absence of large-scale, multidimensional assessment across medical image quality gradients and (2) no systematic confidence calibration analysis. To address these gaps, we present MedQ-Deg, a comprehensive benchmark for evaluating medical MLLMs under image quality degradations. MedQ-Deg provides multi-dimensional evaluation spanning 18 distinct degradation types, 30 fine-grained capability dimensions, and 7 imaging modalities, with 24,894 question-answer pairs. Each degradation is implemented at 3 severity degrees, calibrated by expert radiologists. We further introduce Calibration Shift metric, which quantifies the gap between a model's perceived confidence and actual performance to assess metacognitive reliability under degradation. Our comprehensive evaluation of 40 mainstream MLLMs reveals several critical findings: (1) overall model performance degrades systematically as degradation severity increases, (2) models universally exhibit the AI Dunning-Kruger Effect, maintaining inappropriately high confidence despite severe accuracy collapse, and (3) models display markedly differentiated behavioral patterns across capability dimensions, imaging modalities, and degradation types. We hope MedQ-Deg drives progress toward medical MLLMs that are robust and trustworthy in real clinical practice.

Figures & tables

Figure 1: Illustration of AI Dunning-Kruger Effect. An MLLM correctly identifies a liver lesion in the clean CT image but yields erroneous predictions with similarly high confidence when the image is corrupted while the lesion is still visible.

Benchmark	Publication	Total QA	#Modality	#Capability	#Degrad.
MedMNIST-C [ 11 ]	Arxiv 2024	520k (train & eval.)	9	1 (classification)	17
RobustMedCLIP [ 17 ]	MIUA 2025	137k (train & eval.)	5	1 (classification)	7
Pathology Corruptions [ 51 ]	MICCAI 2022	327k (train & eval.)	1	1 (classification)	9
ROOD-MRI [ 5 ]	NeuroImage 2023	1,475 (eval.)	1	1 (segmentation)	11
VLM Artefacts [ 8 ]	npj Digit. Med. 2025	6,600 (eval.)	3	1 (detection)	5
RobustMed-Bench [ 46 ]	Arxiv 2025	1,500 (eval.)	3	–	6

Table 1: Comparison with related benchmarks. “–” indicates the information is not reported in the original publication.

Figure 2: Overview of the MedQ-Deg benchmark framework. Two orthogonal hierarchies structure the evaluation: a capability hierarchy (left) decomposing clinical competence into 30 fine-grained skills across 6 tasks, and a degradation hierarchy (right) covering 18 degradation types across 7 modalities. The data pipeline (middle) applies each degradation at three expert-calibrated severity degrees (L0–L2).

Figure 3: Model performance across medical image quality degradation. Accuracy of 40 MLLMs evaluated at three severity degrees (L0–L2). X-axis labels are colored by model category: blue for open-source general models, brown for commercial MLLMs, and purple for medical-specialized models.

Commercial MLLMs
		Medical Perception			Clinical Reasoning
Model	Avg.	CU	IP	AR	BS	Diag.	Treat.
GPT-5	64.5 ( $\downarrow$ 3.5)	67.2 ( $\downarrow$ 4.3)	36.5 ( $\downarrow$ 0.9)	67.4 ( $\downarrow$ 4.4)	75.0 ( $\downarrow$ 6.0)	62.1 ( $\downarrow$ 2.9)	78.8 ( $\downarrow$ 2.3)
Gemini-2.5-Pro	58.2 ( $\downarrow$ 3.8)	66.9 ( $\downarrow$ 3.4)	30.0 ( $\downarrow$ 6.5)	72.0 ( $\downarrow$ 5.7)	60.7 ( $\downarrow$ 8.3)	60.1 ( $\downarrow$ 1.3)	59.6 ( $\uparrow$ 2.6)
Gemini-2.5-Flash	50.7 ( $\downarrow$ 2.9)	63.6 ( $\downarrow$ 4.8)	25.5 ( $\downarrow$ 1.2)	71.2 ( $\uparrow$ 0.1)	58.9 ( $\downarrow$ 5.7)	38.9 ( $\downarrow$ 3.2)	46.1 ( $\downarrow$ 2.8)
GPT-5.1	49.6 ( $\downarrow$ 5.9)	65.9 ( $\downarrow$ 3.3)	30.5 ( $\downarrow$ 6.0)	65.8 ( $\downarrow$ 4.2)	50.0 ( $\downarrow$ 9.5)	47.0 ( $\downarrow$ 8.5)	38.5 ( $\downarrow$ 3.9)

Table 2: Capability-level performance (L1&L2) with degradation magnitude. Average accuracy across the mid-level capability dimensions, computed over L1 and L2 degradations. Numbers in parentheses show performance drop from L0 to L1&L2. “CU”: Clinical Understanding. “IP”: Imaging Perception. “AR”: Anatomical Recognition. “BS”: Basic Science. “Diag.”: Diagnosis. “Treat.”: Treatment.

Commercial MLLMs
Model	Avg.	Intensity	Noise	R&B	Artifacts	Motion
Gemini-2.5-Pro	68.2 ( $\downarrow$ 3.7)	71.1 ( $\downarrow$ 0.9)	69.2 ( $\downarrow$ 2.3)	67.2 ( $\downarrow$ 2.0)	65.1 ( $\downarrow$ 5.3)	68.6 ( $\downarrow$ 8.0)
GPT-5	67.5 ( $\downarrow$ 4.2)	72.2 ( $\downarrow$ 4.0)	66.2 ( $\downarrow$ 3.2)	67.5 ( $\downarrow$ 2.5)	63.1 ( $\downarrow$ 6.4)	68.4 ( $\downarrow$ 4.8)
GPT-5.1	65.7 ( $\downarrow$ 4.2)	70.9 ( $\downarrow$ 2.5)	64.2 ( $\downarrow$ 4.8)	65.7 ( $\downarrow$ 4.2)	60.5 ( $\downarrow$ 6.2)	67.2 ( $\downarrow$ 3.4)
Gemini-2.5-Flash	65.4 ( $\downarrow$ 2.5)	67.5 ( $\downarrow$ 0.6)	65.1 ( $\downarrow$ 1.8)	64.8 ( $\downarrow$ 1.0)	60.7 ( $\downarrow$ 5.0)	68.7 ( $\downarrow$ 4.0)
GPT-4.1	65.0 ( $\downarrow$ 2.1)	69.4 ( $\downarrow$ 3.7)	62.7 ( $\downarrow$ 0.8)	65.1 ( $\downarrow$ 2.4)	60.6 ( $\downarrow$ 3.2)	67.4 ( $\downarrow$ 0.6)

Table 3: Degradation-category performance (L1&L2) with degradation magnitude. Average accuracy across five degradation categories, computed over L1 and L2 degradations. Numbers in parentheses show performance drop from L0 to L1&L2. “R&B”: resolution and blur issues.

Figure 4: Comprehensive analysis of model calibration shift across severity degrees. These panels quantify the widening gap between actual accuracy and model certainty under increasing image degradation. As degradation scales from L0 to L2, the stability of perceived performance against collapsing accuracy reveals a fundamental failure in metacognitive awareness across MLLMs.

Panel of Figure 5 — Figure 5: Simulated vs. Real-World Evaluation Consistency. Left: t-SNE projection of BiomedCLIP features from simulated and real degraded images. Real and simulated images co-locate within each modality cluster, confirming distribution alignment. Right: Bump chart tracing per-model ranks across Simulated-Mild, Simulated-Severe, Real-World-Mild, and Real-World-Severe conditions.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Fine-grained Capability	Mid-level Capability	Count	Ratio in Mid-level	Ratio in Total
Disease Diagnosis	Clinical Understanding	12,908	97.74%	62.70%
Severity Grading	Clinical Understanding	138	1.04%	0.67%
Basic Science Understanding	Clinical Understanding	78	0.59%	0.38%
Lesion Grading	Clinical Understanding	68	0.51%	0.33%
Attribute Recognition	Clinical Understanding	10	0.08%	0.05%
Treatment-Understanding	Clinical Understanding	4	0.03%	0.02%

Appendix

Table 4: Capability hierarchy of MedQ-Deg. Row shading indicates high-level capability: blue rows belong to Medical Perception and purple rows belong to Clinical Reasoning .

Name	Parent Category	Count	Ratio in Parent	Ratio in Total	Modality
Artifacts	-	5,322	-	25.85%	All
limited_angle	Artifacts	1,440	27.06%	6.99%	CT
sparse_view	Artifacts	1,440	27.06%	6.99%	CT
bias_field_artifact	Artifacts	936	17.59%	4.55%	MRI
undersampling_artifact	Artifacts	672	12.63%	3.26%	MRI
ghosting_artifact	Artifacts	352	6.61%	1.71%	MRI

Appendix

Table 5: Degradation type statistics.

Commercial MLLMs
Model	L0 $\uparrow$	L1 $\uparrow$	L2 $\uparrow$	L1-L0 $\downarrow$	L2-L0 $\downarrow$
Gemini-2.5-Pro	71.39	69.93	65.20	-1.46	-6.19
GPT-5	70.27	69.09	64.30	-1.18	-5.97
GPT-5.1	68.56	67.34	62.16	-1.22	-6.40
Gemini-2.5-Flash	67.39	67.18	62.00	-0.21	-5.39
GPT-4o	66.98	65.22	59.73	-1.76	-7.25

Appendix

Table 6: Overall accuracy across severity degrees. L0/L1/L2 accuracies and drops (L1-L0, L2-L0).

Figure 7: Performance comparison across seven medical imaging modalities under three severity degrees. Each radar chart shows model performance for a specific modality.

Figure 8: Accuracy difference heatmaps (L0 $-$ Degraded). Red : degradation hurts accuracy; green : no effect or slight gain; white: no change. (a) L0 vs. L1 by degradation type. (b) L0 vs. L2 by degradation type. (c) L0 vs. L1 by fine-grained capability. (d) L0 vs. L2 by fine-grained capability.

Figure 9: Degradation examples (Part 1/3): CT and MRI modalities.

Figure 10: Degradation examples (Part 2/3): X-ray, Ultrasound, and Dermoscopy modalities.

Figure 11: Degradation examples (Part 3/3): Histopathology and Endoscopy modalities.

Explore similar work

Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification
Apr 9, 2026Xun Zhu, Fanbin Mo, Xi Chen +6Multimodal Large Language Models Medical LLMs
Department of Electronic Engineering Tsinghua University Beijing, China · School of Artificial Intelligence BUPT Beijing, China · College of AI Tsinghua University Beijing, China +1
MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging
Apr 15, 2026Zhijie Bao, Fangke Chen, Licheng Bao +4Multimodal Model Evaluation Medical VLMs
School of Data Science, Fudan University, China · Shanghai Innovation Institute, China · School of Integrated Circuits, Zhejiang University, China +2
Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias
Jul 2, 2026Sofiane Ouaari, Kevin Vorwalder, Nico PfeiferVLM Evaluation Image Corruption Robustness
Methods in Medical Informatics, Department of Computer Science, University of Tuebingen, Germany. · Institute for Bioinformatics and Medical Informatics (IBMI), University of Tuebingen, Germany.

Apr 9, 2026J/K move · Enter open · S save

Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification

Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng+5

Department of Electronic Engineering Tsinghua University Beijing, China · School of Artificial Intelligence BUPT Beijing, China · College of AI Tsinghua University Beijing, China+1

Multimodal Large Language Models Medical LLMs Medical Image Classification Multimodal Model Evaluation

Open paper AskPDF arXiv

The rise of multimodal large language models (MLLMs) has sparked an unprecedented wave of applications in the field of medical imaging analysis. However, as one of the earliest and most fundamental tasks integrated into this paradigm, medical image classification reveals a sobering reality: state-of-the-art medical MLLMs consistently underperform compared to traditional deep learning models, despite their overwhelming advantages in pre-training data and model parameters. This paradox prompts a critical rethinking: where exactly does the performance degradation originate? In this paper, we conduct extensive experiments on 14 open-source medical MLLMs across three representative image classification datasets. Moving beyond superficial performance benchmarking, we employ feature probing to track the information flow of visual features module-by-module and layer-by-layer throughout the entire MLLM pipeline, enabling explicit visualization of where and how classification signals are distorted, diluted, or overridden. As the first attempt to dissect classification performance degradation in medical MLLMs, our findings reveal four failure modes: 1) quality limitation in visual representation, 2) fidelity loss in connector projection, 3) comprehension deficit in LLM reasoning, and 4) misalignment of semantic mapping. Meanwhile, we introduce quantitative scores that characterize the healthiness of feature evolution, enabling principled comparisons across diverse MLLMs and datasets. Furthermore, we provide insightful discussions centered on the critical barriers that prevent current medical MLLMs from fulfilling their promised clinical potential. We hope that our work provokes rethinking within the community-highlighting that the road from high expectations to clinically deployable MLLMs remains long and winding.

Abstract

Explore similar work

Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification

MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging

Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias