Can MLLMs Perceive and Describe Medical Image Quality Beyond Scores?

Jiyao Liu; Jinjie Wei; Wanying Qu; Chenglong Ma; Junzhi Ning; Yunheng Li; Ying Chen; Jiafan Zhang

cs.CVOct 2, 2025

Can MLLMs Perceive and Describe Medical Image Quality Beyond Scores?

Authors: Jiyao Liu, Jinjie Wei, Wanying Qu, Chenglong Ma, Junzhi Ning, Yunheng Li, Ying Chen, Jiafan Zhang,

Organizations: Fudan University · Shanghai Artificial Intelligence Laboratory · University of Cambridge

Abstract

Medical Image Quality Assessment (IQA) serves as a critical safety gate for clinical AI, yet existing approaches rely on scalar metrics that fail to capture the descriptive, expert-like evaluation process central to clinical practice. We introduce MedQ-Bench, a comprehensive benchmark for language-based medical IQA evaluation with MLLMs, establishing a perception--description paradigm across 5 imaging modalities and over 40 quality attributes. The benchmark comprises two complementary tasks: MedQ-Perception probes low-level visual attribute discrimination via 2,600 expert-annotated perceptual queries; MedQ-Description covers both no-reference and comparison description sub-tasks with 708 expert-annotated samples, evaluated through a four-dimensional judging protocol. Comprehensive evaluation of 23 state-of-the-art MLLMs reveals that current models exhibit only preliminary and unstable capabilities, and highlights the urgent need for targeted optimization of MLLMs in medical IQA. Project & Dataset: https://github.com/liujiyaoFDU/MedQ-Bench

Figures & tables

Figure 1: Comparison of Description-IQA with score-based IQA paradigm. Unlike purely numerical scores, Description-IQA identifies distortion types and their relative impact, yielding results more consistent with Human-IQA.

Figure 2: MedQ-Bench overview. (1) Comprehensive coverage: 3,308 samples across 5 modalities with 40+ degradation types. (2) Multi-faceted evaluation: perception-description paradigm.

Figure 3: Examples of question types in MedQ-Bench, covering MCQA perception tasks, open-ended description, and pair comparison.

Open-Source General Models
Sub-categories	Perception				Description
Model (variant)	Yes-or-No $\uparrow$	What $\uparrow$	How $\uparrow$	Overall $\uparrow$	Comp. $\uparrow$	Prec. $\uparrow$	Cons. $\uparrow$	Qual. $\uparrow$	Overall $\uparrow$
random guess	50.00%	28.48%	33.30%	37.94%
Non-experts	67.50%	57.50%	57.50%	62.50%	-	-	-	-	-
Human experts	88.50%	77.50%	77.50%	82.50%	1.524	1.483	1.887	1.623	6.502
Qwen2.5-VL-Instruct (7B)	57.89%	48.45%	54.40%	54.71%	0.715	0.670	1.855	1.127	4.367

Table 1: Performance of different models on the MCQA perception and description tasks.

Figure 4: Performance analysis of MLLMs across different evaluation dimensions: (a) Different degradation level performance, (b) general vs modality-specific question.

Open-Source General Models
Model	Comp. $\uparrow$	Prec. $\uparrow$	Cons. $\uparrow$	Qual. $\uparrow$	Overall $\uparrow$
Human experts	1.653	1.682	1.963	1.714	7.003
Qwen2.5-VL-7B	0.714	0.902	1.316	1.143	4.075
Qwen2.5-VL-32B	0.692	0.752	1.895	0.962	4.301
Qwen2.5-VL-72B	0.737	0.977	1.233	1.113	4.060
InternVL3-8B	0.985	1.278	1.797	1.474	5.534

Table 2: Detailed comparison description performance scores across four evaluation dimensions for all models.

Figure 5: Visual comparison of overall description performance patterns across model categories.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Modality	Source Type	Dataset	Samples	% of Modality	Total Samples
CT	Authentic	Radiopaedia	239	27.2%	878
	Simulate	AAPM CT-MAR ( Haneda et al., 2025 )	613	69.8%
	AI-generated	AIsynthesis (subset)	26	3.0%
MRI	Authentic	Radiopaedia	130	15.3%	848
	Authentic	MR-ART ( Nárai et al., 2022 )	68	8.0%
	Authentic	FSL Example MRI Artifacts	43	5.1%

Appendix

Table 3: Comprehensive breakdown of dataset composition.

Figure 6: Interface for the MedQ-MCQA dataset.

Figure 7: Interface for the MedQ-Description dataset.

Source Type	Authentic	Simulate	AI-Generated
Percentage	41.3%	33.9%	24.8%

Appendix

Table 4: Distribution of MedQ-Bench dataset by source type.

Question Type	Percentage
Modality-specific	57.2%
General	42.8%
Total	100.0%

Appendix

Table 5: MedQ-Perception: Distribution of tasks by question type.

Degradation Level	Percentage
No Degradation	23.8%
Mild Degradation	44.6%
Severe Degradation	31.6%
Total	100.0%

Appendix

Table 6: MedQ-Perception: Distribution of degradation severity levels.

Figure 8: Distribution of low-level attributions across imaging modalities and distortion types in MedQ-Bench.

Model	Comp. $\uparrow$	Prec. $\uparrow$	Cons. $\uparrow$	Qual. $\uparrow$	Overall $\uparrow$
Qwen2.5-VL-Instruct (7B)	0.769	0.723	1.878	1.168	4.538
Qwen2.5-VL-Instruct (32B)	1.082	0.959	1.982	1.335	5.358
InternVL3 (8B)	1.005	0.950	1.905	1.421	5.281
InternVL3 (38B)	0.996	0.973	1.932	1.448	5.349
Qwen2.5-VL-Instruct (72B)	0.932	0.923	1.896	1.412	5.163
BiMediX2 (8B)	0.405	0.428	0.315	0.715	1.863

Appendix

Table 7: Few-shot evaluation results on No-reference Description tasks.

Model	CT	Histo.	MRI	Endos.	Retinal
GPT-5	71.47%	65.43%	75.90%	60.89%	70.09%
GPT-4o	72.85%	58.33%	64.75%	60.44%	66.67 %
Grok-4	70.14%	59.37 %	65.93 %	64.49%	61.40%
Gemini-2.5-Pro	67.04%	53.40%	60.79%	60.89%	59.83%
Mistral-Medium-3	65.93%	38.58%	61.51%	65.33%	61.54%
Claude-4-Sonnet	64.27%	54.63%	55.04%	65.33%	65.81%

Appendix

Table 8: Detailed perception accuracy results across five imaging modalities.

Model	Group	Comp.	Prec.	Cons.	Qual. Acc.	Total
GPT-5	Overall	1.293	1.556	1.925	1.564	6.338
	Fine-grained	1.301	1.495	1.903	1.476	6.175
	Coarse-grained	1.267	1.767	2.000	1.867	6.900
GPT-4o	Overall	1.105	1.414	1.632	1.564	5.714
	Fine-grained	1.155	1.388	1.583	1.534	5.660
	Coarse-grained	0.933	1.500	1.800	1.667	5.900

Appendix

Table 9: Detailed numerical results for paired comparison description tasks across models, corresponding to Figure 5 .

Figure 9: Representative QA examples demonstrating typical question-answer patterns in MedQ-Bench across different medical imaging modalities and quality assessment scenarios.

Figure 10: Representative no-reference reasoning image examples demonstrating typical question-answer patterns in MedQ-Bench across different models.

Figure 11: Representative paired image examples demonstrating typical question-answer patterns in MedQ-Bench across different models.

Modality	Error Rate
Endoscopy	39.11%
Histopathology	34.57%
Fundus	29.91%
CT	25.76%
MRI	24.10%

Appendix

Table 10: GPT-5 error rates across imaging modalities on perception tasks, ranked by difficulty.

Metric	Completeness	Preciseness	Consistency	Quality Accuracy
Cohen’s $\kappa$	0.843	0.817	0.856	0.884

Appendix

Table 11: Inter-annotator agreement (Cohen’s

\kappa

) for description task.

Figure 12: Confusion matrices showing alignment between human expert scores and GPT-4o automated evaluation across four evaluation dimensions.

Metric	Completeness	Preciseness	Consistency	Quality Accuracy
$\kappa_{w}$	0.774	0.876	0.840	0.985

Appendix

Table 12: Quadratic weighted Cohen’s

\kappa_{w}

values for human–AI alignment across evaluation dimensions.

Metric	Completeness	Preciseness	Consistency	Quality Accuracy
$\kappa_{w}$	0.702	0.791	0.755	0.952

Appendix

Table 13: Human–AI alignment in the lowest-quality tier (bottom third by human score). Despite challenging cases, alignment remains robust across all evaluation dimensions.

Model	GPT-4o	Claude	Gemini	ICC	ICC 95% CI	MAD (Claude)	MAD (Gemini)
Qwen2.5-VL-Instruct (7B)	4.367	4.376	4.371	0.984	[0.980, 0.987]	0.009	0.004
Qwen2.5-VL-Instruct (32B)	5.272	5.294	5.308	0.992	[0.990, 0.994]	0.022	0.036
InternVL3 (8B)	4.983	4.995	4.968	0.986	[0.982, 0.989]	0.012	0.015
InternVL3 (38B)	4.965	4.977	4.977	0.990	[0.988, 0.992]	0.012	0.012
Qwen2.5-VL-Instruct (72B)	4.982	4.932	4.959	0.988	[0.985, 0.990]	0.050	0.023
BiMediX2 (8B)	1.721	1.692	1.719	0.954	[0.942, 0.963]	0.029	0.002

Appendix

Table 14: Multi-judge validation results showing overall description scores from three independent LLM judges and inter-judge consistency metrics. MAD (Claude) and MAD (Gemini) represent MAD between GPT-4o scores and Claude/Gemini scores respectively. Score differences across judges are minimal, and high ICC values (

>

0.95) confirm excellent agreement, indicating no systematic self-preference bias.

Medical Image Quality Assessment

See all 26 papers

MedQ-Engine: A Closed-Loop Data Engine for Evolving MLLMs in Medical Image Quality Assessment
Mar 20, 2026Jiyao Liu, Junzhi Ning, Wanying Qu +4Medical Image Quality Assessment Medical Image Analysis
Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations
Mar 8, 2026Jiyao Liu, Junzhi Ning, Chenglong Ma +14MLLM Evaluation Medical VLMs
Fudan University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Shanghai Jiaotong University, Shanghai, China +1
Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias
Jul 2, 2026Sofiane Ouaari, Kevin Vorwalder, Nico PfeiferVLM Evaluation Image Corruption Robustness
Methods in Medical Informatics, Department of Computer Science, University of Tuebingen, Germany. · Institute for Bioinformatics and Medical Informatics (IBMI), University of Tuebingen, Germany.

Mar 20, 2026J/K move · Enter open · S save

MedQ-Engine: A Closed-Loop Data Engine for Evolving MLLMs in Medical Image Quality Assessment

Jiyao Liu, Junzhi Ning, Wanying Qu, Lihao Liu+3

Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Medical Image Quality Assessment Medical Image Analysis Multimodal Large Language Models Active Learning

Open paper AskPDF arXiv

Medical image quality assessment (Med-IQA) is a prerequisite for clinical AI deployment, yet multimodal large language models (MLLMs) still fall substantially short of human experts, particularly when required to provide descriptive assessments with clinical reasoning beyond simple quality scores. However, improving them is hindered by the high cost of acquiring descriptive annotations and by the inability of one-time data collection to adapt to the model's evolving weaknesses. To address these challenges, we propose MedQ-Engine, a closed-loop data engine that iteratively evaluates the model to discover failure prototypes via data-driven clustering, explores a million-scale image pool using these prototypes as retrieval anchors with progressive human-in-the-loop annotation, and evolves through quality-assured fine-tuning, forming a self-improving cycle. Models are evaluated on complementary perception and description tasks. An entropy-guided routing mechanism triages annotations to minimize labeling cost. Experiments across five medical imaging modalities show that MedQ-Engine elevates an 8B-parameter model to surpass GPT-4o by over 13% and narrow the gap with human experts to only 4.34%, using only 10K annotations with more than 4x sample efficiency over random sampling.

Abstract

Medical Image Quality Assessment

Explore similar work

MedQ-Engine: A Closed-Loop Data Engine for Evolving MLLMs in Medical Image Quality Assessment

MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias