Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification

Xun Zhu; Fanbin Mo; Xi Chen; Kaili Zheng; Shaoshuai Yang; Yiming Shi; Jian Gao; Miao Li

cs.CVApr 9, 2026

Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification

Authors: Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, +1 more

Organizations: Department of Electronic Engineering Tsinghua University Beijing, China · School of Artificial Intelligence BUPT Beijing, China · College of AI Tsinghua University Beijing, China · Department of Electronic Engineering College of AI, Tsinghua University Beijing, China

Abstract

The rise of multimodal large language models (MLLMs) has sparked an unprecedented wave of applications in the field of medical imaging analysis. However, as one of the earliest and most fundamental tasks integrated into this paradigm, medical image classification reveals a sobering reality: state-of-the-art medical MLLMs consistently underperform compared to traditional deep learning models, despite their overwhelming advantages in pre-training data and model parameters. This paradox prompts a critical rethinking: where exactly does the performance degradation originate? In this paper, we conduct extensive experiments on 14 open-source medical MLLMs across three representative image classification datasets. Moving beyond superficial performance benchmarking, we employ feature probing to track the information flow of visual features module-by-module and layer-by-layer throughout the entire MLLM pipeline, enabling explicit visualization of where and how classification signals are distorted, diluted, or overridden. As the first attempt to dissect classification performance degradation in medical MLLMs, our findings reveal four failure modes: 1) quality limitation in visual representation, 2) fidelity loss in connector projection, 3) comprehension deficit in LLM reasoning, and 4) misalignment of semantic mapping. Meanwhile, we introduce quantitative scores that characterize the healthiness of feature evolution, enabling principled comparisons across diverse MLLMs and datasets. Furthermore, we provide insightful discussions centered on the critical barriers that prevent current medical MLLMs from fulfilling their promised clinical potential. We hope that our work provokes rethinking within the community-highlighting that the road from high expectations to clinically deployable MLLMs remains long and winding.

Figures & tables

Figure 1. Performance comparison of state-of-the-art medical MLLMs and traditional deep learning methods across three medical image classification scenarios. Model categories are distinguished by background color: CNN-based (green), MLP-based (yellow), ViT-based (blue), hybrid-based (red), and MLLM-based (gray).

Figure 2. Evaluation framework for dissecting classification degradation in medical MLLMs.

Figure 3. Layer-wise probing accuracy curves for 14 medical MLLMs across three datasets.

Medical MLLMs	BUSI							COVID19-CT							Chest-Xray
MedVLM-R1-2B	91.42	$\rightarrow$	88.41	$\rightarrow$	81.21	$\rightarrow$	81.33	93.46	$\rightarrow$	84.24	$\rightarrow$	86.11	$\rightarrow$	88.18	71.58	$\rightarrow$	75.29	$\rightarrow$	76.44	$\rightarrow$	75.64
Huatuogpt-Vision-7B	101.88	$\rightarrow$	85.87	$\rightarrow$	83.25	$\rightarrow$	84.00	92.21	$\rightarrow$	82.66	$\rightarrow$	74.73	$\rightarrow$	78.82	78.10	$\rightarrow$	78.03	$\rightarrow$	81.35	$\rightarrow$	78.53
Huatuogpt-Vision-34B	109.59	$\rightarrow$	88.95	$\rightarrow$	79.56	$\rightarrow$	86.00	96.34	$\rightarrow$	81.92	$\rightarrow$	77.58	$\rightarrow$	80.30	69.02	$\rightarrow$	73.81	$\rightarrow$	74.15	$\rightarrow$	73.56
Lingshu-7B	97.69	$\rightarrow$	77.74	$\rightarrow$	83.34	$\rightarrow$	85.33	103.75	$\rightarrow$	83.25	$\rightarrow$	76.81	$\rightarrow$	81.28	66.30	$\rightarrow$	73.78	$\rightarrow$	80.00	$\rightarrow$	73.56
Lingshu-32B	95.03	$\rightarrow$	85.33	$\rightarrow$	81.95	$\rightarrow$	85.33	84.52	$\rightarrow$	79.22	$\rightarrow$	66.71	$\rightarrow$	80.79	65.90	$\rightarrow$	73.59	$\rightarrow$	75.61	$\rightarrow$	76.12
ShizhenGPT-7B-VL	102.14	$\rightarrow$	87.87	$\rightarrow$	87.94	$\rightarrow$	86.00	90.02	$\rightarrow$	80.10	$\rightarrow$	78.06	$\rightarrow$	81.28	66.39	$\rightarrow$	74.17	$\rightarrow$	73.80	$\rightarrow$	73.72

Table 1. Feature health scores (

\bm{FHS_{V}}\rightarrow\bm{FHS_{C}}\rightarrow\bm{FHS_{L}}\rightarrow\bm{P_{\text{final}}}

) for 14 medical MLLMs across three classification datasets. Results in bold denote the dataset best, while results with underlines indicate the dataset worst.

Figure 4. Marginal gains of medical adaptation over general-domain MLLMs: HuatuoGPT-Vision-7B/Lingshu-7B/ShizhenGPT-7B-VL vs. Qwen2.5-VL-7B; MedGemma-4B/MedGemma-1.5-4B vs. Gemma-3-4B.

Models	BUSI		COVID19-CT		Chest-Xray		Average
Models	Acc	F1	Acc	F1	Acc	F1	Gap
HuatuoGPT-Vision-7B	84.00	83.19	78.82	78.82	78.53	85.31
HuatuoGPT-Vision-7B	82.67	80.92	78.33	78.22	80.29	86.32	-0.38%
MedGemma-4B	84.00	83.34	80.79	80.98	70.51	80.91
MedGemma-4B	83.33	82.51	78.82	78.39	74.84	83.24	+0.27%

Table 2. Comparison of frozen vision tower (gray) vs. LoRA-finetuned vision tower on HuatuoGPT-Vision-7B and MedGemma-4B.

Training Paradigm	ShizhenGPT-7B-VL			MedGemma-1.5-4B
Training Paradigm	BUSI	COVID19-CT	Chest-Xray	BUSI	COVID19-CT	Chest-Xray
MLLM SFT	86.00	78.82	73.72	89.33	83.74	70.03
Feature Probing	84.67	79.80	74.52	86.00	81.28	69.87
Vision Tower SFT	89.33	81.28	88.30	93.33	89.10	85.22

Table 3. Accuracy comparison of MLLM SFT, feature probing

\bm{P_{{(V,\text{end})}}}

, and vision tower SFT on ShizhenGPT-7B-VL and MedGemma-1.5-4B.

Figure 5. Comprehension and utilization dynamics in LLM reasoning. Layer-wise probing accuracy of Lingshu-7B and Lingshu-32B.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Medical MLLMs	Params	Base MLLM	Vision Tower	Connector	LLM	Description	Training Stages
MedVLM-R1	2.21B	Qwen2-VL	ViT trained from scratch	MLP	Qwen2-2B	A lightweight medical VLM trained with reinforcement learning using the GRPO algorithm, achieving strong generalization and high efficiency with only 600 training samples.	Stage 1: Uses GRPO on 600 MRI VQA samples (only final answers, no reasoning labels) to optimize format and accuracy rewards, encouraging emergent reasoning without explicit supervision. No separate SFT or multi-stage alignment.
HuatuoGPT-Vision-7B HuatuoGPT-Vision-34B	8.29B 34.75B	Qwen2.5-VL -	Redesigned ViT trained from scratch CLIP ViT-L/14@336	MLP MLP	Qwen2.5-7B Yi-1.5-34B	A medical MLLM trained on PubMedVision (1.3M medical VQA entries), featuring bilingual capabilities and enhanced medical knowledge from HuatuoGPT-II.	Stage 1: Visual-language alignment using 558K general data (LLaVA) and 647K medical data (PubMedVision Alignment VQA), training only the two-layer MLP projector while keeping the CLIP vision encoder and LLM frozen following the LLaVA-1.5 settings. Stage 2: Instruction tuning using 658K general data (LLaVA) and 647K medical data (PubMedVision Instruction-Tuning VQA), updating both the MLP projector and the LLM to enhance instruction-following capabilities following the LLaVA-1.5 settings. Additional: A further 348K bilingual data from a Chinese medical VQA dataset (translated from PubMedVision) and the medical text corpus from HuatuoGPT-II (exact data size not specified) are incorporated to enhance bilingual capability and medical knowledge.
Lingshu-7B Lingshu-32B	8.29B 33.45B	Qwen2.5-VL Qwen2.5-VL	Redesigned ViT trained from scratch Redesigned ViT trained from scratch	MLP MLP	Qwen2.5-7B Qwen2.5-32B	A generalist medical foundation model that progressively infuses extensive medical knowledge and enhances reasoning capabilities through a meticulously curated multi-modal dataset and a multi-stage training paradigm.	Stage 1: Fine-tune the vision encoder and projector (LLM frozen) on ~927K medical image–caption pairs from PMC-OA and ROCO. Stage 2: Perform end-to-end training of the vision encoder, projector, and LLM on ~4.1M image–text pairs, including medical caption datasets (ROCOv2, LLaVA-Med, Quilt-LLaVA, PubMedVision, MIMIC-CXR, FairVLMed, MedICaT, MedPix-2.0, synthesized long captions) and general caption datasets (LLaVA-1.5 Caption, PixMo). Stage 3: Train the vision encoder, projector, and LLM on ~7.1M instruction samples consisting of medical multimodal instruction datasets (PathVQA, PMC-VQA, SLAKE, VQA-Med-2019, VQA-RAD, CheXpert Plus, IU-Xray, etc.), synthesized QA/OCR/CoT data, medical text instruction datasets (MedQA, MedQuAD, MedReason, MedThoughts-8K, ApolloCorpus, HealthCareMagic-100k, etc.), and general instruction data (LLaVA-1.5 instruct, ALLaVA, OpenHermes-2.5). Stage 4: Apply GRPO-based RL with verifiable rewards on ~100K curated medical QA samples derived from datasets such as MIMIC-Ext-MIMIC-CXR-VQA, PMC-VQA, PathVQA, SLAKE, VQA-Med-2019, and VQA-RAD to improve medical reasoning ability.
ShizhenGPT-7B-VL ShizhenGPT-32B-VL	8.29B 33.45B	Qwen2.5-VL Qwen2.5-VL	Redesigned ViT trained from scratch Redesigned ViT trained from scratch	MLP MLP	Qwen2.5-7B Qwen2.5-32B	The first multimodal LLM tailored for Traditional Chinese Medicine (TCM), capable of understanding images, sounds, smells, and pulse signals. Supports TCM’s "Four Diagnostic Methods" (looking, listening/smelling, questioning, and pulse-taking).	Stage 1: Text-only pre-training on 11.9B tokens (6.3B TCM tokens from TCM Web Corpus and TCM Book Corpus + 5.6B general tokens from FineWeb-edu corpus) using full-parameter training of the LLM to inject TCM knowledge while preserving general capabilities. Stage 2: Multimodal pre-training on 3.84B tokens (1.17B TCM image-text tokens, 0.69B general image-text tokens from ShareGPT-4V, 0.03B TCM audio-text tokens, 0.04B general audio tokens, and 1.75B resampled text tokens) jointly training the LLM, vision adapter, and signal adapter for multimodal alignment. Stage 3: Instruction tuning on multimodal datasets (83,629 TCM text instructions, 65,033 vision instructions with 70,638 images, and signal instructions: 99,195 ECG, 57,957 speech, 4,101 pulse, 672 smell, 456 cough, 189 heartbeat) using full-parameter tuning to obtain the final ShizhenGPT.
Hulu-Med-4B Hulu-Med-7B Hulu-Med-14B Hulu-Med-32B	4.83B 8.04B 15.21B 33.21B	- - - -	SigLIP with 2D RoPE SigLIP with 2D RoPE SigLIP with 2D RoPE SigLIP with 2D RoPE	MLP MLP MLP MLP	Qwen3-VL-4B Qwen2.5-7B Qwen3-14B Qwen2.5-32B	A transparent, generalist medical VLM designed to unify language-only, 2D/3D vision-language, and video understanding within a single architecture.	Stage 1: Perform vision-language alignment by training the vision encoder and multimodal projector on 1.4M 2D medical image-caption pairs to align visual features with the LLM embedding space through short caption generation, while keeping the LLM frozen. Stage 2: Perform medical multimodal pretraining by full-model training on 4.9M samples (public medical data and synthetic long captions) to inject medical knowledge and enhance visual understanding via long caption generation and open-ended QA. Stage 3: Perform mixed-modality instruction tuning by continuing training all modules on 10.5M instruction samples (text and multimodal) to develop instruction-following and multi-task capabilities across text, 2D images, 3D volumes, and videos.
MedGemma-4B MedGemma-27B	4.30B 27.43B	Gemma-3 Gemma-3	MedSigLIP MedSigLIP	MLP MLP	Gemma-3-4B Gemma-3-27B	A multimodal medical model built on Gemma-3 with the MedSigLIP vision encoder, designed to process medical images and text jointly for clinical reasoning tasks.	Stage 1: Perform vision encoder enhancement by training the vision tower on approximately 33M medical image–text pairs (including 32.5M internal histopathology patches, 231K MIMIC-CXR, 199K EyePACS, ~60K CT-US1, ~48K MRI-US1, 41K PMC medical images, and dermatology datasets) to obtain MedSigLIP. Stage 2: Perform multimodal pretraining using Gemma-3 general multimodal data mixed with medical image–text data (~33M, with medical data upweighted) while training the LLM and the multimodal connector. Stage 3: Conduct post-training with medical QA datasets (~400K samples) including MedMCQA, MedQA, PubMedQA, HealthSearchQA, LiveQA, AfriMed-QA, and synthetic QA, optimizing the LLM and connector. Additional: The training methodology for the 27B variant was the same as for the 4B variant, with the addition of two training datasets: EHRQA, to improve the model’s inherent EHR understanding, and ChestImaGenome, to enable anatomy localization on chest X-ray images.

Appendix

Table S1. Detailed information for each medical MLLM.

Figure S4. Additional feature probing metrics of MedVLM-R1.

Figure S5. Additional probing metrics of HuatuoGPT-Vision-7B.

Figure S6. Additional probing metrics of HuatuoGPT-Vision-34B.

Figure S7. Additional probing metrics of Lingshu-7B.

Figure S8. Additional probing metrics of Lingshu-32B.

Figure S9. Additional probing metrics of ShizhenGPT-7B-VL.

Figure S10. Additional probing metrics of ShizhenGPT-32B-VL.

Figure S11. Additional probing metrics of Hulu-Med-4B.

Figure S12. Additional probing metrics of Hulu-Med-7B.

Figure S13. Additional probing metrics of Hulu-Med-14B.

Figure S14. Additional probing metrics of Hulu-Med-32B.

Figure S15. Additional probing metrics of MedGemma-4B.

Figure S16. Additional probing metrics of MedGemma-27B.

Figure S17. Additional probing metrics of MedGemma-1.5-4B.

Explore similar work

Mar 8, 2026

MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

Despite impressive performance on standard benchmarks, multimodal large language models (MLLMs) face critical challenges in real-world clinical environments where medical images inevitably suffer various quality degradations. Existing benchmarks exhibit two key limitations: (1) absence of large-scale, multidimensional assessment across medical image quality gradients and (2) no systematic confidence calibration analysis. To address these gaps, we present MedQ-Deg, a comprehensive benchmark for evaluating medical MLLMs under image quality degradations. MedQ-Deg provides multi-dimensional evaluation spanning 18 distinct degradation types, 30 fine-grained capability dimensions, and 7 imaging modalities, with 24,894 question-answer pairs. Each degradation is implemented at 3 severity degrees, calibrated by expert radiologists. We further introduce Calibration Shift metric, which quantifies the gap between a model's perceived confidence and actual performance to assess metacognitive reliability under degradation. Our comprehensive evaluation of 40 mainstream MLLMs reveals several critical findings: (1) overall model performance degrades systematically as degradation severity increases, (2) models universally exhibit the AI Dunning-Kruger Effect, maintaining inappropriately high confidence despite severe accuracy collapse, and (3) models display markedly differentiated behavioral patterns across capability dimensions, imaging modalities, and degradation types. We hope MedQ-Deg drives progress toward medical MLLMs that are robust and trustworthy in real clinical practice.

Jiyao Liu, Junzhi Ning, Chenglong Ma +14

Fudan University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Shanghai Jiaotong University, Shanghai, China +1

Medical VLMs Medical Image Benchmarks Multimodal Model Evaluation Image Corruption Robustness Probability Calibration

Ask PDF

Mar 15, 2026

How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images

Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks. However, their performance on medical tasks, particularly in zero-shot settings where generalization is critical, remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation. In this work, we present a pioneering systematic investigation into the visual grounding capabilities of state-of-the-art medical MLLMs. To disentangle visual grounding from semantic grounding, we design VGMED, a novel evaluation dataset developed with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs. We introduce new quantitative metrics and conduct detailed qualitative analyses. Our study across eight state-of-the-art (SOTA) medical MLLMs validates that they often fail to ground their predictions in clinically relevant image regions. We note that this finding is specific to medical image analysis; in contrast, prior work has shown that MLLMs are capable of grounding their predictions in the correct image regions when applied to natural scene images. Motivated by these findings, we propose VGRefine, a simple yet effective inference-time method that refines attention distribution to improve visual grounding in medical settings. Our approach achieves SOTA performance across 6 diverse Med-VQA benchmarks (over 110K VQA samples from 8 imaging modalities) without requiring additional training or external expert models. Overall, our work, for the first time, systematically validates inadequate visual grounding as one of the key contributing factors for medical MLLMs' under-performance. Additional experiments are included in the Supp.

Guimeng Liu, Tianze Yu, Somayeh Ebrahimkhani +3

Singapore University of Technology and Design, Singapore · Department of Neurology, National Neuroscience Institute, Singapore · Duke-NUS Medical School, Singapore

Medical VLMs Medical Image Grounding VLM Evaluation Medical VQA

Ask PDF

Mar 6, 2026

Multimodal Large Language Models as Image Classifiers

Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and Vision-Language Models (VLMs) report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLM underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLM predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.

Nikita Kisel, Illia Volkov, Klara Janouskova +1

Visual Recognition Group, Czech Technical University in Prague

Multimodal Large Language Models Multimodal Classification Multimodal Model Evaluation Image Classification

Ask PDF