How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

Janet Wang; Yunbei Zhang; Xiao Wang; Jihun Hamm

cs.CVSep 29, 2026

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

Authors: Janet Wang, Yunbei Zhang, Xiao Wang, Jihun Hamm

Organizations: Tulane University · Oak Ridge National Laboratory

Abstract

Medical Vision-Language Models (VLMs) show significant promise for clinical image understanding, offering accurate diagnosis with interpretable reasoning. However, a critical performance gap exists between their strong vision encoders and the full multimodal model: in dermatology, the MedSigLIP encoder outperforms MedGemma by an average of 10.26 percentage points even when both use zero target-task labels; few-shot linear probing provides further evidence of strong visual representations. This gap motivates an investigation of how visual information is used in end-to-end diagnosis and why plausible-sounding predictions can lack grounding in image evidence. Using dermatology as our primary testbed, we systematically investigate three hypotheses for this phenomenon. We further provide a mechanistic analysis of the model's internal attention patterns, showing that a simple describe-then-decide prompting strategy increases vision attention by 30-40% during generation. Task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in our evaluation. To address these challenges, we combine label-free prompting with low-label encoder-assisted reranking while keeping the VLM frozen. We validate the interventions across five VLM backbones in dermatology and provide supporting representation and attention analyses across additional medical modalities.

Figures & tables

Figure 1: A comparison of the zero-shot performance of the MedGemma-4B VLM against the linear probe performance of its vision encoder, MedSigLIP. LP denotes a linear probe on the vision encoder, while VLM denotes a direct query for diagnosis to the full vision-language model. The comparison shows the diagnostic information accessible through linear probing; the label-free control is reported in Table 1 .

	Zero target labels		8-shot
Dataset	VLM	MedSigLIP	Image ICL	LP	Reranking
Derm7pt	16.46	24.18	19.44	28.66	35.56
eSkinHealth	13.35	26.53	18.15	38.80	42.05
Fitzpatrick17k	16.73	28.37	20.72	38.43	37.82
SD-260	10.81	—	—	37.12	42.33
PAD-UFES-20	46.91	55.42	49.58	53.33	80.31

Table 1: The encoder–VLM gap persists without target-task labels. Top-1 accuracy (%). The zero-label columns use no fitted classifier; the 8-shot columns compare image ICL, LP, and LP-assisted Top-5-to-Top-1 prediction. A dash marks an unreported setting. LP and reranking variability and the 1-shot results are retained in Appendix Table 9 .

Prompt context	Fitzpatrick17k	PAD-UFES-20
Names only	16.73	46.22
Class-aligned descriptions	24.01	52.23
Length-matched generic text	16.68	36.36
Shuffled descriptions	12.71	15.64

Table 2: Clinical content matters beyond prompt length. Top-1 accuracy (%) with zero labeled target images. The complete comparison appears in Appendix Table 10 .

Panel of Figure 2 — Figure 2: Clinical context changes the pattern of diagnostic errors. Fitzpatrick17k confusion matrices for (a) direct answering and (b) clinical descriptions. The full four-strategy comparison, including DtD and 2-shot candidate filtering, appears in Appendix Fig. 5 ; class abbreviations are listed in Appendix Table 22 .

Figure 3: Examples of incorrect answers and hallucinated explanations from MedGemma-4B. The model incorrectly diagnoses sarcoidosis (left) and melanoma (middle) as basal cell carcinoma, providing descriptions of features that are not present in the images. In the third example (right), the lesion is manually masked, yet the model generates the same incorrect diagnosis and explanation, indicating it may generate a prediction first and rationalize it with a post-hoc explanation.

Dataset	DA	in-context	DtD	top-5 to top-1	all combined
Derm7pt	16.46 ( $\pm$ 0.17)	26.72 ( $\pm$ 0.82)	30.98 ( $\pm$ 1.37)	35.10 ( $\pm$ 1.95)	38.27 ( $\pm$ 0.81)
eSkinHealth	13.35 ( $\pm$ 0.23)	30.07 ( $\pm$ 1.51)	29.77 ( $\pm$ 0.92)	43.94 ( $\pm$ 1.03)	48.00 ( $\pm$ 0.86)
Fitzpatrick17k	16.73 ( $\pm$ 0.25)	24.01 ( $\pm$ 2.01)	24.75 ( $\pm$ 1.53)	28.32 ( $\pm$ 1.67)	40.97 ( $\pm$ 1.41)
SD-260 (subset)	24.31 ( $\pm$ 0.15)	34.29 ( $\pm$ 1.06)	33.80 ( $\pm$ 1.37)	40.01 ( $\pm$ 2.18)	44.90 ( $\pm$ 0.91)
PAD-UFES-20	46.22 ( $\pm$ 0.17)	52.23 ( $\pm$ 1.26)	62.33 ( $\pm$ 1.58)	75.23 ( $\pm$ 2.07)	84.68 ( $\pm$ 1.80)

Table 3: Ablation study on the effectiveness of each proposed strategy. This table presents Top-1 accuracy for five inference configurations. Clinical descriptions and DtD are label-free; candidate filtering uses target labels. The “DA” column serves as the baseline, using a standard classification prompt. The “in-context” strategy augments the prompt with clinical descriptions; “DtD” requires the model to describe visual features before diagnosis; and “Top-5 to Top-1” is our two-stage strategy where the VLM reranks candidates from a 8-shot linear probe. The final “all combined” column integrates all proposed strategies.

Inference Strategy	Derm7pt	eSkinHealth	Fitzpatrick17k
Direct Answer (Baseline)	12.77 ( $\pm$ 0.12)	23.76 ( $\pm$ 0.21)	10.05 ( $\pm$ 0.08)
+ Top-5 Filtering	29.66 ( $\pm$ 1.45)	40.25 ( $\pm$ 1.82)	58.37 ( $\pm$ 1.15)
+ In-Context Description	31.19 ( $\pm$ 1.33)	44.03 ( $\pm$ 2.01)	62.97 ( $\pm$ 1.28)
+ DtD (Ours)	32.56 ( $\pm$ 0.95)	44.89 ( $\pm$ 1.67)	63.91 ( $\pm$ 1.05)

Table 4: Generalizability on SkinVL (LLaVA-based). Top-1 Accuracy (%) comparison. Our proposed inference pipeline consistently improves performance over the direct answer baseline, demonstrating robustness across different VLM architectures.

Model	Dataset	DA	DtD	Top-5
InternVL3-4B	D7	5.68	6.20	14.83
	F17	10.10	12.23	19.70
InternVL3.5-4B	D7	7.21	7.85	18.32
	F17	9.25	13.15	22.16
biomed-Qwen2.5-VL-3B	D7	20.93	38.50	41.24
	F17	12.14	19.96	22.35

Table 5: The interventions transfer to additional VLM backbones. Top-1 accuracy (%) on Derm7pt (D7) and Fitzpatrick17k (F17). DA: direct answer; DtD: describe-then-decide; Top-5: 8-shot candidate-assisted Top-5-to-Top-1 inference. Full results are listed in Appendix Table 12 .

Panel of Figure 4 — Figure 4: Vision Attention Ratio (VAR) and description quality scores. (a) Per-layer VAR for Fitz17k shows that Describe-then-Decide (DtD) increases vision attention across all layers, with the most pronounced gains at global attention layers 5, 10, 17, 22, 27, and 33. (b) Mean VAR across six medical imaging datasets shows that DtD increases attention allocated to visual tokens by 30–40%. (c) Distribution of 0–5 quality scores for 100 generated descriptions, based on the five clinical criteria described in Sec. 4.3 .

			Linear Probe				VLM
Dataset	Modality	#Cls	1-shot	4-shot	8-shot	16-shot	0-shot
CXR	Radiology	5	24.6 ( $\pm$ 2.1)	54.6 ( $\pm$ 1.5)	59.3 ( $\pm$ 1.2)	60.9 ( $\pm$ 0.9)	36.1 ( $\pm$ 0.5)
PCam	Pathology	2	58.2 ( $\pm$ 2.4)	78.9 ( $\pm$ 1.3)	82.2 ( $\pm$ 0.8)	81.5 ( $\pm$ 1.0)	70.9 ( $\pm$ 0.8)
PathM	Pathology	9	78.2 ( $\pm$ 1.6)	90.9 ( $\pm$ 0.9)	93.5 ( $\pm$ 0.6)	94.3 ( $\pm$ 0.5)	40.6 ( $\pm$ 0.6)
BloodM	Hematology	8	61.3 ( $\pm$ 2.3)	78.1 ( $\pm$ 1.4)	86.3 ( $\pm$ 0.9)	89.0 ( $\pm$ 0.7)	26.8 ( $\pm$ 0.4)
OrganA	CT	11	49.2 ( $\pm$ 2.5)	63.9 ( $\pm$ 1.6)	71.5 ( $\pm$ 1.1)	76.2 ( $\pm$ 0.8)	21.4 ( $\pm$ 0.3)

Table 6: Few-shot linear probe accuracy (%) on frozen MedSigLIP features vs. MedGemma-4B VLM zero-shot classification across medical imaging modalities. LP results are averaged over 5 runs. At 8 shots, LP outperforms the zero-shot VLM on every dataset.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	MedGemma	MedSigLIP	BiomedCLIP	DermLIP
Derm7pt	16.46	24.18	24.55	24.03
eSkinHealth	13.35	26.53	28.22	29.28
Fitzpatrick17k	16.73	28.37	25.71	31.55
PAD-UFES-20	46.91	55.42	46.39	53.06

Appendix

Table 7: Zero-label encoder baselines outperform MedGemma in most comparisons. Top-1 accuracy (%) without fitting a target-task classifier. MedSigLIP outperforms MedGemma on all four datasets.

Dataset	VLM 0-shot	Image ICL	LP	Reranking
Derm7pt	16.46	19.44	28.66	35.56
eSkinHealth	13.35	18.15	38.80	42.05
Fitzpatrick17k	16.73	20.72	38.43	37.82
PAD-UFES-20	46.91	49.58	53.33	80.31

Appendix

Table 8: Image ICL improves the VLM baseline while leaving a gap to LP. Top-1 accuracy (%). The image-ICL, LP, and reranking columns use the 8-shot settings.

	Baseline VLM	Linear Probe		Our Pipeline
Dataset	Zero-Shot	(Vision Encoder Only)		(LP Top-5 $\rightarrow$ VLM Top-1)
	0-shot	1-shot	8-shot	1-shot	8-shot
Derm7pt	16.46 ( $\pm$ 0.01)	12.96 ( $\pm$ 1.75)	28.66 ( $\pm$ 1.38)	20.02 ( $\pm$ 2.39)	35.56 ( $\pm$ 0.99)
eSkinHealth	13.35 ( $\pm$ 0.23)	23.51 ( $\pm$ 1.91)	38.80 ( $\pm$ 1.06)	30.98 ( $\pm$ 2.85)	42.05 ( $\pm$ 1.48)
Fitzpatrick17k	16.73 ( $\pm$ 0.19)	17.63 ( $\pm$ 3.13)	38.43 ( $\pm$ 1.03)	25.35 ( $\pm$ 2.41)	37.82 ( $\pm$ 1.37)
SD-260	10.81 ( $\pm$ 0.13)	11.48 ( $\pm$ 2.27)	37.12 ( $\pm$ 1.46)	22.31 ( $\pm$ 1.95)	42.33 ( $\pm$ 0.92)

Appendix

Table 9: Effectiveness of the proposed two-stage inference pipeline. Comparison of the Baseline VLM (MedGemma zero-shot), the standalone Linear Probe (MedSigLIP), and Our Pipeline which uses the Linear Probe to filter candidates (Top-5) for the VLM (Top-1). All accuracy values are Top-1 (%). Ablation study on different number of shots can be found in Appendix I.4 .

Setting	Fitzpatrick17k	PAD-UFES-20
Names only	16.73	46.22
Class-aligned descriptions	24.01	52.23
Length-matched generic text	16.68	36.36
Shuffled descriptions	12.71	15.64
Top-5-to-Top-1	28.32	75.23
All combined	40.97	84.68

Appendix

Table 10: Class-aligned clinical information improves over prompt-length controls. Top-1 accuracy (%). The first four rows use no labeled target images; the last two include an 8-shot probe.

Data Details				MedGemma-4B						PanDerm
Datasets	#Class	#Train	#Test	Zero-Shot		Fine-tune		Linear Probe		Linear Probe
Datasets	#Class	#Train	#Test	ACC	F1	ACC	F1	ACC	F1	ACC	F1
Derm7pt	14	413	395	16.46 ( $\pm$ 0.02)	17.77	39.50 ( $\pm$ 2.15)	22.49	57.22 ( $\pm$ 0.88)	30.07	58.99 ( $\pm$ 0.76)	57.93
eSkinHealth	24	2,714	2,676	13.35 ( $\pm$ 0.15)	16.33	60.26 ( $\pm$ 1.42)	56.17	65.35 ( $\pm$ 0.54)	42.51	60.64 ( $\pm$ 0.61)	59.83
Fitzpatrick17k	20	3,100	3,100	16.73 ( $\pm$ 0.08)	11.13	47.32 ( $\pm$ 1.10)	45.16	66.32 ( $\pm$ 0.45)	66.90	64.87 ( $\pm$ 0.39)	64.88
SD-260	260	10,362	10,238	10.81 ( $\pm$ 0.05)	7.94	39.45 ( $\pm$ 0.95)	34.63	75.35 ( $\pm$ 0.32)	60.10	72.08 ( $\pm$ 0.28)	60.24

Appendix

Table 11: Classification performance across five dermatology datasets. The table compares the performance for MedGemma-4B under three conditions: zero-shot inference, full-data fine-tuning, and a linear probe on its vision encoder (MedSigLIP). Performance is compared against a linear probe on a competitive vision encoder, PanDerm. Linear probing on both vision encoders significantly outperforms the zero-shot with the VLM, even after fine-tuning. Results are averaged over five runs.

Model	Dataset	DA	DtD	Top-5-to-Top-1
InternVL3-4B	Derm7pt	5.68	6.20	14.83
	Fitzpatrick17k	10.10	12.23	19.70
InternVL3.5-4B	Derm7pt	7.21	7.85	18.32
	Fitzpatrick17k	9.25	13.15	22.16
biomed-Qwen2.5-VL-3B	Derm7pt	20.93	38.50	41.24
	Fitzpatrick17k	12.14	19.96	22.35

Appendix

Table 12: DtD and candidate-assisted inference improve all six added backbone–dataset pairs. Top-1 accuracy (%). Top-5-to-Top-1 uses the 8-shot configuration.

Model	Closed-ended	Open-ended	Token recall
Base MedGemma-4B	70.92	37.50	49.96
Fitzpatrick17k QLoRA	65.88	22.50	27.43

Appendix

Table 13: Dermatology adaptation improves in-domain classification but reduces cross-domain VQA performance. Accuracy and token recall (%) on VQA-RAD; higher is better.

Panel of Figure 5 — Figure 5: Prompting and candidate selection change diagnostic confusion. Original Fitzpatrick17k matrices for (a) direct answer, (b) clinical descriptions, (c) DtD, and (d) Top-5-to-Top-1 with a 2-shot probe. Panels retain the original counts and are enlarged for readability. Class abbreviations appear in Table 22 .

Figure 6: Attention heatmaps. Columns show (a) the original image, (b, d) attention from the DA prompt (baseline), and (c, e) attention from our DtD strategy. We visualize attention for the final diagnosis tokens (b, c) and for the entire response (d, e). The DtD prompt yields attention maps that are more accurately concentrated on the lesion area, both during the final decision (c) and across the full response (e), illustrating the change in attention allocation.

Dataset	Modality	DA	DtD	$\Delta$
Fitzpatrick17k	Dermatology	4.81	4.67	$-$ 0.14
CheXpert	Radiology	4.81	4.61	$-$ 0.20
PatchCamelyon	Pathology	4.74	4.58	$-$ 0.16
PathMNIST	Pathology	4.77	4.69	$-$ 0.08
BloodMNIST	Hematology	4.65	4.57	$-$ 0.08
OrganAMNIST	CT	4.88	4.67	$-$ 0.21

Appendix

Table 14: Mean attention entropy under Direct Answer (DA) vs. Describe-then-Decide (DtD). Lower entropy indicates more focused attention. DtD reduces entropy for all six modalities.

Dataset	Modality	DA	DtD	$\Delta$ (Rel.)
Fitz17k	Dermatology	2.85	3.98	+39.8%
CXR	Radiology	3.02	4.24	+40.2%
PCam	Pathology	2.98	3.95	+32.5%
PathM	Pathology	2.32	3.02	+30.1%
BloodM	Hematology	2.86	3.80	+33.1%
OrganA	CT	2.67	3.60	+34.7%

Appendix

Table 15: Mean Vision Attention Ratio (%) under Direct Answer (DA) vs. Describe-then-Decide (DtD). DtD increases the fraction of attention allocated to visual tokens during generation across the evaluated modalities.

Figure 7: Peak-layer Vision Attention Ratio across datasets. DtD consistently achieves higher peak VAR values compared to DA, with the largest improvements observed in radiology and dermatology.

Figure 8: Vision Attention Ratio (VAR) per layer for six medical imaging datasets under Direct Answer (DA, red dashed) vs. Describe-then-Decide (DtD, green solid). DtD consistently increases vision attention across all layers and modalities, with the most pronounced gains at global attention layers (5, 10, 17, 22, 27, 33).

Figure 9: Abstention test on 200 masked images. (a) DA: the model diagnoses despite masked lesions. (b) DtD: the model abstains more frequently.

Figure 10: Attention entropy heatmap across layers for six medical imaging modalities under Direct Answer (DA) and Describe-then-Decide (DtD) prompting. DtD consistently produces lower entropy (more focused attention) across all modalities and layers.

Figure 11: Example description generated by MedGemma-4B. The model produces a detailed, clinically relevant analysis when prompted to describe visible features before diagnosing.

Type	Prompt
direct answer (baseline)	You are a helpful medical assistant. Look at the given image and determine the most likely dermatology condition from the options: (A) Granuloma annulare (B) Lupus erythematosus (C) Vitiligo. … …
in-context	You are a helpful medical assistant. Look at the given image and determine the most likely dermatology condition from the options: (A) Granuloma annulare, characterized by smooth skin-colored or pink papules arranged in an annular ring with firm ridge. (B) Lupus erythematosus, characterized by scaly plaques with follicular plugging and central scarring on sun-exposed skin; acute malar rash appears as flat red butterfly across cheeks sparing folds. (C) Vitiligo, characterized by sharply bordered milky white macules on face, hands, or genital skin with normal texture and pigment loss. … …
DtD	You are a helpful medical assistant. Look at the given image and describe visible lesion attributes (location, lesion type, shape/border, color, texture). Then determine the most likely diagnosis from the options: (A) Granuloma annulare (B) Lupus erythematosus (C) Vitiligo. … …

Appendix

Table 16: Zero-shot prompts used for evaluation.

Hyperparameter	Search Space	Selected Value
LoRA Rank ( $r$ )	$\{4,8,16\}$	$8$
LoRA Alpha ( $\alpha$ )	$\{8,16,32\}$	$16$
Learning Rate	$\{5e^{-5},1e^{-4},2e^{-4}\}$	$1e^{-4}$
LoRA Dropout	-	$0.05$
Target Modules	-	All Linear Layers
Precision	-	4-bit (QLoRA)

Appendix

Table 17: Hyperparameter search space and selected configuration for LoRA fine-tuning.

Rank ( $r$ )	Alpha ( $\alpha$ )	Learning Rate	Accuracy (%)
4	8	$5\times 10^{-5}$	42.15
4	8	$3\times 10^{-4}$	46.12
4	8	$4\times 10^{-4}$	45.88
4	16	$5\times 10^{-5}$	41.80
4	16	$3\times 10^{-4}$	45.05
4	16	$4\times 10^{-4}$	44.20

Appendix

Table 18: Full grid search results for LoRA fine-tuning on the Fitzpatrick17k dataset. The optimal configuration selected for the final model is highlighted in bold.

Hyperparameter	Value
Vision Encoder	MedSigLIP (Frozen)
Feature Pooling	Global Average Pooling
Precision (Backbone)	bfloat16
Precision (Head)	float32
Optimizer	AdamW
Batch Size	128

Appendix

Table 19: Implementation details and hyperparameters for Linear Probing experiments.

Dataset	0-shot Top-1	1-shot			2-shot			4-shot			8-shot			16-shot
Dataset	0-shot Top-1	Top1	Top5	0-Shot	Top1	Top5	0-Shot	Top1	Top5	0-Shot	Top1	Top5	0-Shot	Top1	Top5	0-Shot
Derm7pt	16.46	12.96	52.25	20.02	18.53	58.03	25.73	21.06	61.42	32.80	28.66	72.35	35.56	38.53	82.68	37.95
eSkinHealth	13.35	23.51	55.75	30.98	27.99	63.84	33.67	35.04	69.70	41.22	38.80	73.96	42.05	43.04	78.41	43.17
Fitzpatrick17k	16.73	17.63	50.58	25.35	23.08	59.50	27.42	31.01	67.67	33.68	38.43	75.88	37.82	45.05	80.99	38.01
SD-260	10.81	11.48	26.89	22.31	17.90	38.61	25.02	26.97	51.92	33.50	37.12	63.61	42.33	49.01	76.72	44.72
PAD-UFES-20	46.91	34.85	93.99	58.33	38.56	96.70	67.60	45.36	98.33	77.85	53.33	98.63	80.31	59.90	98.99	91.11

Appendix

Table 20: Effectiveness of the “Top-5 to Top-1” reranking strategy. This table compares three methods: the baseline zero-shot VLM, a standard k-shot linear probe (LP Top-1), and our two-stage approach. In our method, a k-shot linear probe selects the Top-5 candidates, which are then used by the VLM to produce a final prediction. The results show that our two-stage strategy consistently outperforms the direct linear probe’s Top-1 accuracy in the low-data regime.

	VLM Zero-Shot		Fine-Tuning (FT)		Linear Probe (LP)
Dataset	ACC	F1	ACC	F1	ACC	F1
Derm7pt	12.77	15.05	44.81	22.98	50.16	26.59
eSkinHealth	13.76	10.33	57.74	39.24	62.77	42.90
Fitzpatrick17k	10.05	8.27	42.84	30.49	58.88	46.22

Appendix

Table 21: Performance Gap Analysis on SkinVL. Comparison of the zero-shot VLM performance against full Fine-Tuning (FT) and a Linear Probe (LP) on the vision encoder. Consistent with MedGemma, the Linear Probe significantly outperforms the zero-shot VLM, confirming the under-utilization of visual features.

Skin Condition	Real Training	Real Test	Synthetic
Acne (ACN)	92	91	93
Actinic Keratosis (AK)	88	87	164
Allergic Contact Dermatitis (ACD)	215	215	181
Basal Cell Carcinoma (BCC)	234	234	154
Eczema (ECZ)	102	102	166
Erythema Multiforme (EM)	118	118	155

Appendix

Table 22: Skin Condition Distribution for Fitzpatrick17k

Dataset	# Cls	Train	Test	Class Names
Derm7pt	14	413	395	clark nevus, melanoma, reed or spitz nevus, seborrheic keratosis, basal cell carcinoma, vascular lesion, lentigo, blue nevus, dermal nevus, dermatofibroma, combined nevus, congenital nevus, miscellaneous, recurrent nevus
eSkinHealth	24	2,714	2,676	buruli ulcer, scabies, yaws, prurigo nodularis, tinea corporis, erysipelas, tinea capitis, leprosy, impetigo, necrotizing fasciitis, contact dermatitis, lichen planus, tinea versicolor, folliculitis, chickenpox, acne, vitiligo, abscess, keratosis, herpes zoster, atopic dermatitis, eczema, lipome, mycetoma
Fitzpatrick17k	20	3,100	3,100	psoriasis, squamous cell carcinoma, lichen planus, basal cell carcinoma, allergic contact dermatitis, lupus erythematosus, sarcoidosis, folliculitis, scabies, melanoma, erythema multiforme, granuloma annulare, eczema, pityriasis rosea, mycosis fungoides, acne, actinic keratosis, prurigo nodularis, vitiligo, keloid
SD-260	260	10,362	10,238	abrasion, acne excoriee, acne keloidalis nuchae, acne vulgaris, acrokeratosis verruciformis, actinic solar damage, actinex treatment, actinic cheilitis, actinic keratosis, cutis rhomboidalis nuchae, actinic favre-racouchot, pigmentation, actinic solar elastosis, solar purpura, telangiectasia, actinic wrinkles, acute eczema, allergic contact dermatitis, alopecia areata, anagen effluvium, androgenetic alopecia, angiofibroma, angiokeratoma, angioma, angular cheilitis, aphthous ulcer, apocrine hydrocystoma, arsenical keratosis, atopic dermatitis, balanitis xerotica obliterans, basal cell carcinoma, beaus lines, beckers nevus, behcets syndrome, benign keratosis, blue nevus, bowenoid papulosis, bowens disease, cafe au lait macule, callus, candidiasis, cellulitis, chalazion, cherry angioma, clubbing of fingers, combined nevus, compound nevus, congenital nevus, contact dermatitis, crowes sign, cutanea larva migrans, cutaneous horn, cutaneous leishmaniasis, cutaneous t-cell lymphoma, cutis marmorata, darier-white disease, dermatofibroma, dermatomyositis, dermatosis papulosa nigra, desquamation, digital fibroma, dilated pore of winer, discoid lupus erythematosus, disseminated actinic porokeratosis, drug eruption, dry skin eczema, dyshidrosiform eczema, dysplastic nevus, eccrine poroma, eczema, epidermal nevus, epidermoid cyst, epithelioma adenoides cysticum, erythema ab igne, erythema annulare centrifigum, erythema craquele, erythema multiforme, exfoliative erythroderma, factitial dermatitis, favre-racouchot, fibroma, fibroma molle, fixed drug eruption, follicular mucinosis, follicular retention cyst, fordyce spots, frictional lichenoid dermatitis, ganglion, geographic tongue, granulation tissue, granuloma annulare, green nail, guttate psoriasis, hailey-hailey disease, half and half nail, halo nevus, hand foot mouth disease, herpes gestationis, herpes simplex virus, herpes zoster, hidradenitis suppurativa, hirsutism, histiocytosis x, hyperkeratosis palmaris et plantaris, hypertrichosis, ichthyosis, ichthyosis vulgaris, id reaction, impetigo, infantile atopic dermatitis, insect bite, intradermal nevus, inverse psoriasis, ischemia, junction nevus, keloid, keratoacanthoma, keratolysis exfoliativa of wende, keratosis pilaris, kerion, koilonychia, kyrles disease, leiomyoma, lentigo maligna melanoma, lentigo simplex, leprosy, leukemia cutis, leukocytoclastic vasculitis, leukonychia, lichen planus, lichen sclerosis et atrophicus, lichen simplex chronicus, lichen spinulosis, linear epidermal nevus, lipoma, livedo reticularis, lymphangioma circumscriptum, lymphocytic infiltrate of jessner, lymphocytoma cutis, lymphomatoid papulosis, mal perforans, malignant melanoma, median nail dystrophy, melasma, metastatic carcinoma, milia, molluscum contagiosum, morphea, mucha-habermann disease, mucous membrane psoriasis, myxoid cyst, nail cosmesis, nail dystrophy, nail nevus, nail psoriasis, nail ridging, nail trauma, neurodermatitis, neurofibroma, neurotic excoriations, nevus cell nevus, nevus comedonicus, nevus incipiens, nevus sebaceous of jadassohn, nevus spilus, nummular eczema, onychogryphosis, onycholysis, onychomycosis, onychoschizia, paronychia, pearly penile papules, pediculosis pubis, pemphigus foliaceus, perioral dermatitis, photodermatitis, pilomatrixoma, pincer nail syndrome, pitted keratolysis, pityriasis alba, pityriasis rosea, pityriasis rubra pilaris, pityriasis versicolor, pityrosporum folliculitis, poikiloderma atrophicans vasculare, pomade acne, porokeratosis of mibelli, port wine stain, pseudofolliculitis barbae, pseudorhinophyma, psoriasis, pterygium inversum unguis, pustular psoriasis, pyoderma gangrenosum, pyogenic granuloma, racquet nail, radiodermatitis, rhinophyma, rosacea, scabies, scalp psoriasis, scar, scarring alopecia, schambergs disease, sebaceous gland hyperplasia, seborrheic dermatitis, seborrheic keratosis, skin tag, solar elastosis, solar lentigo, spindle cell nevus, squamous cell carcinoma (scc), stasis dermatitis, stasis edema, stasis ulcer, steroid acne, steroid atrophy, steroid striae, steroid use, stomatitis, strawberry hemangioma, striae, subacute cutaneous lupus erythematosus, subungual hematoma, superficial actinic porokeratosis, syringoma, systemic lupus erythematosus, terrys nails, thermal burn, tick bite, tinea capitis, tinea corporis, tinea cruris, tinea faciale, tinea incognito, tinea manus, tinea pedis, tinea versicolor, toe deformity, traction alopecia, trichilemmal cyst, trichoepithelioma, trichofolliculoma, trichostasis spinulosa, trichotillomania, tuberous sclerosis, twenty nail dystrophy, ulcer, urticaria, uvl burn, varicella, verruca vulgaris, viral exanthem, virilization, vitiligo, von recklinghausens disease, wart, wound infection, xerosis, x-linked ichthyosis
PAD-UFES-20	6	1,134	1,164	actinic keratosis, basal cell carcinoma, melanoma, nevus, seborrheic keratosis, squamous cell carcinoma

Appendix

Table 23: Dataset details.

Condition	Description
acne	Face/chest/back comedones and inflamed papules/pustules ±nodules; red or skin-colored with black/white heads; oily, ±crust.
actinic keratosis	Sun-exposed rough, scaly flat/slightly raised papule <1 cm; pink/red/brown; gritty sandpaper feel.
allergic contact dermatitis	At contact sites, ill-defined pink-red (darker on dark skin) patches/plaques ±vesicles/edema; weepy/crusty/scaly surface.
basal cell carcinoma	Sun-exposed pearly/waxy papule or thin scaly patch with rolled edge ±central ulcer; translucent or brown/black; smooth/shiny ±crust.
eczema	Flexural dry itchy ill-defined patches/plaques ±tiny vesicles; red/pink or purple/gray on dark skin; flaky ±lichenified.
erythema multiforme	Acral target lesions—round 1–3 cm with dark center, pale ring, red outer rim—mostly flat ±blister.

Appendix

Table 24: Skin condition descriptions used in this study.

Explore similar work

Medical Context Distorts Decisions in Clinical Vision Language Models
May 17, 2026David Restrepo, Ira Ktena, Maria Vakalopoulou +2Medical Vision-Language Models Multimodal Clinical Data
1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1
Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
Nov 18, 2025Zehao Liu, Weijieying Ren, Jipeng Zhang +4Skin Lesion Classification Melanoma
Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China
Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs
Jul 20, 2026Binesh Sadanandan, Vahid BehzadanMedical Vision-Language Models Explainable AI Methods
SAIL Lab, University of New Haven, West Haven, CT, USA

May 17, 2026, cs.CVJ/K move · Enter open · S save

Medical Context Distorts Decisions in Clinical Vision Language Models

David Restrepo, Ira Ktena, Maria Vakalopoulou, Stergios Christodoulidis+1

1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK+1

Medical Vision-Language Models Multimodal Clinical Data Clinical Decision Support Visual Evidence Modalities Decisions

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly proposed for clinical decision support, yet their reliability in real-world scenarios that require integrating both visual and textual context from medical records remains poorly characterized. This paper identifies three failure modes: (1) modality over-reliance on text over images, (2) spurious reliance on irrelevant clinical history, and (3) prompt sensitivity across semantically equivalent inputs. We evaluate a diverse set of general-domain and medically-tuned open and closed VLMs on chest x-ray tasks using MIMIC-CXR. By systematically manipulating image-text alignment, clinical history, and prompt formulations, we found that VLM decisions are dominated by the text modality, even when visual evidence is available. Moreover, we observed that VLMs are heavily influenced by irrelevant reports, while minor prompt changes can reverse correct image-based predictions. Our findings underscore the need for explicit safeguards and stress-testing before considering the use of these models in clinical practice.

Abstract

Explore similar work

Medical Context Distorts Decisions in Clinical Vision Language Models

Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models

Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs