Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

Oliver McLaughlin; Daniel Shubin; Carsten Eickhoff; Ritambhara Singh; William Rudman; Michal Golovanevsky

cs.CVApr 10, 2026

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

Authors: Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff, Ritambhara Singh, William Rudman, Michal Golovanevsky

Organizations: Brown University · University of Washington · University of Tübingen · The University of Texas at Austin

Abstract

Vision-language models (VLMs) are increasingly adapted through domain-specific fine-tuning, yet it remains unclear whether this improves reasoning beyond superficial visual cues, particularly in high-stakes domains like medicine. We evaluate four paired open-source VLMs (LLaVA vs. LLaVA-Med; Gemma vs. MedGemma) across four medical imaging tasks of increasing difficulty: brain tumor, pneumonia, skin cancer, and histopathology classification. We find that performance degrades toward near-random levels as task difficulty increases, indicating limited clinical reasoning. Medical fine-tuning provides no consistent advantage, and models are highly sensitive to prompt formulation, with minor changes causing large swings in accuracy and refusal rates. To test whether closed-form VQA suppresses latent knowledge, we introduce a description-based pipeline where models generate image descriptions that a text-only model (GPT-5.1) uses for diagnosis. This recovers a limited additional signal but remains bounded by task difficulty. Analysis of vision encoder embeddings further shows that failures stem from both weak visual representations and downstream reasoning. Overall, medical VLM performance is fragile, prompt-dependent, and not reliably improved by domain-specific fine-tuning.

Figures & tables

Figure 1: Overview of our four tasks. Visually salient tasks (e.g., brain tumor) are reliably solved, while more complex tasks (e.g., pneumonia, skin cancer, Invasive Ductal Carcinoma (IDC)) exhibit instability or near-random performance.

Figure 2: Small prompt changes cause inconsistent decisions in both medical and general VLMs.

Figure 3: Performance (F1 all ) across disease classification tasks for base and medically fine-tuned models, averaged across all prompt variants. Medical fine-tuning yields improvements on some tasks but does not consistently outperform base models.

Model	Setting	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	With one word	45	34	22	71	54
	Without one word	45	35	20	68	62
MedGemma	With one word	68	32	1	60	59
	Without one word	36	27	38	56	42
LLaVA	With one word	58	42	0	38	38
	Without one word	17	11	72	19	12

Table 2: Average performance across all diseases (brain tumor, pneumonia, IDC, and skin cancer), comparing one-word and unconstrained VQA.

	Brain Tumor		Pneumonia		Skin Cancer		IDC
Model	Corr $\uparrow$	F1 $\uparrow$	Corr $\uparrow$	F1 $\uparrow$	Corr $\uparrow$	F1 $\uparrow$	Corr $\uparrow$	F1 $\uparrow$
Gemma	76 $\pm$ 8	80 $\pm$ 5	54 $\pm$ 1	65 $\pm$ 5	50 $\pm$ 2	65 $\pm$ 1	52 $\pm$ 4	45 $\pm$ 29
MedGemma	82 $\pm$ 5	84 $\pm$ 3	70 $\pm$ 8	64 $\pm$ 22	55 $\pm$ 6	67 $\pm$ 2	51 $\pm$ 2	43 $\pm$ 29
LLaVA	70 $\pm$ 3	66 $\pm$ 9	52 $\pm$ 3	35 $\pm$ 15	50 $\pm$ 4	50 $\pm$ 10	51 $\pm$ 3	40 $\pm$ 27
LLaVA-Med	73 $\pm$ 18	78 $\pm$ 11	51 $\pm$ 3	45 $\pm$ 30	42 $\pm$ 5	42 $\pm$ 10	53 $\pm$ 3	50 $\pm$ 22

Table 3: Results from description-based evaluation, where GPT-5.1 diagnoses diseases using VLM-generated descriptions.

Figure 4: t-SNE projections of vision encoder embeddings of the four datasets. Each point represents an image embedding, colored by ground-truth label. Silhouette scores are reported in each subplot to quantify class separability.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Diagnostic Target	Prompt
Pneumonia	Is this image abnormal?
	Is this image normal?
	Does this image contain pneumonia?
	Does this image contain pneumonia? Consider signs of consolidation, air bronchograms, and silhouettes.
	Which does this image show? Choose one answer. A) Pneumonia B) No Pneumonia
	Which does this image show? Choose one answer. A) No Pneumonia B) Pneumonia

Appendix

Table 4: Prompts used for each disease classification task.

Diagnostic Target	Variant	Prompt
Brain Tumor	Role-based	You are a careful radiologist. Given the following brain MRI image description, answer ONLY whether a brain tumor is present. Return a single word: Yes or No.
	Minimal	Task: Decide if a brain tumor is present. Output must be Yes or No.
	Disease-framed	Task: Decide if malignancy is present. Output must be Yes or No.
Pneumonia	Role-based	You are a careful clinician. Given the following lung X-ray description, answer ONLY whether pneumonia is present. Return a single word: Yes or No.
	Minimal	Task: Decide if pneumonia is present. Output must be Yes or No.
	Disease-framed	Task: Decide if disease is present. Output must be Yes or No.

Appendix

Table 5: GPT-5.1 evaluator prompt variants used in the description-based pipeline. Each prompt is prepended to the VLM-generated description. All variants require a binary Yes/No response.

Disease	Model	Corr	Inc	Ref	F1 nr	F1 all
Brain Tumor	Gemma	55.5 $\pm$ 4.8	39.0 $\pm$ 12.0	5.5 $\pm$ 7.8	72.1 $\pm$ 6.5	67.7 $\pm$ 1.5
	MedGemma	76.3 $\pm$ 8.6	15.8 $\pm$ 5.3	7.8 $\pm$ 7.4	86.4 $\pm$ 3.9	79.6 $\pm$ 7.0
	LLaVA	26.3 $\pm$ 27.6	7.2 $\pm$ 10.0	66.5 $\pm$ 37.5	88.5 $\pm$ 13.9	28.0 $\pm$ 30.3
	LLaVA-Med	59.7 $\pm$ 10.0	39.0 $\pm$ 8.9	1.3 $\pm$ 3.3	33.2 $\pm$ 33.0	33.2 $\pm$ 33.0
Pneumonia	Gemma	51.8 $\pm$ 1.7	48.0 $\pm$ 1.8	0.2 $\pm$ 0.4	67.5 $\pm$ 0.8	67.4 $\pm$ 0.8
	MedGemma	43.2 $\pm$ 23.5	30.0 $\pm$ 19.5	26.8 $\pm$ 39.2	45.7 $\pm$ 25.5	39.0 $\pm$ 21.1

Appendix

Table 6: Average performance across prompts for the without answer in one word setting. Values are reported as mean

\pm

standard deviation across prompts. Corr = correct, Inc = incorrect, Ref = refused.

Disease	Model	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Brain Tumor	Gemma	60 $\pm$ 8	35 $\pm$ 14	6 $\pm$ 7	75 $\pm$ 8	70 $\pm$ 3
	MedGemma	79 $\pm$ 7	21 $\pm$ 7	0 $\pm$ 0	83 $\pm$ 5	83 $\pm$ 5
	LLaVA	69 $\pm$ 13	31 $\pm$ 13	0 $\pm$ 0	66 $\pm$ 18	66 $\pm$ 18
	LLaVA-Med	58 $\pm$ 10	42 $\pm$ 10	0 $\pm$ 0	25 $\pm$ 31	25 $\pm$ 31
Pneumonia	Gemma	45 $\pm$ 16	31 $\pm$ 19	24 $\pm$ 34	76 $\pm$ 10	54 $\pm$ 21
	MedGemma	70 $\pm$ 7	30 $\pm$ 7	0 $\pm$ 0	59 $\pm$ 19	59 $\pm$ 19

Appendix

Table 7: Average performance across prompts for the with answer in one word setting. Values are reported as mean with standard deviation in smaller font. Corr = correct, Inc = incorrect, Ref = refused.

Model	Prompt Variant	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	A) Tumor B) No tumor	58	42	0	70	70
	A) No tumor B) Tumor	51	49	0	67	67
	Is the image normal?	52	48	0	67	67
	Is the image abnormal?	51	47	2	68	66
	Tumor present? (consider …)	59	27	14	78	67
	Tumor present?	62	21	17	82	68

Appendix

Table 8: Evaluation results for Brain Tumor without a one-word answer requirement. Corr = correct, Inc = incorrect, Ref = refused.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Model	Prompt Variant	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	A) Tumor B) No tumor	62	35	3	74	72
	A) No tumor B) Tumor	50	48	2	68	66
	Is the image normal?	55	45	0	69	69
	Is the image abnormal?	54	43	3	70	68
	Tumor present? (consider …)	71	23	6	81	76
	Tumor present?	67	14	19	87	70

Appendix

Table 9: Evaluation results for Brain Tumor with a one-word answer requirement. Corr = correct, Inc = incorrect, Ref = refused.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Model	Prompt Variant	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	A) Pneumonia B) No Pneumonia	51	49	0	67	67
	A) No Pneumonia B) Pneumonia	51	49	0	67	67
	Is the image normal?	55	45	0	69	69
	Is the image abnormal?	52	48	0	68	68
	Pneumonia present? (consider …)	52	47	1	68	67
	Pneumonia present?	50	50	0	67	67

Appendix

Table 10: Evaluation results for Pneumonia without a one-word answer requirement. Corr = correct, Inc = incorrect, Ref = refused.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Model	Prompt Variant	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	A) Carcinoma B) No carcinoma	58	42	0	68	68
	A) No carcinoma B) Carcinoma	53	47	0	68	68
	Is the image normal?	26	15	59	62	25
	Is the image abnormal?	27	29	44	62	35
	IDC present? (consider …)	34	26	40	70	42
	IDC present?	13	17	70	59	18

Appendix

Table 11: Evaluation results for IDC without a one-word answer requirement. Corr = correct, Inc = incorrect, Ref = refused.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Model	Prompt Variant	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	A) Carcinoma B) No carcinoma	52	47	1	64	63
	A) No carcinoma B) Carcinoma	52	48	0	68	68
	Is the image normal?	52	48	0	65	65
	Is the image abnormal?	54	46	0	67	67
	IDC present? (consider …)	29	29	42	61	36
	IDC present?	9	8	83	50	8

Appendix

Table 12: Evaluation results for IDC with a one-word answer requirement. Corr = correct, Inc = incorrect, Ref = refused.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Model	Prompt Variant	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	A) Benign B) Malignant	49	51	0	58	58
	A) Malignant B) Benign	46	54	0	61	61
	Is the image normal?	38	35	27	67	49
	Is the image abnormal?	39	36	25	68	51
	Malignant skin cancer present? (consider …)	50	49	1	67	66
	Malignant skin cancer present?	43	45	12	66	58

Appendix

Table 13: Evaluation results for Skin Cancer without a one-word answer requirement. Corr = correct, Inc = incorrect, Ref = refused.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Model	Prompt Variant	Corr $\uparrow$	Inc $\downarrow$	Ref $\downarrow$	F1 nr $\uparrow$	F1 all $\uparrow$
Gemma	A) Benign B) Malignant	46	54	0	63	63
	A) Malignant B) Benign	45	55	0	61	61
	Is the image normal?	39	37	24	68	52
	Is the image abnormal?	28	18	54	76	35
	Malignant skin cancer present? (consider …)	28	25	47	69	37
	Malignant skin cancer present?	4	1	95	89	4

Appendix

Table 14: Evaluation results for Skin Cancer with a one-word answer requirement. Corr = correct, Inc = incorrect, Ref = refused.

\uparrow

indicates higher is better;

\downarrow

indicates lower is better.

Explore similar work

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
Jun 10, 2026Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6Medical VQA Medical VLMs
York University · Vector Institute · University of British Columbia +5
How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
Sep 29, 2026Janet Wang, Yunbei Zhang, Xiao Wang +1Medical Diagnosis VLM Interpretability
Tulane University · Oak Ridge National Laboratory
Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMs
May 23, 2026Wen Ma, Fucheng Niu, Zhiting Fan +3Adversarial Attacks on VLMs Healthcare
Zhejiang University, Zhejiang, China · Guangdong Institute of Intelligence Science and Technology

Jun 10, 2026J/K move · Enter open · S save

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci, Abeer Badawi+5

York University · Vector Institute · University of British Columbia+5

Medical VQA Medical VLMs VLM Reasoning Visually Grounded Reasoning Medical Image Benchmarks

Open paper AskPDF arXiv

High-stakes clinical use of large vision-language models (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers. We introduce OpenMedReason, a large-scale, open multimodal medical reasoning corpus comprising approximately 450K image-question-answer instances whose reasoning traces are primarily derived from curated biomedical, human-authored scientific articles. OpenMedReason provides high-fidelity supervision beyond synthetic chains of thought, covering diverse medical domain vision modalities such as radiological scans, microscopic images, visible light photographs, charts, and others. We complement it with OpenMedReason-Bench, a held-out benchmark that allows fine-grained evaluation of LVLMs along three complementary axes of capability, including perception, medical knowledge, and rationale, enabling diagnostic evaluation beyond final-answer accuracy. OpenMedReason is a rich training resource that exhibits its effectiveness in both supervised fine-tuning (SFT) and reinforcement-based alignment. Training with OpenMedReason yields a 20% average improvement in VQA accuracy over the base model and achieves performance within 4.2% of the strongest comparable-scale medical LVLMs. Fine-grained performance analysis confirms that the gains are not concentrated in any single axis: OpenMedReason improves perception, medical knowledge, and rationale jointly, and its reasoning traces are preferred over those of the base model in 86.1% of pairwise comparisons. We release the code and dataset at huggingface.co/datasets/neginb/OpenMedReason.

Abstract

Explore similar work

OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMs