Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following

Myeongkyun Kang; Soopil Kim; Xiaoxiao Li; Sang Hyun Park

cs.CVMar 19, 2026

Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following

Authors: Myeongkyun Kang, Soopil Kim, Xiaoxiao Li, Sang Hyun Park

Organizations: Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea · Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea · Vector Institute, Toronto, ON M5G 0C6, Canada · Department of Computer Science and Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, Republic of Korea

Abstract

Large vision language models (LVLMs) have demonstrated impressive performance across a wide range of tasks. These capabilities largely stem from visual instruction tuning, which fine-tunes models on datasets consisting of curated image-instruction-output triplets. However, in the medical domain, constructing large-scale, high-quality instruction datasets is particularly challenging due to the need for specialized expert knowledge. To address this issue, we propose an instruction-free tuning approach that reduces reliance on handcrafted instructions, leveraging only image-description pairs for fine-tuning. Specifically, we introduce a momentum proxy instruction as a replacement for curated text instructions, which preserves the instruction-following capability of the pre-trained LVLM while promoting updates to parameters that remain valid during inference. Consequently, the fine-tuned LVLM can flexibly respond to domain-specific instructions, even though explicit instructions are absent during fine-tuning. Additionally, we incorporate a response shuffling strategy to mitigate the model's over-reliance on previous words, facilitating more effective fine-tuning. Our approach achieves state-of-the-art accuracy on multiple-choice visual question answering tasks across SKINCON, WBCAtt, CBIS, and MIMIC-CXR datasets, significantly enhancing the fine-tuning efficiency of LVLMs in medical domains.

Figures & tables

Panel of Figure 1 — Fig. 1 : Conceptual illustrations of (a) instruction tuning, which fine-tunes the model on curated instructions and outputs, and (b) instruction-free tuning, which fine-tunes the model solely on paired textual descriptions (e.g., radiology reports). Instruction tuning requires instruction-image-output triplets constructed by humans or LLMs prior to fine-tuning, whereas instruction-free tuning can be performed on image-description pairs without additional steps. Examples of an image, an instruction-output pair for instruction tuning, and a description used for instruction-free tuning are shown in (c).

Fig. 2 : An illustration of our instruction-free tuning framework. The vision encoder $g$ extracts key-value matrices ${KV}_{v}$ from an image $X_{v}$ , which are then integrated into a prompt $X_{p}$ for the language model $f$ to generate the response $\hat{y}$ . For instruction-free tuning, the text instruction in $X_{p}$ is replaced with the momentum proxy instruction $\bar{t}$ . During supervised fine-tuning, $g$ is updated with autoregressive loss $L$ between the response $\hat{y}$ and the ground truth $y$ . In parallel, (warm-up initialized) $\bar{t}$ is gradually updated via exponential moving average of the proxy instruction $t$ . During inference, $\bar{t}$ is discarded, and a conversational text instruction (e.g., “Describe…” ) is used to generate a natural language response (e.g., “The image depicts…” ).

Fig. 3 : Examples of the medical report for the (a) SKINCON, (b) WBCAtt, (c) CBIS, and (d) MIMIC-CXR datasets.

Fig. 4 : Examples of multiple-choice VQA for the (a) SKINCON, (b) WBCAtt, (c) CBIS, and (d) MIMIC-CXR datasets.

Model	SKINCON	WBCAtt	CBIS	Avg.
BLIP-2 ( w/o FT)	8.7	35.2	4.6	16.1
MedGemma-4B ( w/o FT)	14.3	37.3	39.2	30.2
PubMedVision-7B ( w/o FT)	47.0	36.3	40.6	41.3
Qwen2.5-VL-3B ( w/o FT)	37.7	32.8	36.0	35.5
LLaMA-3.2-11B-Vision ( w/o FT)	39.9	34.6	43.0	39.1
MedGemma-4B (FT)	8.9 $\pm$ 1.76	45.9 $\pm$ 1.18	42.5 $\pm$ 2.23	32.4

TABLE I : Multiple-choice VQA accuracy on the SKINCON, WBCAtt, and CBIS datasets. We compared our method (InstFree) with BLIP-2 [ 32 ] , MedGemma-4B [ 3 ] , PubMedVision-7B [ 4 ] , Qwen2.5-VL-3B [ 17 ] , and LLaMA-3.2-11B-Vision [ 16 ] . FT denotes fine-tuning, and RS denotes response shuffling. Bold indicates the highest accuracy, and underline denotes the second highest.

Panel of Figure 6 — Fig. 5 : Radar charts of the accuracy for each attribute (or question type) across the (a) WBCAtt, (b) CBIS, and (c) MIMIC-CXR datasets. LLaMA-3.2-11B-Vision ( w/o FT) is shown in black, LLaMA-3.2-11B-Vision (FT) in green, InstFree in blue, InstFree w/ RS in red, and MedGemma-4B in yellow-green.

Method	SKINCON	WBCAtt	CBIS	Avg.
Fix	63.9 $\pm$ 1.52	65.1 $\pm$ 0.97	61.7 $\pm$ 1.49	63.5
Update	66.4 $\pm$ 2.55	63.6 $\pm$ 0.83	63.7 $\pm$ 0.49	64.5
InstFree	69.8 $\pm$ 0.75	68.4 $\pm$ 0.61	65.0 $\pm$ 0.37	67.7

TABLE II : Multiple-choice VQA accuracy of instruction-free tuning variants on the SKINCON, WBCAtt, and CBIS datasets. Fix and Update denote models fine-tuned with fixed and continuously updating proxy instructions, respectively.

Model	Accuracy
LLaMA-3.2-11B-Vision (w/o FT)	46.9
MedGemma-4B	44.7
InstFree	58.9 $\pm$ 0.29
InstFree w/ RS	59.3 $\pm$ 0.12

TABLE III : VQA accuracy on the MIMIC-CXR dataset. We compared our method with MedGemma-4B [ 3 ] , an instruction-tuned model trained on the MIMIC-CXR dataset, and LLaMA-3.2-11B-Vision [ 16 ] .

Method	SKN.	WBC.	CBIS	Avg.
InstFree w/ Bal	65.4	63.4	60.8	63.2
InstFree w/ RS (“ ”)	67.9	57.9	67.2	64.3
InstFree w/ RS	75.2	65.2	68.3	69.5
FT w/ RS	74.1	62.0	67.3	67.8
FT w/ Rand	3.1	27.5	27.9	19.5

TABLE IV : Ablation study of response shuffling on the SKINCON (SKN.), WBCAtt (WBC.), and CBIS datasets. Bal denotes balanced sampling; RS (“ ”) and Rand denote models fine-tuned with an incorrect separator and randomized text instructions, respectively.

Panel of Figure 10 — Fig. 6 : Ablation study of (a) the momentum coefficient $\alpha$ and (b) the number of instruction vectors $N$ on the SKINCON, WBCAtt, and CBIS datasets.

Fig. 7 : Qualitative VQA results on the WBCAtt dataset. Correct responses are marked in green, incorrect responses are marked in red, and responses that do not follow instructions are marked in orange.

Explore similar work

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
Apr 10, 2026Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3Medical VLMs VLM Reasoning
Brown University · University of Washington · University of Tübingen +1
OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models
May 27, 2026Xuanzhao Dong, Wenhui Zhu, Xiwen Chen +13Medical Image Analysis Medical VQA
1Arizona State University · 2Clemson University · 3Washington University in St. Louis +5
Boosting Visual Instruction Tuning with Self-Supervised Guidance
Apr 14, 2026Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc +2Large Vision-Language Models VLM Reasoning
Sorbonne Universite, CNRS, ISIR, F-75005 Paris, France · Institut universitaire de France (IUF)

Apr 10, 2026J/K move · Enter open · S save

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff, Ritambhara Singh+2

Brown University · University of Washington · University of Tübingen+1

Medical VLMs VLM Reasoning VLM Robustness Medical Image Classification Prompt Sensitivity

Open paper AskPDF arXiv

Vision-language models (VLMs) are increasingly adapted through domain-specific fine-tuning, yet it remains unclear whether this improves reasoning beyond superficial visual cues, particularly in high-stakes domains like medicine. We evaluate four paired open-source VLMs (LLaVA vs. LLaVA-Med; Gemma vs. MedGemma) across four medical imaging tasks of increasing difficulty: brain tumor, pneumonia, skin cancer, and histopathology classification. We find that performance degrades toward near-random levels as task difficulty increases, indicating limited clinical reasoning. Medical fine-tuning provides no consistent advantage, and models are highly sensitive to prompt formulation, with minor changes causing large swings in accuracy and refusal rates. To test whether closed-form VQA suppresses latent knowledge, we introduce a description-based pipeline where models generate image descriptions that a text-only model (GPT-5.1) uses for diagnosis. This recovers a limited additional signal but remains bounded by task difficulty. Analysis of vision encoder embeddings further shows that failures stem from both weak visual representations and downstream reasoning. Overall, medical VLM performance is fragile, prompt-dependent, and not reliably improved by domain-specific fine-tuning.

Abstract

Explore similar work

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models

Boosting Visual Instruction Tuning with Self-Supervised Guidance