MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

Erfan Nourbakhsh; Ke Yang; Anthony Rios

cs.CLSep 3, 2026

MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

Authors: Erfan Nourbakhsh, Ke Yang, Anthony Rios

Organizations: The University of Texas at San Antonio

Abstract

Medical visual question answering (Med-VQA) is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with MedProb, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals. Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability. Finally, free-text generation exhibits an answer-position bias of up to 10 percentage points, whereas MedProb also has positional bias, however, it is impacted differently than prompting. Our main results target the multiple-choice/multiclass Med-VQA setting; we additionally show the probe can be extended to open-ended generation via a rejection-sampling scoring procedure.

Figures & tables

Figure 1: Prior methods (left) give a medical image and question to a VLM or multi-agent system, which then generates a free-text answer. This can lead to unsupported clinical reasoning. MedProb (right) uses the same input, but keeps the VLM frozen, extracts hidden representations from each layer, and predicts the multiple-choice answer with a linear classifier.

Figure 2: Overview of the MedProb evaluation framework. Left : general-purpose and medical VLMs evaluated, grouped by backbone architecture. Right : prompts consist of a system prompt, medical image, clinical question, and multiple-choice options (A–D), evaluated under three settings. Top : instruction-tuned VLMs are prompted directly (frozen). Middle : MedProb extracts hidden representations from each transformer layer and decodes answers via a linear classifier (frozen). Bottom : fine-tuned VLMs are trained on medical data, then prompted (frozen).

Model	Method	PATH-VQA		SLAKE		VQA-RAD		Average
		F1	Acc	F1	Acc	F1	Acc	F1	Acc
MAM Zhou et al. (2025)	Agentic	51.85	51.80	39.92	59.51	39.31	59.31	39.62	56.87
MMedAgent Li et al. (2024)	Agentic	36.11	37.00	28.30	43.13	59.80	58.17	41.40	46.10
UCAgents Feng et al. (2025)	Agentic	60.55	61.19	78.09	77.10	64.17	71.10	62.36	69.80
HuatuoGPT-Vision-34B Chen et al. (2024a)	Medical VLM	60.61	61.80	77.89	76.86	83.57	75.66	74.02	71.44
Aloe-Vision-72B-AR Guasch-Martí et al. (2026)	Medical VLM	67.70	69.20	86.80	80.24	75.73	79.46	76.74	76.30

Table 1: Performance comparison on medical VQA benchmarks. Bold : best per column; MedProb rows are highlighted in blue; † Trained on PATH-VQA, VQA-RAD, and SLAKE.

Figure 3: Prompting vs. MedProb accuracy for general (Gemma-4B/27B) and medical (MedGemma-4B/27B) VLMs on VQA-RAD and PATH-VQA.

Figure 4: Probing vs. prompting accuracy after single-domain and multi-domain fine-tuning.

Figure 5: Layer-wise probing accuracy for Qwen2.5-VL-7B and MedVLThinker-7B.

Figure 6: Probing vs. prompting accuracy under random, correct-first, and correct-last answer orderings.

General Model	Medical Model	$\Delta$
Gemma-4B	MedGemma-4B	$-$ 4.92
Gemma-27B	MedGemma-27B	$-$ 0.95
Open-Flamingo-9B	Med-Flamingo-9B	$-$ 1.93
InternVL3-1B	BioMed-InternVL3-1B	$+$ 2.36
LLaVA-7B	LLaVA-Med-7B	$+$ 0.17
LLaMA3.2-11B-Vision	BioMed-LLaMA3.2-11B	$-$ 2.30

Table 2: Probing accuracy delta (

\Delta

) between matched general/medical pairs (averaged across benchmarks). Positive: medical better; negative: general better.

Model	Method	SLAKE	VQA-RAD
Llama-3.2-11B-Vision	MedProb	82.4	76.0
	Text-only (image-ablated)	67.2	68.4
	Image-shuffled	66.5	65.8
Qwen3-VL-8B	MedProb	84.1	75.3
	Text-only (image-ablated)	67.7	68.4
	Image-shuffled	65.8	69.1

Table 3: Best-layer probe accuracy (%) with the original image, an image-ablated (blank) control, and an image-shuffled (mismatched) control.

Panel of Figure 7 — Figure 7: Semantic hallucination failure: prompted model identifies real visual features but draws a false clinical conclusion; probe recovers the correct answer from latent activations.

Figure 8: PATH-VQA probe accuracy (%) vs. training-set size (mean

\pm

std over different random seeds).

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Split	Size	Yes/No	Multiple Choice
VQA-RAD	Train	820	753	67
	Validation	205	187	18
	Test	272	251	21
	Total	1297	1191	106
PATH-VQA	Train	9804	9804	0
	Validation	3135	3135	0

Appendix

Table 4: Summary of the number of close-ended examples in the train, validation, and test sets of all visual QA datasets.

Model Name	HF Link
HuatuoGPT-Vision-34B	Link
Aloe-Vision-72B-AR	Link
InfiMed-RL-3B	Link
UniMedVL-14B	Link
Gemma-3-4B-IT	Link
Gemma-3-27B-IT	Link

Appendix

Table 5: Vision-language models and their Hugging Face repositories.

General Model	Medical Model	Medical Training Datasets
Gemma (4B, 27B) Team et al. (2025)	MedGemma (4B, 27B) Sellergren et al. (2026)	MIMIC-CXR, CT-US1, MRI-US1, PMC-OA, EyePACS, PAD-UFES-20, Digital Knee X-ray, internal histopathology ( ${\sim}$ 32.55M patches), internal dermatology (51K images), SLAKE, VQA-RAD.
LLaVA-V0-7B Liu et al. (2023)	LLaVA-Med-7B Li et al. (2023)	PMC-15M (600K caption pairs; 60K GPT-4 instruction pairs across CXR, CT, MRI, histopathology, and gross pathology).
OpenFlamingo-9B Awadalla et al. (2023)	Med-Flamingo-9B Moor et al. (2023b)	Medical Textbook dataset (MTB; ${\sim}$ 0.8M images), PMC-OA (1.3M pairs).
InternVL3-1B Zhu et al. (2025)	BioMed-InternVL3-1B Cheng et al. (2025)	PubMedVision/PMCRefined (500K caption pairs), 144K synthetic visual instruction tasks.
Llama-3.2-11B Vision Grattafiori et al. (2024)	BioMed-Llama-3.2-11B Cheng et al. (2025)	PubMedVision/PMCRefined (500K caption pairs), 144K synthetic visual instruction tasks.
Qwen2-VL-2B Wang et al. (2024)	BioMed-Qwen2-VL-2B Cheng et al. (2025)	PubMedVision/PMCRefined (500K caption pairs), 144K synthetic visual instruction tasks.

Appendix

Table 6: General-to-medical VLMs correspondence and training datasets.

Category	Model	System Prompt	Prompt Format	Sampling
General Models	Gemma (4B, 27B) Team et al. (2025)	✗	✗	⚫
	LLaVA-V0-7B Liu et al. (2023)	⚫	⚫	⚫
	OpenFlamingo-9B Awadalla et al. (2023)	✗	✗	✗
	InternVL3-1B Zhu et al. (2025)	✗	✓	⚫
	Llama-3.2-11B-Vision Grattafiori et al. (2024)	✗	⚫	✗
	Qwen3-VL (2B, 4B, 8B, 30B) Bai et al. (2025a)	✗	✓	✓

Appendix

Table 7: Summary of prompting details available for each evaluated model. ✓ = fully provided; ⚫ = partially provided; ✗ = not provided or not applicable.

Model	Shot	PATH	SLAKE	RAD	Avg.
InternVL3-1B	3	58.20	66.99	68.82	64.67
InternVL3-1B	0	62.80	75.66	65.02	67.83
BioMed-Qwen2-VL-2B	3	37.20	41.93	12.17	30.43
BioMed-Qwen2-VL-2B	0	56.20	63.62	65.02	61.61
Qwen3-VL-2B	3	57.20	74.22	65.02	65.48
Qwen3-VL-2B	0	56.80	77.83	64.64	66.42

Appendix

Table 8: Probing accuracy (%) under 0-shot and 3-shot settings. Bold indicates the better result per model–benchmark pair. The shaded column highlights the average.

Model	Method	PVQ	SLK	RAD	Avg.
LLaVA-V0-7B	Free-text	49.6	45.5	42.6	45.9
	Opt.-likeli.	45.8	53.7	56.2	51.9
	Probing	84.6	68.0	66.9	73.2
Llama-3.2-11B	Free-text	59.8	69.9	66.5	65.4
	Opt.-likeli.	62.2	72.8	72.1	69.0
	Probing	86.2	82.4	75.4	81.3

Appendix

Table 9: Accuracy (%) for free-text prompting, option-likelihood scoring, and MedProb on three representative VLMs. PVQ = PATH-VQA; SLK = SLAKE; RAD = VQA-RAD. Llama = Llama-3.2-11B-Vision.

Model	Dataset	$\boldsymbol{\Delta}$	95% CI	$\boldsymbol{p}$
LLaVA-V0-7B	PATH-VQA	$+35.0$	$[29.6,\;40.4]$	$6.5\!\times\!10^{-32}$
	SLAKE	$+22.4$	$[15.7,\;29.2]$	$4.1\!\times\!10^{-10}$
	VQA-RAD	$+24.3$	$[16.2,\;32.7]$	$5.6\!\times\!10^{-8}$
Llama-3.2-11B	PATH-VQA	$+26.4$	$[21.2,\;31.6]$	$2.4\!\times\!10^{-21}$
	SLAKE	$+12.5$	$[7.7,\;17.3]$	$1.2\!\times\!10^{-6}$
	VQA-RAD	$+8.8$	$[3.3,\;14.3]$	$2.7\!\times\!10^{-3}$

Appendix

Table 10: Paired significance analysis for probing vs. free-text prompting.

\Delta

= probing

-

prompting accuracy (percentage points). Model abbreviations as in Table 9 .

Model	Train Dataset	PATH-VQA	SLAKE	VQA-RAD
InternVL3-1B	OmniMedVQA	55.40	72.29	61.60
	PATH-VQA	82.20	57.35	57.41
	SLAKE	63.00	83.13	58.94
	VQA-RAD	61.40	75.18	72.62
LLaVA-V0-7B	OmniMedVQA	53.60	51.08	53.61
	PATH-VQA	84.60	53.49	55.13

Appendix

Table 11: Probing accuracy (%) across models and training datasets. Bold indicates in-domain (train and test from same benchmark). PATH-VQA-trained probes show severe domain-locking, collapsing on radiology benchmarks. VQA-RAD-trained probes transfer well to SLAKE due to shared radiology domain.

Model	Method	PATH-VQA	SLAKE Binary	VQA-RAD Binary
Uniform random	—	50.00	50.00	50.00
InternVL3-1B	Prompting	62.80	78.31	62.55
InternVL3-1B	MedProb	82.20	85.35	70.52
Qwen3-VL-8B	Prompting	59.00	78.31	69.72
Qwen3-VL-8B	MedProb	86.20	86.48	75.30
MedVLThinker-3B	Prompting	59.60	65.35	69.72

Appendix

Table 12: Binary (yes/no) accuracy (%) for representative models.

Model	Uniform	Prompting	MedProb
InternVL3-1B	25.00	67.75	71.75
Qwen3-VL-8B	25.00	61.75	80.75
MedVLThinker-3B	25.00	63.50	67.50

Appendix

Table 13: True multiclass accuracy (%) on OmniMedVQA (4-way, 400 examples; chance 25%).

Model	Rejection- sampling probe (ours)	Without probe (baseline)	Oracle (select correct candidate)
Qwen2.5-VL-7B-Instruct	0.13	0.09	0.19
Llama-3.2-11B-Vision-Instruct	0.19	0.15	0.27

Appendix

Table 14: Accuracy of rejection sampling with the probe as a reranker over open-ended VLM generations, compared to a single-sample baseline (no probe) and an oracle upper bound.

Model	Dataset	Last-10 (mean $\pm$ std)	Last-10 range	Prompting
Qwen2.5-VL-7B	PATH-VQA	82.06 $\pm$ 0.70	80.40–82.80	62.40
	SLAKE	80.96 $\pm$ 1.06	79.28–82.89	73.01
	VQA-RAD	75.82 $\pm$ 1.79	72.24–77.57	71.10
MedVLThinker-7B	PATH-VQA	82.78 $\pm$ 1.10	81.00–84.40	64.60
	SLAKE	77.06 $\pm$ 1.27	75.66–79.76	65.78
	VQA-RAD	73.95 $\pm$ 1.18	71.86–75.67	65.78

Appendix

Table 15: Accuracy (%) across the final 10 layers (mean

\pm

std, and range) compared to zero-shot prompting. All 10 final layers outperform promptingin every model/dataset combination.

Figure 9: Layer-wise probing accuracy for Gemma-4B and MedGemma-4B. The medical model is competitive from the first layers and establishes a clearer advantage in the deeper part of the network.

Figure 10: Layer-wise probing accuracy for Gemma-27B and MedGemma-27B. The two models remain close overall, with an early advantage for the general model and a more competitive late-layer profile for the medical variant.

Figure 11: Layer-wise probing accuracy for Open-Flamingo-9B and Med-Flamingo-9B. The medical model maintains a consistent advantage across nearly all layers.

Figure 12: Layer-wise probing accuracy for InternVL3-1B and BioMed-InternVL3-1B. The medical model is stronger early, while the general model overtakes it in the deeper layers.

Figure 13: Layer-wise probing accuracy for LLaVA-7B and LLaVA-Med-7B. The two curves are tightly matched, with only small and localized differences across depth.

Figure 14: Layer-wise probing accuracy for LLaMA3.2-11B-Vision and BioMed-LLaMA3.2-11B. The medical model holds a modest but persistent advantage across most of the depth.

Figure 15: Layer-wise probing accuracy for Qwen2-VL-2B and BioMed-Qwen2-VL-2B. The general model has a small early-to-middle advantage and reaches a slightly higher late-layer maximum.

Figure 16: Layer-wise probing accuracy for Qwen2.5-VL-3B and MedVLThinker-RL-3B. The curves stay close early on, but the general model pulls ahead in the deepest layers.

Figure 17: Layer-wise probing accuracy for Qwen2.5-VL-32B and MedVLThinker-RL-32B. The general model leads through most middle and late layers and achieves the higher peak accuracy.

Figure 18: Layer-wise probing accuracy for Qwen3-VL-2B and MediX-R1-2B. The general model quickly establishes a large and growing advantage across the deeper layers.

Figure 19: Layer-wise probing accuracy for Qwen3-VL-4B and MedMo-4B. The general model leads across much of the network, but the medical model rebounds late and reaches a slightly higher maximum.

Figure 20: Layer-wise probing accuracy for Qwen3-VL-8B and MedMo-8B. The general model leads through most early and middle layers, while the medical model catches up in the deeper region.

Figure 21: Layer-wise probing accuracy for Qwen3-VL-30B and MediX-R1-30B. The general model stays ahead across almost the entire network and widens the gap again in the deepest layers.

Ex.	Modality / Task	GT	Prompt	Probe	Failure Pattern
E-1	Histopathology (soft-tissue mass classification)	B (No)	A $\times$	B ✓	Hallucinated pathology narrative
E-2	Brain MRI (imaging modality identification)	B (No)	A $\times$	B ✓	Correct rule, misapplied context

Appendix

Table 16: Examples where MedProb succeeds and prompting fails ( Probe

=

✓ , Prompt

=

\times

Panel of Figure 22 — Figure 22: Semantic hallucination failure: the model accurately describes the visual features but constructs a false pathology narrative and answers incorrectly; the probe at layer 16 is unaffected by this chain-of-thought error.

Panel of Figure 23 — Figure 23: Rule-application failure: the model’s radiological reasoning is internally correct but misattributed to the wrong conclusion; the probe at layer 22 holds the correct answer in its activation pattern regardless of the verbalized reasoning chain.

Ex.	Modality / Task	GT	Prompt	Probe	Failure Pattern
E-3	Brain MRI (ring-enhancement pattern detection)	A (Yes)	B $\times$	B $\times$	Subtle spatial pattern beyond model’s resolution
E-4	Gross pathology (bowel wall skip-lesion detection)	B (No)	A $\times$	A $\times$	Fine-grained structural discrimination failure

Appendix

Table 17: Examples where both MedProb and prompting fail ( Probe

=

\times

, Prompt

=

\times

Panel of Figure 24 — Figure 24: Perceptual ceiling failure: two independently trained models both fail on ring-enhancement detection across different reasoning strategies, with both prompt and probe predicting incorrectly, indicating a task-level perceptual limitation rather than a model-specific artifact.

Panel of Figure 25 — Figure 25: Confident bilateral failure: the model identifies all key pathological features and concludes they are present, but the ground truth is negative; both prompt and probe agree on the wrong answer, indicating representational incapacity for this class of fine-grained bowel pathology.

Ex.	Modality / Task	GT	Prompt	Probe	Failure Pattern
E-5	Lung histology (coarse tissue category recognition)	A (Yes)	A ✓	B $\times$	Direct semantic label, probe misses coarse class
E-6	Gross pathology (cardiac CHF classification)	A (Yes)	A ✓	B $\times$	Compositional clinical reasoning required

Appendix

Table 18: Examples where prompting succeeds but MedProb fails ( Probe

=

\times

, Prompt

=

✓ ).

Panel of Figure 26 — Figure 26: Coarse semantic label failure: prompting trivially names the tissue in one sentence and is correct; the probe fails on the same question, suggesting coarse semantic categories may not be linearly separable in intermediate representation layers.

Panel of Figure 27 — Figure 27: Compositional reasoning failure: the prompted model correctly chains wall thickening $\rightarrow$ fibrosis $\rightarrow$ dilated chambers $\rightarrow$ CHF; the probe cannot replicate this nonlinear multi-step clinical judgment from a single hidden layer.

Cat.	Ex.	GT	Prompt	Probe	Key Failure Mode
P $>$ M	E-1	B	$\times$ A	✓ B	Hallucinated pathology narrative
P $>$ M	E-2	B	$\times$ A	✓ B	Textbook rule misapplied
Both $\times$	E-3	A	$\times$ B	$\times$ B	Ring-enhancement too subtle
Both $\times$	E-4	B	$\times$ A	$\times$ A	Bowel skip-lesion ambiguity
M $>$ P	E-5	A	✓ A	$\times$ B	Coarse semantic label inaccessible to probe
M $>$ P	E-6	A	✓ A	$\times$ B	CHF requires compositional reasoning

Appendix

Table 19: Consolidated summary of all six error analysis examples. Category abbreviations: P

>

M = Probe better than prompting; Both

\times

= Both fail; M

>

P = Prompting better than probe.

Model	Approach	Modality
Model	Approach	CT	Dermoscopy	Fundus Photography	Microscopy Images	MRI	OCT	Ultrasound	X-Ray	Average
Open-Flamingo-9B	Prompting	18.0	26.0	36.0	30.0	26.0	26.0	24.0	40.0	28.0
Open-Flamingo-9B	Probing	62.0	58.0	76.0	56.0	60.0	66.0	82.0	64.0	66.0
Med-Flamingo-9B	Prompting	26.0	20.0	28.0	36.0	40.0	38.0	40.0	36.0	33.0
Med-Flamingo-9B	Probing	80.0	64.0	80.0	60.0	64.0	76.0	76.0	72.0	72.0
InternVL3-1B	Prompting	62.0	70.0	68.0	74.0	74.0	46.0	76.0	72.0	68.0

Appendix

Table 20: Prompting and probing accuracy (%) across eight medical imaging modalities on OmniMedVQA. Bold values highlight the most striking contrasts between the two paradigms, including near-zero prompting scores for large general-purpose models and the rare cases where prompting matches or exceeds probing.

Model	Log. Acc.	MLP Acc.	Log. F1	MLP F1
Llama-3.2-11B-Vision	81.3	80.9	85.9	81.1
Qwen3-VL-8B	81.4	81.6	82.3	71.3

Appendix

Table 21: Average accuracy and macro-F1 (%) across PATH-VQA, SLAKE, and VQA-RAD for the linear logistic probe and the nonlinear MLP probe.

Model	Acc.	Macro-F1	Best Layers
LLaVA-V0-7B	$87.1\pm 0.9$	$87.0\pm 0.9$	18 / 16 / 27
Llama-3.2-11B-Vision	$89.9\pm 0.5$	$89.8\pm 0.5$	20 / 13 / 23
Qwen3-VL-8B	$91.7\pm 0.6$	$91.6\pm 0.6$	21 / 25 / 23

Appendix

Table 22: PATH-VQA sampling sensitivity: mean

\pm

std across seeds 42, 43, and 44. Best Layers lists the optimal layer per seed in the same order.

Figure 28: Probing vs. prompting accuracy (%) on PATH-VQA for 24 of the 28 evaluated models, sorted by probing score.

Figure 29: Probing vs. prompting accuracy (%) on SLAKE for 24 of the 28 evaluated models, sorted by probing score.

Figure 30: Probing vs. prompting accuracy (%) on VQA-RAD for 24 of the 28 evaluated models, sorted by probing score.

Method	PATH-VQA	SLAKE	VQA-RAD	Avg.
Prompting	36.40	63.13	54.37	51.30
MedProb	85.00	83.37	81.37	83.25

Appendix

Table 23: MedGemma-27B accuracy (%), omitted from Table 1 for space.

Explore similar work

Apr 10, 2026

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.

Suyang Xi, Songtao Hu, Yuxiang Lai +4

Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Medical VQA Medical VLMs Latent Visual Reasoning RL for VLMs

Ask PDF

Mar 17, 2026

InViC: Intent-aware Visual Cues for Medical Visual Question Answering

Medical visual question answering (Med-VQA) aims to answer clinically relevant questions grounded in medical images. However, existing multimodal large language models (MLLMs) often exhibit shortcut answering, producing plausible responses by exploiting language priors or dataset biases while insufficiently attending to visual evidence. This behavior undermines clinical reliability, especially when subtle imaging findings are decisive. We propose a lightweight plug-in framework, termed Intent-aware Visual Cues (InViC), to explicitly enhance image-based answer generation in medical VQA. InViC introduces a Cue Tokens Extraction (CTE) module that distills dense visual tokens into a compact set of K question-conditioned cue tokens, which serve as structured visual intermediaries injected into the LLM decoder to promote intent-aligned visual evidence. To discourage bypassing of visual information, we further design a two-stage fine-tuning strategy with a cue-bottleneck attention mask. In Stage I, we employ an attention mask to block the LLM's direct view of raw visual features, thereby funneling all visual evidence through the cue pathway. In Stage II, standard causal attention is restored to train the LLM to jointly exploit the visual and cue tokens. We evaluate InViC on three public Med-VQA benchmarks (VQA-RAD, SLAKE, and ImageCLEF VQA-Med 2019) across multiple representative MLLMs. InViC consistently improves over zero-shot inference and standard LoRA fine-tuning, demonstrating that intent-aware visual cues with bottlenecked training is a practical and effective strategy for improving trustworthy Med-VQA.

Zhisong Wang, Ziyang Chen, Zanting Ye +3

National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University

Medical VQA Medical VLMs Visual Token Compression

Ask PDF

May 10, 2026

LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering

The reasoning gap between large and compact vision-language models (VLMs) limits the deployment of medical AI on portable clinical devices. Compact VLMs of 2--4B parameters can run on resource-constrained hardware but lack the multi-step reasoning capacity needed for interpretable clinical decision support. Existing knowledge distillation methods transfer answers without the reasoning process behind them. Medical visual question answering (VQA) serves as a testbed for this problem, as it requires models to integrate visual evidence with clinical knowledge through structured reasoning chains. We introduce LiteMedCoT-VL, a pipeline that transfers chain-of-thought reasoning from a 235B teacher model to 2B student models through LoRA-based fine-tuning on explanation-enriched training data. All inference is conducted without image captions by default, simulating the clinical scenario in which a physician interprets a medical image directly without an accompanying radiology report. On the PMC-VQA benchmark, LiteMedCoT-VL achieves 64.9% accuracy, exceeding the zero-shot Qwen3-VL-4B baseline of 53.9% by 11.0 percentage points and outperforming all published baselines. This result indicates that a 2B model with reasoning distillation can match or exceed models with twice the parameters. Visual grounding analysis shows that the model relies on image content rather than exploiting textual priors. Our code is publicly available at https://anonymous.4open.science/r/LiteMedCoT-VL.

Runze Ma, Shunbo Jia, Haonan Lyu +2

School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1

CoT Distillation Healthcare Medical VQA Medical VLMs VLM Reasoning Knowledge Distillation

Ask PDF