When Evidence Vanishes: Distilling Visual-Evidence Dependency in Medical Vision-Language Models

Qizhen Lan; Yiling Ma; Ruiqi Chen; Mengchen Fan; Xi Xiao; Lin Zhao; Liang He; Baocheng Geng

cs.CVOct 16, 2026

When Evidence Vanishes: Distilling Visual-Evidence Dependency in Medical Vision-Language Models

Authors: Qizhen Lan, Yiling Ma, Ruiqi Chen, Mengchen Fan, Xi Xiao, Lin Zhao, Liang He, Baocheng Geng,

Organizations: The University of Texas Health Science Center at Houston · Yale University · University of Michigan · University of Alabama at Birmingham ·

Abstract

Compact medical vision-language models (VLMs) can answer clinical questions correctly without reliably using the image, relying instead on question wording or learned priors. Standard knowledge distillation copies a teacher's answer preference on the original image, but it leaves unchecked whether that preference is conditional on the visual evidence required by the question. We study this missing target as medical visual evidence dependency: support for the correct answer should decrease when the matched evidence is absent. We introduce Evidence-Conditioned Dependency Distillation (ECDD), which pairs each training image with a control matched to the evidence level and distills the teacher's drop in answer margin across the pair. A lower bound objective transfers positive dependency by penalizing only shortfalls, while original-image supervision preserves task accuracy. The same formulation applies when the evidence is a queried clinical region, a surgical instrument, or whole-image visual content. Across three medical VQA settings, ECDD keeps original-image accuracy competitive with distillation and alignment baselines while making answer support more sensitive to evidence removal. Under the DME headline objective, for example, the positive-case margin response to target rather than random masking rises from 0.786 with KD to 2.023 with ECDD, while original-image accuracy differs by 0.11 points. These results suggest that medical VLM distillation should transfer both the teacher's answer preference and its dependence on visual evidence.

Figures & tables

Figure 1: Static answer distillation vs. evidence-dependency distillation. Standard distillation copies the teacher’s answer scores on the original image only, leaving the evidence-removed endpoint unseen. ECDD instead compares the original image $x$ with an evidence-ablated control $A(x)$ and distills the teacher’s dependency gap, measured as the drop in the margin between the gold answer and the hardest negative. Here, $y^{+}$ is the gold answer and $y^{-}$ is the strongest wrong answer.

Figure 2: Overview of ECDD. ECDD pairs each original image with a level-matched control in which the relevant visual evidence is absent. The teacher’s original-to-control margin drop is cached as a dependency target, and the student is trained with original-image supervision plus a one-sided dependency constraint that penalizes only shortfalls in evidence dependence.

Task	Method	Qwen-3B				InternVL-2B
Task	Method	Orig Acc $\uparrow$	Evid. Gap $\uparrow$	Rand. Gap $\to 0$	Specific $\Delta m\uparrow$	Orig Acc $\uparrow$	Evid. Gap $\uparrow$	Rand. Gap $\to 0$	Specific $\Delta m\uparrow$
Lesion presence	Teacher	69.39	3.81	2.18	3.094	81.26	16.99	-1.85	4.396
	Base	46.19	1.85	3.16	0.300	77.67	17.32	2.29	2.750
	SFT	62.96	3.05	-0.87	0.990	71.35	7.52	-4.90	1.330
	KD [ 17 ]	80.50	3.81	-0.11	0.786	77.78	12.85	-5.45	1.410
	DBiLD [ 29 ]	80.28	4.68	-0.11	1.050	78.87	14.60	-5.12	1.830

Table 1: DME localized evidence results. Lesion presence has

n{=}918

test questions; lesion localization has

n{=}300

. Specific

\Delta m

is the target-minus-matched-random/distractor margin response, averaged over positive cases (

\Delta m_{+}

) for presence and all cases for localization. Rand. Gap is ranked by distance to zero. The Qwen-3B presence KD row uses the task-matched recipe in Table 5 . Bold and underline mark the best and second-best trained rows; Teacher and Base are unranked diagnostic references.

Method	Orig Acc $\uparrow$	Rand. Acc $\uparrow$	Evid. Gap $\uparrow$	Evid. $\Delta m_{+}\uparrow$	Evid.-Rand. $\Delta m_{+}\uparrow$
Base	68.00	68.67	-2.00	0.328	0.290
SFT	61.00	45.67	27.33	1.210	0.511
KD [ 17 ]	72.67	51.33	36.00	1.495	0.589
Beta-KD [ 30 ]	65.33	46.00	32.00	1.698	0.704
DBiLD [ 29 ]	68.67	47.00	35.00	1.660	0.676
MedAlign [ 8 ]	72.00	53.33	33.00	1.788	0.685

Table 2: Surgical instrument evidence on RoboSurg-VQA Q8 (

n{=}300

). Evidence denotes EndoVis2017 instrument pixels. Rand. Acc uses a same-area random mask. Evid. Gap, Evid.

\Delta m_{+}

, and Evid.-Rand.

\Delta m_{+}

measure instrument-specific evidence dependence. Bold and underline mark the best and second-best trained rows; Base is an unadapted reference and is not ranked.

Method	Orig Acc $\uparrow$	VEA Acc $\downarrow$	VEA Gap $\uparrow$	$\Delta m_{\mathrm{VEA}}\uparrow$	Ret. $m_{\mathrm{VEA}}\downarrow$	ECE / Brier $\downarrow$
Base	62.17	56.72	5.44	0.120	0.694	8.78 / 47.06
SFT	88.55	82.27	6.28	0.751	0.727	2.14 / 17.07
KD [ 17 ]	86.79	81.17	5.62	0.408	0.694	8.84 / 21.34
Beta-KD [ 30 ]	86.91	81.95	4.97	0.604	0.699	2.10 / 19.37
MedAlign [ 8 ]	84.62	79.30	5.32	0.439	0.663	7.28 / 23.65
DBiLD [ 29 ]	84.00	78.67	5.32	0.450	0.661	6.53 / 24.28

Table 3: PathVQA closed yes/no results (

n{=}3{,}362

). VEA Acc evaluates the visual-evidence-absent control against the original label.

\Delta m_{\mathrm{VEA}}

and retained margin measure confidence-level evidence dependence. ECE and Brier are computed on original images. Bold and underline mark the best and second-best trained rows; Base is an unadapted reference and is not ranked.

Figure 3: Representative evidence-control construction. The question and answer candidates stay fixed while visual evidence is removed or degraded at the relevant level: global for PathVQA, lesion-local for DME, and instrument-local for RoboSurg-VQA.

Method	Full Acc. $\uparrow$	Conflict Acc. $\uparrow$
Question-only prior	85.48	– (defines subset)
SFT	88.55	61.89
KD [ 17 ]	86.79	62.70
DBiLD [ 29 ]	84.00	59.84
MedAlign [ 8 ]	84.62	60.25
A3Tune [ 7 ]	85.22	54.30

Table 4: PathVQA prior-conflict evaluation. The conflict subset is selected only by errors of the frozen question-only model. The ECDD row uses the staged PathVQA recipe in Section C.1 .

Variant	Orig. $\uparrow$	ROI Gap $\uparrow$	Rand. Gap $\to 0$	ROI–Rand. $\Delta m_{+}\uparrow$
Matched KD	80.50	3.81	-0.11	0.786
Constant lower bound	80.39	7.52	-0.54	0.971
Two-sided matching	80.17	15.47	0.54	1.755
Random-region endpoint	79.96	2.40	-0.65	0.747
ECDD (ours)	80.39	16.88	0.87	2.023

Table 5: DME ablation under the headline objective. All rows use the same joint CE+KD task objective, optimization rule, and training budget. Only the dependency target or endpoint changes.

Figure 4: Target-region evidence specificity. On the DME lesion-presence VQA, bars show positive-case gold-margin drop under a target ROI mask or a same-area random mask. Across Qwen-3B and InternVL-2B, ECDD increases the target-ROI response over matched KD while keeping the random-mask response small. Teacher is an untuned diagnostic reference.

Method	Orig. Acc. $\uparrow$	Target $\Delta m_{+}\uparrow$	Random $\Delta m_{+}\to 0$	Specific $\Delta m_{+}\uparrow$
SFT	62.96	0.642	-0.103	0.744
Matched KD	80.50	0.746	-0.032	0.778
ECDD	80.39	1.167	-0.033	1.200

Table 6: Transfer to Telea inpainting on DME. Orig. Acc. repeats the reported original-image evaluation in Table 1 ; the remaining columns evaluate the frozen checkpoints with Telea inpainting. ECDD saw zero-filled ROI controls during training, whereas SFT and KD did not use evidence-control supervision. The ECDD gain over KD in specific response is

+0.423

with an image-cluster bootstrap 95% CI of

[0.378,0.467]

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Training control	Evaluation controls
DME	Target ROI zero mask	Target/random mask, blur, Telea
RoboSurg	Instrument zero mask	Instrument/random mask
PathVQA	Global blank	Blank, degraded/global controls

Appendix

Table A1 : Training endpoints and evaluation controls. DME blur and Telea inpainting are held out from training. Random masks are specificity controls.

Metric	ECDD–DBiLD	95% CI	Test
Orig Acc (pp)	$0.11$	$[-0.87,1.09]$	McNemar $p{=}1.00$
ROI Gap (pp)	$12.20$	$[9.59,14.92]$	bootstrap
ROI–Rand. $\Delta m_{+}$	$0.97$	$[0.91,1.03]$	bootstrap

Appendix

Table A2 : Paired DME bootstrap deltas for the Qwen lesion-presence block. Deltas are ECDD minus DBiLD on the same test examples. Accuracy is used as a preservation metric; the larger positive deltas are on the evidence-dependency diagnostics.

Setting	$\lambda_{\mathrm{ECDD}}$	$\rho$	$\tau$
DME, Qwen-3B	0.02	0.5	8
DME, InternVL-2B	0.05	0.7	8
RoboSurg-VQA, Qwen-3B	0.03	0.5	4
PathVQA, Qwen-3B (continuation)	0.05	1.0	8

Appendix

ECDD settings used for the reported comparison rows.

Variant	Orig. Acc.	ROI Gap	Rand. Gap	ROI–Rand. $\Delta m_{+}$
Matched KD	80.50	3.81	-0.11	0.786
Matched KD + anchor	80.28	4.14	-0.76	0.781

Appendix

Table A3 : Reference anchor control with the headline objective.

Endpoint	Teacher Pred. Yes	KD Pred. Yes	ECDD Pred. Yes	ECDD Acc.	ECDD confusion [YY, YN, NY, NN]	ECDD pos. margin median
Original	58.17	47.71	49.35	80.39	[327, 54, 126, 411]	1.000
Target ROI mask	28.00	22.98	5.45	63.51	[48, 333, 2, 535]	-1.250
Same-area random mask	57.84	46.51	48.04	79.52	[317, 64, 124, 413]	1.000
Blank	0.00	0.00	0.00	58.50	[0, 381, 0, 537]	-0.625
ROI blur	35.51	27.78	16.78	71.13	[135, 246, 19, 518]	-0.250

Appendix

Table A4 : DME endpoint prediction distributions for the headline ECDD model. Percentages are computed over 918 test questions. YY and NN denote correct positive and negative predictions; YN and NY denote the two error directions.

Control	Teacher	SFT	Matched KD	ECDD
Target ROI mask	3.254	0.928	0.835	2.072
Same-area random mask	0.161	-0.064	0.049	0.049
Held-out ROI blur	2.164	0.550	0.692	1.215

Appendix

Table A5 : Positive-case gold-margin drop under DME controls. Training uses the target ROI mask; ROI blur and Telea inpainting are held out for evaluation.

Method	Orig. Acc.	Target $\Delta m_{+}$	Random $\Delta m_{+}$	Target–Random $\Delta m_{+}$
SFT	62.96	0.642	-0.103	0.744
Matched KD	80.50	0.746	-0.032	0.778
ECDD	80.39	1.167	-0.033	1.200

Appendix

Table A6 : Transfer to held-out content-aware ROI inpainting on DME. Orig. Acc. repeats the reported original-image evaluation in Table 1 ; the margin responses use the frozen checkpoints under Telea. The random control uses the same inpainting operator and a shifted mask with the same geometry and area, selected to minimize target overlap.

Dataset	Method	Orig. Acc.	Dependency response	ECDD–KD gain
DME	SFT	$63.91\pm 1.12$	$1.176\pm 0.218$	–
	KD	$80.46\pm 0.82$	$0.740\pm 0.051$	–
	ECDD	$80.46\pm 0.91$	$1.954\pm 0.101$	$1.213\pm 0.054$
PathVQA	SFT	$88.65\pm 0.07$	$0.848\pm 0.122$	–
	KD	$86.87\pm 0.21$	$0.444\pm 0.065$	–
	ECDD	$89.81\pm 0.40$	$4.105\pm 0.297$	$3.661\pm 0.241$

Appendix

Table A7 : Three-seed robustness. Reported gains are mean

\pm

sample standard deviation of the seed-wise paired ECDD–KD differences. The dependency-response gains are

+1.213\pm 0.054

+3.661\pm 0.241

, and

+0.125\pm 0.026

on DME, PathVQA, and RoboSurg-VQA, respectively. The corresponding original-accuracy differences are

0.00\pm 0.29

+2.93\pm 0.51

, and

+0.33\pm 0.67

points.

Figure A1 : Single-factor sensitivity for the DME training configuration. Blue reports target-ROI minus random-region positive-case margin response; red reports original-image accuracy. Diamonds mark the headline setting.

Type	$n$	SFT Orig.	ECDD Orig.	SFT Q-only gap	ECDD Q-only gap
Presence	1,561	85.78	87.00	8.71	8.90
Morphology/histology	1,386	94.88	95.17	4.04	24.03
Other	415	77.83	78.80	10.84	11.81

Appendix

Table A8 : PathVQA question-type behavior. The Q-only gap subtracts the accuracy of the same trained VLM with its image input omitted from its original-image accuracy. This endpoint differs from both the blank-image VEA evaluation and the separately trained question-only model used to define the prior-conflict subset.

Figure A2 : PathVQA multi-control VEA diagnostic (

n{=}3{,}362

). Lower retained margin means less original answer support survives the evidence-absent or degraded control. ECDD denotes the staged PathVQA model.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 455 papers

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
Sep 30, 2026Yuanhao Sun, Huawei Ji, Jiaxin Ding +2Counterfactual Evaluation of VLMs VLM Distillation
Shanghai Jiao Tong University, Shanghai, China
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
May 10, 2026Runze Ma, Shunbo Jia, Haonan Lyu +2CoT Distillation Healthcare
School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1

Apr 10, 2026J/K move · Enter open · S save

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan+3

Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Medical VQA Medical VLMs Latent Visual Reasoning RL for VLMs

Open paper AskPDF arXiv

Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.

Abstract

Medical VLMs

Explore similar work

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models

LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering