ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection

Haotian Gan; Yudong Li; Xintian Li; Jiarui Han; Weidong Tang

cs.LGOct 18, 2026

ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection

Authors: Haotian Gan, Yudong Li, Xintian Li, Jiarui Han, Weidong Tang

Organizations: Tsinghua University · The University of Hong Kong · China Agricultural University

Abstract

Medical vision-language models often exhibit hidden stratification, where high overall performance masks failures on subtle visual findings and overlapping clinical conditions. We trace this failure mode to modality imbalance during fine-tuning, where strong linguistic priors can suppress visually grounded learning and induce plausible but weakly grounded rationales. Existing data selection methods usually score image-text pairs as holistic units, without separating visually necessary samples from shortcut-solvable ones. To address this problem, we propose ViCoR, a counterfactual-residual-based data selection method for Med-VLM fine-tuning. ViCoR estimates visual necessity through image-dependent optimization residuals, calibrates the resulting scores with medical supervision compatibility, and constructs a diversity-aware fine-tuning subset. Across two backbones, four external generalization benchmarks, and one multi-disease robustness benchmark, ViCoR-selected 20% subsets achieve the strongest average performance compared with full-data fine-tuning and selection baselines. Gradient-level analysis and perturbation audits further show that ViCoR enriches image-dependent training signals and strengthens image-level and evidence-region dependence. The code will be available at https://github.com/htgan-ai/ViCoR.

Figures & tables

Figure 1: Motivation and effect of ViCoR. ViCoR selects samples whose optimization signals depend on visual evidence, encouraging more visually grounded medical reasoning.

Figure 2: Overview of ViCoR. The pipeline estimates counterfactual residual gradients, calibrates supervision quality, and applies diversity-aware construction to produce the final fine-tuning subset.

Method	GEMeX-ThinkVG					S-Chain					CXR-CounterFact-Single
Method	OMVQA	GMAI	MBLK	DRVD	Avg.	OMVQA	GMAI	MBLK	DRVD	Avg.	OMVQA	GMAI	MBLK	DRVD	Avg.
Qwen3-VL-4B
Full (100%)	82.47	45.54	64.30	56.75	62.27	73.66	52.35	63.37	60.73	62.53	77.59	50.02	60.11	58.81	61.63
Random	83.89	49.98	52.61	60.01	61.62	64.37	46.37	40.45	50.39	50.40	77.67	50.47	60.07	60.06	62.07
GraNd [ 43 ]	84.93	40.12	62.59	60.30	61.99	73.28	42.82	63.29	59.62	59.75	77.08	44.79	58.38	62.42	60.67
LESS [ 20 ]	84.46	42.58	61.57	59.05	61.92	74.33	43.31	63.76	60.33	60.43	78.29	44.79	61.03	62.07	61.55

Table 1: Cross-benchmark generalization across three training sources and two backbones. Benchmarks: OmniMedVQA (OMVQA), GMAI-MMBench (GMAI), MedBLINK (MBLK), and DrVD-Bench (DRVD). Avg. is the four-benchmark mean. Bold and underline mark best and second-best.

Table 4

Subset	$\mathbf{E^{\Delta}}$	$\mathbf{E^{s}}$	$\mathbf{w^{s}}$	$\mathbf{w^{d}}$	$\mathbf{S^{\mathrm{vis}}}$
Full (100%)	3.716	3.840	0.746	1.307	3.472
Random	3.701	3.826	0.746	1.305	3.456
GraNd	4.739	4.622	0.724	1.338	4.352
LESS	4.696	4.574	0.724	1.333	4.285
Adapt- $\infty$	3.255	3.444	0.756	1.304	3.078
Self-Filter	3.911	4.056	0.743	1.310	3.683

Table 4: Gradient-level statistics of selected subsets.

Figure 3: Visual perturbation audits. (a) Image-level dependence under no-image and wrong-image perturbations. (b) Region-level dependence under Ground Truth and Background masking, with bootstrap 95% CIs for the evidence gap.

Configuration	Single-disease				Multi-disease
Configuration	OMVQA	GMAI	MBLK	DRVD	Acc.	Macro-F1
w/o Residual	82.18	42.94	56.59	58.45	43.26	43.12
w/o Shortcut	81.86	41.10	57.76	59.46	44.31	44.12
w/o Direction	82.18	41.10	58.69	59.86	45.13	44.91
w/o Threshold	80.82	50.40	58.22	59.50	45.71	45.60
w/o Calibration	80.68	50.26	56.98	59.95	42.90	42.72

Table 5: Ablation of ViCoR on GEMeX-ThinkVG with Qwen3-VL-4B.

Figure 4: Data efficiency under different selection ratios on GEMeX-ThinkVG with Qwen3-VL-4B.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Calibration Weights
Perturbation	Spearman	Top 20% overlap	Final overlap
$0.5\times$ all calibration weights	0.9951	94.68%	94.80%
$2\times$ all calibration weights	0.9971	95.88%	96.05%
$2\times$ vocabulary weight	0.9873	90.34%	90.90%
$2\times$ answer-concision weight	0.9989	97.98%	98.05%
$2\times$ label-count weight	0.9996	99.87%	99.84%

Appendix

Table 6: Selection stability under configuration perturbations.

Shortcut control	Spearman $\rho$	Top 20% overlap
Blank image	0.9675	88.31%
Shuffled image	0.9681	88.20%
Wrong image	0.9686	88.54%

Appendix

Table 7: Robustness of ViCoR rankings under alternative shortcut controls.

Configuration	Single-disease				Multi-disease
Configuration	OMVQA	GMAI	MBLK	DRVD	Accuracy	Macro-F1
w/ Loss Gap	83.22	40.98	59.17	60.86	41.02	40.57
w/ Joint Gradient	82.17	43.19	59.70	58.76	41.67	41.63
w/ Residual Norm	82.35	41.96	60.25	59.01	44.92	44.82
Calib.+Div. only	82.02	41.28	58.48	58.66	40.82	41.27
Full ViCoR	85.16	51.56	63.68	61.49	46.84	46.65

Appendix

Table 8: Supplementary proxy-signal and reduced-pipeline comparison on GEMeX-ThinkVG with Qwen3-VL-4B.

Score	Top 20% Mean	AUC	Precision@20%	$\Delta$ z-Score
Decision-token loss	0.603	0.566	38.61	0.258
Joint gradient norm	0.655	0.582	39.47	0.230
GraNd score	0.612	0.576	38.18	0.234
Residual norm $E^{\Delta}$	0.689	0.604	40.24	0.303
ViCoR $S^{\mathrm{vis}}$	0.693	0.609	41.94	0.312

Appendix

Table 9: Pre-fine-tuning visual-dependence audit on GEMeX-ThinkVG with the base Qwen3-VL-4B.

Subset	Vocab. Match	Answer Conc.	Label Ctrl.	Answer Len.	Reason Len.	Label Ent.	Rare Cov.	Top Conc.
Full pool	0.4220	0.5019	0.4846	4.6191	112.1705	2.0530	1.0000	0.3569
Random	0.4254	0.5016	0.4922	4.6037	112.1304	2.0544	1.0000	0.3476
GraNd	0.5334	0.6289	0.5746	3.9505	108.0859	1.6018	0.7500	0.5869
LESS	0.5453	0.6355	0.5841	3.9221	108.3751	1.6146	0.7500	0.5781
Adapt- $\infty$	0.4192	0.3869	0.5212	5.1817	117.0334	2.1770	1.0000	0.2592
Self-Filter	0.3617	0.5496	0.3957	4.4513	109.5884	2.0281	0.7500	0.3896

Appendix

Table 10: Supervision quality and label coverage on GEMeX-ThinkVG.

Subset	Area Ratio	Small Rate	Periph. Dist.	Periph. Rate	Hard Rate	Avg. Boxes
Full pool	0.2475	0.2501	0.1800	0.2500	0.3885	1.0585
Random	0.2464	0.2427	0.1705	0.2482	0.3810	1.0574
GraNd	0.2574	0.2276	0.1631	0.2052	0.3454	1.0269
LESS	0.2556	0.2428	0.1661	0.2116	0.3602	1.0254
Adapt- $\infty$	0.2384	0.2569	0.1897	0.2793	0.4131	1.0501
Self-Filter	0.2386	0.2553	0.1862	0.2666	0.4084	1.0353

Appendix

Table 11: Visual evidence geometry on GEMeX-ThinkVG.

Backbone	Method	GEMeX-ThinkVG	S-Chain	CXR-CounterFact-Single
Qwen3-VL-4B	Full	62.27 [61.13, 63.34]	62.53 [61.39, 63.62]	61.63 [60.56, 62.74]
	Random	61.62 [60.41, 62.75]	50.40 [49.27, 51.50]	62.07 [60.94, 63.25]
	GraNd	61.99 [60.86, 63.12]	59.75 [58.67, 60.77]	60.67 [59.52, 61.82]
	LESS	61.92 [60.80, 62.98]	60.43 [59.28, 61.47]	61.55 [60.48, 62.63]
	Adapt- $\infty$	62.96 [61.76, 64.06]	62.54 [61.44, 63.62]	61.99 [60.85, 63.10]
	Self-Filter	64.42 [63.31, 65.50]	61.43 [60.36, 62.49]	62.29 [61.10, 63.42]

Appendix

Table 12: Cross-benchmark average score with 95% CI.

Backbone	Method	Accuracy [95% CI]	Macro-F1 [95% CI]
Qwen3-VL-4B	Full	44.65 [42.05, 47.46]	44.36 [41.77, 47.08]
	Random	40.95 [38.36, 43.69]	40.69 [38.01, 43.43]
	GraNd	43.11 [40.52, 45.71]	42.88 [40.19, 45.46]
	LESS	40.30 [37.78, 42.97]	39.89 [37.39, 42.60]
	Adapt- $\infty$	42.61 [39.94, 45.06]	42.20 [39.40, 44.67]
	Self-Filter	43.05 [40.45, 45.72]	42.82 [40.10, 45.43]

Appendix

Table 13: CheXbench Multiple Disease Identification with 95% confidence intervals.

Backbone	Training Source	ViCoR minus Full	ViCoR minus Best 20% Baseline
Qwen3-VL-4B	GEMeX	$+3.20$ [ $+2.45$ , $+3.95$ ]	$+1.05$ [ $+0.35$ , $+1.75$ ]
Qwen3-VL-4B	S-Chain	$+0.54$ [ $-0.05$ , $+1.13$ ]	$+0.53$ [ $-0.06$ , $+1.10$ ]
Qwen3-VL-4B	CXR Single	$+3.66$ [ $+2.85$ , $+4.45$ ]	$+3.00$ [ $+2.25$ , $+3.75$ ]
InternVL3.5-4B	GEMeX	$+1.95$ [ $+1.15$ , $+2.75$ ]	$+0.23$ [ $-0.45$ , $+0.90$ ]
InternVL3.5-4B	S-Chain	$+2.24$ [ $+1.45$ , $+3.05$ ]	$+2.77$ [ $+1.95$ , $+3.60$ ]
InternVL3.5-4B	CXR Single	$+8.32$ [ $+7.35$ , $+9.25$ ]	$+5.53$ [ $+4.65$ , $+6.40$ ]

Appendix

Table 14: Paired bootstrap differences between ViCoR and competing methods.

Figure 5: Offline selection time on GEMeX-ThinkVG with Qwen3-VL-4B. Right-side labels show backward passes per sample.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 458 papers

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2VLM Evaluation Counterfactual Evaluation
The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community +1
Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
Apr 10, 2026Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3Medical VLMs VLM Reasoning
Brown University · University of Washington · University of Tübingen +1
Aloe-Vision: Robust Vision-Language Models for Healthcare
Jun 25, 2026Jaume Guasch-Martí, Enrique Lopez-Cuena, Martín Suárez-Fernández +3VLM Evaluation VLM Robustness
Barcelona Supercomputing Center (BSC)

Jul 4, 2026J/K move · Enter open · S save

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj, Ashish Vashist+1

The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community+1

VLM Evaluation Counterfactual Evaluation Contrastive Learning Medical VQA Vision-Language Grounding Medical VLMs

Open paper AskPDF arXiv

Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidence or exploit textual shortcuts. We introduce a counterfactual evaluation framework that decouples visual and textual contributions by substituting input images with controlled surrogates blank, pixel-shuffled, image-absent, and CLIP-retrieved hard negatives and derive a suite of grounding metrics including the Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR). We further introduce CORAL (COntrastive Retrieval-Augmented Learning), a 7B-parameter LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that penalises answer invariance under hard-negative image swaps. On a paired controlled evaluation across four closed-form medical VQA benchmarks (PathVQA, PMC-VQA, SLAKE, VQA-RAD; n=400 total), CORAL improves macro accuracy by +6.7 pp (P(Delta>0)=0.988) and reduces VHR by 8.0 pp (P<0.001) over the matched Qwen2.5-VL-7B base; neither MedVLThinker RL variant achieves a significant gain on either metric. Cross-domain diagnostics further reveal that image substitution costs only <=6.5 pp on medical benchmarks versus 48-61 pp on general-domain tasks, situating the grounding gap that CGO targets. We discuss evaluation limitations openly including train/eval benchmark overlap and underpowered secondary metrics and release our framework, training code, and model weights to support reproducible grounding audits of medical VLMs.

Abstract

Medical VLMs

Explore similar work

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

Aloe-Vision: Robust Vision-Language Models for Healthcare