$Δ$Representation: Geometry Supervised Representation Learning of Phenotypes via Counterfactual Reasoning for Medical VLMs

Hao Wang; Qiwei Zeng; Jinghao Lin; Shuchang Ye; Yuezhe Yang; Yige Peng; Haoyuan Che; Jinman Kim

cs.CVOct 7, 2026

$Δ$ Representation: Geometry Supervised Representation Learning of Phenotypes via Counterfactual Reasoning for Medical VLMs

Authors: Hao Wang, Qiwei Zeng, Jinghao Lin, Shuchang Ye, Yuezhe Yang, Yige Peng, Haoyuan Che, Jinman Kim, +1 more

Organizations: The University of Sydney · Shanghai Jiao Tong University · Northeastern University · Jilin University

Abstract

Medical vision-language models (VLMs) have shown increasing potential for radiological image interpretation. Medical VLMs encode radiological images into visual representations that capture both anatomical and phenotypic information for diagnosis. Existing approaches improve pathological phenotype representations through semantic-guided representation alignment. However, pathological phenotypes arise as lesion-specific visual changes superimposed on underlying normal anatomy. Such semantic alignment approaches fail to model the phenotype-specific increment relative to the corresponding normal anatomical representation. To address this gap, we propose \textbf{ $Δ$ Representation}, a visual phenotype representation learning framework based on counterfactual reasoning for medical VLMs. It comprises \textbf{BaseAnatomy}, a geometry-supervised representation learning module, and \textbf{ $Δ$ Phenotype}, a counterfactual incremental representation learning module. BaseAnatomy provides fine-grained geometric supervision through spatial relationships across and within anatomical structures. $Δ$ Phenotype computes the representation increment between lesion representations and their corresponding normal anatomical representations, and supervises increments associated with the same phenotype to cluster in the representation space. Experiments on \textit{ReXGroundingCT} and \textit{LIDC-IDRI} demonstrate that $Δ$ Representation effectively structures pathological phenotype representations and improves lesion grounding and phenotype characterization accuracy in medical VLMs. Code is available at https://anonymous.4open.science/r/deltarep-CF6D.

Figures & tables

Figure 1: BaseAnatomy structures the anatomical visual representation space, while $\Delta$ Phenotype organizes lesion-specific increments relative to counterfactual normal anatomy.

Figure 2: Overview of $\Delta$ Representation. (a) BaseAnatomy shapes an anatomically structured visual space with ideal geometric supervision that encodes anatomical relationships and spatial continuity. (b) $\Delta$ Phenotype constructs counterfactual normal references for lesion patches and learns phenotype-specific visual increments relative to normal anatomy for discriminative phenotype representation learning.

Models	Scale	ReXGroundingCT		LIDC-IDRI
Models	Scale	Grounding Acc.	Phenotype Acc.	Grounding Acc.	Phenotype Acc.
RadFM	14B	10.60	4.95	18.18	2.48
w/ $\Delta$ Representation	$\sim$ 14B	50.53 $\uparrow$ 39.93	38.87 $\uparrow$ 33.92	62.81 $\uparrow$ 44.63	47.11 $\uparrow$ 44.63
LLaVA-Med	7B	43.11	12.37	2.48	5.79
w/ $\Delta$ Representation	$\sim$ 7B	66.78 $\uparrow$ 23.67	51.94 $\uparrow$ 39.57	47.11 $\uparrow$ 44.63	44.63 $\uparrow$ 38.84
Lingshu	7B	77.39	15.19	23.97	42.15

Table 1: Comparison with pretrained medical VLMs and alignment methods on the VQA task.

Figure 3: Anatomical representation analysis of BaseAnatomy. (A–B) Representation distributions before and after BaseAnatomy training. (C) Spatial continuity across anatomical positions. (D) Cross-scale consistency of anatomical representations.

Figure 4: Counterfactual representation analysis. Comparison between inferred counterfactual representations and their corresponding normal representations.

Lesion Data Scale	ReXGroundingCT (VQA)		ReXGroundingCT (RRG)		LIDC-IDRI (VQA)		LIDC-IDRI (RRG)
Lesion Data Scale	Grounding Acc.	Phenotype Acc.	Grounding Acc.	Phenotype Acc.	Grounding Acc.	Phenotype Acc.	Grounding Acc.	Phenotype Acc.
1%	54.06	25.80	10.95	15.55	43.80	54.55	40.50	50.41
5%	72.44 $\uparrow$ 18.38	34.63 $\uparrow$ 8.83	10.95 $\uparrow$ 0.00	15.55 $\uparrow$ 0.00	44.63 $\uparrow$ 0.83	62.81 $\uparrow$ 8.26	40.50 $\uparrow$ 0.00	51.24 $\uparrow$ 0.83
10%	75.27 $\uparrow$ 2.83	39.93 $\uparrow$ 5.30	22.26 $\uparrow$ 11.31	31.45 $\uparrow$ 15.90	60.33 $\uparrow$ 15.70	68.60 $\uparrow$ 5.79	43.80 $\uparrow$ 3.30	53.72 $\uparrow$ 2.48
25%	78.09 $\uparrow$ 2.82	40.64 $\uparrow$ 0.71	23.32 $\uparrow$ 1.06	43.11 $\uparrow$ 11.66	73.55 $\uparrow$ 13.22	77.69 $\uparrow$ 9.09	44.63 $\uparrow$ 0.83	69.42 $\uparrow$ 15.70
50%	79.15 $\uparrow$ 1.06	43.11 $\uparrow$ 2.47	28.27 $\uparrow$ 4.95	50.18 $\uparrow$ 7.07	76.03 $\uparrow$ 2.48	77.69 $\uparrow$ 0.00	47.11 $\uparrow$ 2.48	75.21 $\uparrow$ 5.79

Table 3: Performance on ReXGroundingCT and LIDC-IDRI with different proportions of lesion data for the VQA and RRG tasks.

Figure 5: Representation learning performance of visual phenotypes in different lesion data scales.

Figure 6: Case study of $\Delta$ Representation, illustrating anatomical projection by BaseAnatomy, phenotype-increment projection by $\Delta$ Phenotype, and their contribution to report generation.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Distribution of benchmark instances by dataset and task.

Models	Scale	Atelectasis Phenotype		Consolidation Phenotype		Ground-glass opacity Phenotype		Pulmonary nodule Phenotype
Models	Scale	VQA Acc.	RRG Acc.	VQA Acc.	RRG Acc.	VQA Acc.	RRG Acc.	VQA Acc.	RRG Acc.
RadFM	14B	7.14	4.29	8.14	4.65	6.82	3.41	8.55	4.27
w/ $\Delta$ Representation	$\sim$ 14B	44.29 $\uparrow$ 37.14	17.14 $\uparrow$ 12.86	48.84 $\uparrow$ 40.70	18.60 $\uparrow$ 13.95	36.93 $\uparrow$ 30.11	13.07 $\uparrow$ 9.66	49.15 $\uparrow$ 40.60	14.10 $\uparrow$ 9.83
LLaVA-Med	7B	28.57	24.29	29.07	29.07	26.70	18.75	27.78	23.93
w/ $\Delta$ Representation	$\sim$ 7B	55.71 $\uparrow$ 27.14	45.71 $\uparrow$ 21.43	72.09 $\uparrow$ 43.02	45.35 $\uparrow$ 16.28	54.55 $\uparrow$ 27.84	30.68 $\uparrow$ 11.93	59.40 $\uparrow$ 31.62	39.74 $\uparrow$ 15.81
Lingshu	7B	51.43	1.43	56.98	2.33	40.34	2.27	45.30	2.56

Appendix

Table 4: Disease-wise VQA and RRG performance on ReXGroundingCT. Values are composite accuracies (%), obtained by pooling the Grounding and Phenotype decisions within each task. Gains are absolute percentage points over the native backbone.

(a) Absolute representation quality
Component	Evaluation cohort	Metric	Score
BaseAnatomy	ReX supervised pool	Rotation Pearson $r$ $\uparrow$	0.9742
		Scaling Pearson $r$ $\uparrow$	0.9751
		Translation Pearson $r$ $\uparrow$	0.9750
$\Delta$ Phenotypes	ReX four-phenotype set, $n=379$	Cosine silhouette $\uparrow$	0.3607
		Nearest-prototype accuracy $\uparrow$	75.46%

Appendix

Table 5: High-dimensional representation diagnostics. Panel (a) reports absolute representation-quality measurements. Panel (b) isolates the effect of exact

4\times 4

spatial pooling. Here,

\Delta=4\times-1\times

and pp denotes percentage points.

Explore similar work

Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs
Oct 7, 2026Hao Wang, Qiwei Zeng, Shuchang Ye +5Medical Vision-Language Models
The University of Sydney · Shanghai Jiao Tong University · Northeastern University
EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models
Jun 4, 2026Hao Wang, Qiwei Zeng, Jinghao Lin +6Medical Vision-Language Models Pathology Foundation Models
School of Computer Science, The University of Sydney, Sydney, NSW, Australia · Jilin University, Changchun, China · Northeastern University, Shenyang, China +1
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2Medical Vision-Language Models Medical Visual Question Answering
The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community +1

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs

Hao Wang, Qiwei Zeng, Shuchang Ye, Jinghao Lin+4

The University of Sydney · Shanghai Jiao Tong University · Northeastern University

Medical Vision-Language Models

Open paper AskPDF arXiv

Medical vision-language models (VLMs) have shown increasing potential for clinical image interpretation. However, these models still struggle to interpret multi-phenotype lesions whose diagnosis requires the joint assessment of multiple pathological phenotypes. Existing vision-language alignment methods produce visual representations that fail to preserve anatomical hierarchies and relationships among phenotypic subclasses. This stems from their reliance on semantic supervision, which lacks geometric constraints to preserve these relationships in the visual embedding space. Moreover, the sparsity of lesion-related anatomical and phenotypic representations makes it difficult for medical VLMs to capture important diagnostic evidence. To address these limitations, we propose \textbf{PureVision}, a geometry-supervised visual representation learning framework for multi-phenotype lesion interpretation in medical VLMs. It combines a geometry-supervised representation learning module, \textbf{PureEyes}, and an anatomy-guided evidence aggregation module, \textbf{PureNeurons}. PureEyes provides geometric supervision through ideal spatial distributions that encode anatomical hierarchies and phenotypic subclass relationships. PureNeurons projects visual representations into the learned latent space, using their positions to selectively aggregate lesion-specific anatomical and phenotypic evidence. Experiments on \textit{LIDC-IDRI}, \textit{CBIS-DDSM}, and \textit{3DReasonKnee} demonstrate that PureVision improves lesion grounding and phenotype characterization in visual question answering and radiology report generation. Code is available at: https://anonymous.4open.science/r/purevision-06C2.

Abstract

Explore similar work

Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs

EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs