VTRL-Med: Progressive Visual Refinement with Topology-Guided Regularization for Medical Image Clustering

Guangyu Meng; Pengfei Gu; Peixian Liang; John P. Lalor; Erin Wolf Chambers; Danny Z. Chen

cs.CVOct 23, 2026

VTRL-Med: Progressive Visual Refinement with Topology-Guided Regularization for Medical Image Clustering

Authors: Guangyu Meng, Pengfei Gu, Peixian Liang, John P. Lalor, Erin Wolf Chambers, Danny Z. Chen

Organizations: Dept. of Computer Science and Engineering, University of Notre Dame · Dept. of Computer Science, The University of Texas Rio Grande Valley · Dept. of IT, Analytics, and Operations, University of Notre Dame

Abstract

Deep clustering offers a promising approach for analyzing large unlabeled medical image datasets, yet prevailing methods still face two limitations: they may neglect class-agnostic structural priors that capture object boundaries and regions, and they do not explicitly encode topological patterns (e.g., connectivity and holes). These sources of information play distinct and complementary roles: visual-structural features provide the primary discriminative representation for clustering, while topological features provide additional global geometric cues for residual visual ambiguities. We introduce VTRL-Med, a progressive visual-structural learning framework with topology-guided regularization for medical image clustering. VTRL-Med first adapts the Segment Anything Model (SAM) to extract image features that retain structural cues inherited from mask-supervised pretraining. It then generates reliable pseudo-labels through a two-step filtering process that combines prototype-based selection with k-nearest neighbor consistency validation. Finally, it progressively refines the visual branch while an auxiliary topology branch, constructed from persistent homology (PH) descriptors, provides additional geometric regularization during training. Across 13 medical image benchmarks, VTRL-Med achieves state-of-the-art clustering performance, improving average accuracy by 4.48% over existing methods. On three representative datasets, topology-guided regularization further improves accuracy by 1.3--4.3% over visual refinement alone.

Figures & tables

Figure 1: Motivation for VTRL-Med. (a) Grad-CAM visualizations [ 46 ] show that SimCLR and DINOv3 may under-emphasize region and boundary cues, whereas the adapted SAM encoder attends more strongly to the lesion structure. (b) A Melanocytic Nevus (NV) and a Benign Keratosis-like lesion (BKL) exhibit similar appearance (SSIM [ 54 ] $=0.874$ ; visual-feature cosine similarity $=0.952$ ), but distinct persistent homology signatures (Wasserstein distance [ 11 ] $=0.417$ ). The visual-only model misclusters the NV sample, whereas the topology-guided visual model predicts it correctly. Here, H 0 and H 1 summarize connected components and enclosed structures, respectively.

Figure 2: Illustrating the three-stage VTRL-Med framework. (a) Stage 1: Adapting the SAM image encoder to extract domain-specific visual features and structure-sensitive cues inherited from mask-supervised pretraining. (b) Stage 2: Training the clustering head (ClusHead) with confidence-and-consistency filtering to produce reliable pseudo-labels. (c) Stage 3: Progressively refining the visual branch with auxiliary topology guidance through bootstrap (Phase A) and mutual refinement (Phase B). $K$ denotes the number of clusters.

Dataset	Modality	Classes	Train / Test
PathMNIST	Colon Pathology	9	89,996 / 7,180
DermaMNIST	Dermatoscopy	7	7,007 / 2,005
OCTMNIST	Retinal OCT	4	97,477 / 1,000
PneumoniaMNIST	Chest X-ray	2	4,708 / 624
RetinaMNIST	Fundus Photography	5	1,080 / 400
BreastMNIST	Breast Ultrasound	2	546 / 156

Table 1: Dataset statistics of the 13 medical image benchmarks. MedMNIST v2 images [ 58 ] are natively

224\times 224

; ISIC2019 and Kvasir images are resized to

224\times 224

	PathMNIST			DermaMNIST			OCTMNIST			PneumoniaMNIST			RetinaMNIST			BreastMNIST			BloodMNIST
Method	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI
SimCLR	77.21	83.51	75.66	26.48	16.01	5.73	76.10	53.23	50.94	73.88	20.42	22.68	31.50	8.30	5.03	64.74	10.37	8.20	69.22	70.70	60.99
MoCo-v3	77.53	82.36	74.08	36.06	14.29	9.72	87.00	68.71	70.03	75.64	26.04	26.14	27.75	5.36	0.56	57.05	1.73	4.55	89.39	84.84	82.53
BYOL	77.56	83.13	74.80	56.94	13.07	16.05	88.50	71.18	72.85	86.70	43.32	53.74	35.75	13.76	5.12	54.49	2.99	1.28	86.73	81.58	79.69
DINOv3	76.42	81.23	77.24	59.63	30.54	34.09	85.50	80.86	80.83	91.99	58.84	70.29	42.50	15.45	10.53	60.00	16.23	14.67	90.48	88.18	80.83
SPICE	93.72	88.35	90.13	70.62	36.09	44.29	78.20	64.48	56.85	88.75	62.18	72.38	43.75	9.39	16.08	76.92	11.51	23.04	88.52	82.19	83.14

Table 2: Clustering performances of different methods on 13 medical image datasets. The best and second-best label-free results are bold and underlined . MedMamba is a SOTA supervised method served as an upper bound; all other methods are label-free. Standard deviations and statistical tests are in Supplementary 7 and 8.

	PathMNIST			ISIC2019			Kvasir
Configuration	ACC/NMI/ARI			ACC/NMI/ARI			ACC/NMI/ARI
w/o Stage 1	56.91 /	36.18 /	40.23	36.76 /	17.59 /	20.53	47.37 /	43.04 /	28.91
w/o Stage 2	92.58 /	88.52 /	89.51	55.03 /	22.37 /	35.04	69.29 /	50.31 /	48.18
w/o Stage 3	63.69 /	58.02 /	47.88	51.71 /	20.40 /	19.23	68.15 /	44.41 /	31.09
Raw pixels	45.11 /	45.37 /	30.62	23.35 /	6.93 /	5.29	45.78 /	45.35 /	29.22
ResNet-50 (ImageNet)	73.82 /	74.29 /	67.24	22.66 /	5.41 /	3.67	55.28 /	59.56 /	43.89

Table 3: Component analysis and ablation on 3 datasets. Rows 1–3: stage necessity; rows 4–7: Stage 1 design; rows 8–15: Stage 2 design; rows 16–21: Stage 3 design and full model. † SAM adapted features with pseudo-labeling only (no Stage 3 refinement).

Figure 3: t-SNE visualizations of embeddings for (a) PathMNIST and (b) BloodMNIST. Samples are color-coded: Correct (blue) = both branches predict correctly, Wrong (orange) = both predict incorrectly, Rescued (green stars) = topology branch corrects the visual branch, Degraded (red triangles) = topology branch overrides a correct visual prediction. The right columns show Grad-CAM attention of the visual branch alone (Vis) vs. with topology guidance (Vis+Topo) for representative rescued cases.

Method	Train	Params	FLOPs	Tput.	ACC
	(h) $\downarrow$	(M) $\downarrow$	(G) $\downarrow$	(img/s) $\uparrow$	(%) $\uparrow$
DINOv3-B [ 47 ]	33.41	85.80	17.58	3522.33	60.88
SPICE [ 37 ]	31.41	46.71	7.84	4090.40	77.03
CDC [ 22 ]	40.28	24.56	4.11	6217.12	70.33
MedTrans [ 21 ]	26.82	94.51	4.61	5252.68	65.10
DeepMCAT [ 23 ]	35.43	14.75	15.37	3487.67	67.93

Table 4: Computational costs and clustering accuracy (avg. over 13 datasets), measured on TissueMNIST, with 300 epochs and batch size 256. Tput. = Throughput.

Table 5: VTRL-Med cost breakdown. (a) Training (300 epochs). (b) Inference. Both measured on TissueMNIST, with batch size 256. Tput. = Throughput; PH = Persistent Homology.

Deep Clustering

Deep clustering learns representations and cluster assignments jointly to discover structure in complex, high-dimensional data without labels.

See all 25 papers

TopoCL: Topological Contrastive Learning for Medical Imaging
Mar 15, 2026Guangyu Meng, Pengfei Gu, Peixian Liang +3Medical Image Representation Learning Contrastive Learning
Dept. of Computer Science and Engineering, University of Notre Dame · The University of Texas Rio Grande Valley · Dept. of IT, Analytics, and Operations, University of Notre Dame
Med-DisSeg: Dispersion-Driven Representation Learning for Fine-Grained Medical Image Segmentation
May 14, 2026Zhiquan Chen, Haitao Wang, Guowei Zou +1Image Segmentation Metric Learning
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China
Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations
Oct 10, 2026Jie Gui, Hang Tu, Yunfeng Dou +3Text-Guided Image Segmentation Promptable Image Segmentation
Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University, Hefei, China · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China +1

Mar 15, 2026J/K move · Enter open · S save

TopoCL: Topological Contrastive Learning for Medical Imaging

Guangyu Meng, Pengfei Gu, Peixian Liang, John P. Lalor+2

Dept. of Computer Science and Engineering, University of Notre Dame · The University of Texas Rio Grande Valley · Dept. of IT, Analytics, and Operations, University of Notre Dame

Medical Image Representation Learning Contrastive Learning Topological Representation Learning Medical Image Classification

Open paper AskPDF arXiv

Contrastive learning (CL) has become a powerful approach for learning representations from unlabeled images. However, existing CL methods focus predominantly on visual appearance features while neglecting topological characteristics (e.g., connectivity patterns, boundary configurations, cavity formations) that provide valuable cues for medical image analysis. To address this limitation, we propose a new topological CL framework (TopoCL) that explicitly exploits topological structures during contrastive learning for medical imaging. Specifically, we first introduce topology-aware augmentations that control topological perturbations using a relative bottleneck distance between persistence diagrams, preserving medically relevant topological properties while enabling controlled structural variations. We then design a Hierarchical Topology Encoder that captures topological features through self-attention and cross-attention mechanisms. Finally, we develop an adaptive mixture-of-experts (MoE) module to dynamically integrate visual and topological representations. TopoCL can be seamlessly integrated with existing CL methods. We evaluate TopoCL on five representative CL methods (SimCLR, MoCo-v3, BYOL, DINO, and Barlow Twins) and five diverse medical image classification datasets. The experimental results show that TopoCL achieves consistent improvements: an average gain of +3.26% in linear probe classification accuracy with strong statistical significance, verifying its effectiveness.

Abstract

Deep Clustering

Explore similar work

TopoCL: Topological Contrastive Learning for Medical Imaging

Med-DisSeg: Dispersion-Driven Representation Learning for Fine-Grained Medical Image Segmentation

Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations