TubeMLLM: A Foundation Model for Topology Knowledge Exploration in Vessel-like Anatomy

Yaoyu Liu; Minghui Zhang; Xin You; Hanxiao Zhang; Yun Gu

cs.CVMar 10, 2026

TubeMLLM: A Foundation Model for Topology Knowledge Exploration in Vessel-like Anatomy

Authors: Yaoyu Liu, Minghui Zhang, Xin You, Hanxiao Zhang, Yun Gu

Organizations: Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China · Department of Automation, Shanghai Jiao Tong University, Shanghai, China

Abstract

Modeling medical vessel-like anatomy is challenging due to its intricate topology and sensitivity to dataset shifts. Consequently, task-specific models often suffer from topological inconsistencies, including artificial disconnections and spurious merges. Motivated by the promise of multimodal large language models (MLLMs) for zero-shot generalization, we propose TubeMLLM, a unified foundation model that couples structured understanding with controllable generation for medical vessel-like anatomy. By integrating topological priors through explicit natural language prompting and aligning them with visual representations in a shared-attention architecture, TubeMLLM significantly enhances topology-aware perception. Furthermore, we construct TubeMData, a pionner multimodal benchmark comprising comprehensive topology-centric tasks, and introduce an adaptive loss weighting strategy to emphasize topology-critical regions during training. Extensive experiments on fifteen diverse datasets demonstrate our superiority. Quantitatively, TubeMLLM achieves state-of-the-art out-of-distribution performance, substantially reducing global topological discrepancies on color fundus photography (decreasing the $β_{0}$ number error from 37.42 to 8.58 compared to baselines). Notably, TubeMLLM exhibits exceptional zero-shot cross-modality transferring ability on unseen X-ray angiography, achieving a Dice score of 67.50% while significantly reducing the $β_{0}$ error to 1.21. TubeMLLM also maintains robustness against degradations such as blur, noise, and low resolution. Furthermore, in topology-aware understanding tasks, the model achieves 97.38% accuracy in evaluating mask topological quality, significantly outperforming standard vision-language baselines.

Figures & tables

Figure 1: Unified modeling paradigm of the proposed TubeMLLM. (a) Compared with task-specific I2I models and promptable I2I baselines, TubeMLLM unifies text and image tokens in an MLLM with shared-attention and supports both text and image outputs. (b) TubeMLLM enables better topology, stronger zero-shot cross-modality transfer, and more reliable text guidance. (c.1) Example of topology-aware visual understanding task. (c.2) Example of topology-preserving generation task.

Figure 2: Detailed TubeMLLM architecture and TubeMData. (a) TubeMLLM adopts a Mixture-of-Transformers design with coupled generation transformer branch and understanding transformer branch. Adaptive loss weights are derived from error maps. (b) TubeMData CFP training and testing sample distribution across different topology-centric tasks.

Figure 3: Topology-centric tasks. Bold texts highlight the image modalities and topology priors encoded in language prompts. (a) Topology-refinement generation task. (b) Topology-aware understanding task that select the mask with better topology.

Methods	Dice $\uparrow$	clDice $\uparrow$	$\beta_{0}$ Num $\downarrow$	$\beta_{0}$ Mat $\downarrow$
nnUNet [ 7 ]	74.44	78.48	37.42	36.56
nnUNet w/ clDice [ 10 ]	75.19	79.83	25.98	27.62
nnUNet w/ SR [ 11 ]	74.76	79.68	34.49	30.52
nnUNet w/ cbDice [ 12 ]	74.94	79.34	25.46	30.11
nnUNet w/ CAL [ 13 ]	72.63	77.85	56.91	29.79
FR-UNet [ 8 ]	71.59	76.76	40.13	27.94

Table 1: Results on CFP OOD datasets. T-T denotes topology-centric tasks.

Task	Method	Dice $\uparrow$	clDice $\uparrow$	$\beta_{0}$ Num $\downarrow$	$\beta_{0}$ Mat $\downarrow$
Task: Single Step Refinement(SSR)
SSR	nnUNet [ 7 ]	74.44	78.48	37.42	36.56
SSR	w/ TubeMLLM	76.05	80.43	8.49	26.51
Task: Cross Dataset
ZS	nnUNet [ 7 ]	9.07	11.74	238.26	5.29
ZS	TubeMLLM	67.50	69.17	1.21	7.69

Table 2: Comprehensive quantitative results including refinement, cross-dataset, and low quality input settings. ZS denotes zero-shot setting, FS denotes from sratch setting. GB, GN, LR denotes Gaussian Blur, Gaussian Noise, and Low Resolution.

Figure 4: Qualitative results on CFP and XRA OOD test datasets. $\beta_{0}$ Num denotes the $\beta_{0}$ number error. Regions inside red boxes are highlighted to demonstrate the topological accuracy of TubeMLLM.

Figure 5: Qualitative performance on (a) topology-refinement, (b) zero-shot transfer to XRA dataset and (c) degraded input of CFP datasets. Text colored in green demonstrates superior results or improved performance.

Figure 6: Qualitative and quantitative performance on four understanding tasks. (a-d) Qualitative comparisons between BAGEL and our TubeMLLM on (a) Topological-structure Judgement, (b) Topological-structure Counting, (c) Segmentation Quality Judgement, and (d) Better Segmentation Choosing. (e) Quantitative results demonstrating that TubeMLLM significantly outperforms the baseline across all four tasks.

Figure 7: Illustration of TubeMLLM performance on topology-aware understanding tasks. Texts colored in green represent correct answers. Texts colored in red represent incorrect answers. Bold texts highlight the image modalities and topology priors encoded in language prompts.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Example of topology-preserving generation. Bold texts highlight the image modalities and topology priors encoded in language prompts.

Figure A2 : Example of topological-structure judgement. Bold texts highlight the image modalities and topology priors encoded in language prompts.

Figure A3 : Example of topological-structure counting. Bold texts highlight the image modalities and topology priors encoded in language prompts.

Figure A4 : Example of segmentation topology-quality judgement. Bold texts highlight the image modalities and topology priors encoded in language prompts.

Figure A5 : Qualitative results of TubeMLLM zero-shot generalization on XRA. ZS denotes zero-shot setting, FS denotes from sratch setting.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

Topo-R1: Detecting Topological Anomalies via Vision-Language Models
Mar 13, 2026Meilong Xu, Qingqiao Hu, Xiaoling Hu +6Anomaly Detection VLM Evaluation
Stony Brook University · Massachusetts General Hospital and Harvard Medical School · Stanford University +1
Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation
Feb 27, 2026Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang +3Task Transferability Estimation Medical Imaging Foundation Models
Peking University, Beijing, China · National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China +3
6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models
Dec 3, 2025Leon Mayer, Piotr Kalinowski, Caroline Ebersbach +6VLM Evaluation VLM Robustness
German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany. · Medical Faculty, Heidelberg University, Germany. · Faculty of Mathematics and Computer Science, Heidelberg University, Germany. +9

Mar 13, 2026J/K move · Enter open · S save

Topo-R1: Detecting Topological Anomalies via Vision-Language Models

Meilong Xu, Qingqiao Hu, Xiaoling Hu, Shahira Abousamra+5

Stony Brook University · Massachusetts General Hospital and Harvard Medical School · Stanford University+1

Anomaly Detection VLM Evaluation

Open paper AskPDF arXiv

Topology is critical in tubular structures such as blood vessels, nerve fibers, and road networks, where connectivity and loop structure govern downstream functional analysis. Vision-Language Models (VLMs) are promising candidates for understanding such structures, given their reasoning and grounding capabilities. To probe their topological perception, we systematically evaluate leading closed- and open-source VLMs on localizing and classifying four canonical topological anomalies (broken/spurious connections, missing/extra branches) in tubular-network segmentation masks. They perform nearly at random, indicating that topology-aware perception is largely absent from current general-purpose VLMs. As no existing resource pairs segmentation masks with localized anomaly annotations, we build an automated, multi-domain data-curation pipeline that synthesizes diverse topological perturbations with verifiable Betti-number annotations across graduated difficulty levels, yielding the first systematic benchmark with a large-scale training set and held-out in-distribution (ID) and out-of-distribution (OOD) test suites. Building on this benchmark, we introduce Topo-R1, centered on a topology-aware composite reward that jointly scores localization, classification, and skeleton-level structural fidelity. Supervised fine-tuning cold-starts schema-compliant outputs, and Group Relative Policy Optimization (GRPO) then optimizes the policy against this reward, steering predictions toward topologically meaningful structure rather than superficial pixel overlap. Extensive experiments show that Topo-R1 substantially outperforms general-purpose VLMs and matches or exceeds supervised baselines across ID, OOD, and real-segmentation-output protocols, establishing a strong foundation for VLM-based topological understanding of structured visual data.

Abstract

Medical VLMs

Explore similar work

Topo-R1: Detecting Topological Anomalies via Vision-Language Models

Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models