Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation
Authors: Haoyun Chen, Fenghe Tang, Wenxin Ma, Shaohua Kevin Zhou
Organizations: School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou Jiangsu, 215123, China · State Key Laboratory of Precision and Intelligent Chemistry, USTC, Hefei Anhui 230026, China
Universal medical image segmentation seeks to use a single foundational model to handle diverse tasks across multiple imaging modalities. However, existing approaches often rely heavily on manual visual prompts or retrieved reference images, which limits their automation and robustness. In addition, naive joint training across modalities often fails to address large domain shifts. To address these limitations, we propose Concept-to-Pixel (C2P), a novel prompt-free universal segmentation framework. C2P explicitly separates anatomical knowledge into two components: Geometric and Semantic representations. It leverages Multimodal Large Language Models (MLLMs) to distill abstract, high-level medical concepts into learnable Semantic Tokens and introduces explicitly supervised Geometric Tokens to enforce universal physical and structural constraints. These disentangled tokens interact deeply with image features to generate input-specific dynamic kernels for precise mask prediction. Furthermore, we introduce a Geometry-Aware Inference Consensus mechanism, which utilizes the model's predicted geometric constraints to assess prediction reliability and suppress outliers. Extensive experiments and analysis on a unified benchmark comprising eight diverse datasets across seven modalities demonstrate the significant superiority of our jointly trained approach, compared to universe- or single-model approaches. Remarkably, our unified model demonstrates strong generalization, achieving impressive results not only on zero-shot tasks involving unseen cases but also in cross-modal transfers across similar tasks. Code is available at: https://github.com/Yundi218/Concept-to-Pixel
Figures & tables
Figure 1: Comparison of existing medical image segmentation paradigms and our proposed Concept-to-Pixel framework. Unlike existing paradigms that suffer from limited versatility (A) , limited manual prompts (B) , or reliance on reference quality (C) , our prompt-free model (D) , inspired by clinical diagnostic workflows, explicitly decouples modality semantics and physical geometry via [SEM] and [GEO] . The radar chart in (D) Right demonstrates that our zero-reference method ( Concept-to-Pixel ) significantly outperforms SOTA models ( Spider and SR-ICL ), exhibiting impressive in-domain performance and zero-shot generalization even on completely unseen modalities ( e.g . , X-Ray).
Figure 2: Overview of the proposed Concept-to-Pixel (C2P) framework. (1) The model first extracts shallow (E2) and deep (E5) backbone features to infer modality embeddings via the Style-Content Fusion Module (SCFM), which are dynamically injected into the [SEM] . (2) The universal [GEO] and modality-aware [SEM] interact with visual tokens through bidirectional Cross Attention. (3) The aggregated concepts are then fed into the Token-Guided Dynamic Head (TGDH) to generate instance-specific parameters for the final Dynamic Head to predict precise masks. To guarantee explicit decoupling, the tokens are deeply supervised by physical properties ( LGeo ) and MLLM-distilled knowledge ( LSem ).
Figure 3: Workflow of Geometry-Aware Inference Consensus. [GEO] compares regressed geometry with mask-derived geometry to weight each view and aggregate a robust final prediction.
Methods
Path.
Ultrasound
Endo.
CT
Dermo.
OCT
MRI
Avg
EBHI
TNUI
Breast
Polyp
COVID
ISIC2018
AMDSD
BTD
Specialized Models (One model for one task)
UNet (MICCAI’25) [ 41 ]
94.61
85.11
73.00
87.66
78.62
87.76
85.57
80.92
84.16
AURA-Net (ISBI’21) [ 15 ]
95.08
87.55
79.34
90.53
79.28
89.92
85.91
84.75
86.54
UTANet (AAAI’25) [ 34 ]
94.82
86.79
78.78
89.75
79.14
89.32
85.72
83.57
85.99
Swin-UMamba (MICCAI’24) [ 32 ]
95.14
87.86
81.35
91.42
82.07
90.40
85.51
84.80
87.32
Table 1: Comparison of different methods on multiple datasets (Dice %). The abbreviations Path., Endo., and Dermo. correspond to Pathology, Endoscopy, and Dermoscopy, respectively. The best results are highlighted in bold , and the second-best results are underlined .
Figure 4: Qualitative comparison on representative challenging cases. From left to right: Colon Polyp, ISIC2018, TNUI, Breast, BTD, COVID, EBHI, and AMDSD. The red contours indicate the Ground Truth masks, and the yellow contours denote the model predictions.
Dataset
Modality
nnUNetV2 [ 24 ]
Spider [ 64 ]
SR-ICL [ 9 ]
C2P (ours)
(Zero-Ref)
(64-Ref)
(Self-Ref)
(Zero-Ref)
Generalization to Unseen Datasets (Seen Modalities)
TN3K [ 21 ]
Ultrasound
52.61
74.24
70.59
75.14
PH2 [ 37 ]
Dermoscopy
56.39
90.47
90.70
90.89
ACDC [ 5 ]
MRI
29.17
63.26
59.95
71.87
BUSBRA [ 20 ]
Ultrasound
78.83
85.16
80.70
83.39
Table 2: Zero-shot generalization results on unseen datasets and unseen modalities. C2P is evaluated without any support images, new tokens, or MLLM prompts for unseen data, and compared against specialized (nnUNetv2) and universal baselines (Spider with 64-shot references, SR-ICL with iterative refinement). The best results are highlighted in bold , and the second-best results are underlined . C2P achieves competitive performance on unseen datasets and outperforms all baselines on unseen X-ray and Ultrasound modalities, showing its generalization ability.
ID
Geometry Tokens
Semantic Tokens
Dynamic Head
Inference Strategy
Partial Datasets (%)
Avg Dice (%)
AMDSD
COVID
1
-
-
- (Static)
None
82.19
76.27
82.38
2
-
✓
- (Static)
None
84.93
80.27
87.15
3
✓
-
- (Static)
None
84.98
82.66
87.31
4
-
-
✓
None
84.89
82.45
87.19
5
✓
-
✓
None
85.35
81.34
87.29
Table 3: Ablation studies of Concept-to-Pixel. (Static): Linear Segmentation Head. The best results are highlighted in bold .
Figure 5: In-domain representation analysis. (Left) t-SNE of encoder outputs (E5 features). Without C2P, features from different datasets show entanglements. Our C2P effectively separates the features. (Right) t-SNE of [GEO] and [SEM] . Some geometry tokens are clustered, since samples from different dataset may have similar geometry attributes. In contrast, semantic tokens are clearly separated, indicating that each token has successfully encoded discriminative semantic concepts tied to its corresponding domain.
Figure 6: Generalization Capability Analysis. (Left) t-SNE analysis of [SEM] in unseen dataset. Although our model has never seen the datasets, the result shows clear separation among modalities, while different datasets also belong to different clusters. (Right) Visualization of the attention map between [GEO] and visual tokens. Our [GEO] shows an attention pattern that is highly correlated with the ground truth mask.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Experimental Phase
Modality
Dataset
Segmentation Target
In-Domain Phase (Joint Training & In-Domain Evaluation)
OCT
AMD-SD [ 23 ]
Retinal Lesion (Wet AMD)
MRI (T1)
BTD [ 11 , 12 ]
Brain Tumor
Pathology
EBHI-Seg [ 43 ]
Adenocarcinoma
Ultrasound
TNUI-2021 [ 66 ]
Thyroid Nodule
Ultrasound
BUSI [ 1 ]
Breast Lesion
Endoscopy
Colon Polyp (5 sets) [ 4 , 26 , 44 , 47 , 58 ]
Colorectal Polyp
Appendix
Table 4: Summary of medical datasets evaluating in the experiments. The benchmark is carefully carefully divided into in-domain training/evaluation datasets and zero-shot generalization datasets. Zero-shot datasets are categorized by whether their imaging modalities are seen during training.
Figure 7: Example Prompt Template for Breast Ultrasound (BUSI) used for Offline Semantic Knowledge Generation.
Figure 8: Example of the structured JSON response generated by the MLLM. Corresponding to the input prompt in Fig. 7 , the model successfully outputs a comprehensive clinical analysis strictly following the predefined 9-dimension schema.
MLLM Generator
Modalities
Avg
Pathology
Ultrasound
Endoscopy
CT
Dermoscopy
OCT
MRI
EBHI
TNUI
Breast
Polyp
COVID
ISIC2018
AMDSD
BTD
Doubao-Seed-1.6-flash
95.00
88.14
81.12
92.72
82.64
90.09
85.69
85.13
87.57
Grok 4 Fast
94.94
88.35
80.90
92.66
81.99
90.35
86.10
85.34
87.58
Gemini 3.0 Pro
95.08
88.23
81.02
92.33
82.68
90.40
85.96
85.87
87.70
GPT-4o mini
95.02
88.45
80.78
92.48
82.81
90.40
86.05
85.87
87.73
Appendix
Table 5: Ablation study on the choice of different Multimodal Large Language Models (MLLMs) for semantic knowledge generation (Dice %). The best results are highlighted in bold .
Figure 9: Qualitative zero-shot generalization results. From left to right: ACDC, PH2, TN3K, GlaS, BUSBRA, Covidquex, Montgomery, and ISBI EM. Red indicates Ground Truth, and yellow denotes predictions. C2P (top) shows superior robustness against domain shifts and remarkable generalization to unseen macroscopic modalities ( e.g . , X-Ray). Conversely, its learned compact geometric prior causes expected under-segmentation on the filamentous structures in ISBI EM.
Figure 10: Visualization of the Geometric Token Attention Maps on the in-domain validation set. The high-response areas (red/yellow) demonstrate a strict spatial alignment with the corresponding ground truth masks across highly diverse seen modalities (e.g., OCT, CT, MRI, Ultrasound, Pathology), proving the model’s accurate semantic-to-visual grounding capability.
Figure 11: Visualization of the Geometric Token Attention Maps under zero-shot generalization settings. The model successfully localizes target anatomical structures on completely unseen datasets (e.g., Cardiac MRI, unseen Dermoscopy) and entirely novel macroscopic/microscopic modalities (e.g., Chest X-Ray, Electron Microscopy). This highlights the framework’s strong capability to disentangle universal geometric priors from domain-specific imaging physics.
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Harbin Institute of Technology at Shenzhen, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · University Hospital Tübingen, Tübingen, Germany +2