TP-Seg: Task-Prototype Framework for Unified Medical Lesion Segmentation

Jiawei Xu; Qiangqiang Zhou; Dandan Zhu; Yong Chen; Yugen Yi; Xiaoqi Zhao

cs.CVApr 1, 2026

TP-Seg: Task-Prototype Framework for Unified Medical Lesion Segmentation

Authors: Jiawei Xu, Qiangqiang Zhou, Dandan Zhu, Yong Chen, Yugen Yi, Xiaoqi Zhao

Organizations: School of Artificial Intelligence, Jiangxi Normal University · East China Normal University · Yale University

Abstract

Building a unified model with a single set of parameters to efficiently handle diverse types of medical lesion segmentation has become a crucial objective for AI-assisted diagnosis. Existing unified segmentation approaches typically rely on shared encoders across heterogeneous tasks and modalities, which often leads to feature entanglement, gradient interference, and suboptimal lesion discrimination. In this work, we propose TP-Seg, a task-prototype framework for unified medical lesion segmentation. On one hand, the task-conditioned adapter effectively balances shared and task-specific representations through a dual-path expert structure, enabling adaptive feature extraction across diverse medical imaging modalities and lesion types. On the other hand, the prototype-guided task decoder introduces learnable task prototypes as semantic anchors and employs a cross-attention mechanism to achieve fine-grained modeling of task-specific foreground and background semantics. Without bells and whistles, TP-Seg consistently outperforms specialized, general and unified segmentation methods across 8 different medical lesion segmentation tasks covering multiple imaging modalities, demonstrating strong generalization, scalability and clinical applicability.

Figures & tables

Figure 1: Comparison of framework design and average performance between previous unified models [ 8 , 42 , 51 , 46 ] and our TP-Seg. (1) Single-path shared encoder vs. Dual-path routing encoder; (2) Task-agnostic decoder vs. Prototype-guided foreground-background decoder; (3) Average Dice (%): 80.75 vs. Average Dice (%): 86.63; (4) Average mIoU (%): 82.83 vs. Average mIoU (%): 86.44.

Figure 2: Overall architecture of the proposed TP-Seg framework for unified medical lesion segmentation. Each input image, together with its task embedding, is processed by the task-conditioned routing block (TCRB) for feature extraction, followed by the prototype-guided task decoder (PGTD) for task-aware decoding and final lesion prediction.

Figure 3: Illustration of the task-conditioned routing block (TCRB), which integrates a shared router and a task-specific router within each task-conditioned adapter (TCA) to achieve dynamic feature adaptation before the frozen encoder blocks.

Figure 4: Illustration of the prototype-guided task decoder.

Segmentation Task	Dataset	Modality	#Train	#Val
Wet AMD (WA)	AMD-SD [ 23 ]	OCT	2,346	703
Brain Tumor (BT)	BTD [ 12 , 13 ]	MR-T1	2,298	766
Adenocarcinoma (ADC)	EBHI-Seg [ 33 ]	Pathology image	636	159
Thyroid Nodule (TN)	TNUI 2021 [ 55 ]	Ultrasound	966	276
Colon Polyp (CP)	Five datasets [ 3 , 24 , 34 , 36 , 40 ]	Endoscopy image	1,450	798
Lung Infection (LI)	COVID-19 data [ 17 ]	CT	894	385

Table 1: Data partition of the 8 selected medical lesion segmentation tasks, commonly used by state-of-the-art specialized and unified methods.

Specialized and General Models (One model for one task)
Methods	Publication	Wet AMD		Brain Tumor		Adenocarcinoma		Thyroid Nodule		Colon Polyp		Lung Infection		Breast Lesion		Skin Lesion		Average
Methods	Publication	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$
TRSRD-Net [ 2 ]	ICISN’24	-	-	-	-	-	-	85.40	84.65	-	-	-	-	-	-	-	-	-	-
LDNet [ 48 ]	MICCAI’23	-	-	-	-	-	-	-	-	64.25	74.41	-	-	-	-	-	-	-	-
WeakPolyp [ 43 ]	MICCAI’23	-	-	-	-	-	-	-	-	74.90	80.66	-	-	-	-	-	-	-	-
Inf-Net [ 18 ]	TMI’23	-	-	-	-	-	-	-	-	-	-	43.24	52.85	-	-	-	-	-

Table 2: Quantitative comparison with state-of-the-art methods in terms of Dice and mIoU (%).

\uparrow

indicates that larger scores are better. For specialized models, each task is trained independently, while unified models are trained jointly across the 8 MLS tasks. “Average” denotes the mean performance across all 8 tasks. The top two results are highlighted in green and red .

Figure 5: Visual comparison of TP-Seg with other unified models, including Spider [ 51 ] , SAM2-UNet [ 45 ] ,SegGPT [ 42 ] and SR-ICL [ 8 ] , across the 8 medical lesion segmentation tasks.

(A) Each Component
ID	Settings	Wet AMD		Brain Tumor		Adenocarcinoma		Thyroid Nodule		Colon Polyp		Lung Infection		Breast Lesion		Skin Lesion		Average
ID	Settings	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$	Dice $\uparrow$	mIoU $\uparrow$
A1	Baseline	81.29	82.42	66.39	73.96	92.47	82.28	79.83	78.67	63.71	64.58	80.79	81.74	78.42	78.19	86.64	82.39	78.69	78.02
A2	w/o TCA	83.36	84.55	68.78	76.21	93.76	83.22	80.65	79.74	65.85	66.17	81.37	82.44	79.65	79.43	87.21	82.87	80.08	79.32
A3	w/o PGTD	85.37	85.95	74.22	79.17	94.63	84.21	86.83	85.63	85.33	85.51	84.72	86.39	84.17	84.23	89.11	87.77	85.54	84.85
A4	Full (TP-Seg)	86.53	88.50	75.23	80.56	95.05	85.79	87.66	87.53	86.48	86.59	86.47	88.17	85.49	85.66	90.15	88.69	86.63	86.44

Table 3: Ablation studies on the 8 MLS tasks.

Figure 6: Visual comparison of component-wise predictions.

Average	w/o PGTD	w/o PGTD1	w/o PGTD2	w/o PGTD3	TP-Seg
Dice $\uparrow$	85.54	86.16	86.21	86.41	86.63
mIoU $\uparrow$	84.85	85.77	85.80	86.13	86.44

Table 4: Ablation study on the contribution of individual PGTD modules in terms of Dice (%) and mIoU (%), averaged over all 8 lesion segmentation tasks.

Figure 7: Cosine similarity between foreground prototypes across PGTD modules.

Module	Params	Growth / Task	Growth (%)	FLOPs	Growth / Task	Growth (%)
TCA	3.37 M	+3.37 M	1.27%	1.85 G	0.028	0.01%
PGTD Prototypes	0.46 M	+0.001 M	0.17%	$\approx$ 0	$\approx$ 0	$\approx$ 0
PGTD Decoder	12.03 M	0	0	10.31 G	0	0
Full Model (1 Task)	264.84 M	–	-	254.00 G	–	-
Full Model (8 Tasks)	288.46 M	–	-	254.23 G	–	-

Table 5: Comprehensive cost and scalability analysis of the proposed unified framework.

Figure 8: FG–BG separation scores across 8 MLS tasks and their correlations with segmentation performance metrics (Dice (%) and mIoU (%)). The separation scores range from 0 to 2. A score of 0 (cosine similarity = 1) indicates that FG and BG prototypes are nearly identical and difficult to distinguish, while a score of 2 (cosine similarity = -1) denotes perfectly opposite prototypes, representing the ideal case for clear separation.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation
Mar 18, 2026Haoyun Chen, Fenghe Tang, Wenxin Ma +1Medical Image Segmentation Cross-Modality Medical Image Segmentation
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou Jiangsu, 215123, China +1
UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation
Jul 14, 2026Yunzhou Li, Jiesi Hu, Yanwu Yang +5Interactive Medical Image Segmentation Medical Imaging Foundation Models
Harbin Institute of Technology at Shenzhen, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · University Hospital Tübingen, Tübingen, Germany +2
Adapting Segment Anything Model 3 for Concept-Driven Lesion Segmentation in Medical Images: An Experimental Study
Mar 26, 2026Guoping Xu, Jayaram K. Udupa, Yubing Tong +5Medical Image Segmentation Promptable Image Segmentation
The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX 75390, USA · Medical Image Processing Group, Department of Radiology, University of Pennsylvania, Philadelphia, PA 19104, USA · Division of Digestive and Liver Diseases, Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX 75390, USA

Mar 18, 2026J/K move · Enter open · S save

Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation

Haoyun Chen, Fenghe Tang, Wenxin Ma, Shaohua Kevin Zhou

School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou Jiangsu, 215123, China+1

Medical Image Segmentation Cross-Modality Medical Image Segmentation Zero-Shot Medical Image Segmentation Medical VLMs

Open paper AskPDF arXiv

Universal medical image segmentation seeks to use a single foundational model to handle diverse tasks across multiple imaging modalities. However, existing approaches often rely heavily on manual visual prompts or retrieved reference images, which limits their automation and robustness. In addition, naive joint training across modalities often fails to address large domain shifts. To address these limitations, we propose Concept-to-Pixel (C2P), a novel prompt-free universal segmentation framework. C2P explicitly separates anatomical knowledge into two components: Geometric and Semantic representations. It leverages Multimodal Large Language Models (MLLMs) to distill abstract, high-level medical concepts into learnable Semantic Tokens and introduces explicitly supervised Geometric Tokens to enforce universal physical and structural constraints. These disentangled tokens interact deeply with image features to generate input-specific dynamic kernels for precise mask prediction. Furthermore, we introduce a Geometry-Aware Inference Consensus mechanism, which utilizes the model's predicted geometric constraints to assess prediction reliability and suppress outliers. Extensive experiments and analysis on a unified benchmark comprising eight diverse datasets across seven modalities demonstrate the significant superiority of our jointly trained approach, compared to universe- or single-model approaches. Remarkably, our unified model demonstrates strong generalization, achieving impressive results not only on zero-shot tasks involving unseen cases but also in cross-modal transfers across similar tasks. Code is available at: https://github.com/Yundi218/Concept-to-Pixel

Abstract

Medical Image Segmentation

Explore similar work

Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation

UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation

Adapting Segment Anything Model 3 for Concept-Driven Lesion Segmentation in Medical Images: An Experimental Study