TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation

Qiang Gao; Yi Wang; Yong Zhang; Yong Li; Yongbing Deng; Lan Du; Cunjian Chen

cs.CVApr 13, 2026

TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation

Authors: Qiang Gao, Yi Wang, Yong Zhang, Yong Li, Yongbing Deng, Lan Du, Cunjian Chen

Organizations: Department of Data Science and AI, Monash University, Melbourne, Australia · College of Computer Science, Chongqing University, Chongqing, China · Chongqing University Central Hospital, School of Medicine, Chongqing, China

Abstract

Medical image segmentation remains challenging due to limited fine-grained annotations, complex anatomical structures, and image degradation from noise, low contrast, or illumination variation. We propose TAMISeg, a text-guided segmentation framework that incorporates clinical language prompts and semantic distillation as auxiliary semantic cues to enhance visual understanding and reduce reliance on pixel-level fine-grained annotations. TAMISeg integrates three core components: a consistency-aware encoder pretrained with strong perturbations for robust feature extraction, a semantic encoder distillation module with supervision from a frozen DINOv3 teacher to enhance semantic discriminability, and a scale-adaptive decoder that segments anatomical structures across different spatial scales. Experiments on the Kvasir-SEG, MosMedData+, and QaTa-COV19 datasets demonstrate that TAMISeg consistently outperforms existing uni-modal and multi-modal methods in both qualitative and quantitative evaluations. Code will be made publicly available at https://github.com/qczggaoqiang/TAMISeg.

Figures & tables

Fig. 1: Overview of the proposed TAMISeg framework. Phase I : Consistency-Aware Encoder. The encoder is trained with strong data perturbations, enforcing prediction consistency between original and augmented images to learn robust representations. Phase II : Cross-modal Fine-tuning. The pretrained encoder outputs four hierarchical features ( $f_{1}$ – $f_{4}$ ), which are distilled by a frozen DINOv3 teacher to enhance semantic quality. Clinical language prompts encoded by a frozen CXR-BERT model [ 1 ] guide these features via cross-modal attention, producing aligned maps ( ${f_{t}}_{1}$ – ${f_{t}}_{4}$ ). A Scale-Adaptive Decoder (SAD) then decodes multi-scale features through parallel branches specialized for varying anatomical sizes to generate the final segmentation.

Fig. 2: Overview of the semantic encoder distillation process, where features extracted by the frozen Consistency-Aware Encoder are refined under the supervision of the frozen DINOv3 teacher to obtain more discriminative and semantically enriched representations.

Fig. 3: Architecture of the Scale-Adaptive Decoder (SAD). In this design, ECA and PSA denote the channel-attention and spatial-attention modules, respectively.

Method	Backbone	Params ↓ (M)	FLOPs ↓ (G)	Text	Kvasir-SEG		MosMedData+		QaTa-COV19
					Dice ↑	mIoU ↑	Dice ↑	mIoU ↑	Dice ↑	mIoU ↑
U-Net [ 16 ]	CNN	31.04	54.75	✗	78.94	70.21	64.60	50.73	79.02	69.46
U-Net++ [ 26 ]	CNN	36.01	25.22	✗	80.35	71.85	71.75	58.39	79.62	70.25
nnU-Net [ 10 ]	CNN	41.25	32.71	✗	81.42	73.10	72.59	60.36	80.42	70.81
TransUNet [ 3 ]	Hybrid	104.93	37.22	✗	82.05	74.12	71.24	58.44	78.63	69.13
Swin-UNet [ 2 ]	Hybrid	62.03	42.37	✗	82.47	74.35	63.29	50.19	78.07	68.34

TABLE I: Comparison with state-of-the-art uni-modal and multi-modal segmentation methods on three benchmark datasets using Dice (%) and mIoU (%). Best results are highlighted in bold . The Text column indicates whether a method is uni-modal (✗) or multi-modal (✓).

Panel of Figure 5 — Fig. 4: Qualitative comparison of segmentation results on three benchmark datasets. From top to bottom: Kvasir-SEG, MosMedData+, and QaTa-COV19. Yellow regions denote overlap between prediction and ground truth, while red and green areas indicate over- and under-segmentation, respectively. TAMISeg achieves more accurate and consistent delineations, with larger yellow overlap regions and clearer boundaries across varying lesion sizes and contrast conditions. Model names are color-coded: purple indicates uni-modal methods, and green represents multi-modal methods.

Variant	CMA	SED	SAD	Dice (%)	mIoU (%)
CAE (Baseline)	✗	✗	✗	88.42	81.03
+ CMA	✓	✗	✗	89.26	82.04
+ CMA + SED	✓	✓	✗	90.28	83.12
Full (CMA+SED+SAD)	✓	✓	✓	91.59	84.11

TABLE II: Ablation study on the Kvasir-SEG dataset. Modules are added sequentially to evaluate their individual contributions.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 399 papers

Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation
Sep 24, 2026Md Maklachur Rahman, Md Hasan Al Banna, Saraf Anjum +2Text-Guided Image Segmentation Medical Image Segmentation
Texas A&M University, College Station, TX 77843, USA · Independent Researcher
Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations
Oct 10, 2026Jie Gui, Hang Tu, Yunfeng Dou +3Text-Guided Image Segmentation Promptable Image Segmentation
Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University, Hefei, China · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China +1
SwinTextUNet: Integrating CLIP-Based Text Guidance into Swin Transformer U-Nets for Medical Image Segmentation
Apr 11, 2026Ashfak Yeafi, Parthaw Goswami, Md Khairul Islam +1Medical Image Segmentation Text-Guided Image Segmentation
Department of Electrical and Electronic Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh · Department of Electronics and Communication Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh · Department of Mathematics and Computer Science Hobart and William Smith Colleges Geneva, NY, USA

Sep 24, 2026J/K move · Enter open · S save

Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation

Md Maklachur Rahman, Md Hasan Al Banna, Saraf Anjum, Assame Arnob+1

Texas A&M University, College Station, TX 77843, USA · Independent Researcher

Text-Guided Image Segmentation Medical Image Segmentation

Open paper AskPDF arXiv

Textual descriptions can reduce ambiguity in medical image segmentation by specifying the finding and location to be delineated. Existing text-guided methods mainly improve where image and language features interact but generally retain a single learned update pathway across all image-text pairs. We propose MRSeg, a parameter-efficient framework that uses each image-text pair to route the adaptation of visual and textual features before dense prediction. Frozen ConvNeXt-Tiny and PubMedBERT encoders provide multiscale visual features and clinical text tokens. A joint router uses the deepest visual feature and pooled text to predict a sparse mixture over low-rank adapter bases. The resulting route is shared across separate adapter banks for two visual scales and text, coordinating their adaptation while keeping the feature-specific parameters separate. Region Bridge uses text-derived queries to aggregate dense visual tokens into latent regions, refines these regions through self-attention and text cross-attention, and redistributes the refined information back to the feature maps. Finally, a multiscale decoder combines refined semantic features with shallow image evidence. On QaTa-COV19 and MosMedData+, MRSeg achieves 90.90/83.32 and 81.53/68.82 Dice/mIoU, respectively, with 7.11M trainable parameters and 7.60 GFLOPs. Code: https://github.com/maklachur/MRSeg.

Abstract

Image Segmentation

Explore similar work

Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation

Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations

SwinTextUNet: Integrating CLIP-Based Text Guidance into Swin Transformer U-Nets for Medical Image Segmentation