Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations
Authors: Jie Gui, Hang Tu, Yunfeng Dou, Wen Sha, Xiuquan Du, Shuo Li
Organizations: Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University, Hefei, China · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China · Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA
Specifying segmentation targets with natural language provides flexible semantic conditioning for medical image segmentation. However, existing methods still struggle to reliably ground high-level textual semantics into accurate pixel-level spatial prompts, especially under low-contrast lesions, ambiguous boundaries, and visually similar anatomical contexts. In this paper, we propose RADAR (Residual Aggregation and Dense Alignment Representations), a parameter-efficient framework for text-guided medical image segmentation. RADAR explicitly decouples and synergizes the medical semantic anchoring capability of BiomedCLIP with the prompt-driven mask decoding ability of MedSAM2, while enabling low-cost task adaptation via LoRA. Specifically, Guided Residual Depth Aggregation adaptively aggregates cross-layer features under semantic guidance to alleviate delayed textual injection and deep semantic dilution. Structured Dense Representation Alignment leverages a frozen BiomedCLIP to construct crop-level target, boundary, and context anchors, improving the local discriminability of dense features and the reliability of automatic prompts. Multi-modal Prompt Generation further converts cross-modal semantic responses into composite text, point, and mask prompts for MedSAM2 decoding. Experiments on three public medical image segmentation datasets show that RADAR achieves competitive performance with only 0.74M trainable parameters, demonstrating its effectiveness in parameter-efficient adaptation, semantic grounding, and dense representation enhancement. The source code is available at https://github.com/JGuiidev/RADAR.
Figures & tables
Figure 1: Challenge and limitations of existing methods, and our solution.
Figure 2: Overview of RADAR. GRDA performs text-conditioned cross-layer routing, SDRA provides training-only dense alignment, and MPG converts cross-modal responses into sparse and dense prompts for MedSAM2. The SDRA teacher is removed at inference.
Method
Type
Trainable Params ↓
FLOPs ↓
QaTa-COV19
MosMedData+
Kvasir
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
U-Net [ 14 ]
Visual
31.04M
41.91G
76.95
66.97
30.75
8.68
40.57
30.41
42.55
14.81
54.82
41.38
54.33
20.06
UNet++ [ 40 ]
Visual
9.16M
26.72G
76.60
66.29
35.00
9.54
65.35
50.78
23.28
5.81
61.74
49.13
55.25
19.20
Swin-UNet [ 41 ]
Visual
27.17M
5.92G
77.47
68.05
26.28
7.55
63.98
50.61
22.45
5.89
83.90
75.98
19.51
6.37
TransUNet [ 15 ]
Visual
105.28M
49.36G
77.38
67.27
30.21
8.01
63.52
49.26
23.83
5.81
86.19
78.50
23.28
6.64
LGMS [ 42 ]
V-L
43.95M
11.18G
83.33
74.33
15.78
4.20
71.83
58.22
13.74
3.29
90.62
84.45
16.74
4.55
Table 1: Quantitative comparison across three datasets. Dice and mIoU are reported in percentages (%), while HD95/ASSD are reported in pixels. Params denotes the number of trainable parameters. “Visual”, “V-L”, and “V-L+S” denote pure vision models, vision-language models, and VLM-SAM cascades, respectively. BioCLIP denotes BiomedCLIP. CLIP-SAM2, BioCLIP-SAM2, and CLIP-MedSAM2 correspond to RADAR backbone variants equipped with the full GRDA+SDRA configuration. Best and second-best results are bolded and underlined , respectively.
Figure 3: Qualitative comparison on representative QaTa-COV19, MosMedData+, and Kvasir samples covering low-contrast lesions, small targets, and ambiguous boundaries. Green, red, and blue indicate true positive, false negative, and false positive pixels, respectively. RADAR shows improved target localization, background suppression, and boundary recovery across these challenging cases.
Variant
MPG
GRDA
Dense Mod.
SDRA
MosMedData+
Kvasir
Text
Point
Mask
Dice (%) ↑
mIoU (%) ↑
Dice (%) ↑
mIoU (%) ↑
MedSAM2-only
–
–
–
–
–
–
69.19
55.33
88.11
81.07
+ Text
✓
–
–
–
–
–
69.86
55.68
88.73
82.17
+ Text + Point
✓
✓
–
–
–
–
71.14
57.47
89.93
83.62
Full MPG
✓
✓
✓
–
–
–
71.47
57.67
90.36
84.29
Full MPG + GRDA
✓
✓
✓
✓
–
–
72.78
59.40
91.35
85.61
Table 2: Main component ablation study on MosMedData+ and Kvasir. Dense Mod. denotes the modification of the last BiomedCLIP ViT block for dense feature extraction.
Backbone
Variant
GRDA
SDRA
MosMedData+
Kvasir
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
CLIP-SAM2
w/o GRDA & SDRA
×
×
71.86
57.90
19.06
5.06
86.11
79.08
21.73
6.25
GRDA only
✓
×
73.84
60.35
17.41
4.36
87.57
81.03
19.62
5.58
SDRA only
×
✓
73.01
59.33
17.91
4.72
87.79
81.21
19.63
5.47
GRDA + SDRA
✓
✓
74.58
61.28
14.71
3.74
88.25
82.26
16.75
4.91
BioCLIP-SAM2
w/o GRDA & SDRA
×
×
71.70
57.85
18.71
5.08
88.91
82.56
21.46
6.01
Table 3: Cross-backbone ablation study of GRDA and SDRA on MosMedData+ and Kvasir. BioCLIP denotes BiomedCLIP.
Variant
Score
GRDA branches
MosMedData+
Kvasir
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Aggregation score design
Base-only
Base
S-I, B-V, B-T
75.13
62.01
13.37
3.28
91.75
86.15
17.60
4.53
Multiplicative
Base × Text
S-I, B-V, B-T
75.35
62.32
13.22
3.37
91.90
86.21
16.69
4.24
Text-only
Text
S-I, B-V, B-T
75.22
62.11
12.83
3.20
92.00
86.44
15.48
3.93
GRDA branch placement
Table 4: Ablation of GRDA score design and branch placement on MosMedData+ and Kvasir. S-I, B-V, and B-T denote the MedSAM2 image, BiomedCLIP visual, and text encoders, respectively. Base and Text denote feature-intrinsic and text-conditioned scores in B-V.
Figure 4: Qualitative visualization of SDRA. The columns show the medical text, BiomedCLIP dense features without SDRA, BiomedCLIP dense features with SDRA, and ground-truth masks.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Modality
Task
Training
Validation
Testing
Total
QaTa-COV19 [ 36 ]
X-ray
Chest infection
5716
1429
2113
9258
MosMedData+ [ 37 , 38 ]
CT
Chest infection
2183
273
273
2729
Kvasir [ 39 ]
Colonoscopy
Polyp segmentation
810
90
100
1000
Appendix
Table 5: Statistics of the dataset splits used for training, validation, and testing.
Text type
QaTa-COV19
MosMedData+
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Original
84.12
75.32
14.27
4.09
75.47
62.43
13.16
3.26
Empty
76.90
66.92
33.90
10.13
67.80
54.40
20.49
5.23
Incomplete
82.01
71.95
20.32
5.67
73.72
60.53
13.77
3.60
Mild Noise
83.57
74.72
17.54
4.85
75.02
61.97
13.40
3.28
Synonym
83.26
74.32
19.95
5.39
74.56
61.41
13.79
3.33
Appendix
Table 6: Robustness to different test-time text conditions on QaTa-COV19 and MosMedData+. The model checkpoint and input images are fixed, and only the textual descriptions are changed. Best and second-best results are bolded and underlined , respectively.
Method
Trainable Params
MosMedData+
Kvasir
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
RADAR-SimMask
0.74M
74.81
61.68
14.05
3.33
91.00
84.98
19.49
4.72
RADAR-Extrema
0.74M
75.19
61.98
13.71
3.57
91.55
85.75
16.84
4.31
RADAR-LMPG
0.91M
75.10
61.97
13.38
3.29
92.24
86.74
16.73
4.03
RADAR
0.74M
75.47
62.43
13.16
3.26
92.15
86.65
14.79
3.91
Appendix
Table 7: Comparison of different prompt-generation strategies on MosMedData+ and Kvasir. Best and second-best results are bolded and underlined , respectively.
Setting
LoRA injection layers
Trainable Params ↓
MosMedData+
Kvasir
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
Dice ↑
mIoU ↑
HD95 ↓
ASSD ↓
All layers
S-I 0–11, B-V/B-T 0–11
1.23M
74.67
61.44
14.29
3.49
92.29
86.92
14.92
3.76
BioCLIP late-6
S-I 0–11, B-V/B-T 6–11
0.86M
73.52
60.19
14.35
3.88
92.03
86.63
15.17
3.90
BioCLIP last-2
S-I 0–11, B-V/B-T 10–11
0.62M
74.78
61.51
13.77
3.53
91.52
85.62
17.45
4.56
MedSAM2 late-6
S-I 6–11, B-V/B-T 0–11
0.69M
74.22
60.94
12.97
3.28
91.98
86.48
16.74
4.32
MedSAM2 late-4
S-I 8–11, B-V/B-T 0–11
0.67M
72.17
58.43
13.20
3.63
91.89
86.86
16.88
4.45
Appendix
Table 8: Ablation study on LoRA injection layers. S-I, B-V, and B-T denote the MedSAM2 image encoder, BiomedCLIP visual encoder, and BiomedCLIP text encoder, respectively. The MedSAM2 decoder LoRA layers are fixed to S-D 0–1 in all settings. Best and second-best results are bolded and underlined , respectively.
Figure 5: Extended visualization of SDRA on QaTa-COV19 and MosMedData+. The columns show the medical text, BiomedCLIP dense features without and with SDRA, text-image similarity maps, generated point prompts, predictions, and ground-truth masks.
Textual descriptions can reduce ambiguity in medical image segmentation by specifying the finding and location to be delineated. Existing text-guided methods mainly improve where image and language features interact but generally retain a single learned update pathway across all image-text pairs. We propose MRSeg, a parameter-efficient framework that uses each image-text pair to route the adaptation of visual and textual features before dense prediction. Frozen ConvNeXt-Tiny and PubMedBERT encoders provide multiscale visual features and clinical text tokens. A joint router uses the deepest visual feature and pooled text to predict a sparse mixture over low-rank adapter bases. The resulting route is shared across separate adapter banks for two visual scales and text, coordinating their adaptation while keeping the feature-specific parameters separate. Region Bridge uses text-derived queries to aggregate dense visual tokens into latent regions, refines these regions through self-attention and text cross-attention, and redistributes the refined information back to the feature maps. Finally, a multiscale decoder combines refined semantic features with shallow image evidence. On QaTa-COV19 and MosMedData+, MRSeg achieves 90.90/83.32 and 81.53/68.82 Dice/mIoU, respectively, with 7.11M trainable parameters and 7.60 GFLOPs. Code: https://github.com/maklachur/MRSeg.
Md Maklachur Rahman, Md Hasan Al Banna, Saraf Anjum +2
Texas A&M University, College Station, TX 77843, USA · Independent Researcher
Medical image segmentation remains challenging due to limited fine-grained annotations, complex anatomical structures, and image degradation from noise, low contrast, or illumination variation. We propose TAMISeg, a text-guided segmentation framework that incorporates clinical language prompts and semantic distillation as auxiliary semantic cues to enhance visual understanding and reduce reliance on pixel-level fine-grained annotations. TAMISeg integrates three core components: a consistency-aware encoder pretrained with strong perturbations for robust feature extraction, a semantic encoder distillation module with supervision from a frozen DINOv3 teacher to enhance semantic discriminability, and a scale-adaptive decoder that segments anatomical structures across different spatial scales. Experiments on the Kvasir-SEG, MosMedData+, and QaTa-COV19 datasets demonstrate that TAMISeg consistently outperforms existing uni-modal and multi-modal methods in both qualitative and quantitative evaluations. Code will be made publicly available at https://github.com/qczggaoqiang/TAMISeg.
Qiang Gao, Yi Wang, Yong Zhang +4
Department of Data Science and AI, Monash University, Melbourne, Australia · College of Computer Science, Chongqing University, Chongqing, China · Chongqing University Central Hospital, School of Medicine, Chongqing, China
Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.
Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3