Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations

Jie Gui; Hang Tu; Yunfeng Dou; Wen Sha; Xiuquan Du; Shuo Li

cs.CVOct 10, 2026

Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations

Authors: Jie Gui, Hang Tu, Yunfeng Dou, Wen Sha, Xiuquan Du, Shuo Li

Organizations: Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University, Hefei, China · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China · Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA

Abstract

Specifying segmentation targets with natural language provides flexible semantic conditioning for medical image segmentation. However, existing methods still struggle to reliably ground high-level textual semantics into accurate pixel-level spatial prompts, especially under low-contrast lesions, ambiguous boundaries, and visually similar anatomical contexts. In this paper, we propose RADAR (Residual Aggregation and Dense Alignment Representations), a parameter-efficient framework for text-guided medical image segmentation. RADAR explicitly decouples and synergizes the medical semantic anchoring capability of BiomedCLIP with the prompt-driven mask decoding ability of MedSAM2, while enabling low-cost task adaptation via LoRA. Specifically, Guided Residual Depth Aggregation adaptively aggregates cross-layer features under semantic guidance to alleviate delayed textual injection and deep semantic dilution. Structured Dense Representation Alignment leverages a frozen BiomedCLIP to construct crop-level target, boundary, and context anchors, improving the local discriminability of dense features and the reliability of automatic prompts. Multi-modal Prompt Generation further converts cross-modal semantic responses into composite text, point, and mask prompts for MedSAM2 decoding. Experiments on three public medical image segmentation datasets show that RADAR achieves competitive performance with only 0.74M trainable parameters, demonstrating its effectiveness in parameter-efficient adaptation, semantic grounding, and dense representation enhancement. The source code is available at https://github.com/JGuiidev/RADAR.

Figures & tables

Figure 1: Challenge and limitations of existing methods, and our solution.

Figure 2: Overview of RADAR. GRDA performs text-conditioned cross-layer routing, SDRA provides training-only dense alignment, and MPG converts cross-modal responses into sparse and dense prompts for MedSAM2. The SDRA teacher is removed at inference.

Method	Type	Trainable Params $\downarrow$	FLOPs $\downarrow$	QaTa-COV19				MosMedData+				Kvasir
Method	Type	Trainable Params $\downarrow$	FLOPs $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$
U-Net [ 14 ]	Visual	31.04M	41.91G	76.95	66.97	30.75	8.68	40.57	30.41	42.55	14.81	54.82	41.38	54.33	20.06
UNet++ [ 40 ]	Visual	9.16M	26.72G	76.60	66.29	35.00	9.54	65.35	50.78	23.28	5.81	61.74	49.13	55.25	19.20
Swin-UNet [ 41 ]	Visual	27.17M	5.92G	77.47	68.05	26.28	7.55	63.98	50.61	22.45	5.89	83.90	75.98	19.51	6.37
TransUNet [ 15 ]	Visual	105.28M	49.36G	77.38	67.27	30.21	8.01	63.52	49.26	23.83	5.81	86.19	78.50	23.28	6.64
LGMS [ 42 ]	V-L	43.95M	11.18G	83.33	74.33	15.78	4.20	71.83	58.22	13.74	3.29	90.62	84.45	16.74	4.55

Table 1: Quantitative comparison across three datasets. Dice and mIoU are reported in percentages (%), while HD95/ASSD are reported in pixels. Params denotes the number of trainable parameters. “Visual”, “V-L”, and “V-L+S” denote pure vision models, vision-language models, and VLM-SAM cascades, respectively. BioCLIP denotes BiomedCLIP. CLIP-SAM2, BioCLIP-SAM2, and CLIP-MedSAM2 correspond to RADAR backbone variants equipped with the full GRDA+SDRA configuration. Best and second-best results are bolded and underlined , respectively.

Figure 3: Qualitative comparison on representative QaTa-COV19, MosMedData+, and Kvasir samples covering low-contrast lesions, small targets, and ambiguous boundaries. Green, red, and blue indicate true positive, false negative, and false positive pixels, respectively. RADAR shows improved target localization, background suppression, and boundary recovery across these challenging cases.

Variant	MPG			GRDA	Dense Mod.	SDRA	MosMedData+		Kvasir
Variant	Text	Point	Mask	GRDA	Dense Mod.	SDRA	Dice (%) $\uparrow$	mIoU (%) $\uparrow$	Dice (%) $\uparrow$	mIoU (%) $\uparrow$
MedSAM2-only	–	–	–	–	–	–	69.19	55.33	88.11	81.07
+ Text	$\checkmark$	–	–	–	–	–	69.86	55.68	88.73	82.17
+ Text + Point	$\checkmark$	$\checkmark$	–	–	–	–	71.14	57.47	89.93	83.62
Full MPG	$\checkmark$	$\checkmark$	$\checkmark$	–	–	–	71.47	57.67	90.36	84.29
Full MPG + GRDA	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	–	72.78	59.40	91.35	85.61

Table 2: Main component ablation study on MosMedData+ and Kvasir. Dense Mod. denotes the modification of the last BiomedCLIP ViT block for dense feature extraction.

Backbone	Variant	GRDA	SDRA	MosMedData+				Kvasir
Backbone	Variant	GRDA	SDRA	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$
CLIP-SAM2	w/o GRDA & SDRA	$\times$	$\times$	71.86	57.90	19.06	5.06	86.11	79.08	21.73	6.25
	GRDA only	$\checkmark$	$\times$	73.84	60.35	17.41	4.36	87.57	81.03	19.62	5.58
	SDRA only	$\times$	$\checkmark$	73.01	59.33	17.91	4.72	87.79	81.21	19.63	5.47
	GRDA + SDRA	$\checkmark$	$\checkmark$	74.58	61.28	14.71	3.74	88.25	82.26	16.75	4.91
BioCLIP-SAM2	w/o GRDA & SDRA	$\times$	$\times$	71.70	57.85	18.71	5.08	88.91	82.56	21.46	6.01

Table 3: Cross-backbone ablation study of GRDA and SDRA on MosMedData+ and Kvasir. BioCLIP denotes BiomedCLIP.

Variant	Score	GRDA branches	MosMedData+				Kvasir
Variant	Score	GRDA branches	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$
Aggregation score design
Base-only	Base	S-I, B-V, B-T	75.13	62.01	13.37	3.28	91.75	86.15	17.60	4.53
Multiplicative	Base $\times$ Text	S-I, B-V, B-T	75.35	62.32	13.22	3.37	91.90	86.21	16.69	4.24
Text-only	Text	S-I, B-V, B-T	75.22	62.11	12.83	3.20	92.00	86.44	15.48	3.93
GRDA branch placement

Table 4: Ablation of GRDA score design and branch placement on MosMedData+ and Kvasir. S-I, B-V, and B-T denote the MedSAM2 image, BiomedCLIP visual, and text encoders, respectively. Base and Text denote feature-intrinsic and text-conditioned scores in B-V.

Figure 4: Qualitative visualization of SDRA. The columns show the medical text, BiomedCLIP dense features without SDRA, BiomedCLIP dense features with SDRA, and ground-truth masks.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Modality	Task	Training	Validation	Testing	Total
QaTa-COV19 [ 36 ]	X-ray	Chest infection	5716	1429	2113	9258
MosMedData+ [ 37 , 38 ]	CT	Chest infection	2183	273	273	2729
Kvasir [ 39 ]	Colonoscopy	Polyp segmentation	810	90	100	1000

Appendix

Table 5: Statistics of the dataset splits used for training, validation, and testing.

Text type	QaTa-COV19				MosMedData+
Text type	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$
Original	84.12	75.32	14.27	4.09	75.47	62.43	13.16	3.26
Empty	76.90	66.92	33.90	10.13	67.80	54.40	20.49	5.23
Incomplete	82.01	71.95	20.32	5.67	73.72	60.53	13.77	3.60
Mild Noise	83.57	74.72	17.54	4.85	75.02	61.97	13.40	3.28
Synonym	83.26	74.32	19.95	5.39	74.56	61.41	13.79	3.33

Appendix

Table 6: Robustness to different test-time text conditions on QaTa-COV19 and MosMedData+. The model checkpoint and input images are fixed, and only the textual descriptions are changed. Best and second-best results are bolded and underlined , respectively.

Method	Trainable Params	MosMedData+				Kvasir
Method	Trainable Params	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$
RADAR-SimMask	0.74M	74.81	61.68	14.05	3.33	91.00	84.98	19.49	4.72
RADAR-Extrema	0.74M	75.19	61.98	13.71	3.57	91.55	85.75	16.84	4.31
RADAR-LMPG	0.91M	75.10	61.97	13.38	3.29	92.24	86.74	16.73	4.03
RADAR	0.74M	75.47	62.43	13.16	3.26	92.15	86.65	14.79	3.91

Appendix

Table 7: Comparison of different prompt-generation strategies on MosMedData+ and Kvasir. Best and second-best results are bolded and underlined , respectively.

Setting	LoRA injection layers	Trainable Params $\downarrow$	MosMedData+				Kvasir
Setting	LoRA injection layers	Trainable Params $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$	Dice $\uparrow$	mIoU $\uparrow$	HD95 $\downarrow$	ASSD $\downarrow$
All layers	S-I 0–11, B-V/B-T 0–11	1.23M	74.67	61.44	14.29	3.49	92.29	86.92	14.92	3.76
BioCLIP late-6	S-I 0–11, B-V/B-T 6–11	0.86M	73.52	60.19	14.35	3.88	92.03	86.63	15.17	3.90
BioCLIP last-2	S-I 0–11, B-V/B-T 10–11	0.62M	74.78	61.51	13.77	3.53	91.52	85.62	17.45	4.56
MedSAM2 late-6	S-I 6–11, B-V/B-T 0–11	0.69M	74.22	60.94	12.97	3.28	91.98	86.48	16.74	4.32
MedSAM2 late-4	S-I 8–11, B-V/B-T 0–11	0.67M	72.17	58.43	13.20	3.63	91.89	86.86	16.88	4.45

Appendix

Table 8: Ablation study on LoRA injection layers. S-I, B-V, and B-T denote the MedSAM2 image encoder, BiomedCLIP visual encoder, and BiomedCLIP text encoder, respectively. The MedSAM2 decoder LoRA layers are fixed to S-D 0–1 in all settings. Best and second-best results are bolded and underlined , respectively.

Figure 5: Extended visualization of SDRA on QaTa-COV19 and MosMedData+. The columns show the medical text, BiomedCLIP dense features without and with SDRA, text-image similarity maps, generated point prompts, predictions, and ground-truth masks.

Text-Guided Image Segmentation

Text-guided image segmentation maps natural-language descriptions or concepts to pixel-level regions in images.

See all 94 papers

Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation
Sep 24, 2026Md Maklachur Rahman, Md Hasan Al Banna, Saraf Anjum +2Text-Guided Image Segmentation Medical Image Segmentation
Texas A&M University, College Station, TX 77843, USA · Independent Researcher
TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation
Apr 13, 2026Qiang Gao, Yi Wang, Yong Zhang +4Image Segmentation Medical Image Segmentation
Department of Data Science and AI, Monash University, Melbourne, Australia · College of Computer Science, Chongqing University, Chongqing, China · Chongqing University Central Hospital, School of Medicine, Chongqing, China
MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Feb 23, 2026Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3Medical Image Segmentation Text-Guided Image Segmentation
Concordia University, Montreal, Canada

Sep 24, 2026J/K move · Enter open · S save

Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation

Md Maklachur Rahman, Md Hasan Al Banna, Saraf Anjum, Assame Arnob+1

Texas A&M University, College Station, TX 77843, USA · Independent Researcher

Text-Guided Image Segmentation Medical Image Segmentation

Open paper AskPDF arXiv

Textual descriptions can reduce ambiguity in medical image segmentation by specifying the finding and location to be delineated. Existing text-guided methods mainly improve where image and language features interact but generally retain a single learned update pathway across all image-text pairs. We propose MRSeg, a parameter-efficient framework that uses each image-text pair to route the adaptation of visual and textual features before dense prediction. Frozen ConvNeXt-Tiny and PubMedBERT encoders provide multiscale visual features and clinical text tokens. A joint router uses the deepest visual feature and pooled text to predict a sparse mixture over low-rank adapter bases. The resulting route is shared across separate adapter banks for two visual scales and text, coordinating their adaptation while keeping the feature-specific parameters separate. Region Bridge uses text-derived queries to aggregate dense visual tokens into latent regions, refines these regions through self-attention and text cross-attention, and redistributes the refined information back to the feature maps. Finally, a multiscale decoder combines refined semantic features with shallow image evidence. On QaTa-COV19 and MosMedData+, MRSeg achieves 90.90/83.32 and 81.53/68.82 Dice/mIoU, respectively, with 7.11M trainable parameters and 7.60 GFLOPs. Code: https://github.com/maklachur/MRSeg.

Abstract

Text-Guided Image Segmentation

Explore similar work

Multimodal Routing and Region Refinement for Language-Guided Medical Image Segmentation

TAMISeg: Text-Aligned Multi-scale Medical Image Segmentation with Semantic Encoder Distillation

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation