SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation

Wei Tang; Xuejing Liu; Yanpeng Sun; Zechao Li

cs.CVMar 18, 2026

SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation

Authors: Wei Tang, Xuejing Liu, Yanpeng Sun, Zechao Li

Organizations: School of Computer Science and Engineering, Nanjing University of Science and Technology, No. 200 Xiaolingwei Road, Nanjing 210094, China. · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China. · NExT++ Lab, School of Computing, National University of Singapore, Singapore 119077, Singapore.

Abstract

The Segment Anything Model (SAM) excels at general image segmentation but has limited ability to understand natural language, which restricts its direct application in Referring Expression Segmentation (RES). Toward this end, we propose SSP-SAM, a framework that fully utilizes SAM's segmentation capabilities by integrating a Semantic-Spatial Prompt (SSP) encoder. Specifically, we incorporate both visual and linguistic attention adapters into the SSP encoder, which highlight salient objects within the visual features and discriminative phrases within the linguistic features. This design enhances the referent representation for the prompt generator, resulting in high-quality SSPs that enable SAM to generate precise masks guided by language. Although not specifically designed for Generalized RES (GRES), where the referent may correspond to zero, one, or multiple objects, SSP-SAM naturally supports this more flexible setting without additional modifications. Extensive experiments on widely used RES and GRES benchmarks confirm the superiority of our method. Notably, our approach generates segmentation masks of high quality, achieving strong precision even at strict thresholds such as [email protected]. Further evaluation on the PhraseCut dataset demonstrates improved performance in open-vocabulary scenarios compared to existing state-of-the-art RES methods. The code and checkpoints are available at: https://github.com/WayneTomas/SSP-SAM.

Figures & tables

Fig. 1: (a) SAM struggles with text prompts, especially the free-form languages in the Referring Expression Segmentation, resulting in poor segmentation mask. (b) SSP-SAM seamlessly transform images and languages to Semantic-Spatial Prompts to guide SAM for referent segmentation.

Fig. 2: Illustration of referring expression segmentation (RES) settings. (a) Classic RES focuses on single-target segmentation. (b) and (c) Generalized RES (GRES) introduces multi-target and no-target scenarios, which require not only a comprehensive understanding of all objects in the scene but also a contextual interpretation of both the image and the referring expression.

Method	Type	Core Contribution
Grounded SAM [ 33 ]	Pre-SAM	Generates language-driven prompts from external models to guide SAM.
CLIP Surgery [ 23 ]	Pre-SAM	Samples points from attention regions to refine SAM mask proposals.
TAS [ 24 ]	Post-SAM	Generates mask proposals via SAM and selects the best using CLIP cross-modal matching.
Semi-SAM [ 19 ]	Post-SAM	Refines SAM-generated masks using CLIP to improve alignment with text.
DIT [ 20 ]	One-stage	Fine-grained layer-wise adjustments to SAM for improved segmentation.
Prompt-RIS [ 27 ]	One-stage	Text-guided SAM, yet still heavily dependent on CLIP-generated geometric prompts.

TABLE I: Summary of Representative SAM-based RES methods.

Fig. 3: An illustration of SSP-SAM, where SAM is equipped with Semantic-Spatial Prompt (SSP) encoder for RES. The multi-modal features of inputs are extracted by CLIP, and their attention scores are adapted to generate Semantic-Spatial referent features. These features, along with the special tokens, are processed by a prompt generator to identify key information about the referent for the segmentation process. Additionally, an auxiliary REC task is utilized to improve RES performance.

Fig. 4: An illustration of the attention adapter. It consists of a visual attention adapter and a linguistic attention adapter. The images and texts embedding are extracted from CLIP and then feed to the visual and linguistic attention adapter to obtain the enhanced referent features.

Method	Image Encoder	Text Encoder	RefCOCO			RefCOCO+			RefCOCOg		ReferIt
Method	Image Encoder	Text Encoder	val	testA	testB	val	testA	testB	val	test	test
gIoU
Non-SAM based
VLT [ 46 ]	DN53	Bi-GRU	65.65	68.29	62.73	55.50	59.20	49.36	52.99	56.65	-
CRIS [ 5 ]	RN101	GPT-2	70.47	73.18	66.10	62.27	68.08	53.68	59.87	60.36
SeqTR [ 46 ]	DN53	Bi-GRU	71.70	73.31	69.82	63.04	66.73	58.97	64.69	65.74	-

TABLE II: Comparison with state-of-the-art methods on four referring image segmentation benchmarks. gIoU and cIoU denote the IoU metrics averaged per instance and accumulated over the entire dataset, respectively. The methods marked with

{\ddagger}

are trained on the combined RefCOCO/+/g dataset. The methods marked with

\dagger

are adapted methods. The methods shown in gray indicate MLLM-based models trained on large-scale external datasets with much larger text encoders. DN53 refers to DarkNet-53, and RN101 refers to ResNet-101, respectively. The highest performance is marked in bold, and the second-highest performance is marked with an underline.

Method	RefCOCO val
Method	[email protected]	[email protected]	[email protected]	gIoU	cIoU
LAVT [ 6 ]	34.30	75.28	84.46	74.46	72.73
ReLA [ 34 ]	34.99	77.71	85.82	75.61	73.82
CGFormer [ 7 ]	38.77	78.69	87.23	76.93	74.75
MagNet [ 48 ]	38.77	80.01	87.79	77.43	75.24
SSP-SAM-224	46.11	81.42	87.18	77.91	75.12

TABLE III: Comparisons of Pr@X results on the RefCOCO val split under classic RES settings. The best performance is highlighted in bold, and the second-best is underlined. Our method tends to generate higher-precision masks, as evidenced by its superior performance under the stricter [email protected] metric.

Non-SAM based
Method	val			testA			testB
Method	gIoU	cIoU	N-acc.	gIoU	cIoU	N-acc.	gIoU	cIoU	N-acc.
MattNet [ 37 ]	48.24	47.51	41.15	59.30	58.66	44.04	46.14	45.33	41.32
LTS [ 41 ]	52.70	52.30	-	62.64	61.87	-	50.42	49.96	-
VLT [ 44 ]	52.00	52.51	47.17	63.20	62.19	48.74	50.88	50.52	47.82
CRIS [ 5 ]	56.27	55.34	-	63.42	63.82	-	51.79	51.04	-

TABLE IV: Generalized referring image segmentation (GRES) results on gRefCOCO dataset. gIoU and cIoU denote the IoU metrics averaged per instance and accumulated over the entire dataset, respectively. N-acc. is short for the No-target accuracy. (ft) denotes the model is finetuned on the training set of gRefCOCO.

Method	gRefCOCO val
Method	[email protected]	[email protected]	[email protected]	gIoU	cIoU
CGFormer [ 7 ]	22.43	56.57	68.93	63.01	62.28
ReLA [ 34 ]	24.68	68.33	74.20	63.60	62.42
SSP-SAM-224	44.00	60.55	66.49	68.87	61.28
SSP-SAM-336	44.77	61.54	67.33	69.09	62.56

TABLE V: Comparisons of Pr@X results on the gRefCOCO validation split under the GRES settings. The best performance is highlighted in bold, and the second-best is underlined. Our method consistently tends to produce higher-precision masks, as reflected by its strong performance under the strict [email protected] metric.

Training Set	PhraseCut test
Training Set	CRIS	LAVT	RISAM	SSP-SAM-224	SSP-SAM-336
RefCOCO	15.53	16.68	22.82	24.66	24.69
RefCOCO+	16.30	16.64	21.68	25.75	24.91
RefCOCOg	16.24	16.05	22.47	27.00	28.29

TABLE VI: Comparisons in the open-vocabulary scenario on the PhraseCut test split. The best performance is highlighted in bold, and the second-best is underlined. The evaluation metric is the default gIoU . Our method demonstrates improved generalization compared to other approaches.

Method	Params.	Inference Time (ms)	Std. (ms)
Layer-wise DIT-B	212.87M	52.57	2.66
LISA-Vicuna-7B	288.26M	1324.46	98.98
SSP-SAM-224	18.37M	26.68	2.10
SSP-SAM-336	18.73M	53.00	1.26

TABLE VII: Comparison of the time cost with state-of-the-art methods on the testB split of RefCOCO dataset.

SSP Backbone	RefCOCO
SSP Backbone	val	testA	testB
CLIP	76.24	79.30	71.77
EVA-CLIP	76.81	79.23	72.53
SigLIP2	76.81	79.44	72.86

TABLE VIII: Ablation on different SSP backbones for the SSP-SAM-224 framework on RefCOCO dataset. All models use SAM-H as the image encoder.

Pre-training	RES	Aux.	RefCOCO
Pre-training	RES	Aux.	val	testA	testB
	✓		76.03	78.55	71.60
	✓	✓	76.24	79.30	71.77
✓	✓	✓	77.91	80.67	73.71

TABLE IX: The ablation study of different training strategies on RefCOCO dataset. Aux. is the abbreviation for the auxiliary task.

Method	Freeze	RefCOCO
Method	Freeze	val	testA	testB
SSP-SAM-224*	✓	76.74	79.61	72.35
SSP-SAM-224	✗	77.91	80.67	73.71
SSP-SAM-336*	✓	78.13	80.89	73.80
SSP-SAM-336	✗	79.37	81.95	75.52

TABLE X: The ablation study on the impact of freezing SAM mask decoder on RefCOCO dataset.

*

indicates the model with a frozen SAM decoder during training.

P.G.	V.A.D.	L.A.D.	RefCOCO
P.G.	V.A.D.	L.A.D.	val	testA	testB
✓			42.30	43.13	41.18
	✓		61.27	63.83	58.97
		✓	66.09	70.24	61.78
	✓	✓	67.87	71.24	63.11
✓	✓	✓	77.91	80.67	73.71

TABLE XI: The ablation study of different modules on RefCOCO dataset. “P.G.”, “V.A.D.” and “L.A.D.” are the abbreviation for Prompt Generater, Visual Attention Adapter and Linguistic Attention Adapter, respectively.

P.G. Type	RefCOCO
P.G. Type	val	testA	testB
MLP	67.87	71.24	63.11
DETR-like decoder	72.54	76.13	67.82
Default (encoder-based)	77.91	80.67	73.71

TABLE XII: Ablation study of different types of prompt generator on RefCOCO dataset. “P.G.” stands for Prompt Generator.

Method	G.L.M.	RefCOCO
Method	G.L.M.	val	testA	testB
SSP-SAM-224 †		75.38	78.54	70.79
SSP-SAM-224	✓	76.24	79.30	71.77

TABLE XIII: Ablation study of Gaussian-like mapping on RefCOCO dataset. “G.L.M.” indicates the presence of the Gaussian-like mapping and “

\dagger

” indicates its removal.

Layers	RefCOCO			PhraseCut
Layers	val	testA	testB	test
1	74.57	77.04	69.45	23.48
2	75.56	78.16	71.64	25.05
4	76.27	78.75	71.61	24.81
6	76.24	79.30	71.77	24.66
8	76.43	79.18	72.14	24.59

TABLE XIV: The ablation study for the numbers of transformer encoder layers in the prompt generator. We train the model on RefCOCO dataset and test on RefCOCO and the open-vocabulary PhreaseCut.

Number	RefCOCO			PhraseCut
Number	val	testA	testB	test
2	76.21	78.68	71.88	25.22
16	76.56	78.81	71.82	25.31
32	76.24	78.88	72.07	25.39
128	76.24	79.30	71.77	24.66
256	76.47	79.01	72.41	24.17

TABLE XV: The ablation study for the numbers of [RES] tokens in the prompt generator. We train the model on RefCOCO dataset and test on RefCOCO and the open-vocabulary PhreaseCut.

Method	w/o spatial cues		w/ spatial cues
Method	gIoU	cIoU	gIoU	cIoU
SSP-SAM-224	70.63	60.49	73.23	62.99
SSP-SAM-336	76.53	68.79	80.88	72.70

TABLE XVI: Comparison of SSP-SAM under different spatial cues settings on 25 random samples from RefCOCO testB split.

Fig. 5: Qualitative segmentation results of SSP-SAM on classic RES datasets (RefCOCO, RefCOCO+, RefCOCOg, ReferIt) and the open-vocabulary PhraseCut dataset. SSP-SAM generates masks with sharper boundaries and finer details. Predicted masks are shown in red, and ground truth in green.

Fig. 6: Qualitative results of SSP-SAM on the gRefCOCO dataset under the GRES scenario. Our method produces segmentation masks with sharper edges and finer details. Predicted masks are shown in red, and ground truth masks are shown in green.

Fig. 7: Failure cases of SSP-SAM on classic RES datasets (RefCOCO, RefCOCO+, RefCOCOg, ReferIt), the Generalized RES dataset (gRefCOCO), and the open-vocabulary PhraseCut dataset. Predicted masks are shown in red, and ground truth in green.

Fig. 8: Qualitative results of the adapted visual attention. The salient objects within the images receive higher attention scores through the visual attention adapter.

Fig. 9: Qualitative results of the adapted linguistic attention. The discriminative phrases within the referring expressions receive higher attention scores through the linguistic attention adapter.

Referring Expression Segmentation

Referring expression segmentation maps natural-language descriptions to the pixels or 3D points of the objects they identify.

See all 28 papers

Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation
Apr 9, 2026Weiming Zhang, Dingwen Xiao, Songyue Guo +5Referring Expression Segmentation Text-Guided Image Segmentation
The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University
DRAgent: Discriminative Reasoning Agent for Referring Expression Segmentation
Aug 24, 2026Yujie Qi, Luyan ZhangReferring Expression Segmentation Text-Guided Image Segmentation
School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, 310018, China. · Khoury College of Computer Sciences, Northeastern University, Vancouver, BC V6B 1Z3, Canada.
VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation
Aug 2, 2026Savindu Dilshan WickramasinghePromptable Image Segmentation Efficient VLM Inference
University of Moratuwa · Department of Electronic and Telecommunication Engineering, University of Moratuwa Moratuwa, Sri Lanka

Apr 9, 2026J/K move · Enter open · S save

Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation

Weiming Zhang, Dingwen Xiao, Songyue Guo, Guangyu Xiang+4

The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University

Referring Expression Segmentation Text-Guided Image Segmentation Referring Expression Comprehension Promptable Image Segmentation

Open paper AskPDF arXiv

Referring Expression Segmentation (RES) aims to segment image regions described by natural-language expressions, serving as a bridge between vision and language understanding. Existing RES methods, however, rely heavily on large annotated datasets and are limited to either explicit or implicit expressions, hindering their ability to generalize to any referring expression. Recently, the Segment Anything Model 3 (SAM3) has shown impressive robustness in Promptable Concept Segmentation. Nonetheless, applying it to RES remains challenging: (1) SAM3 struggles with longer or implicit expressions; (2) naive coupling of SAM3 with a multimodal large language model (MLLM) makes the final results overly dependent on the MLLM's reasoning capability, without enabling refinement of SAM3's segmentation outputs. To this end, we present Tarot-SAM3, a novel training-free framework that can accurately segment from any referring expression. Specifically, Tarot-SAM3 consists of two key phases. First, the Expression Reasoning Interpreter (ERI) phase introduces reasoning-assisted prompt options to support structured expression parsing and evaluation-aware rephrasing. This transforms arbitrary queries into robust heterogeneous prompts for generating reliable masks with SAM3. Second, the Mask Self-Refining (MSR) phase selects the best mask across prompt types and performs self-refinement by leveraging rich feature relationships from DINOv3 to compare discriminative regions among ERI outputs. It then infers region affiliation to the target, thereby correcting over- and under-segmentation. Extensive experiments demonstrate that Tarot-SAM3 achieves strong performance on both explicit and implicit RES benchmarks, as well as open-world scenarios. Ablation studies further validate the effectiveness of each phase.

Abstract

Referring Expression Segmentation

Explore similar work

Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation

DRAgent: Discriminative Reasoning Agent for Referring Expression Segmentation

VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation