SAPNet++: Evolving Point-Prompted Instance Segmentation with Semantic and Spatial Awareness

Zhaoyang Wei; Xumeng Han; Xuehui Yu; Xue Yang; Guorong Li; Zhenjun Han; Jianbin Jiao

cs.CVFeb 25, 2026

SAPNet++: Evolving Point-Prompted Instance Segmentation with Semantic and Spatial Awareness

Authors: Zhaoyang Wei, Xumeng Han, Xuehui Yu, Xue Yang, Guorong Li, Zhenjun Han, Jianbin Jiao

Organizations: the School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Science (UCAS), Beijing, 100049, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University (SJTU), Shanghai, 200240, China. · the School of Computer Science and Technology, University of Chinese Academy of Science (UCAS), Beijing, 100049, China.

Abstract

Single-point annotation is increasingly prominent in visual tasks for labeling cost reduction. However, it challenges tasks requiring high precision, such as the point-prompted instance segmentation (PPIS) task, which aims to estimate precise masks using single-point prompts to train a segmentation network. Due to the constraints of point annotations, granularity ambiguity and boundary uncertainty arise the difficulty distinguishing between different levels of detail (eg. whole object vs. parts) and the challenge of precisely delineating object boundaries. Previous works have usually inherited the paradigm of mask generation along with proposal selection to achieve PPIS. However, proposal selection relies solely on category information, failing to resolve the ambiguity of different granularity. Furthermore, mask generators offer only finite discrete solutions that often deviate from actual masks, particularly at boundaries. To address these issues, we propose the Semantic-Aware Point-Prompted Instance Segmentation Network (SAPNet). It integrates Point Distance Guidance and Box Mining Strategy to tackle group and local issues caused by the point's granularity ambiguity. Additionally, we incorporate completeness scores within proposals to add spatial granularity awareness, enhancing multiple instance learning (MIL) in proposal selection termed S-MIL. The Multi-level Affinity Refinement conveys pixel and semantic clues, narrowing boundary uncertainty during mask refinement. These modules culminate in SAPNet++, mitigating point prompt's granularity ambiguity and boundary uncertainty and significantly improving segmentation performance. Extensive experiments on four challenging datasets validate the effectiveness of our methods, highlighting the potential to advance PPIS.

Figures & tables

Panel of Figure 1 — Fig. 1 : The motivation behind SAPNet++ stems from two key challenges of point annotation: (a) Granularity Ambiguity : SAM-generated masks often assign higher scores to non-target categories ( e.g . , the clothes instead of the person in the yellow dashed box). Additionally, segmentation struggles to separate individual targets within the same category (group issue) and overlooks local details due to MIL’s preference under point labels for foreground-dominant regions (local issue) in the green dashed box. Within the blue dashed box,our proposed S-MIL tackles these issues by selecting proposals that capture the target’s complete semantic information. (b) Boundary Uncertainty : Despite resolving granularity ambiguity and related issues, compared to the ground truth, most proposal generators still yield proposals of varying quality, leading to “boundary uncertainty” where masks do not fully encompass the targets. We address that by refining predicted masks during segmentation to achieve highly satisfactory results even with imprecise supervision.

Fig. 2 : SAPNet++ is structured into three key branches: the proposal selection mechanism branch for the initial proposal selection using proposal selection and SASD I, the selection refinement mechanism branch for refining proposals through SASD II and semantic matching, and the SEG branch for segmentation and further refinement. i). Proposal Selection Mechanism (PSM) Branch: The process starts with generating category-agnostic mask proposals using point prompts within a visual foundation model. The PSM branch employs multi-instance learning and a point-guided strategy to construct the PSM for the input box proposals. It combines completeness scores from SASD I with confidence scores to capture global object semantics, initially filtering box proposals to obtain the $box_{psm}$ . ii). Selection Refinement Mechanism (SRM) Branch: Emulating typical MIL approaches, this branch utilizes positive and negative proposal generator to create high-quality positive and regulatory negative bags to enhance the semantic matching capabilities of the SRM. These bags are processed through SASD II and refined using box mining strategy, resulting in the refined $box_{srm}$ . iii). Segment Branch : We use the final selected pseudo box to obtain corresponding mask proposals through IoU matching. Given that these mask proposals are less precise than ground truth, the branch employs Affinity Refinement at both pixel and high-level semantic space to construct soft pseudo masks alongside mask proposals for enhanced segmentation supervision.

Panel of Figure 3 — Fig. 3 : (a) The PSM weights scores from category, instance, and point-guided strategies to preliminarily filter proposals. (b) The SRM further enhances the selection capability by introducing negative examples and employs box mining strategy to address the local issues. (c) The SASD predicts a completeness score for each proposal in PSM and SRM, re-weighting them with the previous scores.

Fig. 4 : Multi-level Affinity Refinement integrates pixel-level color and texture information with high-dimensional semantic information from the backbone, refining segmentation masks through global and local affinity.

Method	Ann.	Backbone	sched.	Arch.	$\rm mAP$	$\rm mAP_{50}$	$\rm mAP_{75}$	$\rm mAP_{s}$	$\rm mAP_{m}$	$\rm mAP_{l}$
Fully-supervised instance segmentation models.
Mask R-CNN [ 1 ]	$\mathcal{M}$	ResNet-50	1x	Mask R-CNN	34.6	56.5	36.6	18.3	37.4	47.2
YOLACT-700 [ 6 ]	$\mathcal{M}$	ResNet-101	4.5x	YOLACT	31.2	54.0	32.8	12.1	33.3	47.0
PolarMask [ 71 ]	$\mathcal{M}$	ResNet-101	2x	PolarMask	32.1	53.7	33.1	14.7	33.8	45.3
SOLO [ 4 ]	$\mathcal{M}$	ResNet-50	1x	SOLO	33.1	53.5	35.0	12.2	36.1	50.8
SOLOv2 [ 2 ]	$\mathcal{M}$	ResNet-50	1x	SOLOv2	34.8	54.9	36.9	13.4	37.8	53.7

TABLE I : Mask annotation (

\mathcal{M}

), image annotation (

\mathcal{I}

), box annotation (

\mathcal{B}

) and point annotation (

\mathcal{P}

) performance on COCO-17 val. “Ann.” is the annotation type and “sched.” means schedule. † indicates applying MRCNN refinement. The multi-scale augment training for re-training segmentation methods on 3x schedule for a fair comparison, and other experiments are on the single-scale training. SwinMR is Swin-Transformer-Mask R-CNN. SwinMR and Mask2Former use multi-scale augment strategies for SOTA. SAPNet (MPS) refers to adding MPS strategy in the appendix during training.

Method	Ann.	Backbone	$\rm AP_{25}$	$\rm AP_{50}$	$\rm AP_{75}$
Mask R-CNN [ 1 ]	$\mathcal{M}$	R-50	78.0	68.8	43.3
Mask R-CNN [ 1 ]	$\mathcal{M}$	R-101	79.6	70.2	45.3
BoxInst [ 9 ]	$\mathcal{B}$	R-50	-	59.1	34.2
BoxInst [ 9 ]	$\mathcal{B}$	R-101	-	61.4	37.0
DiscoBox [ 10 ]	$\mathcal{B}$	R-50	71.4	59.8	35.5
DiscoBox [ 10 ]	$\mathcal{B}$	R-101	72.8	62.2	37.5

TABLE II : Instance segmentation performance on the VOC2012

test

set. For a fair comparison, the segmentation network is trained on a multi-scale augment dataset using our and previous networks on Resnet101. † indicates applying MRCNN refinement.

Method	Ann.	BB.	COCO17 ( $\rm AP$ )	VOC12 ( $\rm AP_{50}$ )
FPN [ 89 ]	$\mathcal{B}$	R-50	37.4	75.3
CASD [ 90 ]	$\mathcal{I}$	VGG-16	12.8	53.6
CASD [ 90 ]	$\mathcal{I}$	R-50	13.9	56.8
OD-WSCL [ 91 ]	$\mathcal{I}$	VGG-16	13.6	56.2
OD-WSCL [ 91 ]	$\mathcal{I}$	R-101	14.4	-
UFO 2 [ 92 ]	$\mathcal{P}$	VGG-16	13.0	41.0

TABLE III : Object detection performance on the COCO 2017 and VOC 2012 val sets.

Method	Ann.	Pretrain	Detection		Segmentation
Method	Ann.	Pretrain	$\rm AP$	$\rm AP_{50}$	$\rm AP$	$\rm AP_{50}$
FPN [ 89 ]	$\mathcal{B}$	ImageNet	37.3	65.4	-	-
FPN [ 89 ]	$\mathcal{B}$	COCO	40.3	65.3	-	-
Mask R-CNN [ 1 ]	$\mathcal{M}$	ImageNet	39.4	66.0	34.2	61.0
Mask R-CNN [ 1 ]	$\mathcal{M}$	COCO	40.9	66.2	36.4	61.8
CondInst [ 5 ]	$\mathcal{M}$	ImageNet	-	-	33.1	58.1

TABLE IV : Object detection and instance segmentation Performance on the Cityscape

val

set. The “pretrain” means models are pretrained on ImageNet or COCO dataset.

Fig. 5 : Visualization of instance segmentation results on the COCO train2017 dataset.

Method	Ann.	BB.	AP	AP 50	AP 75	AP S	AP M	AP L
Mask R-CNN [ 1 ]	$\mathcal{M}$	R-50	34.2	57.5	36.2	19.6	41.4	47.9
PolarMask [ 71 ]	$\mathcal{M}$	R-50	27.2	48.5	27.3	-	-	-
CondInst [ 5 ]	$\mathcal{M}$	R-50	31.8	56.4	31.5	15.3	41.0	50.2
SOLO [ 4 ]	$\mathcal{M}$	R-50	23.5	43.1	22.6	7.4	30.5	43.3
SOLOv2 [ 2 ]	$\mathcal{M}$	R-50	28.9	50.6	28.8	11.6	37.8	48.3
SOLOv2 [ 2 ]	$\mathcal{M}$	R-101	32.6	54.4	33.4	13.6	42.4	54.6

TABLE V : Instance segmentation results on remote sensing image dataset iSAID

val

. The input size is 800

\times

800 for all models with 1x training schedule.

PS	PDG	SR	PNPG	BMS	SASD	MLAR	$\rm mAP$
✓							26.8
✓	✓						27.5
✓	✓	✓					27.7
✓	✓	✓	✓				29.6
✓	✓	✓	✓	✓			30.7
✓	✓	✓	✓	✓	✓		31.3

TABLE VI : The effect of each component in SAPNet++: proposal selection (PS), point distance guidance (PDG), positive, negative proposals generator (PNPG), selection refinement (SR), box mining strategy (BMS), Spatial-aware Self-distillation (SASD) and Multi-level Affinity Refinement (MLAR).

train stage on coco	type	sched.	$\rm AP$	$\rm AP_{50}$	$\rm AP_{75}$
SAM-top1	two stage	1x	24.6	41.9	25.3
SAPNet+SOLOv2	two stage	1x	30.2	49.8	31.5
SAPNet (MPS)	end to end	1x	31.2	51.8	32.3
SAPNet	end to end	1x	30.7	51.2	31.8
SAPNet++	end to end	1x	32.4	53.5	34.2

TABLE VII : The experimental comparisons of segmenters in COCO dataset, SAM-top1 is the highest scoring mask generated by SAM.

Fig. 6 : The comparative visualization between SAM-top1 and SAPNet++ is presented, showcasing SAM’s segmentation outcomes in green masks and our results in yellow. The orange and red bounding boxes highlight the respective mask boundaries.

SASD		$\rm AP$	$\rm AP_{50}$	$\rm AP_{75}$	$\rm AP_{s}$	$\rm AP_{m}$	$\rm AP_{l}$
SASD-I	SASD-II	$\rm AP$	$\rm AP_{50}$	$\rm AP_{75}$	$\rm AP_{s}$	$\rm AP_{m}$	$\rm AP_{l}$
		30.7	51.2	31.8	11.9	34.7	47.1
✓		30.9	51.3	32.0	12.0	34.8	47.2
	✓	31.1	51.7	32.6	12	35.0	47.3
✓	✓	31.3	52.0	32.6	12.6	35.2	48.0

TABLE VIII : Meticulous ablation experiments in SASD

Affinity-level	AP	AP 50	AP 75	AP S	AP M	AP L
✗	30.7	51.2	31.8	11.9	34.7	47.1
low-level	31.0	51.4	32.2	12.1	35.0	47.2
high-level	31.3	51.9	32.4	12.7	35.2	47.9
low & high-level	31.9	52.5	33.4	13.1	36.1	48.5

TABLE IX : The effectiveness of low-level pixel space and high-level semantic features on SAPNet++.

Global affinity	Local affinity	Cascade	AP	AP 50	AP 75
✗	✗	0	30.7	51.2	31.8
✓		0	31.0	51.6	32.1
	✓	1	31.2	51.7	32.3
✓	✓	1	31.6	52.1	33.0
✓	✓	2	31.9	52.5	33.4
✓	✓	3	31.7	52.3	33.2

TABLE X : The effectiveness of global and local affinity.

Fig. 7 : The visualization of segmentation using different affinity strategies reveals their varied impacts. with its wide propagation range, global affinity effectively manages discontinuities in large-scale masks but struggles with maintaining local consistency, often missing finer details at edges and corners. Conversely, local affinity, with its limited propagation distance, underperforms in areas where mask parts are distant. Combining both global and local affinity yields smoother segmentation masks. Using a cascading approach further improves mask completeness, striking a balance between broad coverage and detailed accuracy.

Method	$Gap_{local}$	$Gap_{group}$	$mIoU_{box}$	$mIoU_{mask}$
SAM	28.4	24.7	58.3	55.6
MIL	19.9	18.6	63.8	58.8
SAPNet	13.1	14.1	69.1	65.9
SAPNet++	12.4	13.3	70.4	67.7

TABLE XI : Experimental analysis with group and local issues.

Segmenters	BB.	Sch.	Detection		Segmentation
Segmenters	BB.	Sch.	$\rm AP$	$\rm AP_{50}$	$\rm AP$	$\rm AP_{50}$
Solov2	R-101	3x	38.9	62.0	35.7	57.6
Swin	Swin-S	3x	40.1	63.4	35.8	59.5
Mask2former	Swin-S	3x	43.1	63.5	39.5	62.8
ConvNeXt-V2	ConV2-B	3x	46.0	66.8	39.7	63.1

TABLE XII : Advanced segmenters on COCO.

Type	Image tag	Point	Box	Mask
Cost (per Image)	1.5s/image	1.87s	34.5s	239.7s
Cost (per Object)	–	0.9s	7s	79.2s

TABLE XIII : Annotation cost comparison for different supervision types.

Supervision Type	Method	mAP	Rel. Ann. Cost (vs. Point/Image)
Image Tag (Weakest)	BESTIE	14.3 mAP	1.25 $\times$ faster
Point (Our Method)	SAPNet++	32.4 mAP	Base Cost
Box	Box2Mask	32.6 mAP	18.4 $\times$ slower
Mask (Full Supervision)	Mask2Former	38.7 mAP	128 $\times$ slower

TABLE XIV : Tradeoff Across Different Supervision Methods.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

SRPR-Net: Semantic and Relational Prompt Refinement for Automated SAM-based Instance Segmentation
Sep 21, 2026Lufei Liu, Guojie Li, Suncheng Xiang +1Instance Segmentation Promptable Image Segmentation
Ruijin-XJTLU Intelligent Medicine Institute, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China; and Xi’an Jiaotong-Liverpool University, Suzhou, China · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China
PinPoint: Prompting with Informative Interior Points
May 26, 2026Pouya Sadeghi, Shawn He, Pedro Pablo Guerrero Vela +3Image Segmentation Promptable Image Segmentation
University of Waterloo · 2Critical ML, University of Waterloo · 3Apple
Dense Point-to-Mask Optimization with Reinforced Point Selection for Crowd Instance Segmentation
Apr 2, 2026Hongru Chen, Jiyang Huang, Jia Wan +1Instance Segmentation Crowd Counting
Harbin Institute of Technology, Shenzhen · City University of Hong Kong

Sep 21, 2026J/K move · Enter open · S save

SRPR-Net: Semantic and Relational Prompt Refinement for Automated SAM-based Instance Segmentation

Lufei Liu, Guojie Li, Suncheng Xiang, Fan Zhang

Ruijin-XJTLU Intelligent Medicine Institute, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China; and Xi’an Jiaotong-Liverpool University, Suzhou, China · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China

Instance Segmentation Promptable Image Segmentation Segment Anything Models

Open paper AskPDF arXiv

Instance segmentation is a fundamental computer vision task with diverse real-world applications. Recently, prompt-driven foundation models have shown promising generalization. However, automated prompting remains limited by insufficient semantic guidance and inter-instance modeling. To address this challenge, we propose a novel architecture, named Semantic Relational Prompt Refinement Network (SRPR-Net), for automated SAM-based instance segmentation. A sequential prompt refinement mechanism is introduced to enrich detector geometry with visual-language semantics and then incorporate same-image instance dependencies, enabling context-aware box adjustment before SAM segmentation. Experiments on multiple standard benchmarks demonstrate that SRPR-Net achieves consistent improvements in segmentation performance over existing state-of-the-art approaches. The code is publicly available at https://github.com/JeremyXSC/SRPR-Net.

Abstract

Instance Segmentation

Explore similar work

SRPR-Net: Semantic and Relational Prompt Refinement for Automated SAM-based Instance Segmentation

PinPoint: Prompting with Informative Interior Points

Dense Point-to-Mask Optimization with Reinforced Point Selection for Crowd Instance Segmentation