Organizations: the School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Science (UCAS), Beijing, 100049, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University (SJTU), Shanghai, 200240, China. · the School of Computer Science and Technology, University of Chinese Academy of Science (UCAS), Beijing, 100049, China.
Single-point annotation is increasingly prominent in visual tasks for labeling cost reduction. However, it challenges tasks requiring high precision, such as the point-prompted instance segmentation (PPIS) task, which aims to estimate precise masks using single-point prompts to train a segmentation network. Due to the constraints of point annotations, granularity ambiguity and boundary uncertainty arise the difficulty distinguishing between different levels of detail (eg. whole object vs. parts) and the challenge of precisely delineating object boundaries. Previous works have usually inherited the paradigm of mask generation along with proposal selection to achieve PPIS. However, proposal selection relies solely on category information, failing to resolve the ambiguity of different granularity. Furthermore, mask generators offer only finite discrete solutions that often deviate from actual masks, particularly at boundaries. To address these issues, we propose the Semantic-Aware Point-Prompted Instance Segmentation Network (SAPNet). It integrates Point Distance Guidance and Box Mining Strategy to tackle group and local issues caused by the point's granularity ambiguity. Additionally, we incorporate completeness scores within proposals to add spatial granularity awareness, enhancing multiple instance learning (MIL) in proposal selection termed S-MIL. The Multi-level Affinity Refinement conveys pixel and semantic clues, narrowing boundary uncertainty during mask refinement. These modules culminate in SAPNet++, mitigating point prompt's granularity ambiguity and boundary uncertainty and significantly improving segmentation performance. Extensive experiments on four challenging datasets validate the effectiveness of our methods, highlighting the potential to advance PPIS.
Figures & tables
Fig. 1 : The motivation behind SAPNet++ stems from two key challenges of point annotation: (a) Granularity Ambiguity : SAM-generated masks often assign higher scores to non-target categories ( e.g . , the clothes instead of the person in the yellow dashed box). Additionally, segmentation struggles to separate individual targets within the same category (group issue) and overlooks local details due to MIL’s preference under point labels for foreground-dominant regions (local issue) in the green dashed box. Within the blue dashed box,our proposed S-MIL tackles these issues by selecting proposals that capture the target’s complete semantic information. (b) Boundary Uncertainty : Despite resolving granularity ambiguity and related issues, compared to the ground truth, most proposal generators still yield proposals of varying quality, leading to “boundary uncertainty” where masks do not fully encompass the targets. We address that by refining predicted masks during segmentation to achieve highly satisfactory results even with imprecise supervision.
Fig. 2 : SAPNet++ is structured into three key branches: the proposal selection mechanism branch for the initial proposal selection using proposal selection and SASD I, the selection refinement mechanism branch for refining proposals through SASD II and semantic matching, and the SEG branch for segmentation and further refinement. i). Proposal Selection Mechanism (PSM) Branch: The process starts with generating category-agnostic mask proposals using point prompts within a visual foundation model. The PSM branch employs multi-instance learning and a point-guided strategy to construct the PSM for the input box proposals. It combines completeness scores from SASD I with confidence scores to capture global object semantics, initially filtering box proposals to obtain the boxpsm . ii). Selection Refinement Mechanism (SRM) Branch: Emulating typical MIL approaches, this branch utilizes positive and negative proposal generator to create high-quality positive and regulatory negative bags to enhance the semantic matching capabilities of the SRM. These bags are processed through SASD II and refined using box mining strategy, resulting in the refined boxsrm . iii). Segment Branch : We use the final selected pseudo box to obtain corresponding mask proposals through IoU matching. Given that these mask proposals are less precise than ground truth, the branch employs Affinity Refinement at both pixel and high-level semantic space to construct soft pseudo masks alongside mask proposals for enhanced segmentation supervision.
Fig. 3 : (a) The PSM weights scores from category, instance, and point-guided strategies to preliminarily filter proposals. (b) The SRM further enhances the selection capability by introducing negative examples and employs box mining strategy to address the local issues. (c) The SASD predicts a completeness score for each proposal in PSM and SRM, re-weighting them with the previous scores.
Fig. 4 : Multi-level Affinity Refinement integrates pixel-level color and texture information with high-dimensional semantic information from the backbone, refining segmentation masks through global and local affinity.
Method
Ann.
Backbone
sched.
Arch.
mAP
mAP50
mAP75
mAPs
mAPm
mAPl
Fully-supervised instance segmentation models.
Mask R-CNN [ 1 ]
M
ResNet-50
1x
Mask R-CNN
34.6
56.5
36.6
18.3
37.4
47.2
YOLACT-700 [ 6 ]
M
ResNet-101
4.5x
YOLACT
31.2
54.0
32.8
12.1
33.3
47.0
PolarMask [ 71 ]
M
ResNet-101
2x
PolarMask
32.1
53.7
33.1
14.7
33.8
45.3
SOLO [ 4 ]
M
ResNet-50
1x
SOLO
33.1
53.5
35.0
12.2
36.1
50.8
SOLOv2 [ 2 ]
M
ResNet-50
1x
SOLOv2
34.8
54.9
36.9
13.4
37.8
53.7
TABLE I : Mask annotation ( M ), image annotation ( I ), box annotation ( B ) and point annotation ( P ) performance on COCO-17 val. “Ann.” is the annotation type and “sched.” means schedule. † indicates applying MRCNN refinement. The multi-scale augment training for re-training segmentation methods on 3x schedule for a fair comparison, and other experiments are on the single-scale training. SwinMR is Swin-Transformer-Mask R-CNN. SwinMR and Mask2Former use multi-scale augment strategies for SOTA. SAPNet (MPS) refers to adding MPS strategy in the appendix during training.
Method
Ann.
Backbone
AP25
AP50
AP75
Mask R-CNN [ 1 ]
M
R-50
78.0
68.8
43.3
Mask R-CNN [ 1 ]
M
R-101
79.6
70.2
45.3
BoxInst [ 9 ]
B
R-50
-
59.1
34.2
BoxInst [ 9 ]
B
R-101
-
61.4
37.0
DiscoBox [ 10 ]
B
R-50
71.4
59.8
35.5
DiscoBox [ 10 ]
B
R-101
72.8
62.2
37.5
TABLE II : Instance segmentation performance on the VOC2012 test set. For a fair comparison, the segmentation network is trained on a multi-scale augment dataset using our and previous networks on Resnet101. † indicates applying MRCNN refinement.
Method
Ann.
BB.
COCO17 ( AP )
VOC12 ( AP50 )
FPN [ 89 ]
B
R-50
37.4
75.3
CASD [ 90 ]
I
VGG-16
12.8
53.6
CASD [ 90 ]
I
R-50
13.9
56.8
OD-WSCL [ 91 ]
I
VGG-16
13.6
56.2
OD-WSCL [ 91 ]
I
R-101
14.4
-
UFO 2 [ 92 ]
P
VGG-16
13.0
41.0
TABLE III : Object detection performance on the COCO 2017 and VOC 2012 val sets.
Method
Ann.
Pretrain
Detection
Segmentation
AP
AP50
AP
AP50
FPN [ 89 ]
B
ImageNet
37.3
65.4
-
-
FPN [ 89 ]
B
COCO
40.3
65.3
-
-
Mask R-CNN [ 1 ]
M
ImageNet
39.4
66.0
34.2
61.0
Mask R-CNN [ 1 ]
M
COCO
40.9
66.2
36.4
61.8
CondInst [ 5 ]
M
ImageNet
-
-
33.1
58.1
TABLE IV : Object detection and instance segmentation Performance on the Cityscape val set. The “pretrain” means models are pretrained on ImageNet or COCO dataset.
Fig. 5 : Visualization of instance segmentation results on the COCO train2017 dataset.
Method
Ann.
BB.
AP
AP 50
AP 75
AP S
AP M
AP L
Mask R-CNN [ 1 ]
M
R-50
34.2
57.5
36.2
19.6
41.4
47.9
PolarMask [ 71 ]
M
R-50
27.2
48.5
27.3
-
-
-
CondInst [ 5 ]
M
R-50
31.8
56.4
31.5
15.3
41.0
50.2
SOLO [ 4 ]
M
R-50
23.5
43.1
22.6
7.4
30.5
43.3
SOLOv2 [ 2 ]
M
R-50
28.9
50.6
28.8
11.6
37.8
48.3
SOLOv2 [ 2 ]
M
R-101
32.6
54.4
33.4
13.6
42.4
54.6
TABLE V : Instance segmentation results on remote sensing image dataset iSAID val . The input size is 800 × 800 for all models with 1x training schedule.
PS
PDG
SR
PNPG
BMS
SASD
MLAR
mAP
✓
26.8
✓
✓
27.5
✓
✓
✓
27.7
✓
✓
✓
✓
29.6
✓
✓
✓
✓
✓
30.7
✓
✓
✓
✓
✓
✓
31.3
TABLE VI : The effect of each component in SAPNet++: proposal selection (PS), point distance guidance (PDG), positive, negative proposals generator (PNPG), selection refinement (SR), box mining strategy (BMS), Spatial-aware Self-distillation (SASD) and Multi-level Affinity Refinement (MLAR).
train stage on coco
type
sched.
AP
AP50
AP75
SAM-top1
two stage
1x
24.6
41.9
25.3
SAPNet+SOLOv2
two stage
1x
30.2
49.8
31.5
SAPNet (MPS)
end to end
1x
31.2
51.8
32.3
SAPNet
end to end
1x
30.7
51.2
31.8
SAPNet++
end to end
1x
32.4
53.5
34.2
TABLE VII : The experimental comparisons of segmenters in COCO dataset, SAM-top1 is the highest scoring mask generated by SAM.
Fig. 6 : The comparative visualization between SAM-top1 and SAPNet++ is presented, showcasing SAM’s segmentation outcomes in green masks and our results in yellow. The orange and red bounding boxes highlight the respective mask boundaries.
SASD
AP
AP50
AP75
APs
APm
APl
SASD-I
SASD-II
30.7
51.2
31.8
11.9
34.7
47.1
✓
30.9
51.3
32.0
12.0
34.8
47.2
✓
31.1
51.7
32.6
12
35.0
47.3
✓
✓
31.3
52.0
32.6
12.6
35.2
48.0
TABLE VIII : Meticulous ablation experiments in SASD
Affinity-level
AP
AP 50
AP 75
AP S
AP M
AP L
✗
30.7
51.2
31.8
11.9
34.7
47.1
low-level
31.0
51.4
32.2
12.1
35.0
47.2
high-level
31.3
51.9
32.4
12.7
35.2
47.9
low & high-level
31.9
52.5
33.4
13.1
36.1
48.5
TABLE IX : The effectiveness of low-level pixel space and high-level semantic features on SAPNet++.
Global affinity
Local affinity
Cascade
AP
AP 50
AP 75
✗
✗
0
30.7
51.2
31.8
✓
0
31.0
51.6
32.1
✓
1
31.2
51.7
32.3
✓
✓
1
31.6
52.1
33.0
✓
✓
2
31.9
52.5
33.4
✓
✓
3
31.7
52.3
33.2
TABLE X : The effectiveness of global and local affinity.
Fig. 7 : The visualization of segmentation using different affinity strategies reveals their varied impacts. with its wide propagation range, global affinity effectively manages discontinuities in large-scale masks but struggles with maintaining local consistency, often missing finer details at edges and corners. Conversely, local affinity, with its limited propagation distance, underperforms in areas where mask parts are distant. Combining both global and local affinity yields smoother segmentation masks. Using a cascading approach further improves mask completeness, striking a balance between broad coverage and detailed accuracy.
Method
Gaplocal
Gapgroup
mIoUbox
mIoUmask
SAM
28.4
24.7
58.3
55.6
MIL
19.9
18.6
63.8
58.8
SAPNet
13.1
14.1
69.1
65.9
SAPNet++
12.4
13.3
70.4
67.7
TABLE XI : Experimental analysis with group and local issues.
Segmenters
BB.
Sch.
Detection
Segmentation
AP
AP50
AP
AP50
Solov2
R-101
3x
38.9
62.0
35.7
57.6
Swin
Swin-S
3x
40.1
63.4
35.8
59.5
Mask2former
Swin-S
3x
43.1
63.5
39.5
62.8
ConvNeXt-V2
ConV2-B
3x
46.0
66.8
39.7
63.1
TABLE XII : Advanced segmenters on COCO.
Type
Image tag
Point
Box
Mask
Cost (per Image)
1.5s/image
1.87s
34.5s
239.7s
Cost (per Object)
–
0.9s
7s
79.2s
TABLE XIII : Annotation cost comparison for different supervision types.
Supervision Type
Method
mAP
Rel. Ann. Cost (vs. Point/Image)
Image Tag (Weakest)
BESTIE
14.3 mAP
1.25 × faster
Point (Our Method)
SAPNet++
32.4 mAP
Base Cost
Box
Box2Mask
32.6 mAP
18.4 × slower
Mask (Full Supervision)
Mask2Former
38.7 mAP
128 × slower
TABLE XIV : Tradeoff Across Different Supervision Methods.
Instance segmentation is a fundamental computer vision task with diverse real-world applications. Recently, prompt-driven foundation models have shown promising generalization. However, automated prompting remains limited by insufficient semantic guidance and inter-instance modeling. To address this challenge, we propose a novel architecture, named Semantic Relational Prompt Refinement Network (SRPR-Net), for automated SAM-based instance segmentation. A sequential prompt refinement mechanism is introduced to enrich detector geometry with visual-language semantics and then incorporate same-image instance dependencies, enabling context-aware box adjustment before SAM segmentation. Experiments on multiple standard benchmarks demonstrate that SRPR-Net achieves consistent improvements in segmentation performance over existing state-of-the-art approaches. The code is publicly available at https://github.com/JeremyXSC/SRPR-Net.
Lufei Liu, Guojie Li, Suncheng Xiang +1
Ruijin-XJTLU Intelligent Medicine Institute, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China; and Xi’an Jiaotong-Liverpool University, Suzhou, China · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China
Modern referring image segmentation pipelines couple a vision-language model (VLM) for grounding with a promptable segmenter such as the Segment Anything Model (SAM) for mask generation. Prior training-free instances of this recipe consistently trail fine-tuned and reinforcement-learning (RL)-tuned specialists, and it has been unclear whether the gap comes from the VLM's grounding, SAM's capacity, or the prompt. We show that the gap is dominated by prompt ambiguity: a VLM-proposed bounding box (bbox) leaves SAM to guess which pixels inside the bbox belong to the object the expression denotes. Interior points are the natural disambiguator, but where they fall matters; prior work relies on naively sampled points that land on boundaries, distractors, and background clutter, and can even hurt performance compared to the bbox alone. Supervised and RL-tuned methods close this gap by training a VLM to predict better points; we show that this training is unnecessary. At a matched budget of five interior points, replacing naive sampling with stable, informative point selection improves cumulative Intersection-over-Union (cIoU) by 12-18 points across RefCOCO/+/g, with every model fixed. We turn this observation into PinPoint, a deterministic, training-free point selector that fuses four visual cues into a consensus map, selects compact, spatially diverse points away from boundaries, and uses the frozen VLM to label each point. Without any task-specific training, PinPoint matches supervised and RL-tuned specialists on the same stack while issuing only two VLM calls per query.
Pouya Sadeghi, Shawn He, Pedro Pablo Guerrero Vela +3
University of Waterloo · 2Critical ML, University of Waterloo · 3Apple
Crowd instance segmentation is a crucial task with a wide range of applications, including surveillance and transportation. Currently, point labels are common in crowd datasets, while region labels (e.g., boxes) are rare and inaccurate. The masks obtained through segmentation help to improve the accuracy of region labels and resolve the correspondence between individual location coordinates and crowd density maps. However, directly applying currently popular large foundation models such as SAM does not yield ideal results in dense crowds. To this end, we first propose Dense Point-to-Mask Optimization (DPMO), which integrates SAM with the Nearest Neighbor Exclusive Circle (NNEC) constraint to generate dense instance segmentation from point annotations. With DPMO and manual correction, we obtain mask annotations from the existing point annotations for traditional crowd datasets. Then, to predict instance segmentation in dense crowds, we propose a Reinforced Point Selection (RPS) framework trained with Group Relative Policy Optimization (GRPO), which selects the best predicted point from a sampling of the initial point prediction. Through extensive experiments, we achieve state-of-the-art crowd instance segmentation performance on ShanghaiTech, UCF-QNRF, JHU-CROWD++, and NWPU-Crowd datasets. Furthermore, we design new loss functions supervised by masks that boost counting performance across different models, demonstrating the significant role of mask annotations in enhancing counting accuracy.
Hongru Chen, Jiyang Huang, Jia Wan +1
Harbin Institute of Technology, Shenzhen · City University of Hong Kong