Semantic segmentation requires dense pixel-level annotations, which are costly and time-consuming to acquire. To address this, we present SeSAM, a framework that uses a foundational segmentation model, i.e. Segment Anything Model (SAM), with weak labels, including coarse masks, scribbles, and points. SAM, originally designed for instance-based segmentation, cannot be directly used for semantic segmentation tasks. In this work, we identify specific challenges faced by SAM and determine appropriate components to adapt it for class-based segmentation using weak labels. Specifically, SeSAM decomposes class masks into connected components, samples point prompts along object skeletons, selects SAM masks using weak-label coverage, and iteratively refines labels using pseudo-labels, enabling SAM-generated masks to be effectively used for semantic segmentation. Integrated with a semi-supervised learning framework, SeSAM balances ground-truth labels, SAM-based pseudo-labels, and high-confidence pseudo-labels, significantly improving segmentation quality. Extensive experiments across multiple benchmarks and weak annotation types show that SeSAM consistently outperforms weakly supervised baselines while substantially reducing annotation cost relative to fine supervision.
Figures & tables
Figure 1: Our SeSAM framework , utilizing additional instance priors, integrates with various weak labels (points, scribbles, and coarse annotations) and enhances semantic segmentation quality (see road and car on the left) while remaining cost-effective (right plot). For instance, using scribbles achieves 94% of fine-supervised performance while requiring only 2% of the annotation budget.
Figure 2: SAM challenges. Left : As an instance-based model, SAM performs poorly when segmenting a class comprising multiple instances, see the class “person”. Splitting the class mask into individual instances addresses this challenge. Middle : Confidence-based sampling ( Kweon and Yoon, 2024 ) yields a suboptimal mask; in contrast, our proposed sampling strategy achieves comprehensive mask coverage, resulting in improved segmentation. Right : SAM score-based mask selection may not give the best desired segmentation (see mask in red box). Our mask selection strategy using intersection with the coarse mask determines the desired mask granularity (see mask in green box).
Figure 3: SeSAM framework for coarse supervision . First, pseudo-labels for training images are generated by the teacher model. Next, these pseudo-labels are then refined following the key steps 1-3, namely instance separation, point sampling and mask selection (see Section 3 ). To improve mask quality, we augment coarse labels with high-confidence pseudo-labels before applying SAM (step 0). The resulting refined labels, called SeSAM labels, are then used to train the student network.
Scribble
Point
Method
DeepLab
mIoU
WvF
mIoU
WvF
Pan et al. ( Pan et al., 2021 )
V2
74.6
96.0
-
-
SPML ( Ke et al., 2021 )
V2
74.2
95.5
-
-
A2GNN ( Zhang et al., 2021 )
V2
74.3
95.5
66.8
85.9
Supervised
V3+
69.7
87.6
61.6
77.4
Fixmatch ( Sohn et al., 2020 )
V3+
71.3
89.6
68.4
86.0
Table 1: Comparisons on PASCAL VOC 2012 using scribble and point labels with DeepLabv3+ and DeepLabV2 segmentation networks (ResNet-101 backbone). WvF ( Ke et al., 2021 ) refers to weak vs fine supervision performance ratio. AGMM ∗ : Our re-implementation.
SegFormer
DeepLabV3+
Annotation
Method
Cityscapes
ADE20k
PASCAL
Cityscapes
ADE20k
Point
Supervised
51.8
26.5
61.5
52.2
29.5
Fixmatch ( Sohn et al., 2020 )
53.1
27.2
63.6
52.9
29.8
AGMM ∗ ( Wu et al., 2023 )
53.4
-
56.6
52.4
-
SAM baseline
51.0
29.3
59.0
50.0
35.1
Ours
58.5
32.6
67.5
61.7
36.3
Table 2: Comparison on Cityscapes ( Cordts et al., 2016 ) , ADE20k ( Zhou et al., 2017 ) , and PASCAL VOC 2012 ( Everingham et al., 2010 ) with SegFormer ( Xie et al., 2021 ) and DeepLabV3+. AGMM ∗ denotes our reimplementation of AGMM ( Wu et al., 2023 ) . The SAM baseline naïvely uses weak labels as prompts to SAM (see Section 4.1 ). SegFormer uses MiT-B0, whereas DeepLabV3+ uses ResNet-101 ( He et al., 2016 ) .
Figure 4: Annotation cost vs. performance plot . We compare the performance at different budgets for each of the weak labels (point, scribble and coarse), and compare the cost-effectiveness of our method with fine annotation (bottom-left). In particular, scribble supervision reaches performance comparable to fine-label supervision using only 6% of the annotation budget. Whereas, in a low-budget setting (109 hrs), it outperforms fine annotations by 18.5%.
Figure 5: Qualitative performance comparison . We conduct a qualitative performance comparison of our framework with supervised models using various weak labels, including point, scribble, and coarse annotations. Overall, our method demonstrates superior class segmentation, particularly with enhanced prediction along boundaries (see car).
Table 3: Ablation studies of SeSAM. We study the contribution of the proposed components, instance mask separation, and refined-mask-based point sampling. Base refers to the FixMatch ( Sohn et al., 2020 ) -based segmentation network. Experiments on Cityscapes with SegFormer-B0.
Table 4: Analysis of SeSAM design choices. We study the effect of different point sampling strategies, mask selection strategies, and prompt types. Experiments on Cityscapes with SegFormer-B0.
SAM ( Kirillov et al., 2023 )
SAM 2 ( Ravi et al., 2025 )
SAM 3 ( Carion et al., 2025 )
Precision
92.2
92.8
92.1
Recall
62.7
61.8
61.9
F1
74.7
74.2
73.9
Table 5: Comparison across SAM versions. We compare the mask generation quality of different SAM versions using precision and recall. Although newer versions (SAM 2 and SAM 3) introduce additional capabilities beyond image segmentation ( Ravi et al., 2025 ; Carion et al., 2025 ) , the original SAM provides the best trade-off between precision and recall for image-based segmentation. Experiments on Cityscapes dataset.
Training time (sec/iter)
GPU memory (Gb)
Performance (mIoU)
Fixmatch ( Sohn et al., 2020 )
0.24
22.3
68.9
AGMM ( Wu et al., 2023 )
1.56
60.8
70.6
Ours
0.81
22.7
72.4
Table 6: Runtime and memory comparison on Cityscapes using DeepLabv3+. We report training time per iteration, GPU memory usage, and mIoU.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
# of iterations
1
2
4
8
16
Cityscapes mIoU
69.3
69.4
69.9
68.2
68.2
Appendix
Table 7: Ablation experiment with frequency of applying SAM during training. We observe that applying SAM too frequently, i.e, every training iteration does not improve performance compared to training once every few iterations (once in 4 iterations). Further applying SAM in even fewer iterations (once in 16 iterations) results in performance degradation.
Classwise performance comparison: Cityscapes
road
sidewalk
building
wall
fence
pole
tra. light
tra. sign
vege.
terrain
sky
person
rider
car
truck
bus
train
motor.
bicycle
mIoU
mIoU*
point
94.9
66.6
85.9
42.2
37.9
45.1
34.6
59.6
86.3
44.4
88.9
67.0
38.4
87.1
38.4
55.6
41.1
38.1
60.2
58.5
64.8
scribble
96.8
76.2
90.0
58.3
48.5
48.0
59.7
72.7
90.4
56.7
93.1
76.3
52.2
92.2
66.6
80.9
71.7
56.7
71.5
71.5
77.6
coarse
95.7
71.7
89.3
54.6
47.5
53.9
54.7
68.7
89.4
44.5
93.9
74.6
50.6
92.0
70.3
76.1
70.1
55.6
70.2
69.9
75.9
Fine
98.0
83.4
91.7
60.3
55.3
59.8
67.4
76.1
92.2
64.2
94.6
79.3
56.2
94.2
73.8
83.6
77.0
61.6
75.6
76.0
80.8
Appendix
Table 8: Classwise performance comparison on Cityscapes. We compare the classwise performance of SeSAM for different weak labels with fine labels. In general, scribble performs better than other weak labels for all of the classes. Further, we also compare mIoU (mIoU ∗ ) performance by removing problematic classes with wrong ground-truth labels( Table 9 ). By removing such classes, we observe further narrowing of the performance gap with fine labels.
Figure 14
Figure 8: Why points for prompting? Point prompts generate the best segmentation masks using SAM (left). Whereas, bounding-box prompting forces SAM to generate labels within the box and does not completely label the object (middle). Mask prompt fails to generate meaningful mask (right) as also discussed in ( , 2024 ) .
Figure 9: Inconsistency between coarse and fine labels . In this example, we see the class wall is incorrectly labeled as fence in coarse annotation. Similarly, vegetation is wrongly labeled as road in coarse annotation.
Coarse vs Fine label mismatch in Cityscapes
road
sidewalk
building
wall
fence
pole
tra. light
tra. sign
vege.
terrain
sky
person
rider
car
truck
bus
train
motor.
bicycle
mIoU †
val
98.8
93.2
96.8
61.1
71.6
87.9
86.5
94.4
97.4
67.4
97.6
96.2
86.1
99.2
94.9
98.2
90.0
87.2
93.5
89.4
train
99.0
91.7
97.4
62.7
74.3
84.9
87.6
91.6
96.3
70.1
97.5
97.3
89.0
98.9
86.7
94.6
95.8
92.4
91.4
89.4
Appendix
Table 9: Inconsistency between coarse and fine annotations. We perform an experiment to understand the label mismatch between manually labeled coarse and fine labels. We compute intersection of fine and coarse labels over coarse labels (mIoU † for both train and val sets. We observe classes wall, fence and terrain to be severely affected, also shown in Fig. 9
Figure 10: Image label as weak supervision. Best performing image label WSSS method SAM2CAM ( Kweon and Yoon, 2024 ) fails to predict correct segmentation labels for Cityscapes dataset ( Class ‘road’ is wrongly predicted as ‘bus’). In such real world datasets, classes co-occur quite often (e.g. car and road in all images) making it difficult to obtain correct CAM for class localization.
Figure 11: Top: Contrast corruption on Cityscapes image for different severity levels. Bottom: Performance comparison for Contrast and Fog corruptions across severity levels. SAM remains robust with minimal F1 drop up to severity 3.
Figure 20
Figure 14: Qualitative performance comparison on Cityscapes. We conduct a qualitative performance comparison of our framework with supervised models using various weak labels, including point, scribble, and coarse supervision. Overall, our method demonstrates superior class segmentation, particularly with enhanced prediction along boundaries (see car in row 1, 3 and 4, wall and bicycle in row 2).
Figure 15: Qualitative performance comparison on ADE20k. We conduct a qualitative performance comparison of our framework with supervised models using various weak labels, including point, scribble, and coarse supervision. Overall, our method demonstrates superior class segmentation, particularly with enhanced prediction along boundaries (see house in row 1 and chair in row 3).
Weakly supervised semantic segmentation (WSSS) trains dense pixel-level segmentation models from partial or coarse annotations such as bounding boxes, scribbles, or image-level tags. While recent work leverages foundation models such as the Segment Anything Model (SAM) to generate pseudo-labels, these approaches typically depend on heuristic prompt choices and offer limited ways to incorporate prior knowledge or heterogeneous labels. We address this gap by taking a neurosymbolic perspective: integrating differentiable fuzzy logic with deep segmentation models. Weak annotations and domain-specific priors are unified as continuous logical constraints that fine-tune SAM under weak supervision. The refined foundation model then produces improved pseudo-labels, from which we train a second-stage prompt-free segmentation model. Experiments on Pascal VOC 2012 and the REFUGE2 optic disc/cup segmentation dataset show that our logic-guided fine-tuning yields higher-quality pseudo-labels, leading to state-of-the-art segmentation accuracy that often exceeds densely supervised baselines.
Weakly supervised semantic segmentation aims to achieve pixel-level predictions using image-level labels. Existing methods typically entangle semantic recognition and object localization, which often leads models to focus exclusively on sparse discriminative regions. Although foundation models show immense potential, many approaches still follow the tightly coupled optimization paradigm, struggling to effectively alleviate pseudo-label noise and often relying on time-consuming multi-stage retraining or unstable end-to-end joint optimization. To address the above challenges, we present ModuSeg, a training-free weakly supervised semantic segmentation framework centered on explicitly decoupling object discovery and semantic assignment. Specifically, we integrate a general mask proposer to extract geometric proposals with reliable boundaries, while leveraging semantic foundation models to construct an offline feature bank, transforming segmentation into a non-parametric feature retrieval process. Furthermore, we propose semantic boundary purification and soft-masked feature aggregation strategies to effectively mitigate boundary ambiguity and quantization errors, thereby extracting high-quality category prototypes. Extensive experiments demonstrate that the proposed decoupled architecture better preserves fine boundaries without parameter fine-tuning and achieves highly competitive performance on standard benchmark datasets. Code is available at https://github.com/Autumnair007/ModuSeg.
Qingze He, Fagui Liu, Dengke Zhang +2
South China University of Technology, Guangzhou, China · Pengcheng Laboratory, Shenzhen, China
Weakly Supervised Semantic Segmentation (WSSS) learns pixel-level predictions from image-level tags. Recent work focuses on improving coarse CAMs extracted from large vision-language models (commonly CLIP), but does little to improve their accuracy along segment boundaries. That job is instead delegated to a post-processing method like DenseCRF. However, because DenseCRF relies on low-level colour cues, it can flip correct labels to incorrect ones when neighbouring pixels share similar colours. SAM has recently been adopted as a natural alternative, yet it simply takes on DenseCRF's role as an intermediate "refinement" step that outputs one-hot pseudo-labels in prior work. By discarding the valuable uncertainty in CAMs, these one-hot pseudo-labels turn borderline errors into confidently wrong targets. Our key insight is that CAMs should supervise training alongside SAM boundaries, each through its own loss, rather than being fused together into a single hard target. Inspired by CRF potentials, we propose a framework that disentangles soft pseudo-labels as unary supervision and binary edge maps as pairwise supervision. We realize our framework in a single-stage model, DS-CRF, using CAMs from dino.txt and boundaries from SAM. DS-CRF sets a new state-of-the-art of 56.5% mIoU on MS COCO.
Joshua Li, Yuri Boykov
Cheriton School of Computer Science University of Waterloo