cs.CVApr 13, 2026
SaveDo Instance Priors Help Weakly Supervised Semantic Segmentation?
Organizations: Max Planck Institute for Informatics, Saarland Informatics Campus · University of Technology Nuremberg
Abstract
Semantic segmentation requires dense pixel-level annotations, which are costly and time-consuming to acquire. To address this, we present SeSAM, a framework that uses a foundational segmentation model, i.e. Segment Anything Model (SAM), with weak labels, including coarse masks, scribbles, and points. SAM, originally designed for instance-based segmentation, cannot be directly used for semantic segmentation tasks. In this work, we identify specific challenges faced by SAM and determine appropriate components to adapt it for class-based segmentation using weak labels. Specifically, SeSAM decomposes class masks into connected components, samples point prompts along object skeletons, selects SAM masks using weak-label coverage, and iteratively refines labels using pseudo-labels, enabling SAM-generated masks to be effectively used for semantic segmentation. Integrated with a semi-supervised learning framework, SeSAM balances ground-truth labels, SAM-based pseudo-labels, and high-confidence pseudo-labels, significantly improving segmentation quality. Extensive experiments across multiple benchmarks and weak annotation types show that SeSAM consistently outperforms weakly supervised baselines while substantially reducing annotation cost relative to fine supervision.
Figures & tables
Figure 1: Our SeSAM framework , utilizing additional instance priors, integrates with various weak labels (points, scribbles, and coarse annotations) and enhances semantic segmentation quality (see road and car on the left) while remaining cost-effective (right plot). For instance, using scribbles achieves 94% of fine-supervised performance while requiring only 2% of the annotation budget.
Figure 2: SAM challenges. Left : As an instance-based model, SAM performs poorly when segmenting a class comprising multiple instances, see the class “person”. Splitting the class mask into individual instances addresses this challenge. Middle : Confidence-based sampling ( Kweon and Yoon, 2024 ) yields a suboptimal mask; in contrast, our proposed sampling strategy achieves comprehensive mask coverage, resulting in improved segmentation. Right : SAM score-based mask selection may not give the best desired segmentation (see mask in red box). Our mask selection strategy using intersection with the coarse mask determines the desired mask granularity (see mask in green box).
Figure 3: SeSAM framework for coarse supervision . First, pseudo-labels for training images are generated by the teacher model. Next, these pseudo-labels are then refined following the key steps 1-3, namely instance separation, point sampling and mask selection (see Section 3 ). To improve mask quality, we augment coarse labels with high-confidence pseudo-labels before applying SAM (step 0). The resulting refined labels, called SeSAM labels, are then used to train the student network.
| Scribble | Point | ||||
| Method | DeepLab | mIoU | WvF | mIoU | WvF |
| Pan et al. ( Pan et al., 2021 ) | V2 | 74.6 | 96.0 | - | - |
| SPML ( Ke et al., 2021 ) | V2 | 74.2 | 95.5 | - | - |
| A2GNN ( Zhang et al., 2021 ) | V2 | 74.3 | 95.5 | 66.8 | 85.9 |
| Supervised | V3+ | 69.7 | 87.6 | 61.6 | 77.4 |
| Fixmatch ( Sohn et al., 2020 ) | V3+ | 71.3 | 89.6 | 68.4 | 86.0 |
Table 1: Comparisons on PASCAL VOC 2012 using scribble and point labels with DeepLabv3+ and DeepLabV2 segmentation networks (ResNet-101 backbone). WvF ( Ke et al., 2021 ) refers to weak vs fine supervision performance ratio. AGMM ∗ : Our re-implementation.
| SegFormer | DeepLabV3+ | |||||
| Annotation | Method | Cityscapes | ADE20k | PASCAL | Cityscapes | ADE20k |
| Point | Supervised | 51.8 | 26.5 | 61.5 | 52.2 | 29.5 |
| Fixmatch ( Sohn et al., 2020 ) | 53.1 | 27.2 | 63.6 | 52.9 | 29.8 | |
| AGMM ∗ ( Wu et al., 2023 ) | 53.4 | - | 56.6 | 52.4 | - | |
| SAM baseline | 51.0 | 29.3 | 59.0 | 50.0 | 35.1 | |
| Ours | 58.5 | 32.6 | 67.5 | 61.7 | 36.3 | |
Table 2: Comparison on Cityscapes ( Cordts et al., 2016 ) , ADE20k ( Zhou et al., 2017 ) , and PASCAL VOC 2012 ( Everingham et al., 2010 ) with SegFormer ( Xie et al., 2021 ) and DeepLabV3+. AGMM ∗ denotes our reimplementation of AGMM ( Wu et al., 2023 ) . The SAM baseline naïvely uses weak labels as prompts to SAM (see Section 4.1 ). SegFormer uses MiT-B0, whereas DeepLabV3+ uses ResNet-101 ( He et al., 2016 ) .
Figure 4: Annotation cost vs. performance plot . We compare the performance at different budgets for each of the weak labels (point, scribble and coarse), and compare the cost-effectiveness of our method with fine annotation (bottom-left). In particular, scribble supervision reaches performance comparable to fine-label supervision using only 6% of the annotation budget. Whereas, in a low-budget setting (109 hrs), it outperforms fine annotations by 18.5%.
Figure 5: Qualitative performance comparison . We conduct a qualitative performance comparison of our framework with supervised models using various weak labels, including point, scribble, and coarse annotations. Overall, our method demonstrates superior class segmentation, particularly with enhanced prediction along boundaries (see car).
Table 3: Ablation studies of SeSAM. We study the contribution of the proposed components, instance mask separation, and refined-mask-based point sampling. Base refers to the FixMatch ( Sohn et al., 2020 ) -based segmentation network. Experiments on Cityscapes with SegFormer-B0.
Table 4: Analysis of SeSAM design choices. We study the effect of different point sampling strategies, mask selection strategies, and prompt types. Experiments on Cityscapes with SegFormer-B0.
| SAM ( Kirillov et al., 2023 ) | SAM 2 ( Ravi et al., 2025 ) | SAM 3 ( Carion et al., 2025 ) | |
| Precision | 92.2 | 92.8 | 92.1 |
| Recall | 62.7 | 61.8 | 61.9 |
| F1 | 74.7 | 74.2 | 73.9 |
Table 5: Comparison across SAM versions. We compare the mask generation quality of different SAM versions using precision and recall. Although newer versions (SAM 2 and SAM 3) introduce additional capabilities beyond image segmentation ( Ravi et al., 2025 ; Carion et al., 2025 ) , the original SAM provides the best trade-off between precision and recall for image-based segmentation. Experiments on Cityscapes dataset.
| Training time (sec/iter) | GPU memory (Gb) | Performance (mIoU) | |
| Fixmatch ( Sohn et al., 2020 ) | 0.24 | 22.3 | 68.9 |
| AGMM ( Wu et al., 2023 ) | 1.56 | 60.8 | 70.6 |
| Ours | 0.81 | 22.7 | 72.4 |
Table 6: Runtime and memory comparison on Cityscapes using DeepLabv3+. We report training time per iteration, GPU memory usage, and mIoU.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| # of iterations | 1 | 2 | 4 | 8 | 16 |
| Cityscapes mIoU | 69.3 | 69.4 | 69.9 | 68.2 | 68.2 |
Table 7: Ablation experiment with frequency of applying SAM during training. We observe that applying SAM too frequently, i.e, every training iteration does not improve performance compared to training once every few iterations (once in 4 iterations). Further applying SAM in even fewer iterations (once in 16 iterations) results in performance degradation.
| Classwise performance comparison: Cityscapes | |||||||||||||||||||||
| road | sidewalk | building | wall | fence | pole | tra. light | tra. sign | vege. | terrain | sky | person | rider | car | truck | bus | train | motor. | bicycle | mIoU | mIoU* | |
| point | 94.9 | 66.6 | 85.9 | 42.2 | 37.9 | 45.1 | 34.6 | 59.6 | 86.3 | 44.4 | 88.9 | 67.0 | 38.4 | 87.1 | 38.4 | 55.6 | 41.1 | 38.1 | 60.2 | 58.5 | 64.8 |
| scribble | 96.8 | 76.2 | 90.0 | 58.3 | 48.5 | 48.0 | 59.7 | 72.7 | 90.4 | 56.7 | 93.1 | 76.3 | 52.2 | 92.2 | 66.6 | 80.9 | 71.7 | 56.7 | 71.5 | 71.5 | 77.6 |
| coarse | 95.7 | 71.7 | 89.3 | 54.6 | 47.5 | 53.9 | 54.7 | 68.7 | 89.4 | 44.5 | 93.9 | 74.6 | 50.6 | 92.0 | 70.3 | 76.1 | 70.1 | 55.6 | 70.2 | 69.9 | 75.9 |
| Fine | 98.0 | 83.4 | 91.7 | 60.3 | 55.3 | 59.8 | 67.4 | 76.1 | 92.2 | 64.2 | 94.6 | 79.3 | 56.2 | 94.2 | 73.8 | 83.6 | 77.0 | 61.6 | 75.6 | 76.0 | 80.8 |
Table 8: Classwise performance comparison on Cityscapes. We compare the classwise performance of SeSAM for different weak labels with fine labels. In general, scribble performs better than other weak labels for all of the classes. Further, we also compare mIoU (mIoU ∗ ) performance by removing problematic classes with wrong ground-truth labels( Table 9 ). By removing such classes, we observe further narrowing of the performance gap with fine labels.
Figure 8: Why points for prompting? Point prompts generate the best segmentation masks using SAM (left). Whereas, bounding-box prompting forces SAM to generate labels within the box and does not completely label the object (middle). Mask prompt fails to generate meaningful mask (right) as also discussed in ( , 2024 ) .
Figure 9: Inconsistency between coarse and fine labels . In this example, we see the class wall is incorrectly labeled as fence in coarse annotation. Similarly, vegetation is wrongly labeled as road in coarse annotation.
| Coarse vs Fine label mismatch in Cityscapes | ||||||||||||||||||||
| road | sidewalk | building | wall | fence | pole | tra. light | tra. sign | vege. | terrain | sky | person | rider | car | truck | bus | train | motor. | bicycle | mIoU | |
| val | 98.8 | 93.2 | 96.8 | 61.1 | 71.6 | 87.9 | 86.5 | 94.4 | 97.4 | 67.4 | 97.6 | 96.2 | 86.1 | 99.2 | 94.9 | 98.2 | 90.0 | 87.2 | 93.5 | 89.4 |
| train | 99.0 | 91.7 | 97.4 | 62.7 | 74.3 | 84.9 | 87.6 | 91.6 | 96.3 | 70.1 | 97.5 | 97.3 | 89.0 | 98.9 | 86.7 | 94.6 | 95.8 | 92.4 | 91.4 | 89.4 |
Table 9: Inconsistency between coarse and fine annotations. We perform an experiment to understand the label mismatch between manually labeled coarse and fine labels. We compute intersection of fine and coarse labels over coarse labels (mIoU for both train and val sets. We observe classes wall, fence and terrain to be severely affected, also shown in Fig. 9
Figure 10: Image label as weak supervision. Best performing image label WSSS method SAM2CAM ( Kweon and Yoon, 2024 ) fails to predict correct segmentation labels for Cityscapes dataset ( Class ‘road’ is wrongly predicted as ‘bus’). In such real world datasets, classes co-occur quite often (e.g. car and road in all images) making it difficult to obtain correct CAM for class localization.
Figure 11: Top: Contrast corruption on Cityscapes image for different severity levels. Bottom: Performance comparison for Contrast and Fog corruptions across severity levels. SAM remains robust with minimal F1 drop up to severity 3.
Figure 14: Qualitative performance comparison on Cityscapes. We conduct a qualitative performance comparison of our framework with supervised models using various weak labels, including point, scribble, and coarse supervision. Overall, our method demonstrates superior class segmentation, particularly with enhanced prediction along boundaries (see car in row 1, 3 and 4, wall and bicycle in row 2).
Figure 15: Qualitative performance comparison on ADE20k. We conduct a qualitative performance comparison of our framework with supervised models using various weak labels, including point, scribble, and coarse supervision. Overall, our method demonstrates superior class segmentation, particularly with enhanced prediction along boundaries (see house in row 1 and chair in row 3).
More in this topic
Weakly Supervised Semantic Segmentation
Weakly supervised semantic segmentation learns pixel-level class masks from image-level labels, points, scribbles, or other sparse annotations.
See all 33 papers